Title: Sample-Efficient Multi-Round Generative Data Augmentation for Long-Tail Instance Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: instance segmentation

data synthesis

diffusion

copy-paste augmentation
Abstract: Data synthesis has become increasingly crucial for long-tail instance segmentation tasks to mitigate class imbalance and high annotation costs. Previous methods have primarily prioritized the selection of data from a pre-generated image object pool, which frequently leads to the inefficient utilization of generated data. To address this inefficiency, we propose a **collaborative** approach that incorporates feedback from an instance segmentation model to guide the augmentation process. Specifically, the diffusion model uses feedback to generate objects that exhibit high uncertainty. The number and size of synthesized objects for each class are dynamically adjusted based on the model state to improve learning in underrepresented classes. This augmentation process is further strengthened by running **multiple rounds**, allowing feedback to be refined throughout training. In summary, **multi-round collaborative augmentation (MRCA)** enhances sample efficiency by providing optimal synthetic data at the right moment. Our framework requires **only 6\%** of the data generation needed by state-of-the-art methods while outperforming them.

Sample-Efficient Multi-Round Generative Data Augmentation for Long-Tail Instance Segmentation

Abstract

Data synthesis has become increasingly crucial for long-tail instance segmentation tasks to mitigate class imbalance and high annotation costs. Previous methods have primarily prioritized the selection of data from a pre-generated image object pool, which frequently leads to the inefficient utilization of generated data. To address this inefficiency, we propose a collaborative approach that incorporates feedback from an instance segmentation model to guide the augmentation process. Specifically, the diffusion model uses feedback to generate objects that exhibit high uncertainty. The number and size of synthesized objects for each class are dynamically adjusted based on the model state to improve learning in underrepresented classes. This augmentation process is further strengthened by running multiple rounds , allowing feedback to be refined throughout training. In summary, multi-round collaborative augmentation (MRCA) enhances sample efficiency by providing optimal synthetic data at the right moment. Our framework requires only 6% of the data generation needed by state-of-the-art methods while outperforming them.

1 Introduction

The advent of diffusion models has made generating high-quality images significantly easier with lower cost [9, 23, 30, 31]. These advancements naturally opened new possibilities for generating training data in data-hungry vision tasks like object detection [16, 17] and instance segmentation [4, 21, 27] 1 . Diffusion models can synthesize images conditioned on textual descriptions, enabling the creation of diverse and detailed datasets with labels [6, 22]. Especially for long-tail instance segmentation, where class imbalance and high annotation costs pose significant challenges, diffusionbased image synthesis has become a crucial solution.

Copy-paste-based methods, originally intended for augmenting training data using real image objects [8, 15], now have been extended to utilize high-quality synthetic image objects generated by diffusion models. That is, these recent methods first generate a large pool of image objects and then augment the training data by pasting them to real images [11, 37, 41]. The advantage arises from generating objects in a training-free manner, at a comparably lower cost than layout-based methods that train the diffusion model with the training data as shown in Figure 1a [5, 40]. However, some studies have shown that continuously training with synthetic data can eventually degrade model performance [12, 13]. Such finding has naturally led to research on active and curriculum learning strategies that can incrementally select most informative synthetic objects from a pre-generated object pool as shown in Figure 1b [26, 41]. Nevertheless, preemptively constructing the object pool before training may be suboptimal, as it can result in a significant number of unused objects and may not include object characteristics that could address the model's deficiencies encountered during training.

gen

⨁

Generator

train

train

Instance

Segmentation

Model

印

gen

Generator

train

Real Data

Syn Objects

Instance

Segmentation

Model

paste

reject

血

(b) Training-free generative augmentation (existing).

gen

Generator

train

Real Data

round r+1

Instance

Segmentation

Model

paste

round r

Feedback: Rare class? Increase difficulty, number, and scale!

round r-1

Syn Objects

To solve this problem, we propose multi-round collaborative augmentation between a generator and an instance segmentation model to guide the generator to build an augmented training set which can achieve high accuracy with a smaller number of synthetic objects-i.e., toward being sample-efficient .

Collaborative Augmentation To effectively improve the power of the instance segmentation model using synthetic images, we transfer its feedback to the augmentation process. As illustrated in Figure 1c, the feedback is exploited in two distinct places: (i) in the generation of synthetic objects and (ii) in the pasting of these objects to real images. For the generation step, the classifier gradients are employed to produce more challenging objects for the instance segmentation model's classifier, akin to prior research on image classification [1, 19]. Also, the class-wise validation accuracy and classifier weights are used to ascertain an adequately mandated quantity of synthetic objects for each class, thereby enhancing the classifier's capacity to learn from underrepresented or difficult classes. For the pasting step, the class-wise validation accuracy is employed to assign greater space to underperforming classes, as larger objects are more effective in enhancing the performance of instance segmentation [28]. In short, this two-fold feedback approach facilitates the object generator in supplying the most beneficial data to the instance segmentation model.

Multi-Round This novel collaborative augmentation is boosted through multiple rounds . It is well-known that the most useful training instances vary depending on the training stage, as witnessed by curriculum learning [3]. Thanks to the collaborative augmentation, the feedback from the instance segmentation model can be effectively transferred to the object generator. Then, it is more effective to transfer such feedback periodically throughout the training stage. As a result, the generation and pasting of synthetic objects occur incrementally , resulting in a reduced overall quantity of synthetic objects. Regarding each incremental generation, better customized images can be supplied for the training of the instance segmentation model at the appropriate time.

In summary, our proposed M ultiR ound C ollaborative A ugmentation ( MRCA ) achieves higher or comparable performance with only 6% of object generation compared to state-of-the-art methods [7, 37, 41] in generative augmentation for long-tail instance segmentation on the LVIS 1.0 [18] dataset, particularly outperforming in rare classes. Extensive ablation experiments show the effectiveness of the two-fold feedback components of MRCA. The source code is publicly available at https: //github.com/kaist-dmlab/MRCA.

2 Related Work

Data Synthesis for Instance Segmentation Data synthesis for instance segmentation can be broadly categorized into two main approaches. The former involves cut-and-paste (or copy-pastebased) methods, where objects from a source image are extracted with precise boundaries and pasted onto destination images to create new synthesized images. This approach originates from cut-andpaste augmentation [8] and copy-paste augmentation [15], which were initially designed for pasting real objects to enhance data diversity.

Building upon these foundational methods for the former approach, X-Paste [37] extends the basic cut-and-paste method by incorporating objects from multiple sources, including both real and diffusion-based synthetic objects, to further diversify training examples. More recently, BSGAL [41] builds upon X-Paste by integrating instance-wise data selection strategies based on gradient and loss information, ensuring that pasted objects contribute effectively to model training. Additionally, DiverGen [11] focuses on diversity-centered instance generation, optimizing synthetic data to enhance representation learning by promoting diversity among generated objects.

The latter approach involves layout-based methods, where bounding boxes or segmentation masks are given a priori for generating images for object detection. GeoDiffusion [5] and ODGEN [40] generate bounding box coordinates derived from the dataset and condition the diffusion model to produce objects accordingly. InstaGen [14] instead synthesizes images and creates bounding box annotation by training an instance-level grounding head. However, these methods cannot be extended to instance segmentation tasks because of the high complexity of segmentation masking. Even for layout-based methods that provide segmentation masks as a condition [25], the diverse geometric knowledge of the diffusion model cannot be fully utilized owing to the fixed segmentation mask.

Diffusion with Guidance The diffusion model generates the data following a data distribution p ( x ) by learning the denoising process. Recent diffusion models are based on classifier-guidance [6] or classifier-free guidance [22] that learns the data distribution conditioned on class labels p ( x | y ) . That is, ∇ x log p θ,γ ( x | y ) = ∇ x log p θ ( x ) + γ ∇ x log p ( y | x ) , where p θ ( x ) is the generative model's data distribution of an image x parametrized by θ , p ( y | x ) is the classifier's probability of a class y given x , and γ is a guidance weight controlling classifier influence.

ControlNet [36] proposed a more structured approach by injecting additional spatial conditions into the diffusion process without retraining the base model, allowing fine-grained control over outputs. More recently, Universal Guidance [2] emerged as a general framework that can direct diffusion models using arbitrary signals through plug-and-play mechanisms, broadening the flexibility and applicability of guided generation. Altogether, these methods have significantly expanded the controllability and reliability of diffusion-based generative models.

3 Methodology: MRCA

3.1 Problem Formulation

An instance segmentation model is trained using a training set D real = { ( x i , y i ) }| T i =1 , where x i is an image, y i is a set of annotations for the image x i , and T is the total number of images. Here, y i = { y i,j | y i,j = ( bb i,j , m i,j , c i,j ) }| | y i | j =1 , where bb i,j , m i,j and c i,j denote the bounding box, segmentation mask, and class of the j -th object.

We aim to augment the original training set D real through data synthesis to increase data diversity, rectify class imbalance, and enhance model generalization. To achieve this goal, a set of image objects O syn = { o i | o i = ( I i , y i ) }| B i =1 is created using a generative model G with a given label y i from the set of classes as well as a dichotomous segmentation model S , where B represents the given budget of data synthesis. Two functions 2 are internally used for this augmentation process A ( D real ) = D aug , as shown in Figure 2.

Gen ( B ) : It generates B image objects to compose O syn , where the number of objects per class is automatically determined by the learning progress of the instance segmentation model.

Map-Paste ( o ) : For each object o ∈ O syn , it first selects the real image and annotations, ( x, y ) ∈ D real , that o will be pasted. Consequently, the mapping between objects and images, P , is constructed. For each pair of an object and an image ( o, ( x, y )) ∈ P , it then pastes o into x and accordingly updates the corresponding set of annotations y . That is, it returns an augmentation ( x ′ , y ′ ) from ( x, y ) ∈ D real and o ∈ O syn .

As a result, an augmented training set D aug = { ( x ′ i , y ′ i ) } | T i =1 is used instead of D real . Note that the number of images is fixed to N whereas the number of annotations in these images is increased by B , following the common convention [11, 37, 41]. It is also possible to expand the set of images that are used for augmentation, and we leave it as a topic of future work.

Generator

Round

𝒓𝒓

Round

𝒓𝒓

+

𝟏𝟏

…

Round

𝑵𝑵

Instance

Segmentation

Model

Round

𝒓𝒓 - 𝟏𝟏

Model

Guidance

"Coconut"

"Seahorse"

"Bulldozer"

Gen(B)

Paste

Map-Paste(o)

Train

Feedback to the

Next Round

Class-wise Metrics

Instance

Segmentation

Model

Dreal

Daug

S

S

(x,y)

(x',y')

M

y

M

C

Let's consider two instance segmentation models trained on D real and D aug , respectively. The performance difference between these models on the test set D test indicates the quality( Q ) of the data augmentation. Therefore, the primary challenge is to find an augmentation process A B that maximizes the quality of data augmentation within a specified budget B with respect to D real . Specifically, let M real and M aug denote the instance segmentation models trained on D real and D aug , respectively, and L denote the loss function. Then, the objective is formalized as finding A B such that

3.2 Multi-Round Collaborative Augmentation

To address the challenge, we propose the framework of multi-round collaborative augmentation (MRCA), which exploits the feedback from an instance segmentation model through multiple rounds . Figure 3 shows its overall procedure. MRCA consists of a warm-up stage and N rounds, where each round comprises one or several epochs of the instance segmentation model training. We initially run the warm-up stage by training the instance segmentation model with real data and generating objects with no feedback. After the warm-up stage or a round is completed, the feedback from the instance seg-

Instance

Segmentation

Model

Object

Generator

Warm-up

Round 1

Real Data

Object

Generator

Syn Data

Round N

Object

Generator

Syn Data

Final Result

Trained

Model

Syn Data

Training

Feedback

Map-Paste(·)

Instance

Segmentation

Model

Instance

Segmentation

Model

mentation model is applied to both Gen( · ) and Map-Paste( · ) of the next round. Therefore, the instance segmentation model and the object generator collaboratively augment the training data. See Appendix A.1 for the pseudocode of this augmentation pipeline.

For every round, the training of the instance segmentation model and the generation of image objects are done simultaneously on multiple GPUs. However, the training is dependent on the generation since the generated objects are required for the augmentation of the training set. To overcome the delay caused by the dependency, the generated objects are passed to the next round for training. In summary, at the round r , we generate objects with the feedback from the previous round r -1 and provide the generated objects to the next round r +1 .

Budget Constraint We set the generation budget B equal to the maximum number of generations possible in the training time of each round. By doing so, both the generation and training steps can overlap on different GPUs and thus are processed without delays.

Collaborative Augmentation Given the budget constraint, our main idea is to leverage feedback from the instance segmentation model M during the augmentation process. The intuition is that the

model M holds valuable information that can help generate the most useful objects in Gen( · ) and guide optimal placement during Map-Paste( · ), thereby using the budget more effectively. Employing model feedback for generative data augmentation has been shown effective in classification tasks by directly aligning augmentation with the model's objective as in previous studies [1, 34]. We formalize this widely-accepted idea as Assumption 3.1 with specific conditions.

Assumption 3.1 (Collaborative Augmentation Improves Quality) . Let A denote a data augmentation applied to a dataset D , and let f ϕ denote the feedback from a model M ϕ trained on the dataset D . Then, under the conditions of ( C 1 ) sufficient model capacity, ( C 2 ) controlled distribution shift, and ( C 3 ) proper feedback direction, the augmentation guided by f ϕ has higher quality than the unguided augmentation. That is,

Conditions for the Assumption We first define the information metric I D ( M ϕ ) as a measure of how well the model M ϕ captures the distribution of the dataset D . It can be measured by the average accuracy, i.e., I D ( M ϕ ) = 1 |D| ∑ ( x, y ) ∈D 1 | y | ∑ y ∈ y 1 [ ∃ ˆ y ∈ M ϕ ( x ) s.t. IoU ( y, ˆ y ) ≥ 0 . 5] .

( C 1 ) Sufficient model capacity: Model capacity can be represented by the empirical risk R emp , which is defined as the average loss L over the dataset D . Formally, R emp ( M ϕ ) = 1 |D| ∑ ( x i , y i ) ∈D L ( M ϕ ( x i ) , y i ) . To ensure that the model retains available learning capacity, the

empirical risk should be decreasing over time, i.e., d R emp ( M ( t ) ϕ ) dt < 0 .

( C 2 ) Controlled distribution shift: The knowledge gained through augmentation should surpass the adverse effect of distribution shift caused by the augmentation. Formally, given the data distribution

p D and the model's learned distribution p ( t ) M ϕ over time t , dD KL ( p D || p ( t ) Mϕ ) dt < 0 .

( C 3 ) Proper feedback direction: Given the information from the model M ϕ on the dataset D , the feedback f ϕ should properly guide the augmentation process toward quality maximization. For example, the feedback should be applied in the direction that increases diversity and uncertainty of the dataset (proper direction), not decreasing them (improper direction). That is, [ I D ( M ϕ ) ≥ I D ( ∅ )] ∧ ( C 3) → Q ( A ( D ; f ϕ )) ≥ Q ( A ( D )) .

Under the sufficient model capacity and controlled distribution shift conditions, the average accuracy of the model M ϕ on the dataset D increases through training according to the definitions. That is, ( C 1) ∧ ( C 2) →I D ( M ϕ ) ≥ I D ( ∅ ) . Combining it with the proper feedback direction condition ( C 3) , we obtain the overall guarantee that Q ( A ( D ; f ϕ )) ≥ Q ( A ( D )) holds under these conditions.

Similarly, as the model is trained by the data with higher quality of augmentation, the quality of the feedback from the model will increase accordingly. We incrementally formalize this concept as Lemma 3.2.

Lemma 3.2 (Collaborative Augmentation with a Further Trained Model Improves Quality) . Let A be a data augmentation applied to a dataset D , and let f ϕ r be the model feedback obtained from a model trained for r rounds. Then, under Assumption 3.1 with the three conditions ( C 1 , C 2 , and C 3 ), the augmentation guided by a further trained model f ϕ r provides higher quality than the one guided by f ϕ r -1 . That is,

Proof. By Assumption 3.1, under sufficient model capacity ( C 1 ) and controlled distribution shift ( C 2 ), I D ( M ϕ r ) ≥ I D ( M ϕ r -1 ) . Also, more information guides augmentation of higher quality under proper feedback direction ( C 3 ), so ( C 1) ∧ ( C 2) ∧ ( C 3) → Q ( A ( D ; f ϕ r )) ≥ Q ( A ( D ; f ϕ r -1 )) .

Multi-Round Augmentation We further claim that applying feedback in multiple rounds is superior to a one-time feedback under certain conditions. The high-level idea is that iterative augmentation incorporates feedback from a recently trained model instance, ensuring that the augmented data aligns with the model's evolving needs. We formalize this statement in Theorem 3.3.

Theorem 3.3 (Multi-Round Collaborative Augmentation Improves Quality) . Let A be a data augmentation applied to a real dataset D 0 = D real , and let f ϕ r denote the model feedback obtained

from a model M ϕ r trained on the dataset available at the round r . Consider a multi-round augmentation, where augmentation is performed over N rounds. The augmented dataset at the round r with cumulative budget r N B is defined as

Then, multi-round collaborative augmentation has higher quality compared to a single-round collaborative augmentation. That is,

Proof. The detailed proof is provided in Appendix B.

To ensure that the theoretical claims hold in practice, we specifically design MRCA to keep the assumption better. First, we run the training process until the model validation accuracy saturates to verify that the current round has more information than the previous round, I D ( M ϕ r ) ≥ I D ( M ϕ r -1 ) . While this statement is usually true since the model learns from newly generated objects, learning on only a portion of the dataset sometimes leads to biases and information loss. Therefore, we train the model for multiple epochs between rounds to incorporate the entire dataset. Second, the feedback f ϕ r itself should be properly applied to the augmentation process toward the direction that maximizes the quality of augmentation. Since the mechanism of how feedback improves augmentation quality solely depends on its design, we explain the details of how each feedback guides the augmentation process toward quality maximization in the next section.

3.3 Feedback on Object Generation

Class-Wise Budget Optimization Class-wise budget optimization dynamically adjusts the number of synthetic objects for each class based on the model's state. Specifically, the number of objects to create for a class c ∈ { 1 , · · · , c max } at the round r , B r ( c ) , is computed as

where C r,c is the classifier weight, A r,c is the validation accuracy, and S r,c is the number of objects in the augmented training set. The scale factor α r scales the budget for each round so that the sum of B r ( c ) is equal to B . Eq. (6) targets generation on classes with low accuracy, low coverage, and scarce data. It is straightforward how budget is weighed on classes with low validation accuracy and a low number of training instances for sample efficiency. The classifier weight C r,c is defined as the weight vector corresponding to a class c in the final classification weight matrix at the round r . The classifier weight difference term ∥ C r -2 ,c -C r -3 ,c ∥ reflects how much the class has learned between the rounds, indicating room for improvement. Therefore, we provide more budget to classes with larger weight differences to maximize the learning efficiency.

The two-round gap between the left-hand term and the right-hand term in Eq. (6) results from two factors. One round gap occurs because the generation uses metrics from the model trained on the previous round. Another round gap is required to earn time for generating objects. This intentional gap allows efficient pipelining of generation with feedback, resulting in minimal halts.

Feedback-Guided Object Generation To imbue each object with richer class information, we incorporate feedback from the instance segmentation model M ϕ into the diffusion process. Specifically, the diffusion sampling is steered by the gradients of M ϕ with respect to our chosen criterion C . By using entropy as the criterion, we synthesize objects that become progressively more difficult for the model to learn over successive rounds. Mathematically, the diffusion process is conditioned as

where γ is the scale of classifier guidance and ω is the scale of feedback guidance. Compared to previous work [1] on a classifier model, computing feedback guidance from an instance segmentation model requires an additional technique. A general instance segmentation model produces multiple objects' bounding boxes and segmentation masks from a given image which cannot be directly used for guidance. Fortunately, we aim to generate a single object per image in our method. Therefore, we set the whole image as the box proposal and compute the remaining layers of the instance segmentation model. The detailed algorithm is provided in Appendix A.2.

3.4 Feedback on Map-Paste of Objects

Recall the two stages of the Map-Paste process defined in Section 3.1. The first stage is Map , where we create a set of mappings P between o ∈ O syn and ( x, y ) ∈ D real . The second stage is Paste , where each mapped object in ( o, ( x, y )) ∈ P is scaled by s , which is adaptively determined by class accuracy, and is pasted onto the mapped image to create the final augmented image ( x ′ , y ′ ) . We aim to maximize the learning of the instance segmentation task through optimizing each stage.

Quota-Balanced Unique Mapping Previous studies [37, 41] adopt a purely random assignment of synthetic objects to real images, P = { ( o, ( x, y )) | ∀ ( x, y ) ∈ D real , o ∼ Uniform ( O syn ) } . While random mapping serves as a strong baseline, it becomes suboptimal under a tight object generation budget. Furthermore, random mapping hinders our class-wise budget optimization since some generated objects may never be selected.

To address these limitations, we introduce quota-balanced unique mapping . In this scheme, once an object o is drawn from the synthetic object pool O syn , it is removed so that each synthetic object contributes exactly once toward its class quota. When O syn is exhausted, it is refilled with the initially generated object pool. This scheme ensures that, across the entire augmentation process, each class is represented according to its budget ratio prescribed in Section 3.3.

Accuracy-Based Object Resizing When pasting an object, existing approaches [37, 41] draw object size from the distribution of each class in the real dataset, i.e., s ∼ N ( µ c , σ 2 c ) , where µ c and σ 2 c are the mean and variance of objects of a class c in D real . While such an approach helps match the overall scale between synthetic and real objects, it fails to boost learning for rare or low-accuracy classes, since their typically smaller object sizes remain hard to distinguish and learn from.

To remedy this imbalance, we introduce a resizing scheme based on the validation accuracy of each class. Specifically, objects from low-accuracy classes are scaled larger to enhance the visibility of their features, thereby facilitating better feature extraction. In contrast, objects from high-accuracy classes are generated at smaller scales to reduce their influence during training and prevent the model from overfitting to already well-represented classes. This simple yet effective strategy balances the contribution of each class by providing richer visual cues for underrepresented classes, enhancing the model's ability to recognize rare objects. The detailed algorithm is provided in Appendix A.3.

4 Experiments

4.1 Experiment Settings

Datasets and Implementation Details Instance segmentation and object detection experiments are mainly conducted on the LVIS v1.0 dataset [18] as in the relevant literature, with supporting experiments on the Pascal VOC dataset [10] and the Open Images V5 dataset [24]. CenterNet2 [39], with a ResNet-50 [20] backbone, is used as our main instance segmentation model, implemented in Detectron2 [33]. The Swin-L [27] backbone is also used for comparison with baseline methods. Stable Diffusion 3 Medium [9] is used as our generation model, and BiRefNet [38] is used for segmentation on the generated objects. After generation or segmentation, CLIP [29] is used to filter out the low-quality objects whose score is smaller than 0 . 25 .

Our model is trained for 10 rounds, with each round consisting of 9,000 iterations and a batch size of 16. Our multi-round collaborative augmentation serves as the sole modification to the baseline training pipeline. We evaluate performance using average precision (AP) for both bounding box detection and instance segmentation while also analyzing results across different class frequencies ( rare , common , and frequent classes) as defined in LVIS. We use 8 NVIDIA GeForce RTX 3090 GPUs, where 4 GPUs are used for training the model, 3 GPUs for generating objects, and 1 GPU for segmentation. Only for the Swin-L experiment, we use NVIDIA A40 GPUs instead to fit the model in the GPU memory. To run both training and generation without waiting for another process, the number of object generations per round is empirically set to 6 × 1203 (number of classes in LVIS). Besides, γ and ω in Eq. (7) are set to 5.0 and 0.03, respectively. Further details about the experiments are provided in Appendix C.

Compared Methods We compare our MRCA with strong generative copy-paste methods, including X-Paste [37], BSGAL [41], and DiverGen [11]. The numerical results of the baselines are borrowed from the BSGAL and DiverGen papers owing to the same experiment settings.

4.2 Main Experiment Results

Table 1 and Table 2 show the mean average precision (mAP) on the instance segmentation and object detection tasks on the LVIS v1.0 dataset for the ResNet-50 [20] and Swin-L [27] backbones. The superscripts 'box' and 'mask' denote the mAP for object detection and instance segmentation, respectively. The subscripts 'r', 'c', and 'f' signify rare, common, and frequent classes, respectively. Overall, MRCA outperforms the state-of-the-art generative augmentation methods while generating only 6% as many synthetic objects. These results indeed demonstrate the sample efficiency of MRCA, because its multi-round collaborative augmentation delivers the most essential training data at the appropriate time. Furthermore, the performance improvement for rare classes is more pronounced than for other classes, because the rare classes receive superior treatment due to increased generation budgets (§3.3 ) and expanded space (§3.4). See Appendix D for augmented image examples.

4.3 Ablation Study and Further Analysis

Effect of the Feedback on Generation and Map-Paste The ablation study is conducted on the feedback components: class-wise budget optimization, model gradient-based generation, and object resizing. We apply each component individually and collectively in a multi-round

manner to confirm its efficacy. Table 3 shows that each individual component contributes to performance gain without interfering with other components.

Effect of the Class-Wise Budget Optimization

Components We also conduct a fine-grained ablation study on class-wise budget optimization while keeping other main methods (model gradient and resizing) enabled, as shown in Table 4. When using only the number of objects, we observe notable improvements for rare classes.

The validation accuracy term further supports common and frequent classes with low accuracy. Nonetheless, some low-accuracy classes continue to produce similar objects across rounds. To mitigate this issue, we incorporate the classifier weight, which reflects whether gradient feedback is being updated over rounds. As evidenced by the improvements in the result, both validation accuracy and classifier weight contribute effectively as intended.

Effect of the Generation Model In Table 5, we compare the performance when using different generative models, Stable Diffusion 1.5 [30] and Stable Diffusion 3 Medium [9]. The hyperparameters of the diffusion models are set equal, with classifier guidance scale as 5.0, generation

step as 30, and image resolution as 512 × 512. The recent version of Stable Diffusion generates higher-quality objects, leading to better results. When using the same version, Stable Diffusion 1.5, the superiority of MRCA over BSGAL is still maintained in terms of the mAP for all and rare classes.

Effect of the Number of Rounds We investigate the effectiveness of applying feedback in a multi-round manner in Table 6. The number of training rounds is adjusted from 1 to 30, given a fixed number of total training iterations. As shown in Theorem 3.3, when the number of rounds increases from 1 to 10, the quality of instance segmentation and object detection improves, indicating that the quality of the augmentation improves accordingly. Over 10 rounds, however, the performance drops because the model does not learn sufficiently between the short training rounds; that is, a single round is unable to cover a whole epoch over in such conditions.

Effect of the Curriculum Learning The proposed MRCA generates objects and trains with the curriculum divided by rounds. To check the effect of the curriculum learning strategy, we separately train a model from scratch given the

objects generated over all rounds of MRCA. Table 7 shows the difference depending on the existence of the curriculum learning. Although training without the curriculum has more object quantity at the beginning, the performance becomes lower in most cases. This result suggests that while the objects generated by MRCA can be used for the input to any training pipeline, the optimal strategy is to use the curriculum learning strategy provided by our method.

Analysis on the Generation Budget In previous experiments, we empirically chose a generation budget B of 72k to efficiently pipeline our generation and training processes, ensuring no bubbles. This additional test examines the

effect of larger budgets on the mAP as well as the training time excluding the warm-up stage. Table 8 shows that the training time is proportional to the number of object generations, because multi-round training needs to wait for the object generation to be used for augmentation. Only a small performance improvement is observed with an increased generation budget, decreasing the sample efficiency. This limitation of low scalability is further discussed in Appendix E.

Analysis on the Feedback Guidance Scale The feedback guidance scale, ω in Eq. (7), has a significant impact on the image quality. Table 9 supports that feedback guidance can largely affect the generated image quality. The default value ω = 0 . 03 usually offers the best performance. The performance drops significantly for

a large ω because excessively large model guidance overwhelms the trained diffusion denoising gradients. Its effective range can be quickly identified with qualitative analysis, as a high ω produces unnatural artifacts, while a low ω results in images nearly identical to unguided outputs.

Analysis on the Classifier Guidance Scale Table 10 presents the effect of the classifier guidance scale, γ in Eq. (7). A higher γ amplifies the influence of the conditioning signal (e.g., text prompt or class label) on the generation process, which strengthens fidelity and prompt alignment but pushes samples toward a narrower region of

the data distribution, thus reducing diversity among outputs. Conversely, when γ is too low, the diffusion model behaves closer to its unconditional form. Therefore, the generated samples exhibit higher variability but lower fidelity and semantic accuracy, leading to visibly degraded quality. The default value of 5 . 0 , which has been adopted in prior studies [37, 41], is also found to be effective in our experiments by stabilizing the tradeoffs.

Experiments on Additional Datasets We conduct only object detection on Pascal VOC 2012[10] because the annotations only support semantic segmentation. For Open Images V5[24], due to the large size of the dataset, we use a pareto sampling method from previous work [32] to create its long-tailed version.

Again, MRCA boosts the performance through our novel data augmentation pipeline in these datasets.

4.4 Qualitative Analysis

Round-Wise Object Visualization and Entropy Figure 4 visualizes a portion of generated objects and their average entropy for each round. As shown in Figure 4a, the generated objects maintain their class characteristics over all rounds. We assess the entropy of objects with respect to two instances of the model: (1) the warmed-up model instance, which is trained solely on real training data, and (2) the feedback model instance, which is the model that guides the generation of the synthetic images in the corresponding round. Given that the guidance scale is constant, the entropy (uncertainty) respect to the feedback model M ϕ r remains similar across rounds, as shown in Figure 4b. However, from the perspective of the initial model M ϕ 0 , the feedback received from more advanced models in subsequent rounds results in the generation of increasingly uncertain or challenging objects, as depicted in Figure 4c. This tendency is also evident when examining the average entropy across all classes in Figure 4d. Overall, MRCA forms an effective easy-to-hard curriculum.

Dove

Dragonfly

Dustpan

0

1

2

3

4

5

6

7

8

9

Round

2.2

2.4

2.6

2.8

3.0

3.2

Dove

Dragonfly

Dustpan

0

1

2

3

4

5

6

7

8

9

Round

2.2

2.4

2.6

2.8

3.0

3.2

Dove

Dragonfly

Dustpan

0

1

2

3

4

5

6

7

8

9

Round

2.4

2.5

2.6

2.7

Entropy via Warm-up Model

Entropy via Feedback Model

(b) Mean entropy of objects respect to the feedback model ( M ϕ r ).

(c) Mean entropy of objects respect to the warm-up model ( M ϕ 0 ).

(d) Class average entropy respect to the warm-up and feedback models.

Figure 4: Examples of generated objects for each round (a) , with their class-wise entropy respect to the feedback model (b) , the warm-up model (c) , and their comparison over average of all classes (d) .

5 Conclusion

The proposed multi-round collaborative augmentation (MRCA) framework enables generating objects that are continuously more informative to the instance segmentation model with a considerably small amount of data generations (e.g., 6%). The instance segmentation model consistently guides the diffusion generation process to create objects that elicit greater uncertainty from the model, hence generating more informative samples, while employing class-wise budget optimization to determine the appropriate number of generations for each class. Then, accuracy-based scaling effectively places the generated objects in real images to focus on learning of rare classes. Experiment results demonstrate that MRCA outperforms the state-of-the-art generative copy-paste augmentation methods for instance segmentation. In conclusion, we assert that the MRCA framework presents a highly promising avenue for enhancing the performance of instance segmentation.

References

Reyhane Askari-Hemmat, Mohammad Pezeshki, Florian Bordes, Michal Drozdzal, and Adriana RomeroSoriano. Feedback-guided data synthesis for imbalanced classification. arXiv preprint arXiv:2310.00158 , 2023.

Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 843-852, 2023.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the International Conference on Machine Learning (ICML) , pages 41-48, 2009.

Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(5):1483-1498, 2019.

Kai Chen, Enze Xie, Zhe Chen, Yibo Wang, Lanqing Hong, Zhenguo Li, and Dit-Yan Yeung. Geodiffusion: Text-prompted geometric control for object detection data generation. arXiv preprint arXiv:2306.04607 , 2023.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) , 34:8780-8794, 2021.

Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion-based augmentation. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) , 36:79024-79034, 2023.

Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy synthesis for instance detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 1301-1310, 2017.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the International Conference on Machine Learning (ICML) , 2024.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision , 88:303-338, 2010.

Chengxiang Fan, Muzhi Zhu, Hao Chen, Yang Liu, Weijia Wu, Huaqi Zhang, and Chunhua Shen. Divergen: Improving instance segmentation by learning wider data distribution with more diverse generative data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3986-3995, 2024.

Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7382-7392, 2024.

Haoyang Fang, Boran Han, Shuai Zhang, Su Zhou, Cuixiong Hu, and Wen-Ming Ye. Data augmentation for object detection via controllable diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1257-1266, 2024.

Chengjian Feng, Yujie Zhong, Zequn Jie, Weidi Xie, and Lin Ma. Instagen: Enhancing object detection by training on synthetic dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14121-14130, 2024.

Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2918-2928, 2021.

Ross Girshick. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 1440-1448, 2015.

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 580-587, 2014.

Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5356-5364, 2019.

David S Hayden, Mao Ye, Timur Garipov, Gregory P Meyer, Carl Vondrick, Zhao Chen, Yuning Chai, Eric Wolff, and Siddhartha S Srinivasa. Generative data mining with longtail-guided diffusion. arXiv preprint arXiv:2502.01980 , 2025.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 2961-2969, 2017.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) , 33:6840-6851, 2020.

Ilya Krylov, Sergei Nosov, and Vladislav Sovrasov. Open images v5 text annotation and yet another mask text spotter. In Proceedings of the Asian Conference on Machine Learning , pages 379-389, 2021.

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 22511-22521, 2023.

Yijun Liang, Shweta Bhardwaj, and Tianyi Zhou. Diffusion curriculum: Synthetic-to-real generative curriculum learning via image-guided diffusion. arXiv preprint arXiv:2410.13674 , 2024.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 10012-10022, 2021.

Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin. Libra r-cnn: Towards balanced learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 821-830, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML) , pages 8748-8763, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Tong Wu, Qingqiu Huang, Ziwei Liu, Yu Wang, and Dahua Lin. Distribution-balanced loss for multi-label classification in long-tailed datasets. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 162-178, 2020.

Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https: //github.com/facebookresearch/detectron2, 2019.

Teresa Yeo, Andrei Atanov, Harold Benoit, Aleksandr Alekseev, Ruchira Ray, Pooya Esmaeil Akhoondi, and Amir Zamir. Controlled training data generation with diffusion models. arXiv preprint arXiv:2403.15309 , 2024.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 6023-6032, 2019.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 38363847, 2023.

Hanqing Zhao, Dianmo Sheng, Jianmin Bao, Dongdong Chen, Dong Chen, Fang Wen, Lu Yuan, Ce Liu, Wenbo Zhou, Qi Chu, et al. X-paste: Revisiting scalable copy-paste for instance segmentation using clip and stablediffusion. In Proceedings of the International Conference on Machine Learning (ICML) , pages 42098-42109. PMLR, 2023.

Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral reference for high-resolution dichotomous image segmentation. arXiv preprint arXiv:2401.03407 , 2024.

Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Probabilistic two-stage detection. arXiv preprint arXiv:2103.07461 , 2021.

Jingyuan Zhu, Shiyu Li, Yuxuan Andy Liu, Jian Yuan, Ping Huang, Jiulong Shan, and Huimin Ma. Odgen: Domain-specific object detection data generation with diffusion models. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) , 37:63599-63633, 2025.

Muzhi Zhu, Chengxiang Fan, Hao Chen, Yang Liu, Weian Mao, Xiaogang Xu, and Chunhua Shen. Generative active learning for long-tailed instance segmentation. In Proceedings of the International Conference on Machine Learning (ICML) . PMLR, 2024.

A Algorithm Pseudocode

A.1 Multi-Round Collaborative Augmentation Pipeline

The overall procedure of MRCA is explained in Algorithm 1, composed of two phases: the warm-up phase and the multi-round phase. At the warm-up phase, the instance segmentation model M ϕ is trained with real training data in parallel with the generation of objects by G . When both processes terminate, the multi-round phase starts. Similar to the warm-up phase, the multi-round phase also undergoes parallel training and generation. The difference lies in applying the feedback from the previous round's model to both object generation and training data augmentation. For the final round, the object generation does not proceed since there are no more training rounds left to use them. Eventually, MRCA returns the trained instance segmentation model and the set of generated objects.

Algorithm 1 Multi-round collaborative augmentation pipeline

̸

A.2 Diffusion with Feedback from the Instance Segmentation Model

The feedback-guided diffusion-based object generation is explained by two hierarchical algorithms. At a higher level, Algorithm 2 describes where the feedback is applied in the diffusion process. A general diffusion process is composed of multiple denoising steps, where the scheduled denoising is applied sequentially. We apply the feedback guidance selectively after the scheduled denoising to keep the uncertainty in range so that the generated images remain in the context of the input class. In our experiments, the feedback guidance is applied every 5 steps out of 30 steps, for a total of 6 times per image. The hyperparameter ω in Line 6 is set to 0.03.

Algorithm 2 Denoising the diffusion model with a criterion function

Algorithm 3 explains in more detail how the instance segmentation model and the criterion function compute the gradients for the diffusion model. The key idea is setting the whole image as the object proposal for extracting the features, since we generate a single object per image. Lines 2-5 explain the criterion function computation line by line.

Algorithm 3 Applying the criterion function to the instance segmentation model

1: Given intermediate stable diffusion model latent state z ′ t -1 , stable diffusion vae decoder v dec , instance segmentation model M ϕ consisting of backbone(feature extractor) b ϕ and classifier c ϕ , and criterion function C

2: ˜ x t - 1 = v dec ( z ′ t - 1

) ▷ Create an image from latent variable

3: feat = b (˜ x -

ϕ t 1 ) ▷ Extract features from detector's backbone

4: logits = c ϕ ( feat ( prop

)) ▷ With whole image as proposal, classify corresponding features

5: grads = C ( logits, z ′ )

t - 1 ▷ Calculate gradients respect to criterion function

6: Return gradients on latent state grads

A.3 Object Resizing Based on Class-Wise Evaluation

Algorithm 4 provides a more detailed explanation of the resizing method. Given the average precision A c of a specific class, we assign a scale factor to a class, which will cause its objects to be resized in accordance with the scale factor. For three values of A c , 0%, ¯ A (mean), and 100%, A c = 0% → scale = s max , A c = ¯ A → scale = 1 , A c = 100% → scale = 1 s max . For other values of A c , we linearly interpolate in log scale.

Algorithm 4 Computing a scale factor based on average precision (AP)

1: function COMPUTE_SCALE_FACTOR( A c , ¯ A )

2: Input: A c (Average precision of a class c ), ¯ A (Mean average precision of all classes), s max (Maximum scale)

3: Output: Computed scale factor

4: log_scale_min ← ln( s 2 max )

5: log_scale_max ← ln ( 1 s 2 max )

6: if A c < ¯ A then

7: log_scale ← log_scale_min ( ¯ A -A c ¯ A )

8: else

9: log_scale ← log_scale_max ( A c -¯ A 100 -¯ A )

10: end if

11: return scale = exp( log_scale )

12: end function

B Proof of Theorem 3.3

We begin by expressing the quality of the final dataset in the multi-round setting

Consider the 1 N B objects created at a round r . Since the feedback from the recent model f ϕ r -1 is more informative than f ϕ r -2 , I D ( M ϕ r -1 ) ≥ I D ( M ϕ r -2 ) , by Lemma 3.2,

Similarly, continuing for 1 N B objects at rounds r -1 , r -2 , · · · , 1 ,

▷ Log scale values for interpolation

▷ Scale at A c = 0%

▷ Scale at A c = 100%

▷ Piecewise linear interpolation in log scale

By multiplying all inequalities from Eq. (9) to Eq. (11),

C More Details for Experiments

Object Edge Smoothing When pasting objects, we do not employ any methods for smoothing object edges, such as Gaussian blurring or alpha/Poisson blending. Based on previous work [15] and our own evaluation, the edge handling methods have no significant effect on the overall performance.

Reason for Using the ResNet Backbone Since the overall performance trend remains consistent across different backbone models (ResNet and Swin-L), we choose a lighter model to enable more extensive ablation studies within a given time. Notably, the performance improvement reported in Table 1 and Table 2 is even larger when using the Swin-L backbone, supporting that our ResNetbased ablation results do not overstate the effectiveness of MRCA. The ResNet backbone is more memory-efficient and can be trained on a 24GB RTX 3090, whereas the Swin-L backbone requires a higher-memory setup, such as a 48GB A6000 or larger. We believe that both models are important, as they represent a trade-off between model size and performance, which is an important consideration for practical deployment, particularly in on-device scenarios.

Overlap of Augmented Objects The augmented objects can be overlapped in our experiment setting. When an object is overlapped, its object mask is adjusted to eliminate the obscured portion. This scheme is a common practice in the relevant studies [7, 37, 41].

Unrealistic Augmented Objects Our approach follows the line of research established by XPaste [37], BSGAL [41], and DiverGen [11], which leverage synthetically generated, often unrealistic, map-pasted images to improve performance. While the augmented images may not share the same distribution as the original images in terms of object size, they still contribute positively to instance segmentation performance by incorporating knowledge from the generation model. The MRCA framework further enhances this effect by explicitly optimizing both object generation and pasting strategies. It is worth noting that the use of visually unrealistic images for data augmentation has been widely observed to be effective in improving downstream performance. For instance, also in image classification, CutMix [35] often produces implausible examples such as images featuring a cat's head on a dog's body.

D Visualization of Augmented Images

We visualize some of the augmented images in Figure 5. During training, we paste 1-3 generated objects onto an image from the real LVIS training set. We keep the number of pasted objects small to minimize the deviation in the average number of objects per image compared to real images. The sizes of the objects are managed with our accuracy-based resizing scheme (§3.4). For example, in the first image, the class 'cockroach' has low accuracy, and therefore an object of that class is pasted with a larger size compared to its actual size from real images.

E Limitations

The current implementation of MRCA shows low scalability, meaning that generating more objects with the current framework does not lead to a large increase in performance. This downside is due to our method's focus on uncertainty, which is effective in generating small-scale datasets since there is a low chance of overlap between objects. However, generating large-scale datasets without a diversity-focused method may generate similar objects, resulting in low scalability. Increasing the diversity of the generated objects effectively addresses the scalability issue, as reducing overlap between objects allows for the acquisition of more information. DiverGen [11], a prior study that focuses on the diversity of the generated dataset, supports our earlier argument with scalable results.

122

L69

684

654

697

697

630

630

295

295

360

704

1116

631

697

697

1024

Poile

153

153

153

149

782

49

281

1054

844

697

630

707

1044

1149

1036

Furthermore, the current framework only works with a single generator to efficiently use feedback. Previous studies, such as BSGAL [41] and DiverGen [11], demonstrate that using multiple generation models for generating objects can enhance data quality by increasing diversity. Therefore, employing multiple generation models in our multi-round augmentation framework remains a future task. Such an approach can increase diversity in another dimension, as well as solve the scalability issue.

We also note that scaling of rare classes can lead to undesirable effects in domains where object size carries critical contextual meaning. For instance, in medical imaging or other domains with consistent scales, resizing objects may distort important spatial cues and disrupt the natural size distribution. To mitigate such risks, resizing strategies should ideally be avoided or applied only within narrow bounds when using MRCA in these settings. However, in real-world image datasets such as LVIS [18], PASCAL[10], and OpenImages [24], where object scales vary greatly even within the same category, we did not observe notable negative impacts on detection performance. By carefully constraining the maximum scaling coefficient, we ensured that objects did not become unnaturally large or small while maintaining overall realism. Moreover, as the representation of rare classes improves through successive training rounds, their resulting scale distribution tends to align more closely with natural proportions, further reducing the potential adverse effects of scaling.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims are clearly provided in the abstract and introduction of the paper with its scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are provided at a separate appendix section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions and theorems are completely provided.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the details are provided in the experiment section and the attached appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code for data generation and training and testing are provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training and test details are provided in the experiment section and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: This paper does not provide error bars, but we conduct extensive experiments over many ablation settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resources and their training time are provided in the experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: This paper is conducted under the guidance of Code Of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The societal impact of this paper is intractable.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper do not release data or models that pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: This paper uses LVIS v1.0, Pascal VOC 2012, OpenImagesV5 datasets, Stable Diffusion 1.5 and 3 generation models, Detectron2 and X-paste code, and properly cite them.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper do not introduce any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subject.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subject.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: This paper does not involve LLM as our core method development.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy (https://neurips.cc/Conferences/2025/LLM) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multi-round collaborative augmentation (MRCA) framework for long-tail instance segmentation. By integrating feedback from an instance segmentation model into both object generation and placement, MRCA adaptively synthesizes challenging and informative training samples. It uses diffusion models guided by model uncertainty and class-wise accuracy, and dynamically adjusts generation budgets. Experiments on LVIS show that MRCA achieves superior performance with only 6% of the synthetic data used by prior methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n\n1. The paper is well-organised with clear structure.\n2. The paper integrates two-fold model feedback (class-wise budget optimization + gradient-guided generation) in a multi-round loop, which is a principled extension of curriculum learning to generative augmentation.\n3. The paper effectively disentangles contributions of each feedback component (budget optimization, gradient-based generation, resizing).\n\n## Weaknesses\n\n1. Theoretical analysis is superficial. Assumption 3.1 merely states that feedback from a trained model improves augmentation quality, but does not formalize under what conditions this holds (e.g. model capacity limits, data distribution shifts). Furthermore, The paper references information gain $\\mathcal{I}\\_{\\mathcal{D}}(M\\_{\\phi})$ but does not define how it is computed or measured in practice.\n2. Feedback guidance hyperparameter sensitivity. Performance is sensitive to the feedback guidance scale (ω). While the paper provides analysis, tuning such parameters can be non-trivial in practice.\n\nQUESTIONS:\nHow do you ensure that the augmented data and the original data share the same distribution? From my perspective, the map-pasted pictures seem not quite realistic.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Multi-Round Collaborative Augmentation (MRCA), a novel generative data augmentation framework designed to enhance sample efficiency for long-tail instance segmentation tasks. MRCA addresses the inefficient utilization of pre-generated data by incorporating feedback from an instance segmentation model to guide the augmentation process. It dynamically adjusts the number and size of synthetic objects for each class based on the model's state, improving learning for underrepresented classes. This multi-round process allows feedback to be refined throughout training , enabling MRCA to outperform state-of-the-art methods while requiring only 6% of their data generation.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths\n\n1. **Methodological Soundness and Intuitive Design:** The proposed Multi-Round Collaborative Augmentation (MRCA) framework is conceptually strong and highly intuitive. By incorporating feedback from the instance segmentation model to guide the generative process, MRCA ensures that synthetic data is optimally tailored to the model's evolving needs, which makes inherent sense for improving learning efficiency. The dynamic adjustment of object generation and resizing based on model uncertainty and class accuracy directly targets areas of weakness, providing a logically sound approach to data augmentation.\n2. **Empirical Validation and Significant Performance Gains:** The paper provides compelling experimental evidence demonstrating MRCA's effectiveness. Notably, MRCA outperforms state-of-the-art generative augmentation methods on the LVIS v1.0 dataset while requiring only 6% of the data generation. This substantial reduction in generated objects coupled with superior performance, particularly in challenging rare classes, clearly validates the method's sample efficiency and practical utility.\n3. **Exemplary Presentation and Clarity:** The paper is exceptionally well-structured and clearly written, enhancing its readability and impact. The figures, such as Figure 1 illustrating the novelty of MRCA and Figure 4 visualizing round-wise object generation and entropy, are highly informative and visually clear, effectively conveying complex ideas. The use of tables to present detailed experimental results and ablation studies further contributes to the paper's comprehensiveness and allows for easy understanding of the contributions of individual components.\n\nweakness\n\n1. **Limited Impact of Core Feedback Mechanisms Compared to Simpler Design:** Table 3  in the ablation study suggests that the more complex feedback mechanisms, such as \"Class Budget\" and \"Model Gradient,\" contribute less to overall performance improvement than the seemingly simpler \"Resizing\" component. This observation raises questions about the relative efficacy and necessity of intricate feedback loops, as the straightforward resizing based on accuracy  appears to be a more dominant factor in the gains. Further analysis or alternative designs for the feedback-guided generation could be explored to maximize the impact of these sophisticated components.\n2. **Insufficient Granularity in Ablation of Equation (6) Components:** The paper's current ablation study does not fully explore the individual contributions of the terms within Equation (6), which governs the Class-Wise Budget Optimization. Specifically, it would be beneficial to ablate the impact of including or excluding components like the classifier weight difference term and the validation accuracy . A more detailed breakdown would provide clearer insights into which specific feedback signals are most influential for dynamic budget allocation and why, thereby strengthening the empirical foundation of this crucial part of the methodology.\n\nQUESTIONS:\n**Effectiveness of Guidance from Noisy Intermediate States:** The paper's method applies guidance from the instance segmentation model (Mϕ) to noisy intermediate diffusion states. A key question is whether this guidance remains effective early in the diffusion process, when latent states are still heavily dominated by noise. Features extracted from such noisy representations might be unreliable, potentially leading to suboptimal or counterproductive signals for guiding object generation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript presents a collaborative, multi-round approach (MCRA) using feedback from an instance segmentation model, to guide data augmentation after synthesis. This method is claimed to require only a small fraction (about 6%) of the amount of synthetic generated data compared to other current methods, while improving average precision for bounding box and instance segmentation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n - (Quality) Comprehensive ablation experiments isolating effects of feedback, generation model, curriculum learning, guidance scale and generation budget\n - (Originality) Integrates well-known ideas on model feedback and guided learning to the segmentation task\n\n\nWeaknesses\n - (Quality) Ablation on classifier guidance appears missing\n - (Significance) While the requirement for fewer synthetic examples is claimed as a major benefit of MCRA, no ablation analysis on the effect of reduced synthetic examples (i.e. 72k instead of 1200k) is performed for the competing methods; this is important as Table 8 suggests potential early saturation of the impact of additional synthetic examples\n - (Significance) Performance improvements appear task-specific, and findings (e.g. on optimal hyperparameters) may not be fully generalizable\n\nQUESTIONS:\n1. In Section 3.1, it might be clarified as to whether the same generated image (from the frozen generator $G$) may be associated with different segmentation $S$ in different rounds, due to the effect of the evolving segmentation model.\n\n2. In Section 3.1, it might be clarified as to whether there is any processing/blurring (gradient?) of object edges, for the Map-Paste() function.\n\n3. In Section 3.2, the proof of Lemma 3.2 (i.e. \"increased information\" of D necessarily guides augmentation of higher quality) appears somewhat informal. The concept of \"information\" applied here, and how increased dataset information implies improved augmentation, could be explained or derived in greater detail.\n\n4. In Section 3.2, \"as round pass\" might be \"as rounds pass\".\n\n5. In Section 3.3, it is stated that entropy is used as the criterion to steer diffusion sampling. It might be clarified as to how entropy is measured for data instances (also with respect to the warmup and feedback models, as explored in Section 4.4).\n\n6. From Section 4.2 (Tables 1 and 2), the Swin-L backbone significantly outperforms the ResNet-50 backbone. Given this, was there a reason to use the ResNet-50 backbone as the baseline main instance segmentation model, since that could exaggerate the potential gains from MCRA in practical use?\n\n7. For the augmentation examples in Appendix C, it might be clarified as to whether augmented objects can be overlapped.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework called Multi-Round Collaborative Augmentation (MRCA) to tackle the challenges of class imbalance and annotation costs in long-tail instance segmentation. MRCA operates by forming a feedback loop between a diffusion-based generative model and an instance segmentation network, selectively generating additional synthetic objects for classes that are either underrepresented or challenging for the model. By iteratively incorporating these synthetic samples across multiple rounds, MRCA progressively enhances the training dataset. In each round, both the number and size of generated objects are adaptively adjusted according to the current performance of the segmentation model, aiming to maximize sample efficiency. On benchmarks such as LVIS, MRCA achieves higher performance than existing state-of-the-art methods using only 6% as much synthetic data, and ablation studies demonstrate the contribution of each core component.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- Among prior works addressing long-tailed instance segmentation with generative models, directly leveraging feedback from the instance segmentation model to guide the generative process is a fresh and interesting idea. The proposed framework is novel and, to my knowledge, has not been seen in previous literature.\n- The paper conducts thorough ablation studies and analyses for each key component (such as class-wise budget allocation, model gradient feedback, and accuracy-based resizing), which provides valuable insights into the contribution of each part.\n- The authors also provide reproducible code, which is a strong point in terms of transparency and research usability.\n\nWeaknesses: \n- Depending on the relative speeds of instance segmentation training and object generation, the required round gap in Equation 6 may need to be increased. For example, if the segmentation model is lightweight and trains much faster than object generation, a larger gap (e.g., 3 or more rounds) might be needed to prevent the training process from stalling. In this situation, the feedback used to guide object generation would come from an older model rather than the most recent checkpoint. This could reduce the method’s ability to adapt quickly to the model’s current weaknesses, and might weaken the core advantage of feedback-driven augmentation, especially in fast training scenarios. \n- Limitation in accuracy-based resizing strategy is that some rare classes are inherently small in real life (e.g., coin, ant..) Enlarging these objects during training may harm the model’s ability to recognize them in realistic contexts where their small size is important.\n- While the paper shows impressive sample efficiency given a fixed generative budget (e.g., 6%), I wonder whether constraints on synthetic data generation are as significant in practiceas reducing human annotation costs. The importance of being sample-efficient with generative models may not be as high as with real human-labeled data, so I am somewhat uncertain about the broader impact or significance of the proposed efficiency gains in this context.\n\nQUESTIONS:\n1) In cases where the instance segmentation model trains much faster than object generation, have the authors observed any decline in the effectiveness of feedback-driven augmentation when the round gap is increased? \n2) For rare classes that are inherently small in real images, have the authors experienced any negative effects on detection performance from scaling these objects up during training? I’m also curious if the authors have considered or tried any alternative resizing strategies for these cases.\n3) While MRCA demonstrates strong performance with only 6% of the synthetic data compared to previous methods, I am curious whether further improvements are possible if the generation budget is increased to match that of methods like BSGAL or X-Paste+CLIP. (~1200k of #Gen Objects) Have the authors explored the performance of MRCA under higher generation budgets?\n\nLIMITATIONS:\nThe accuracy-based resizing strategy involves artificially enlarging rare or low-accuracy objects rather than preserving their natural size distribution. This may disturb the true data size distribution, which could be particularly problematic in domains like medical imaging, where the physical size of objects often carries important diagnostic meaning. In such cases, this approach might introduce bias or lead to unintended side effects in model training and deployment.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-03"}