Fetch and Forge: Efficient Dataset Condensation for Object Detection

Authors: ding  qi
jian  li
jinlong  peng
bo  zhao
shuguang  dou
jialin  li
jiangning  zhang
yabiao  wang
chengjie  wang
cairong  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:36:31

Abstract: Dataset condensation (DC) is an emerging technique capable of creating compact synthetic datasets from large originals while maintaining considerable performance. It is crucial for accelerating network training and reducing data storage requirements. 
However, current research on DC mainly focuses on image classification, with less exploration of object detection.
This is primarily due to two challenges: (i) the multitasking nature of object detection complicates the condensation process, and (ii) Object detection datasets are characterized by large-scale and high-resolution data, which are difficult for existing DC methods to handle.
As a remedy, we propose DCOD, the first dataset condensation framework for object detection. It operates in two stages: Fetch and Forge, initially storing key localization and classification information into model parameters, and then reconstructing synthetic images via model inversion. 
For the complex of multiple objects in an image, we propose Foreground Background Decoupling to centrally update the foreground of multiple instances and Incremental PatchExpand to further enhance the diversity of foregrounds.
Extensive experiments on various detection datasets demonstrate the superiority of DCOD. Even at an extremely low compression rate of 1\%, we achieve 46.4\% and 24.7\% $\text{AP}_{50}$ on the VOC and COCO, respectively, significantly reducing detector training duration.

Main paper: 
 introduction: 
 In the past decade, the field of deep learning has witnessed the emergence of high-performance models, driven by the development of convolutional and Transformer networks [10, 24, 8, 5] . These models, while benefiting from large-scale datasets, also encounter significant challenges such as data storage and the demand for extensive training resources. Dataset condensation (or distillation) [27, 32, 2] has emerged in response, aiming to synthesize a small amount of data that approximates the training effectiveness of original datasets, thus offering hope to alleviate this dilemma.Current dataset condensation (DC) research mainly focuses on image classification and is divided into two main frameworks: Meta-learning and Data-matching [11] . As shown in Figure 1 (a), the Meta-learning framework [27, 17, 34, 15] optimizes synthetic datasets by minimizing performance risks on the original dataset's validation set, using a bi-level optimization process. In contrast, Figure 1 (b) illustrates the Data-matching framework, which aligns gradients [32] , feature distributions [31] , or training trajectories [2] to simulate the original data's impact during different model training stages. While both frameworks above make significant contributions to the field, they share common limitations. The complexity and computational expense of their bi-level optimization process restrict their application, typically confining them to smaller datasets (e.g., CIFAR10, CIFAR100, Tiny-ImageNet) and simpler network architectures (e.g., ConvNet-3, AlexNet, VGG-11, ResNet-18). This limitation is particularly pronounced in object detection tasks, which are inherently more complex than image classification. Specifically, i) the requirement in object detection for simultaneous localization and classification of multiple objects in an image considerably complicates the condensation process; and ii) the large-scale, high-resolution nature of detection dataset images, along with the complexity of detection network structures, poses significant challenges in extending existing DC methods.In this study, we propose DCOD, the first dataset condensation framework for object detection. DCOD is distinct in its ability to flexibly generate foreground objects of varying sizes, shapes, and categories at any position within an image, as demonstrated in Figure 2 . Additionally, DCOD streamlines the process by eliminating the complexities of traditional bi-level optimization, enhancing its compatibility with complex detection networks and large-scale, high-resolution data. DCOD operates in two stages: Fetch and Forge. As in Figure 1 (c), in the first stage Fetch, we train an object detector on the original dataset following a standard procedure, during which key information from the original data is stored within the detector. Then, in the second stage, we employ the model inversion to synthesize images from the trained detector. Specifically, we freeze the detector's weights and input initialization images and labels (including position, size, and category) randomly sampled from the original dataset to capture the instance distribution of real originals. During the inversion process, we propose the Foreground Background Decoupling (F BD ) to enhance attention to the foreground areas through random erasure guided by a coarse mask and Incremental PatchExpand (I P E ) expands a single image into multiple patches, each guided by different target labels, thus synthesizing a richer variety of instances. Finally, we utilize the detector's loss function as the guiding task loss for dataset condensation. To ensure the quality of the generated images, we implement two types of regularization: pixel-level alignment and feature-level alignment. Our method underwent rigorous evaluation on the MS COCO [13] and Pascal VOC [7, 6] datasets, achieving top-tier results that highlight its significant effectiveness. With some problems left open and a considerable improvement room existing, we hope this pilot study will attract more community interest.The primary contributions of this paper can be summarized as follows:• We propose DCOD, the first dataset condensation framework for object detection datasets.Utilizing a two-stage process of Fetch and Forge, it can simultaneously condense crucial localization and classification information from the original dataset.• For the multi-object distribution characteristic of object detection datasets, we propose the Foreground Background Decoupling strategy and Incremental PatchExpand which notably boost the diversity of multi-instances, all within a constrained storage budget. • Extensive experiments on various detection datasets underscore the superiority of DCOD.Remarkably, even at an extremely low compression rate of 1%, we achieve 46.4% and 24.5% AP50 on the VOC and COCO datasets respectively, significantly reducing the training duration for object detection.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction clearly outline the primary contributions and scope of the paper. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Dataset Condensation. Dataset condensation (or distillation) aims to compress large-scale original data into a small amount of synthetic data, accelerating network training while maintaining comparable performance. Current research primarily focuses on the field of image classification and can be divided into two frameworks: meta-learning and data matching. In the meta-learning framework, methods such as DD [27] , KIP [17] , RFAD [15] , and FRePo [34] update model parameters in the inner loop and synthetic data in the outer loop. In the data matching framework, techniques like DC [32] , DSA [30] , and IDC [9] utilize gradient matching, comparing neural network weight gradients from training on both real and synthetic data. Distribution matching approaches, exemplified by DM [31] , CAFE [26] , DataDAM [22] , and IDM [33] , align real and synthetic data distributions using Maximum Mean Discrepancy (MMD), often through a single optimization process but potentially limiting the performance. Trajectory matching, such as MTT [2] , optimizes synthetic data by pre-calculating and storing training trajectories of expert networks from the real dataset. A critical downside of MTT is the significant storage requirement for these trajectories.Recent research suggests compressing original data into models rather than synthetic data. DiM [25] minimizes the difference in predicted logits between real and generated images, using a generator to store original dataset information. SRe 2 L [29] indicates that key dataset information can be preserved in deep neural network training. Inspired by this, we argue that the localization and classification information of the detection data can be learned and preserved by the detector, allowing the key information to be recovered to reconstruct the synthetic image. Unlike SRe 2 L, we focus on object detection datasets with multiple instances in a single image. We propose a foreground-background decoupling strategy and incremental PatchExpand to enhance the updating of synthesized multiple instances.Diverging from the traditional focus on image classification, our study pioneers dataset condensation in object detection. We introduce a streamlined approach that separates the traditional bi-level optimization into a two-stage Fetch and Forge process. This strategy effectively realizes dataset condensation in object detection tasks with increased efficiency.High Training Overhead in Object Detection. Object detection methods are broadly divided into two categories: one-stage and two-stage detectors. One-stage detectors, such as YOLO series [19, 20] , SSD [14] , DSFD [12] , are valued for their speed and simplicity. Conversely, two-stage detectors like Faster R-CNN [21] detection, highlighting the growing importance of dataset condensation advancements. To address this issue, we introduce the first dataset condensation framework for object detection.3 Dataset Condensation for Object DetectionQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We describe it in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 Dataset condensation reduces a large real dataset T into a smaller synthetic dataset S. We first extend the existing DC [27, 32] optimization framework to accommodate detection task. For object detection, each image is annotated with multiple bounding boxes and their associated class labels. The real dataset T , containing N labeled images, is given by T = {(X i , A i )} N i=1 , where A i = {(x j , y j , w j , h j , c j )} ni j=1 , with x j , y j , w j , h j representing the coordinates and sizes of the j-th bounding box, and c j being the class label. The synthetic dataset S, comprising M images where M ≪ N , is formulated as S = {( X i , A i )} M i=1 , where A i = {( x j , y j , w j , h j , c j )} ni j=1 represents the synthetic bounding boxes and their corresponding class labels. The optimization process for dataset condensation is twofold: Model Parameters Update (Inner Loop): The model parameters θ are optimized over the synthetic dataset S to minimize the loss function L S (θ):EQUATIONSynthetic Dataset Update (Outer Loop): The synthetic dataset S is refined to minimize the loss on the real dataset T using the model parameters θ S ( S) obtained from the inner loop:EQUATIONHowever, this optimization process underlying this procedure involves a complex bi-level optimization scheme, which leads to high computational costs. The challenge intensifies when dealing with intricate model structures in the inner-loop or large synthetic datasets in the outer-loop. In the context of object detection, where high-resolution images and complex network structures are the norm, employing such bi-level optimization becomes even more impractical. This emphasizes the need for more streamlined and practical methods in dataset condensation, especially for tasks like object detection that involve high computational demands.Pioneering the application of dataset condensation to object detection, we develop the Dataset Condensation for Object Detection (DCOD) framework, which is a novel two-stage process depicted in Figure 3 . This method diverges from the traditional bi-level optimization that compresses information directly into synthetic data. Instead, we optimize the model and synthetic data separately. Subsequent sections will delve into the optimization objectives for the Fetch and Forge stages.Stage-I: Fetch. For object detection tasks, the real dataset is given by T = {(X i , A i )} N i=1 , with A i encompassing bounding boxes and class labels. By training an object detector ψ θ parameterized with θ on the original dataset T , we capture key information crucial for the tasks of localization and recognition within images. The optimization can be expressed as:EQUATIONwith L det representing the composite loss function that combines localization loss L loc and classification loss L cls :EQUATIONminimizing L det adjusts θ to detect objects with higher precision.Stage-II: Forge. In the forge stage, we borrow principles from model inversion [16, 28, 3] , utilizing a well-trained detection model to guide the embedding of essential information back into the synthetic images. To achieve this, it is essential to recognize the unique aspects of detection tasks: an image typically contains multiple foreground objects with different positions, sizes, and shapes. Therefore, we propose Foreground Background Decoupling (F BD ) and Incremental PatchExpand (I P E ). Finally, we use the detection loss as the task loss to guide the update of the targets' corresponding areas with the foregrounds of various categories.Foreground-Background Decoupling. We initialize the synthetic set S = {( X i , A i )} M i=1 by randomly selecting a subset of images and their corresponding targets from the original dataset, which includes position coordinates and category labels. The primary objective of object detection is to distinguish the foreground from the background and then classify the foreground objects. Therefore, we prioritize synthesizing the foreground. Additionally, while the background is treated as a single category in detection tasks, we adopt a background suppression strategy to avoid blending different contextual semantics (e.g., integrating sky or ocean information into a grassy background).To this end, we propose Foreground-Background Decoupling (F BD ) to separately handle the updating of the foreground and background areas, as follows:EQUATIONWe first separate the foreground and background regions using a binary mask based on the bounding box coordinates from the original labels. For the background (x back ), we apply a suppression strategy controlled by a hyperparameter α, which limits updates to preserve contextual information. For the foreground (x fore ), we employ a random erasure method R E , ensuring the model focuses more on refining foreground pixels during the inversion phase. By strategically applying suppression and enhancement, this approach prioritizes important visual information, optimizing the model's ability to generate high-fidelity synthesized images.Incremental PatchExpand. Unlike classification datasets that contain only one object, images in detection datasets sometimes contain multiple instances of several different categories. Inspired by curriculum learning, we propose Incremental PatchExpand (I P E ) to learn more instances during training. By adopting this approach, we increase the number of divisions, thereby introducing complexity into the synthetic data. We first standardize all images to a uniform size via padding. Subsequently, we employ a procedure, as defined by the function: Apply F BD and I P E for synthetic set S=F BD (I P E ( S))EQUATION6:Compute task loss L det using Equ. 47:Compute R reg using Equ. 8, where compute R pixel and R f eature using Equ. 9 and Equ. 118: Calculate L = L det (ψθ T ( X), A) + R reg 9:Update S ← S -η s ∇ s L 10: end for Output: Condensed synthetic dataset S which divides the images into k × k patches and each patch is then optimized for different targets, enhancing the diversity in object position, size, and shape. The incremental introduction of more patches allows the model to gradually adapt to various complexities and learn effectively across a broader range of scenarios.Optimization with Pixel and Feature Regularization. The synthetic images X are optimized by solving the following minimization problem:EQUATIONwhere L det denotes the detection loss and θ signifies the parameters of the model applied to X. Here, A represents the set of annotations for the targets present in X. The term R reg , a regularization term, is crucial for preserving the intrinsic qualities of the original dataset within the synthetic images and is defined as the sum of pixel and feature regularization:EQUATIONThe pixel-level regularization R pixel comprises two terms:EQUATIONwith R T V promoting spatial smoothness through the Total Variation regularization:EQUATIONand R l2 minimizing the squared Euclidean norm of pixel values. The coefficients α T V and α l2 are hyperparameters that balance the Total Variation and L 2 terms, respectively.Feature-level regularization R f eature aims to align the feature statistics of the synthetic image X with the batch normalization (BN) layers of the pre-trained model, as follows:EQUATIONwhere µ l ( X) and σ 2 l ( X) represent the mean and variance of the activations at the l-th BN layer of X, while µ BN l and σ BN l correspond to the statistics derived from the pre-trained model. This feature regularization is fundamental for the consistency of the feature distribution and the utility of X in training object detectors. At last, we summarize the complete two-stage process in Algorithm 1.Table 1 : The Performance comparison results on Pascal VOC, with the compression ratios of 0.5%, 1%, and 2%. The methods compared include Random [18] , Uniform, K-Center [23] and Herding [1, 4] . The models used in both the compression and evaluation phases are YOLOv3-SPP. Ratio (%): the ratio of condensed images to the whole training set. We train the network from scratch 5 times on the distilled dataset and evaluate them on the original test dataset to get the x ± std.Implementation Details. The initialization of the synthetic set involves random sampling from the original images, adhering to the specified compression ratio. The compression rate in classification tasks is typically set to around 1%. In our task, for VOC, we evaluate at compression rates of 0.5%, 1%, and 2%, while for COCO, we report at 0.25%, 0.5%, and 1%. During the standard image synthesis process, we set the image learning rate at 0.002. The weight for the task loss is set at 1, while R f eature is assigned a weight of 0.1. The α T V and α l2 is established at 1 and 0.001, respectively. The α in background suppression strategy is set to 0.7. All experiments were performed on a single V100 gpu.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Answer: [No] Justification: We commit to releasing the complete code later. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experimental results: 
 Counterpart Methods. As we are the first dataset condensation methods for object detection, we refer to the comparison settings of general DC in classification [27, 32] and choose three core-set selection methods: Random [18] , K-center [23] , and Herding [1, 4] . The Random method randomly selects real images from the original training set to form a subset. To adapt the K-center and Herding for object detection tasks, we implement a straightforward modification by using a pre-trained feature extractor (ResNet50) to process the features of entire images and employing the L2 norm to measure distances. Additionally, considering class balance, we also employ the Uniform, which is commonly used in practical applications within the field of object detection. Comparison Results. Table 1 and 2 provide the comparative results of our dataset condensation method on the VOC and COCO datasets, underlining its superior performance at all compression rates. According to the official ultralytics/yolov3 implementation, the detector achieves an mAP of 46.5% and AP 50 of 76.4% on VOC when trained with the full dataset, and 35.6% mAP on COCO, which serves as an approximate upper limit of performance. On VOC, using a 2% compression rate, our method still reaches an AP 50 of 50.7%. On COCO, at a 1% compression rate, the mAP and AP 50 reach 12.1% and 24.7% respectively. Compared to random sampling, at the lowest compression rate of 0.5% on VOC, our method increases mAP and AP 50 by 9.3% and 22.1% respectively, and at COCO's lowest compression rate of 0.25%, our method improves mAP and AP 50 by 3.7% and 7.5% respectively. Moreover, while the Uniform method shows slight improvement over Random, it still falls short of the performance achieved by our approach. The Coreset method, focusing only on the overall image features and neglecting instance-level details, shows poor performance. Our method also demonstrates comprehensive advantages in multi-size object detection metrics like AP s , AP m , AP l , indicating that our DCOD significantly enhances the diversity of foreground positions, sizes, and shapes.Effectiveness of Each Component. Table 3 shows that using model Inversion from Forge as our baseline, our proposed augmentations I P E and F BD improve performance by 1.7% and 9.4% respectively. Combined, they yield an 11.5% overall improvement. Notably, the baseline alone underperforms compared to random sampling. We found that without these augmentations, the detection loss quickly converges in early iterations, missing key positional and classification details. Thus, incorporating both I P E and F BD is crucial to enhance the diversity and effectiveness of the synthetic images.Cross-architecture Generalization. We verify the performance of condensed data on new architectures, and in Table 4 , our approach is able to maintain good generalization at 1% compression. However, we observe a larger performance drop at lower compression rates. This is due to the fact that the one-stage detector and two-stage detector architectures differ significantly, preserving limited information when too few images are synthesized.Sensitivity of Hyper-Parameters. Figure 4 shows how different α T V regularization weights affect our method's performance, using the VOC dataset with a 1% compression rate. The results indicate that an optimal regularization weight enhances performance, while no regularization or too much regularization reduces it. This happens because lack of regularization introduces excessive noise, and too much regularization causes excessive smoothing in the synthetic images, both of which degrade the quality of the images. Performance upper bound As shown in Figure 7 , we compare the performance variations of the Random baseline method and the DCOD method as the compression ratio increases. The performance of the full dataset, marked by a gray line, serves as the theoretical upper bound. When the ratio is below 5%, DCOD shows a significant advantage over the random method, while as the ratio exceeds 20%, the performance of both methods converge.GT Iter 1000 Iter 0 Iter 2000 Iter 3000 Iter 500Discussion on Initialization We discuss the impact of initialization sampling on the performance of the synthetic dataset in this section. As shown in Table 5 , the Random method, which is the primary choice for the experiments in this paper, is simple and performs well. Initialization strategies based on K-center and Herding lead to a decline in performance. This is these core-set methods fail to account for the complex factors in detection datasets, such as class distribution and object size distribution, resulting in significant differences between the synthetic and original datasets. Using noise as the initialization method, due to the absence of any prior information from the original dataset, causes a notable drop in the effectiveness of the synthetic dataset.We present a comparative analysis of synthetic images across various iterations, as shown in Figure 5 . Notably, there is a progressive enhancement in the foreground scores with advancing iterations, indicative of the incremental integration of category-specific beneficial information into the images. Specifically, for sheep in the first row, the scores at iter500 and iter1000 start lower than the initial image but surpass initial scores after iter2000. This trend implies that early images are initially disrupted by noise from the update process, but as iterations progress, pixel, and feature regularization help maintain image clarity.Additionally, we also showcase the visualization results of synthetic images containing multiple instances, as shown in Figure 6 , demonstrating that DCOD effectively supports the generation of multiple instances required in detection tasks. 
conclusion, limitations and future work: 
 In this study, we introduce the first dataset condensation framework for object detection, DCOD. It is a two-stage framework where, in the first stage, Fetch stores information vital for detection from the original dataset into model parameters. In the second stage, Forge, the images are then reconstructed through model Inversion. We propose foreground background decoupling and incremental Patch-Expand, to improve the efficiency and diversity of the synthetic images. Extensive experiments are conducted on two standard detection benchmarks, Pascal VOC and MS COCO, to demonstrate that DCOD can significantly maintain superior performance even at an extremely low compression rate.We acknowledge the limitations of our work from two perspectives. First, due to the significant differences among object detectors, we did not extend our approach to more complex detectors like DETR, as this might require more specialized designs to accommodate their structures. Second, the performance across different architectures is still insufficient and remains an area that needs improvement. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The potential positive societal impacts are described in Abstract and Introduction. In our view, this work has no potential negative social impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: The paper addresses the dataset condensation task for object detection, a more challenging problem than classification. It introduces a two-stage method, Fetch+Forge, with a clear description and thorough experimental evaluation. Reviewers generally agree this is a novel approach, though concerns about writing clarity and evaluation quality were noted. The final ratings varied from Borderline Accept to Accept. The meta-review confirms that most issues raised have been resolved, particularly for reviewer SZRX. Given the clear method and significant results, the AC recommends accepting the paper.

Reviews: review: paper_summary: paper_summary: This manuscript introduces, as far as I’m aware, the first method to do object detection dataset condensation. The method accomplishes this by first training a detection model on the original dataset called the Fetch stage, then uses this trained model to synthesize a condensed dataset through model inversion and regularization, called the Forge stage. Two modules are introduced to aid in inversion: Forward Background Decoupling (FBD) and Incremental PatchExpand (IPE). FBD induces separate processing on background and foreground pixels; background pixels are down weighted and foreground pixels are randomly masked. IPE progressively divides images into smaller and smaller patches as training progresses and processes different targets in each of these patches. A total variation regularizer is used to ensure synthesized image smoothness and a batch norm feature regularizer is used to ensure the batch norm mean and variance statistics are similar between real and synthetic images. Results on generated datasets outperforms random subsamples of the larger real dataset.
strength_weakness: strength_weakness: **Weaknesses:**

*W1:* The distribution of classes varies quite a bit at small/medium/large object sizes for both VOC and MS-COCO. It’s possible that K-Center and/or Herding sampling strategies conform to this distribution, but if not, I expect that this would be a strong baseline and should be compared against.

*W2:* In line 159, it states “the foreground is derived from the bounding box coordinates of the current targets”. This needs to be more precise. How exactly is it derived? Is a union taken over all bounding boxes of the current batch and those pixels used as the foreground?

*W3:* In line 167, it states “inspired by course learning”. Is this meant to be curriculum learning? 

*W4:* “Expand” in Eq. 6 is never defined. Are image patches upsampled be the same size as the original image? 

*W5:* The rate of increase for $k$ in the IPE module is never provided. 

*W6:* Does $\tilde{A}$ stay the same as the images sampled at initialization to run synthesis on? If so, how are these samples selected? This seems like it would drastically affect performance.

*W7:* line 218: What is “the Uniform”? I have worked in object detection for years and this term isn’t something I recognize. A citation here is needed.

*W8:* Specify how error bars were computed. Are these the standard deviation over multiple runs of dataset condensation?



**Additional comments, not affecting my rating:**

*C1:* The description of object detectors in line 99-102 isn’t quite correct. Fast models here are the YOLO series and SSD models. These are fast because they don’t include additional processing for small objects, e.g., an FPN. Faster R-CNN and RetinaNet are more accurate because they include an FPN. 

*C2:* On line 222: what open-source implementation is used? This should have a citation.

*C3:* Line 230: “The Coreset method” I assume this is referring to K-Centers, but terminology should stay consistent.

*C4:* It would be interesting to compare dataset condensations across more similar architectures. For instance, it should be easy to train a dataset condensed on RetinaNet or Faster-RCNN with FCOS, ATSS, or TOOD.

*C5:* It would be interesting to see Figure 5 on multi-class images.
questions: questions: This manuscript is interesting and if the authors address the weaknesses I’ve listed, I’ll be happy to raise my score from a 4 to a 5 or above. The most involved item I’d like to see is W1. This should hopefully be a relatively straightforward calculation followed by evaluation on already trained models. W2-8 are items that need to be added to the manuscript to ensure the method can be reproducible given the manuscript text.

Additionally, I’ve provided additional comments on how to improve the manuscript at the end of the weaknesses section, but these have not factored into my score.
limitations: limitations: The manuscript has addressed any concerns I’d have regarding limitations.
paper_summary: paper_summary: The paper looks at the task of dataset condensation for object detection. While there have been many works looking at dataset condensation for classification this is the first work to look at it for detection which is more challenging as each image can contain multiple objects of different categories. This is an important problem as it reduces the training requirements. The proposed method operated in two stages, first training a model on the full data. Followed by model inversion to get synthetic images which capture the most important classes and regions. The authors also propose new techniques to handle the multi object nature of images and show good results on VOC and COCO.
strength_weakness: strength_weakness: The paper has the following weaknesses:

1. In object detection and recognition multiple works have show that the context (background) plays an important role. [Noise or Signal: The Role of Image Backgrounds in Object Recognition]. The Fetch a Forge paper continuously suppresses background (L155-1156) and considers background not important. Is there backing of these claims and why would context (background) not help?
2. Lacking discussion about the input initialization. What effect do different input initializations have. Is there a way to find an optimal initialization? I think the initialization part needs to be discussed in more detail.
3. Section 3.1 seems less relevant given the paper is talking about object detection not classification. It should be modified to describe the setup in terms of detection data.
4. Missing discussion of what happens with different initializations during the Fetch part. Do you get similar performance and visualization of the condensed data?
5. Missing analysis of whether during condensation for object detection location is more important or classification while performing inversion to get the condensed synthetic samples.
6. The authors should also report at what percent of condensed data they can meet the real number. That is important to get some idea of compression to number of images tradeoff and what the best performance this method can give.

Small clarifications:
1. Its confusing why incremental patch expand is needed if the network is already trained during fetch step.
2. Missing ablation regarding the Reg loss in Eq 7 and also within it the importance of pixel vs feature regularization.
L112 For the complex of -> For the complex case/scenario
L167 course -> coarse
questions: questions: I think the paper is well written and in general easy to follow and sound. At the same time I have mentioned a few clarification and discussions in the weaknesses and would hope the authors answer some of them. Especially around background, initializations and upper bound with more images. Overall I feel the problem is important and the authors propose a first step towards solving it.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper attempts to generalize dataset condensation to object detection. It proposes Fetch and Forge. Fetch: training a normal object detector on the original dataset as usual. Forge: the synthetic images are sampled from the original data and optimized by the detection loss. Experiments on VOC and COCO show the effectiveness of the method.
strength_weakness: strength_weakness: 1. Notation issue: see $\mathcal{I}_{PE}$ in Eqn. 6 and Step 5 of Alg.1.

2. Why is row 3 of Table 3 blank?

3. Implementation Details are not comprehensive. Is it the training epoch for IPE set to $M$/batchsize? And I would suggest not calling it a training epoch.

4. What would happen if we increase the condensation ratio, e.g., 5%, 10%, 20%, 50%?

5. Typos: it should be ''ratio'' in Table 3.
questions: questions: see weaknesses above
limitations: limitations: see weaknesses above
paper_summary: paper_summary: 1) The paper introduces a dataset compression technique and applies it for the first time in object detection tasks.
 2) For large-scale object detection tasks, the condensation of datasets can not only shorten model training time but also save a lot of computational resources.
3) The method proposed in the paper is simple and effective, which can effectively solve the problems raised.
strength_weakness: strength_weakness: 1.Algorithm needs to be optimized.
2.The description of the second stage of DCOC is not clear enough.
questions: questions: 1、In the experimental section, it is possible to add several more datasets for comparison of condensing methods.
2、Why is the author's two-stage model framework more effective than a complex bidirectional compression optimization framework? Please discuss in detail.
3、It cannot be seen from Figure 3 what the target in the annotation refers to.
4、Formula 5's introduction is not specific enough.
5、How is each patch optimized for different goals and increased complexity in the IPE module?
6、Some annotations are missing in Tables 1 and 2, such as how the results of the entire dataset are obtained. What does "±" mean?
7、Based solely on the experimental results, due to the lack of comparison with similar experiments, the experimental results lack some persuasiveness.
8、How is the synthesized image updated under the guidance of loss?Suggest adding composite images with multiple instances in the visualization result graph.
9、Lack of reference to Figure 5 in the visualization section.
limitations: limitations: 1.Due to the significant differences among object detectors, you did not extend your approach to more complex detectors like
 DETR, as this might require more specialized designs to accommodate their structures. 
2. The performance across different architectures is still insufficient and remains an area that needs improvement.

Comments: title: Gentle Reminder
comment: Thank you for increasing your score! Your insights have been incredibly helpful, and we are excited to incorporate the changes based on your suggestions into our paper.

Thanks again for your support and valuable input!
Dear Reviewer SZRX,

We hope our response addressed your questions. As the discussion period comes to a close, we eagerly await your valuable feedback. We would greatly appreciate it if you could consider improving your rating.

Thank you for your valuable time!

Sincerely,

Authors of Paper #8952
Thanks for the authors' response it helped clarify my concerns and adding these to the paper will definitely make it stronger. I appreciate the authors' efforts in the rebuttal. I will improve my rating to weak accept.
Ratio = 100% means the whole-dataset training, which is the upper-bound of dataset condensation, the performance is 46.5% mAP. Thanks for the good suggestion! We will add a plot to depict the mAP growth w.r.t. condensation ratio in the final version.
About Q4: What is the mAP when ratio=100%? This may be the upper bound of dataset condensation. I also suggest adding a plot that depicts the mAP growth w.r.t. condensation ratio.
We appreciate your attention to detail.

Yes, the number of training iterations is 3,000. 

In Figure 3, we recorded the synthetic images within a batch, as indicated by "mAP for current Batch" on the right column of the images. This mAP was calculated by directly predicting on the current batch of synthetic images using a pre-trained YOLOv3-SPP model. However, in the experiments described in the paper, we trained a detector from scratch on the full synthetic set and then tested it on the original test set. Thank you for your reminder; we will include all the specific details of Figure 3 in the revised version. 

we apply learning rate decay using the `CosineAnnealingLR` scheduler, with `eta_min` set to 0.0005.
Thanks for the quick reply. So, is the training iteration 3K? As shown in Fig. 3 of the supplementary PDF, mAP goes for 94+ at 3000 iterations. While in the main paper, there are no such experimental settings that can exceed 94 mAP. 
So I wonder what is the experimental setup corresponding to this figure. Also, is there any learning rate decay?
### Stage-II: Forge Phase
We are pleased to provide a more detailed explanation:

**Initialization**: Randomly sample original images and their corresponding targets from the real dataset to form the initial synthetic set $\widetilde{\mathcal{S}}$, using the detector $\psi_{\theta_\\mathcal{T}}$ trained in Stage-I (with weights frozen).

**Training (Inversion)**: The optimization goal is to update the pixels of the synthetic images.

1. Divide $\widetilde{\mathcal{S}}$ into $k \times k$ patches using the IPE module. The FBD module constructs a binary mask to differentiate between foreground and background, applies random erasure to the foreground, and sets the update weight for background pixels to $\alpha=0.7$.
2. Input $F_{BD}$($I_{PE}$($\widetilde{\mathcal{S}}$)) into the detector, and calculate the loss according to Equation 7 in the paper: $L = L_{\text{det}}(\psi_{\theta_{\mathcal{T}}}(\widetilde{X}), \widetilde{A}) + R_{\text{reg}}$.
3. Update the pixels: $\widetilde{\mathcal{S}} \leftarrow \widetilde{\mathcal{S}} - \eta_s \nabla_s L$.

Under the guidance of the task loss, the synthetic images gradually stabilize, as shown in Figure 3 and 4 of the supplementary PDF.
Thanks for the response. Most of my concerns are addressed.
I have one question on Q3.
Please describe the training details for Stage-II: Forge.
Thank you for the in-depth feedback, which undoubtedly helps us improve the paper. We are grateful for your willingness to raise the final score.
Q1 & Q6: Glad to see the stronger baseline also improves your method!

Q5 & Q8: Please include this information in the revised manuscript. I’d recommend putting it in the Implementation Details section.

Q7: No need to add a citation for uniform sampling. This make sense to me now and I’m satisfied as long as this is clear in the paper.

Rebuttal PDF Figures: I think your manuscript would benefit by adding these figures in the appendix with appropriate reference to these figures in the main paper.

Overall, I’m satisfied and will raise my final score.