When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration
Authors: Suorong Yang
Peng Ye
Furao Shen
Dongzhan Zhou
Venue: ICML 2025 poster (2025)
Keywords: Dynamic data selection
dataset pruning
data augmentation
Primary area: deep_learning->everything_else
TL;DR: We propose a novel online data training framework that, for the first time, unifies dynamic data pruning and augmentation to enhance both training efficiency and model generalization.

Abstract: Dynamic data selection aims to accelerate training with lossless performances.
However, reducing training data inherently limits data diversity, potentially hindering generalization.
While data augmentation is widely used to enhance diversity, it is typically not optimized in conjunction with selection.
As a result, directly combining these techniques fails to fully exploit their synergies.
To tackle the challenge, we propose a novel online data training framework that, for the first time, unifies dynamic data selection and augmentation, achieving both training efficiency and enhanced performance.
Our method estimates each sample's joint distribution of local density and multimodal semantic consistency, allowing for the targeted selection of augmentation-suitable samples while suppressing the inclusion of noisy or ambiguous data.
This enables a more significant reduction in dataset size without sacrificing model generalization.
Experimental results demonstrate that our method outperforms existing state-of-the-art approaches on various benchmark datasets and architectures, e.g., reducing 50% training costs on ImageNet-1k with lossless performance.
Furthermore, our approach enhances noise resistance and improves model robustness, reinforcing its practical utility in real-world scenarios.

When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration
Suorong Yang" 12 Peng Ye 23 Furao Shen
Dongzhan Zhou?
Abstract
parameter-extensivee it is necessary to utilize even larger datasets, introducing challenges like reduced training effi- ciency. Moreover, large-scale datasets often include redun- dant or noisy samples (Xia et al., 2023b; Northcutt et al., 2021; Wang et al., 2018), which can compromise training effectiveness. To address these challenges, various data se- lection strategies have been proposed reduce dataset size and enhance data efficiency while maintaining model per- formance. These methods can be broadly categorized into static data selection (Tan et al.. 2024; Zhang et al., 2024; Xia et al., 2023b) and dynamic data selection (or pruning) (Qin et al., 2024; Raju et al., 2021; He al., 2024). Static selec- tion identifies a fixed subset of data before training begins, whereas dynamic pruning continuously selects the most influential samples during training. While these methods can effectively reduce training costs without degrading per- formance, the reduced training data volume often leads to reduced data diversity. Consequently, the generalization of deep models is limited, and lossless performance is typically achieved with relatively high selection ratios.
Dynamic data selection aims to accelerate train- ing with lossless performance. However, reduc- ing training data inherently limits data diversity, potentially hindering generalization. While data augmentation is widely used to enhance diversity, it is typically not optimized in conjunction with selection. As a result, directly combining these techniques fails to fully exploit their synergies. To tackle the challenge, we propose a novel online data training framework that, for the first time, unifies dynamic data selection and augmentation, achieving both training efficiency and enhanced performance. Our method estimates each sam- ple's joint distribution of local density and mul- timodal semantic consistency, allowing for the targeted selection of augmentation-suuie suitaable sam- ples while suppressing the inclusion of noisy ambiguous data. This enables a more significant reduction in dataset size without sacrificing model generalization. Experimental results demonstrate that our method outperforms existing state-of-the art approaches on various benchmark datasets and architectures, e.g., reducing 50% training costs ImageNet-IK Ik with lossless performance. Fur- thermore, our approach enhances noise resistance and improves model robustness, reinforcing its practical utility in real-world scenarios.
In fact, to address the issue of data diversity, data augmen- tation is commonly used in model training, which can also improve generalization (Xu et al., 2023; Yang et al., 2022b). However, effectively combining data selection and augmen- tation remains a challenge. Existing data selection methods typically prioritize representative and challenging samples, which are not specifically designed with augmentation in mind. Although augmentation can increase the diversity of selected data and further improve model robustness, apply- ing it to complex samples may introduce ambiguity noise, potentially increasing training difficulty (Gong et al., 2021; Yang et al., 2024b). Therefore, integrating data selection and augmentation a unified framework presents a promis- ing yet underexplored direction to balance efficiency and generalization.
1. Introduction
Deep learning has thrived with the growing availability of large-scale datasets. As models become more complex and
"This work was done during his internship at Shanghai Artifi cial Intelligence Laboratory. 'National Key Laboratory for Novel Software Technology. Nanjing University "Shanghai Artificial Intelligence Laboratory "The Chinese University of Hong Kong. Correspondence to: Furao Shen <frshen @ nju.edu.cn Dongzhan Zhou <zhoudongzhan @ pjlab.org.cn>.
In this study, we propose novel framework that integrates dynamic data selection with augmentation, enabling uni- fied approach to enhance training efficiency and model gen- eralization. During model training, low-density samples of- ten correspond to underlearned or insufficiently represented data points, such as classification boundaries. Applying augmentation transformations these samples reinforces
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

When Dynamic Data Selection Meets Data Augmentation
Selected Data
Original Dasaset
0.6
Original Dataset
Seletted without DA
Selected data
05-
0.3
Aug.
2.3
0.3
0.2
00
2
Figure 1. Illustration of the distribution of our selected data points using the T-SNE algorithm (left) and the density histograms without (mid) and with (right) augmenting the selected data on CIFAR-IO. 10. The selection ratio is 10%.
model learning and improves robustness. However, noisy or outlier samples typically exhibit relatively low density, increasing the risk of introducing noise. To address this, we introduce a semantic consistency distribution derived from the pre-trained multimodal model CLIP (Radford et al., 2021). By prioritizing samples with high sparsity and strong semantic alignment, our approach leverages the joint distri- bution of density and semantic consistency for effective and robust sample selection.
consistency, ensuring effective sample selection and reduc- ing noise and ambiguity. 3) Extensive experiments across diverse datasets and architectures demonstrate superior ac- curacy and generalization ability, particularly in noisy and challenging scenarios, validating its practical applicability.
2. Related Work
2.1. Data Selection
As illustrated in the left sub-figure of Fig. 1, the selected data points predominantly cluster around boundary regions among clusters. Meanwhile, the density histogram in Fig. reveals a more balanced distribution after augmentation, with fewer low- and high-density samples and more data points converging toward the moderate-density regions. Compared to the distribution without DA, this redistribu- tion highlights our framework's is ability to enhance sparse regions, improving model generalization across the entire data distribution.
The primary goal of data selection is enhance data- efficient learning, which can be broadly categorized into dataset distillation (Lei & Tao, 2023; Du et al., 2023; Sun et al., 2024; Liu & Wang, 2024), and static or dynamic data selection (or pruning) (Tan et al., 2024; Xia et al., 2023b; Sorscher et al., 2022; Qin et al., 2024). Dataset distillation focuses on synthesizing a small representative dataset that preserves the performance of training on the full dataset. In contrast, following dynamic data selection without synthe- sizing new data in this work, we propose new data training framework that unifies dynamic data selection and augmen- tation to achieve enhanced model training acceleration.
Experimental results across benchmark datasets and deep ar- chitectures demonstrate the effectiveness of our method. On large-scale datasets such as Tiny-ImageNet (Chrabaszez et al., 2017) and ImageNet- Ik (Deng et al., 2009), our method significantly accelerates training while maintain- ing or even improving generalization. For instance, on ImageNet- Ik, our approach doubles the training efficiency while achieving comparable performance with the entire dataset. Moreover, our framework exhibits strong cross- architecture and cross-scenario generalization, effectively mitigating the impact of noisy data and enhancing versatility in real-world applications. On Tiny- ImageNet, our approach outperforms leading baselines by at least 3% in accuracy under noisy conditions, further demonstrating its reliability.
Static data selection identifies fixed subset of the training dataset before training begins. Existing methods can be categorized into selection with importance criteria, dataset distribution- bassed methods, and iitimizzation-bssed meth- ods. Selection with importance criteria computes per- sample importance scores and selects the most informative samples. This includes: 1) the expectation of f--norm er- ror vector and the gradient norm (EL2N and GraNd) (Paul et al., 2021), 2) the change in the optimal empirical risk when sample is removed (Tan et al., 2024), 3) the num- ber of forgetting events in the whole training process (For- getting) (Toneva et al., 2018), 4) the impact of including or excluding sample on the model's classification abil- ity (Feldman & Zhang, 2020). Dataset distributionnbassed methods select samples based on the geometric distribu- tion of the dataset. Herding (Welling, 2009) chooses sam- ples based on their distance from the corresponding class centers. D2 (Maharana et al., 2023) defines sample diffi-
Our main contributions are summarized as follows: 1) We propose novel training framework that dynamically inte- grates data selection and augmentation, significantly acceler- ating training while maintaining model performance. 2) We introduce a joint distribution based on density and semantic
2

---

When Dynamic Data Selection Meets Data Augmentation
culty by incorporating the diffieulty of its neighboring ex- amples. The work (Ramalingam et al., 2023) applies greedy k-center to select the coreset with good data coverage, and CCS (Zheng et al., 2023) balances the sample distribution and importance in selection. Similarly, Moderate-DS (Xia et al., 2023b) selects samples that are closer to the median score, aiming to balance diversity and representativennss. Optimization -based methods formulate selection as an op- timization problem using techniques such as scalable self- supervised pruning metrics (Sorscher et al., 2022), influence function (Yang et al., 2023a), bi-level optimization (Killam- setty et al., 2021), gradient matching (Mirzasoleiman et al., 2020b), convex optimization (Mirzasoleiman et al., 2020a), facility location function (Yang et al., 2023b), temporal dual- depth scoring (Zhang et al., 2024), and submodularity (lyer
image each epoch, including Augmmnt (Cubuk et al., 2019), Fast-AutoAAugment (Lim et al., 2019), RandAug- ment (Cubuk et al., 2020), TrivialAugment (Müller & Hutter, 2021). SelectAugment (Lin et al., 2023), EntAugment (Yang et al., 2024b), and MADAug (Hou et al., 2023), etc. Beyond these, generative data augmentation further enriches data by synthesizing new samples using generative models (Moreno- Barea et al., 2020). Recent studies also emphasize represen- tation consistency (Atienza, 2022) and address distribution gaps between clean and augmented data (He et al., 2019), pointing to new challenges in effective DA design.
3. The Proposed Method
3.1. Overview of the Proposed Method
et al., 2021: Nohyun et al., 2023).
As shown in Fig. 2. we propose novel data training frame- work that integrates dynamic data selection with augmenta- tion to enhance both training efficiency and generalization. Our framework employs two complementary distributions: 1). a density distribution, dynamically estimated by a task- specific model (c.g., ResNet), which identifies underrep- resented samples for augmentation and training, and (2) a semantic consistency distribution, computed using a frozen pre-trained multimodal model (CLIP), which quantifies the alignment between an image and its corresponding textual label. Low-density regions highlight underrepresented sam- ples but may also include noisy or ambiguous instances. To address this issue, the semantic consistency distribution acts as a strong complement, filtering samples with weak semantic alignment. By combining both distributions, we construct joint distribution that captures the relationship between sample informativeness and semantic correctness. Consequently, samples with higher joint distribution scores are prioritized to be selected and augmented for training.
Dynamic data selection identifies informative samples throughout training, allowing the dataset to adapt as the model learns. The work (Raju et al., 2021) proposes UCB and e-greedy algorithms to estimate the uncertainty value as- sociated with each training sample, selecting subset of the data that exhibits the highest levels of uncertainty. Similarly, the work (He et al., 2024) also employs both prediction uncertainty and training dynamics to guide the selection process, ensuring that the most informative samples are re- tained throughout training. The work (Liu & Mirzasoleiman, 2022) proposes a data-efficient framework for training neu- ral networks and achieves promising results. SAS (Joshi & Mirzasoleiman, 2023) improves data efficiency in SSL by proving and selecting the most beneficial data for con- trastive training. Moreover, InfoBatch (Qin et al., 2024) proposes method for unbiased dynamic data selection that accelerates training by pruning less informative samples to retain their relevance in model optimization, which al- lows for more efficient training without compromising the model's performance.
3.2. How to Identify Samples for Augmentation
Given dataset D that follows an underlying distribution P(D), the optimization objective of our dynamic data selec- tion module at time is to select a subset D., containing at most & samples. The goal is to minimize the expected loss over the distribution P(D). with the following optimiza- tion: D, = arg min E.-r(D) L (2.0A(b)) where
2.2. Data Augmentation
Data augmentation (DA) improves the generalization of deep neural networks by increasing the diversity of training samples (Yang al., 2024a). Existing DA methods can be divided into image erasing/mixxng-hant based and automatic augmentation methods (Xu et al., 2023). Image erasing and laxing-based. augmentation erase some sub-regions in im- ages or mix random information from two or more images for augmentation to create new samples, respectively. These methods include Cutout (DeVries & Taylor, 2017), Ran- domErasing (Zhong et al., 2020), HaS (Singh & Lee, 2017), AdvMask (Yang al., 2022a), Mixup (Zhang al., 2018), GuidedMixup (Kang& & Kim, 2023), and GradSalMix (Hong et al., 2023), etc. In addition, based on pre-defined or opti- mized image transformation policies, automatic DA meth- ods randomly apply one or multiple transformations to each
b.C..lb..skk
= represents a test sample, L is the loss function, 0z. the empirical risk minimizer on D., and A represents the augmentation operations applied to the selected subset.
Our method prioritizes low-density samples because such re- gions in the feature space often correspond to underlearned or insufficiently represented data points. Focusing on these samples and applying augmentation operations helps the model capture distinctive features. By augmenting these sparse samples, we compensate for their underrepresenta-
3

---

When Dynamic Data Selection Meets Data Augmentation
Semantic Consistency
Pretrained VLM
Pros
Pp
Nearest Neighbor
Image Encoder
Text
Selected
Augmentation
Encoder
Ahg aantna
Selected samples
Joint distribution
[Image]
[abel]
Arch.
Augmented semples: clarifying the boundary.
Augmented samples: filling the blank within the cluster.
1:56
Density estimation of epoch t
Figure 2. The framework of our proposed data training method: The core idea of our framework is construct joint distribution that integrates both the density and semantic consistency distributions, enabling the prioritization low-density, semantically consistent samples. After augmentation, augmented sparse samples in intra-cluster regions help to fill the underrepresented spaces, while samples located around the decision boundaries between clusters differentiate the classification decision more clearly, thus improving generalization.
tion, improving the model's ability to generalize across diverse data regions. To efficiently determine the den- sity of data points during online training, we exploit an online approximate nearest neighbor search architecture (HNSW) (Malkov & Yashunin, 2018) to query the nearest k neighbors of each sample v, denoted (x)).. The density of sample is then estimated the mean of the & distance between x and its neighbors:
multimodal consistency supervision as an additional crite- rion for assessing sample reliability. This complements the density-basee selector by filtering out samples that exhibit low cross-modal consistency.
To implement this, we leverage pre-trained CLIP model to embed images and text into shared multimodal space, enabling semantic alignment assessment. How- ever, CLIP's zero-shot generalization is limited to domain- specific datasets, making necessary to adapt the embed- dings to the target domain. Instead performing computa- tionally expensive fine-tuning, we incorporate lightweight adapters of MLP for both the image and text encoders (Poth et al., 2023; Yang et al., 2024c). The lightweight architec- ture of adapters ensures efficient adaptation while preserving CLIP's pretrained knowledge.
Px.
||; - 2y (EN
Σ
(I)
where higher values indicate low-density samples. The density scores are then normalized using Min-Max scal- ing to obtain (1)).. For augmentation, we employ slight augmentation, which generates neighboring samples within low-density areas while preserving local structure.
To measure the cross-modal consistency, we compute the cosine similarity the encoded image and text features:
Nevertheless, low-density regions may also contain chal- lenging, outlier, or noisy samples. Continuously selecting these samples for training can substantially complicate train- ing, especially on real-world datasets that inevitably include noise. To address this issue and improve the practical effec- tiveness of our method, we introduce a multimodal semantic consistency constraint that simultaneously refines the selec- tion low-density samples. This ensures that augmentation applied meaningful data, improving both robustness and efficiency in the training process.
con(x;) = (EI Er(y.))
(2)
where Fy and Er are visual and textual encoders, respec- tively. The consistency scores are normalized via Min-Max scaling to approximate the consistency distribution Poo (I)), where higher values indicate stronger semantic aligmment. Since the image-label alignment is derived from pretrained vision-language model and remains independent of the train- ing process, we precompute the consistency distribution heforehand. This enables direct use during sample selection, eliminating additional computational overhead in online training.
3.3. Multimodal Consistency Estimation for Robust Data Selection
Noisy data arising from incorrect labels, corrupted images, outlier samples- reflects fundamental mismatch be- tween the semantic content of x and its corresponding label To detect and filter such inconsistencies, the data cleaner evaluates the joint distribution p(z, y), which captures the plausibility of an image-label pair. Given the inherent cor- relation and multimodal nature of and y, we introduce
3.4. Augmenter
To integrate structural sparsity and semantic consistency, we define joint distribution that combines both density and consistency distributions:
(oc((F;) = Pp( Poon (x)n
(3)

---

When Dynamic Data Selection Meets Data Augmentation
Table 1. The accuracy (%) comparison tate-of-the-art baselines. All methods trained with ResNet-18 CIFAR-10/100 and
ResNet-50 on Tiny- Imageet Note that some results could not be computed due to the unavailability of open-source code and parameter settings, making it impossible to reproduce. Random" means randomly selecting samples in each epoch.
Dataset
CIFAR-IO
CIFAR
Tiny-Imun ImageNet
Whole Dataset
95.6
78.2
45.0
Selection Ratio (%)
30
50
70
30
50
70
30
50
70
Random
90.2
92.3
93.9
69.7
72.1
73.8
29.8
37.2
42.2
Herding (Welling. 2009)
80.1
88.0
92.2
69.6
71.8
73.1
29.4
31.6
39.8
EI.2N (Paul et al., 2021)
91.6
95.0
95.2
69.5
72.1
77.2
26.6
37.1
44.0
GraNd (Paul et al., 2021)
91.2
94.6
95.3
68.8
71.4
74.6
29.7
36,3
43.2
Glister (Killamsetty et al., 2021)
90.9
94.0
95.2
70.4
73.2
76.6
30.1
39.5
43.9
Forgetting (Toneva et al., 2018)
91.7
94.1
94.7
69.9
73.1
75.3
28.7
33.0
41.2
Modcrate-DS (Xia ct al., 2023b)
91.5
94.1
95.2
70.2
73.4
77.3
30.6
38.2
42.8
Self-sup. prototypes (Sorscher et al., 2022)
91.0
94.0
95.2
70.0
71.7
76.8
27.7
37,9
43,4
MoSo (Tan al., 2024)
91.1
94.2
95.3
70.9
73.6
77.5
31.2
38.5
43.4
DP (Yang ul., 2023a)
90.8
93.8
94.9
73.1
77.2
Random
93.0
94.5
94.8
74.4
75.3
77.3
41.5
42.8
43.1
UCB (Raju al., 2021)
93.9
94.7
95.3
75.3
77.3
e-Greedy (Raju et al., 2021)
94.1
94.9
95.2
74.8
76.4
InfoBatch (Qin et al., 2024)
94.7
95.1
95.6
76.5
78.1
78.2
42.2
43.2
43.8
Ours
94.9
95.5
96.0
77.6
78.9
79.5
44,9
47.0
49,4
Here, Pp evolves dynamically with model training, allowing sample selection to adapt to the current model training state. During online training, samples with higher joint distribu- tion scores are prioritized for augmentation and training, ensuring that both underrepresented and semantically mean- ingful samples are utilized effectively.
making our method scalable for large datasets. Furthermore, the data augmentation, standard pipeline in model train- ing. introduces negligible overhead.
4. Experiment
4.1. Experiment Setup
We employ TrimialAugment as our augmenter, which is widely used and offers a computationally efficient aug- mentation strategy. During augmentation, only single lightweight transformation per image is applied. This brings two key advantages: i). introduces negligible computation overhead to the online training process, making it highly efficient. fi). Since each image undergoes only one trans- formation with slight magnitude, the augmented samples remain within their original local feature space. This is well-suited for the objective of our dynamic data selection framework: filling intra-cluster gaps and enhancing deci- sion boundaries within clusters. Thus, the consistency of selected samples is preserved while the data diversity in sparse regions is enhanced. Consequently, training using these augmented samples improves model performance. In addition, we provide the augmentation operation list used in Table in the Appendix, which includes image transfor- mations such as translation, rotation, equalization, etc.
Datasets and network architectures. In line with pre- vious works (Tan et al., 2024; Xia al., 2023b; Qin et al., 2024), we evaluate the effectiveness of our pro- posed method using widely adopted benchmark datasets, including CIFAR-10/18 (Krizhevsky et al., 2009). Tiny- ImageNet (Chrabaszez et al., 2017), and ImageNet- Ik (Deng et al., 2009). addition, we evaluate the ro- bustness of our method in noisy datasets. To further assess the generalization ability of our method, we extend the evaluation to more challenging datasets, such as ImageNet- A/O (Hendrycks et al., 2021b), ImageNet-Hard (Taesiri et al., 2024), and ImageNet-R (Hendrycks et al., 202 Additionally, we evaluate the generalization of our method across different deep architectures. Specifically, we con- duct experiments using both ResNet-based, such as ResNet- 18/50, and ViT-based models, such as ViT-B/L and Swin- Transformer, to demonstrate the robustness and scalability of our approach across diverse models.
Complexity analysis. The computational costs of our framework, when integrated into online training, are pri- marily associated with density estimation. Specifically, both querying and updating within the HNSW graph operates with a complexity of (l)g(n))) where is the total num- ber of data points. Let T denote the total number of training epochs; then, the total cost is O(T log(n)). Since T & n, the overall computational complexity remains (llg(n)),
Comparison with state-of-the-are We compare with our method both static and dynamic data selection meth- ods, including 1) Random, 2) EL.2N (Paul et al., 2021), 3) GraNd (Paul et al., 2021), 4) Herding (Welling, 2009), 5) Forgetting (Toneva et al., 2018), 6) Moderate-DS (Xia et al., 2023b). 7) Self-sup. prototypes (Sorscher et al., 2022), 8) MoSo (Tan et al., 2024), 9) DP (Yang et al., 2023a), 10)
5

---

When Dynamic Data Selection Meets Data Augmentation
Table Results ImageNet- with 60% selection ratio using ResNer-50 an 8-A100 server. Note that due the high computational costs and device memory costs (Xia et al., 2023b), Glister and CG-Score are not reported. Some results are from (Qin et al., 2024). Time the wall clock time; Overall (a is total GPU hour, where n the node number.
Method
Herding
EL2N
GraNd
Forgetting
SSP
Modorate
MoSo
UCB
Infobatch
Glister
CG-Score
Ours
Whole Dataset
Acc. (%))
71.1
72.3
71.0
72.5
70.0
13.1
74,0
75.8
76.5
76,9
76.4
Time (b)
10.5
10.5
10.5
10.5
10.5
10.5
10.5
10.5
10.5
10.5
10.5
10,5
17.5
Overhead (b)
>17.5
>17.5
217.
>17.
2-4.0
217.
217.5
0.03
0,0028
0.53
Overall (n"h)
>224.
>224.
>224.0
>224.
>276.0
>224.
52240
84.0
84.0
88.2
140.0
78
Table Experimental results on Tiny- ImageNet with noisy and corrupted data using ResNet-50. The noisy ratio 20%.
Table Experimental results Tiny- ImageNet with data augmentation. The selection ratios 30%, 50%, and 70%.
76
50% less
74-
Method
Noisy
Corrupted
Whole Dataset
52.0
72
Selection Ratio (%))
20
30
30
Selection Ratio
30%
50%
70%
Random
17.8 23.9
20.0
25.9
Random
29.8
37.2
42.2
Accuracy
70
Hending
19.0 24.2
35.0
30.6
Herding
31.6
39.2
45.6
Modcrate-DS
19.6 25.0
23.3
29.1
EL2N
32,0
40.1
45.9
EL2N
13.9 18.6
18.6
24.4
GraNd
32.2
40.5
46.2
66-
GraNd
18.3 23.7
20.0
26.7
Forgetting
13.2 21.8
18.5
25.5
Glister
33.1
42,2
46.5
64
Forgetting
27.2
36,2
44.2
Our
Self-sup. prototypes
15.1 21.0
20.2
26.9
CG-Score
8.4 15.3
16.4
24.4
Moderate-DS
33.8
41.5
46.6
62-
Randonn
Glister
21.6 25.5
21.2
22.0
Self-sup. proto.
33.4
41.1
46.6
20
30
50
60
70
80
90
100
MoSo
7.4 11.3
23.1
28.8
MoSo
32.6
41.5
45.9
Training Samples (%)
Random
33.8
36.5
35.1
36.9
Random*
42.1
43.9
45.2
Figure 3. The performance on ImageNet-
InfoBatch
34.9
37.1
35.1
38.1
InfoBatch
43.2
45.9
48.3
Ik across various selection ratios with 4-A100-GPU server.
Ours
35.9
39.6
39.1
42.0
Ours
44.9
47.0
49.4
UCB (Raju et al., 2021), 11) c-Greedy (Raju et al., 2021), 12) Glister (Killamsetty et al., 2021), and 13) InfoBatch (Qin et al., 2024).
tion with relatively higher selection ratios on these datasets, such as over 60% CIFAR-I0/100 and over 70% Tiny- ImageNet.
Notably, our approach outperforms the other methods at the same selection ratios. On Tiny-ImageNet, a large-scale dataset, our method yields an average performance improve- ment of at least 2.7% while maintaining the same training costs. As the training data volume increases, this perfor- mance gap becomes
Implementation details. To ensure consistency with prior work (Qin et al., 2024; Xia et al., 2023b), we follow simi- lar experimental settings. Specifically, we use the OneCy- cle scheduler with the SGD/LARS optimizer for model training, a momentum of 0.9, weight decay of 5e-4, and cosine annealing. We employ TrivialAugment (Mäller & Hutter, 2021) in our framework. For fairness, we adopt the annealing and re-scaling techniques introduced in (Qin et al., 2024), which standardize the dynamic dataset prun- ing process across all methods compared. Moreover, we use InfoNCE loss to fine-tune adapters for 15 epochs on datasets. Since InfoBatch uses soft pruning with dynamic number of selected samples, report its performance using the same number of forward passes as in our method.
more pronounced, further high- lighting the efficiency and effectiveness of our framework.
4.3. ImageNet-Ik Results
Table 2 presents the evaluation results of our method on the ImageNet-Ik dataset with 60% selection ratio. Our approach outperforms the full dataset by achieving a nearly 40% training cost reduction, resulting in a reduction of up to 56 hours in training overhead with a 0.5% accuracy im- provement. Meanwhile, since most static data selection methods require training surrogate models to determine the sample's influence throughout model training, the computa- tion overheads are relatively much higher than ours. Thus, the results highlight that our method outperforms static data selection methods in both performance and computational efficiency. It also surpasses dynamic pruning methods in terms of final accuracy with comparable efficiency. These findings underscore the generality and competitiveness of
4.2. Performance Comparison
As shown in Table 1, we evaluate the performance of our method by training ResNet- 18 CIFAR CIFAR10/100 10/100 and ResNet- 50 on Tmay-Im across different selection ratios. Our method achieves comparable performance to models trained on the full dataset, even when only 50% of the data is used on CIFAR-10/100 and 30% on ImageNe. In con- trast, existing methods typically achieve lossless data selec-
6

---

When Dynamic Data Selection Meets Data Augmentation
Table Generalization of models trained with our method on ImageNee-Hard, ImageNet-A, ImageNet-R, R, and ImageNet-O. We report AUPR (%) for ImageNet-O and accuracy (%) on others. All models are ResNet-50.
Sclection (atio(%)
20
30
50
70
80
90
Whole Dataset
ImageNct-A
1.9112
2.1
2.9
3.1
3.410.3
3,4403
3,5
3.1
ImageNet-R
37.2
38.5123
39,3+1.1
39,813s
39.91377
40.6r44
41.0;48
36.2
ImageNet-O
15.4+22
15.8126
16.1
16.3125
16,3125
16.4r26
16.5ç27
13.2
ImtaeNet-Hard
14.2,0.5
15.3106
15,9-12
16.5tus
16.7120
17.2/15
17,5
14.7
our approach large-scale datasets.
the corrective power of multimodal alignment, our method provides a reliable and efficient solution for robust deep learning in practical, noisy data environments, demonstrat- ing its practical significance.
Further analysis of the performance across different selec- tion ratios on ImageNet-I1 I is shown in Fig. 3. The results show that our method achieves lossless performance with only 50% of the training data. When using 20% of the data, performance drops by about 2% while nearly 80% of the training overhead is eliminated. Compared to ran- dom selection, which suffers a significant accuracy drop as the selection ratios decrease, our method maintains robust performance even with reduced data. Similarly, most exist- ing baseline methods typically require at least 60% of the training data to achieve similar lossless performance. This demonstrates that our framework further lowers the data requirement for maintaining full performance.
4.5. Effect of Data Augmentation on Model Performance
To further assess the effectiveness of our method, we com- pare its performance against several baseline methods using TrivialAugment for data augmentation, as shown in Table 4. Our method consistently outperforms the other approaches across various selection ratios.
While data augmentation enhances the performance of all methods, our approach consistently achieves superior results at different selection ratios. This indicates that our method is not simply straightforward combination of data augmen- tation and data selection. Instead, it effectively identifies the most beneficial samples for augmentation, leading to signifi- cant performance improvements. By selectively amplifying the impact of data augmentation, our method optimizes model performance, demonstrating its ability to leverage augmentation more effectively than other approaches.
4.4. Robustness to Noisy Scenarios
In real-world scenarios, training data is often polluted by corrupted and mislabeled images (Xia et al., 2023a; Wang al., 2018), which can significantly degrade model perfor- mance. Specifically, we simulate mislabeled data (noisy) by flipping portion of the labels to incorrect ones using symmetric label noise. Meanwhile, we introduce five types of realistic distortions to simulate corrupted data, namely Gaussian noise, random occlusion, resolution variations, fog, and motion blur. The examples of corrupted data are shown in the Appendix. To evaluate the practical relevance of our data training framework in such noisy environments, we assess the robustness of our method compared to existing state-of-the- art methods.
4.6. Generalization on Hard Benchmarks
To evaluate the generalization capabilities of our pro- posed framework, we conduct experiments on challeng- ing benchmark datasets, including ImageNet-Hard (Taesiri et al., 2024), ImageNet-R (Hendrycks et al., 2021a), and ImageNet-A/O (Hendrycks et al., 2021b). Specifically, we pre-train ResNet-50 models using data selected through our method across various selection ratios and then test their performance on these challenging benchmark datasets. Following standard evaluation settings, we report the area under the precision-recall curve (AUPR) for ImageNet-O and classification accuracy for the other datasets.
As shown in Table 3, our approach consistently outperforms the compared methods, demonstrating superior robustness against both mislabeled and corrupted data. Specifically, our method achieves a 4% improvement over competing methods corrupted datasets, even with 20% noise ratio on ImageNe. Our framework excels in these sce- narios due to its ability to combine low-density sample selection with multimodal semantic alignment. While noisy data is typically sparse and low-density, our method's 's ro- bust integration of multimodal semantics offers a powerful mechanism for mitigating noise and highlighting meaning- ful patterns in the data. This approach allows to maintain high robustness without sacrificing data efficiency.
As shown in Table our method maintains or even enhances generalization performance on these challenging datasets, despite using fewer training samples. The results demon- strate that reducing the dataset size with our framework does not compromise generalization ability. Meanwhile, as the selection ratios increase, our method achieves superior generalization compared to training on the full dataset.
By prioritizing sparse, low-density samples and leveraging
7

---

When Dynamic Data Selection Meets Data Augmentation
Ciaa Deta
X New Dela
Sclected Doan Dut
X Se Kany fro
Table Experiment results on more advanced architectures, in- cluding VTT-B, VIT-L, and Swin-T I on ImageNet-Ik IK with a 4-A100 GPU server. Overhead represents GPU hours (h), and S, refers to
the selection ratio.
(%)%
50
60
70
80
Full Dataset
ViT-B
82.6
82,9
83.2
83.2
83.3
82.5
VIT-I
85.2
85.3
85.6
85.7
85.7
84.6
Swin-T
84.1
84.1
84.2
84.2
84.3
84.2
(e) Random
(1) Ous
Figure 5. Visualization of the selection results on noisy Tiny-
350
ImageNet with 20% noise ratio. The selection ratio is 20%.
Training Costs (h)
300
Savings: 100h
250
the number of selected noisy points is minimized.
200
150
4.9. Further Analysis of the Overheads
100
Sch
Although our method introduces negligible overheads into online training, our framework incorporates adapter fine- tuning and feature embedding via CLIP models and corre- sponding adapters before model training begins. As shown in Table 7, analyze these pre-computation overheads of the adapter fine-tuning and feature embedding via CLIP models. It can be observed that these one-time overheads before model training are negligible compared to standard target model training. Once computed, no further compu- tation required during online training across selection ratios.
VIT-B VT--L
Swin-T
Figure 4. The overall cost savings achieved by our method on ViT- based architectures with lossless performance. The experiment is conducted on ImageNet-IV IK with a a 4-A100-GPU server.
4.7. Generalization on Different Architectures
To evaluate the scalability of our proposed method, we conduct experiments on advanced architectures, including ViT-B, VIT-L (Dosovitskiy et al., 2020), and Swin-T' (Liu al., 2021). Specifically, we train these architectures using our framework across various selection ratios.
4.10. Ablation Study
Effect of different modules in our framework. In Ta- ble 8, we systemically evaluate the effectiveness of differ- ent components within our proposed framework on Tiny- ImageNet using ResNet-50 across various selection ratios.
As shown Table 6, our framework sichtecturee-annssiie achieving robust generalization across these different mod- els. even with reduced selection ratios. Notably, the lossless performance can be achieved with only 50% of the training data. The results underscore that method generalizes ResNet-based and Transformer-bases architectures, all with reduced training costs.
When only the density distribution P, is used, performance is lower, as low-density samples often include sparse and outlier data, which can introduce ambiguity into the train- ing process. However, when both the density distribution Pp and consistency distribution Peon are combined, perfor- mance improves, demonstrating that incorporating semantic consistency helps mitigate the negative effects of density- based selection. Further performance gains are achieved by including the augmentation module, which boosts accu- racy by a significant margin. This shows that augmentation plays crucial role in improving performance, especially when the selected data points under P, and Poon are more suited for augmentation, enhancing the model's generaliza- tion. The results indicate that removing any module from our framework leads to substantial drop in performance.
Additionally, in Fig. 4, we present the practical training costs on these architectures and the lossless cost savings achieved by our framework. can be seen that our proposed framework can significantly save hundreds of hours on large- scale architecture training.
4.8. Visualization of the Selection Robustness
In Fig. 1, illustrate our selection results on clean datasets, showing that the selected data points mainly cluster around boundary regions among clusters. To better understand our selection effectiveness, in Fig. 5, we further illustrate our selection results on the noisy Ting-ImageNet dataset with a 20% noise ratio. can be seen that compared to the baseline, our method can effectively filter out noisy samples:
5. Conclusion
This paper proposes a novel data training framework that unifies dynamic data selection and data augmentation for
8

---

When Dynamic Data Selection Meets Data Augmentation
Proceedings of the IEEE/CVE winter conference on ap-
Table Overheads of fine-tuning and feature embedding hefore
plications of computer vision, pp. 372-381, 2022.
model training large-scale datasets with 1-V100 GPU server.
Dataset
Fine-tuning
Embedding
Overall training
Chrabaszez, P., Loshchilov, I., and Hutter, F. downsam- pled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv: 1707::7007.880 707.08819, Aug 2017.
Tiny- ImageNet
0.39h
0.03h
21.0h
ImageNet-Ik
1.25h
0.17h
84.0h
Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation strategies from data. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 113-123, 2019.
Table Effect of density distribution, consisteney distribution, and augmenter Imy-eeeeeeet using ResNet-55. We report test accuracy (%). The selection ratios (%) are 30%, 50%, and 70%.
Pp
Prom
aug.
30%
50%
70%
39.0
40.7
42,5
Cubuk, E. D., Zoph, B., Shlens, J., and Lc, Q. Randaugment: Practical automated data augmentation with a reduced search space. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Proc. Adv. Neural Inf. Process. Syst. volume 33, pp. 18611-18624, 2020.
42.0
45.6
45.8
41.6
45.9
48.3
42.5
46.3
49.3
41.5
43.1
44.3
v
41.1
45.1
48.5
43.5
47.5
50.2
Deng, J., Dong, W., Socher, R., L...J., Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. 2009 IEFE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
more enhanced model training acceleration. Unlike existing selection methods, our proposed approach identifies sam- ples suitable for data augmentation. By combining this with augmentation, our framework can improve model gener- alization with reduced training costs. As a result, we can achieve lossless training acceleration with fewer data and enhanced generalization using the same volume of data. Extensive experiments demonstrate the effectiveness and ef- ficiency our method, especially in terms of generalization across large-scale datasets and more challenging scenarios.
De Vries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1rXiv:/70..045555 2017.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner. T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Acknowledgement
Du, J., Jiang, Y., Tan, V. Y., Zhou, J. T., and Li, H. Minimiz- ing the accumulated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 3749-3758, 2023.
This work is supported by the STI 2030-Major Projects of China under Grant 2021ZD0201300 the Fundamental Research Funds for the Central Universities under Grant 2024300394, the National Natural Science Foundation of China under Grant 62276127. This work is supported by the Shanghai Municipal Science and Technology Major Project. This work is supported by Shanghai Artificial Intelligence Laboratory.
Feldman, V. and Zhang, C. What neural networks mem- orize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 13:2881-2899 2020.
Gong, C., Wang. D., Li, M., Chandra, V., and Liu, Q. Keep- augment: A simple information information-preserving data augmen- tation approach. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 1055-1064, 2021.
Impact Statement
We propose unified framework for dynamic data selection and augmentation that improves training efficiency and gen- eralization in deep learning. By reducing data and computa- tional requirements, our method promotes more sustainable and accessible AI development, especially for large-scale models and resource-constrained settings.
He, M., Yang. S., Huang, T., and Zhao, B. Large-scale dataset pruming with dynamic uncertainty. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7713-7722, 2024.
He, Z., Xie, L., Chen, X., Zhang, Y., Wang, Y., and Tian, Q. Data augmentation revisited: Rethinking the distribution gap between clean and augmented data, arXiv preprint arXiv: 099.09148, 2019.
References
Atienza, R. Improving model generalization by agreement of learned representations from data augmentation. In

---

When Dynamic Data Selection Meets Data Augmentation
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F,, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of generalization. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 1.40, 2021a.
Lin, S., Zhang, Z., Li, X., and Chen, Z. Selectaugment: Hierarchical deterministic sample selection for data aug- mentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1604-1612, 2023.
Liu, S. and Wang, X. Mgdd: A meta generator for fast dataset distillation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021b.
Liu, T. Y. and Mirzasoleiman, B. Data-efficient augmen- tation for training neural networks. Advances in Neural Information Processing Systems, 35:5124-51533 2022.
Hong, T., Wang, Y., Sun, X., Lian, F., Kang, Z., and Ma, J. Gradsalmix: Gradient saliency-based mix for image data augmentation. In 2023 IEEE International Conference on Multimedia and Expo (ICME), pp. 1799-1804. IEEE, 2023.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang. Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Hou, C., Zhang, J., and Zhou, T. When to learn what: Model- adaptive data augmentation curriculum. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1717-1728, 2023.
Maharana, A., Yadav, P., and Bansal, M. D2 pruning: Mes- sage passing for balancing diversity and difficulty in data pruning. arXiv preprint arXiv:2310.079.1 2023.
Iyer, R., Khargoankar, N., Bilmes, J., and Asanani, H. Sub- modular combinatorial information measures with appli- cations in machine learning. In Algorithmic Learning Theory, pp. 722-754. PMLR, 2021.
Malkov, Y. A. and Yashunin, D. A. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4))824- 836, 2018
Joshi, S. and Mirzasoleiman, Data-efficient contrastive self-supervised learning: Most beneficial examples for supervised learning contribute the least. In Interna- tional conference on machine learning, pp. 15356-15370. PMLR, 2023.
Mirzasoleiman, B., Bilmes, J., and Leskovec, J. Coresets for data-efficient training of machine learning models. International Conference on Machine Learning, pp. 6950-6960. PMLR, 2020a.
Kang, M. and Kim, S. Guidedmixup: an efficient mixup strategy guided by saliency maps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1096-1104, 2023.
Mirzasoleiman, B., Bilmes, J., and Leskovec. J. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960. PMLR, 2020b.
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., and Iyer, R. Glister: Generalization based data subset selection for efficient and robust learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8110-8111 8118, 2021.
Moreno-Ban Barea, F. J., Jerez, M., and Franco, L. Improving classification accuracy using data augmentation on small data sets. Expert Systems with Applications, 161:113696 2020.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Müller, S. G. and Hutter, F. Trivialaugment: Tuning-free yet saae--of-the-art data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 774-782, 2021.
Lei, S. and Tao, D. A comprehensive survey to dataset distillation. arXiv preprint arXiv:2301.05603, 2023.
Lim. S., Kim, I., Kim, T., Kim, C., and Kim, S. Fast au- toaugment. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alchéé Buc, F., Fox, E., and Garnett, R. (eds.), Proc. Adv. Neural Inf. Process. Syst., volume 32. Curran As- sociates, Inc., 2019. URL https: httLttttt///pcceeeeee neurips. cc httpp//sccc/aaaer////////eee Bad007cE50422bh444af6499333233
Nohyun, K., Choi, H., and Chung, H. W. Data valuation without training of a model. In The Eleventh International Conference on Learning Representations, 2023.
Northcutt, C. G., Athalye, A., and Mueller, J. Pervasive la- bel errors in test sets destabilize machine learning bench- marks. arXiv preprint arXiv:2 103. 10.. 14749, 2021.
pdf.
10

---

When Dynamic Data Selection Meets Data Augmentation
Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 18596-20607 2021.
Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv: 1rXiv:181.005/555 2018.
Poth, C., Sterz, H., Paul, I., Purkayastha, S., Engländer, L., Imhof, T., Vulić, I., Ruder, S., Gurevych, I., and Pfeiffer, J. Adapters: A unified library for parameter- efficient and modular transfer learning. arXiv preprint arXiv:2311.11077 2023.
Wang. Y., Liu, W., Ma, X., Bailey, J., Zha, H., Song. L., and Xia, S.-T. Iterative learning with open-set noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8688-8696, 2018.
Welling, M. Herding dynamical weights to learn. In Pro-
Qin, 7,, Wang, K., Zheng, Z., Gu, J., Peng, X., Zhao Pan, Zhou, D., Shang, L.. Sun, B., Xie, X., and You, Y. Infobatch: Lossless training speed up by unbiased dy- namic data pruning. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview net httr:/orrrmmmiicc1kkkssssk
ceedings of the 26th Annual International Conference on Machine Learning, pp. 1121-1128, 2009,
Xia, X., Han, B., Zhan, Y., Yu, J.. Gong, M., Gong, C., and Liu, T. Combating noisy labels with sample selection by mining high-diserepancy examples. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1833-1843, 2023a.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp.
Xia, X., Liu, J., Yu, J., Shen, X., Han, B., and Liu, T. Mod- erate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2023b.
PMLR, 2021.
Raju, R. S., Daruwalla, K., and Lipasti, M. Accelerating deep learning with dynamic data pruning. arXiv preprint arXiv:2/11. 126221, 2021.
Xu, M., Yoon, S., Fuentes, A., and Park, D. S. A comprehen- sive survey of image augmentation techniques for deep learning. Pattern Recognition, 137:109347, 2023.
Ramalingam, S., Awasthi, P., and Kumar, S. weighted k- center algorithm for data subset selection. arXiv preprint arXiv:2312. 10602, 2023.
Yang. S., Li, J., Zhao, J., and Shen, F. Advmask: A sparse adversarial attack based data augmentation method for image classification, 2022a.
Singh, K. K. and Lee, Y. J. Hide-and-seek: Forcing network to be meticulous for weakly-supervised object and action localization. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pp. 3544-3553. IEEE, Oct 2017.
Yang. S., Xiao, W., Zhang, M., Guo, S., Zhao, J., and Shen, F. Image data augmentation for deep learning: A survey, arXiv preprint arXiv:2204.0866
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. S. Beyond neural scaling laws: beating power law scaling via data pruning. In Oh, H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
2022b.
Yang, S., Xie, Z., Peng, H., Xu, M., Sun, M., and Li, P. Dataset pruning: Reducing training data by examining generalization influence. In International Conference on Learning Representations, 2023a.
Sun, P.. Shi, B., Yu, D., and Lin, T. On the diversity and re- alism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9390- 9399, 2024.
Yang, S., Guo, S., Zhao, J., and Shen, F. Investigating the effectiveness of data augmentation from similarity and diversity: An empirical study. Pattern Recognition, 148: 110204, 2024a.
Taesiri, M. R., Nguyen, G., Habchi, S., Bezemer, C.-P., and Nguyen, A. Imagenet-hard: The hardest images remain- ing from study of the power of zoom and spatial biases in image classification. Advances in Neural Information Processing Systems, 36, 2024.
Yang, S., Shen, F,, and Zhao, J. Entaugment: Entropy- driven adaptive data augmentation framework for image classification. arXiv preprint arXiv:2409.06299 2024b.
Yang, S., Ye, P., Ouyang, W. Zhou, D., and Shen, F. A clip-powered framework for robust and generalizable data selection. arXiv preprint ar arXiv:2410. 11215, 2024c.
Tan, H., Wu, S., Du, F., Chen, Y., Wang. Z., Wang, F., and Qi, X. Data pruning via moving-one sowing-one-samplee -ne-sample-out. Advances in Neural Information Processing Systems, 36, 2024.
Yang, Y., Kang. H., and Mirzasoleiman, B. Towards sustain- able learning: Coresets for data-efficient deep learning.

---

When Dynamic Data Selection Meets Data Augmentation
In International Conference Machine Learning, pp- 39314-39330. PMLR, 2023b.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paze D. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018. URL httpt:ttp//........ /ettps://openreviiie net forum? id id-r1Ddpl-Rb.
Zhang. X., Du, J., Li, Y., Xie, W., and Zhou, Spanning training progress: Temporal dual-depth scoring (Idds) for enhanced dataset pruning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26223-26253 2024.
Zheng. H., Liu, R., Lai, F., and Prakash, A. Coverage-centrri coreset selection for high pruning rates. In The Eleventh International Conference Learning Representations, 2023.
Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang, Ran- dom erasing data augmentation. In Proc. AAAI, vol- ume 34, pp. 13001-13308, 2020.
12

---

When Dynamic Data Selection Meets Data Augmentation
A. More Implementation Details
A.I. Augmentation Space
Table Augmentation operations and their magnitudes.
Operation
Value Range
Magnitude-based
Identity
X
ShearX
[0.0, 0.99
ShearY
[0.0, 0.99
TranslateX
30.0,
Translate
[0.0,32.0]
Rotate
0.0,
Brightness
[..9,
Color
[0.0,
Contrast
0.99
Sharpness
[0.0, 0.99
Posterize
[2.8]
Solarize
225.00, 0.0]
AutoContrast
Equalize
B. Discussion and Future Work
In this paper, we propose a novel online data training framework that unifies dynamic data selection and data augmentation to achieve enhanced model training acceleration. this section, we discuss some potential limitations and future work for our method. 1). Our proposed method based on the pretrained CLIP model to estimate the sample semantic consistency. While this exhibits superior effectiveness in general datasets, further applying our method special tasks, such as medical imaging, where pretrained multimodal models are unavailable mismatched, is worth exploring in future work. 2). We employ our framework on image classification tasks and demonstrate its superior effectiveness. Future work should extend the applications to more real-world tasks, such as object detection and segmentation.
13


Reviews:
[{"review_id": "QZzg6ci9z0", "reviewer": "Reviewer_JbtQ", "summary": "This paper combines data augmentation and dynamic data selection. The main idea is to augment examples that the model is uncertain about, while filtering noisy examples using semantic consistency. The experimental results show that with only 50% training compute, equal performance can be achieved to training on the full dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer JbtQ,\n\nWe sincerely thank you for the comments and constructive suggestions. We appreciate your recognition of our work's strengths, e.g., reasonable experimental designs and well-supported claims. \n\nFor the comments, we provide our response as follows.\n\n- **Q1: Comparison with Data-Efficient Augmentation (DEA) [a].**\n- **A1**: Thank you for suggesting additional references and comparisons. We acknowledge the theoretical significance of Data-Efficient Augmentation, particularly its principled approach to rigorously determining which samples are most useful for training.\n\nIn response to your suggestion, we compared our method with the suggested data-efficient augmentation using the same amount of training data and number of training iterations, across standard benchmarks (Table A-1), and under challenging noisy and corrupted scenarios (Table A-2). For ImageNet-1k, we utilized the reported results from [a].\n\nThe results show that our method consistently achieves higher accuracy than DEA across selection ratios and datasets, achieving at least 3% higher accuracy on ImageNet-1k, and notable gains on CIFAR-10 and Tiny-ImageNet.\nMoreover, as shown in Table A-2, our method demonstrates stronger robustness to both noisy and corrupted conditions.\n\n\n\n**Table A-1**：Comparison with [a] on CIFAR-10 with ResNet-18 and Tiny-ImageNet and ImageNet-1k with ResNet-50. \n|||C-10|||T-IN|||IN1k||\n|-|-|-|-|-|-|-|-|-|-|\n|Selection Ratio (%)|30|50|70|30|50|70|10|30|50|\n|[a]|83.4|88.1|88.6|31.9|36.9|43.5|68.5|72.0|73.3|\n|Ours|**94.9**|**95.5**|**96.0**|**44.9**|**47.0**|**49.4**|**71.5**|**75.0**|**76.5**|\n\n\n**Table A-2**: Comparison with [a] under noisy and corrupted conditions using Tiny-ImageNet with ResNet-50.\n||Noisy||Corrupted||\n|-|-|-|-|-|\n|Selection Ratio (%)|20|30|20|30|\n|[a]|25.8|28.2|31.8|31.9|\n|Ours|**35.9**|**39.6**|**39.1**|**42.0**|\n\n[a] Liu, Tian Yu, and Baharan Mirzasoleiman. \"Data-efficient augmentation for training neural networks.\" NeruIPS 2022.\n\n- **Q2: Suggested References.**\n\n- **A2:** Thank you for suggesting more related works.\nSince the revised manuscript can not be updated at the current stage, we will include these references in Section 2.1 in the final version. Specifically, \n\n    - Sec 2.1, paragraph 3: add references \"*The work (Liu \\& Mirzasoleiman, 2022) proposes a theoretically rigorous approach to determine impactful data for training. SAS (Joshi \\& Mirzasoleiman, 2023) improves data efficiency in SSL by proving and selecting the most beneficial data for contrastive training.*\"\n\nWe hope these additions and clarifications address your comments.", "claims_and_evidence": "Yes, the claims are well supported.", "methods_and_evaluation_criteria": "I think a key baseline is missing: (Data-Efficient Augmentation) Liu, Tian Yu, and Baharan Mirzasoleiman. \"Data-efficient augmentation for training neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 5124-5136.\nThis paper is perhaps the most relevant baseline for comparison as it similarly selects a subset of data for augmentation. It proposes a more theoretically rigorous approach to determine which examples are most useful for training. While uncertainty-based methods can sometimes prioritize difficult but unlearnable samples or even noisy samples, and semantic consistency might mitigate this issue, using model uncertainty as a heuristic for sample importance remains limited. The aforementioned paper instead leverages model gradients to identify the samples that contribute most significantly to learning, providing both theoretical guarantees and empirical validation of this approach's effectiveness.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Looks reasonable.", "relation_to_broader_scientific_literature": "Discussed in essential references not discussed.", "essential_references_not_discussed": "Missing a reference to some key works exploring similar ideas: \n- (Data-Efficient Augmentation) Liu, Tian Yu, and Baharan Mirzasoleiman. \"Data-efficient augmentation for training neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 5124-5136.\nThe paper above is perhaps the most relevant baseline for this paper as it too selects a subset of data to augment. \nOther Relevant Work:\n-  (Submodular Data Selection) Joshi, Siddharth, and Baharan Mirzasoleiman. \"Data-efficient contrastive self-supervised learning: Most beneficial examples for supervised learning contribute the least.\" International conference on machine learning. PMLR, 2023.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "I would highly recommend the authors to include the missing baseline and references. I think this would strengthen the paper significantly.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vRoAaghxPj", "reviewer": "Reviewer_bQXA", "summary": "Data selection---eliminating unhelpful samples---plays a crucial role in machine learning. While selecting high-value samples improves training efficiency without degrading performance, it can reduce data diversity and harm model generalization. To address this, this paper proposes a unified framework that integrates data selection and data augmentation, achieving both reduced computational cost and enhanced performance. Moreover, instead of naively combining these two techniques, they estimate each sample's local density and semantic alignment in a joint distribution with the help of multimodal model CLIP. Empirical results show that our approach accelerates training while improving model generalization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bQXA,\n\nWe sincerely thank you for the careful review and insightful comments/questions. We appreciate your recognition of our work's strengths, e.g., novelty and empirical effectiveness.\n\nFor the comments and questions, we provide our responses here:\n\n- **Q1: The impact of data selection on generalization.**\n- **A1:** Thank you for your insightful comment. \nTo assess the impact of data selection on generalization, we have evaluated models trained on the selected data across various benchmark datasets (Tab. 1/3/4/5/6, Fig. 3,), which are commonly used to assess generalization performance. The results present a consistent trend across all baselines, as the selection ratio decreases, model test performance degrades, indicating reduced generalization. Despite this overall degradation, our method effectively alleviates this degradation by integrating data augmentation (DA) and selection.\n\n- **Q2: Clarification on the data used for training.**\n- **A2:** We would like to emphasize that only the augmented data is used for training. This follows common practice in many widely used DA and selection methods (e.g., TrivialAug, AutoAugment, InfoBatch, Moderate, etc.), where only the augmented version of selected data is used for training.\nRetaining the original and augmented versions would **double** the number of training data used for model forward and backward passes, thereby doubling the computation costs. This would **substantially undermine the training efficiency** that our method is designed to achieve.\n\n    While only augmented data is used for training, our augmentation strategy is carefully designed to preserve semantic consistency. Please refer to **A3** and **A4** for details.\n\n- **Q3: Semantic alignment before and after augmentation.**\n- **A3:** Thanks for the insightful comments. While our method does not explicitly verify each augmented data post-augmentation, our DA strategy is designed to prioritize semantic alignment.\nSpecifically, each selected data is augmented using **only one operation** with its magnitude contained within a **moderate, predefined range** (Sec. A.1).\nIn contrast, widely-used methods, such as AutoAug, Fast-AA, and RandAug, typically apply **2-3 operations per sample** with stronger transformation strengths, emphasizing more diverse training data rather than maintaining semantic consistency.\n\n    To further address your concern, we investigate the semantic alignment with a vanilla pretrained CLIP model using cosine similarity. Tab. B-1 shows that **the augmented data maintains a strong semantic consistency** with the original data.\n    \n    While our current method does not include explicit post-augmentation verification to maintain high efficiency, we agree that incorporating a lightweight verification mechanism, without compromising efficiency, is a promising direction for future work, which is also a key challenge in DA research.\n\n**Table B-1**: Semantic alignment between original and augmented data points.\n||C-10|C-100|\n|-|-|-|\n|Avg. Sim.|0.94|0.93|\n|Std.|0.06|0.05|\n\n- **Q4: Clarification on local structure after augmentation.**\n- **A4:** To further assess the local structural stability, we investigate changes in each sample's local nearest neighbors before and after augmentation. As shown in Tab. B-2, the proportion of altered neighbors is extremely low (<= 0.3%), validating that our DA introduces only minimal changes to local structure.\n\n    However, it is important to note that the feature space evolves continuously during training as the model updates. Thus, even if the augmented data remains semantically and structurally stable, it is **necessary to update the embedding space** to capture the model's current training state for dynamic data selection. This is also the core principle of dynamic data selection approaches.\n**Table B-2**: The ratio of changes in local nearest neighbors after augmentation.\n||Change Ratio|\n|-|-|\n|C-10|0.2%|\n|C-100|0.3%|\n- **Q5: Clarification on the embedding generators.**\n- **A5:**  As clarified in Sec. 3.1, only the CLIP model is used to derive the semantic consistency distribution for filtering, which captures the intrinsic representativeness of training data. Meanwhile, as introduced in lines 207-213, to adapt CLIP to the target domain, we use a lightweight adapter to enable domain-specific knowledge transfer while preserving CLIP's strong alignment capabilities.\nOn the other hand, the task model (e.g., ResNet or ViT series) is used to estimate the evolving density distribution during training. Thus, our framework minimizes the inherent biases.\n\n- **Q6: Details of augmentation operations.**\n- **A6:** For each sample, we apply **only one random augmentation operation** from Sec A.1. Following common practice in data augmentation (e.g., AutoAug, TrivialAug), we control the number of applied operations and the applied strength via a predefined, bounded magnitude range, ensuring consistency and avoiding overly distorted augmentation.", "claims_and_evidence": "1. I agree that data selection can improve training efficiency and maintain comparable performance, but it inherently **reduces data diversity, which could impact generalization**. However, the claim that it harms model generalization lacks supporting evidence. Are there any preliminary results to validate this statement? And how to set up such experiments to present or measure model generalization.\n\n2. In the proposed framework, if a data point is selected for augmentation, is the model trained on both the original and augmented versions, or only on the augmented one? From my understanding, only the augmented version is used, but wouldn’t retaining the original sample help preserve semantic completeness?\n\n3. The proposed method does not explicitly control the outcomes of data augmentation. How can we ensure that post-augmentation samples maintain strong semantic alignment with the original data points and do not negatively impact model training? Introducing a verification mechanism for post-processing might be beneficial.\n\n4. Furthermore, if the augmentation process preserves local structure (as mentioned in lines 210–211, right-hand side), is it necessary to continuously update the feature space during training to identify low-density samples? How does the relationship between a sample and its neighbors change before and after augmentation?", "methods_and_evaluation_criteria": "Their evaluation criteria is reasonable to me, and most of the experimental setups align with standard practices in the data selection field.", "theoretical_claims": "There is no major parts for theoretical claims but i have walked through their problem setup. It looks correct to me.", "experimental_designs_or_analyses": "Yes, their experimental design makes sense to me. However, in their setup, the CLIP model and a ResNet-18 model are used as **embedding generators for filtering**, while the model being trained is from the **ViT series**. Could this lead to differences in the embedding space or quality? Additionally, would there be inherent biases due to the different pretraining datasets used for these models?", "relation_to_broader_scientific_literature": "The key contribution of this paper is highlighting the downside of data selection—namely, **the reduction of data diversity and its potential impact on model generalization**. To address this, they propose **an integrated framework** that combines data selection and data augmentation in a unified approach, which is novel to data selection literature.", "essential_references_not_discussed": "There are no major additional references to include, as the baseline methods used for comparison are commonly employed in the data selection literature.", "other_strengths_and_weaknesses": "I suggest investigating the domain shift before and after data augmentation. If a significant shift exists, should we consider training with the original data points as well? If there is no significant shift, would it be necessary to repeatedly update the embedding space during training?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See my questions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zfrDNpiDyB", "reviewer": "Reviewer_GA1s", "summary": "The authors propose a unified framework combining dynamic data selection and data augmentation to accelerate model training. An online nearest neighbour search is used to find low-density samples along with a semantic consistency score from a pre-trained CLIP model to filter out noisy data. The targeted augmentation of the filtered data helps fill data gaps and sharpen decision boundaries in sparse regions, improving model generalization and robustness. The authors claim to achieve lossless training acceleration with fewer data points using a similar amount of data.\n\n## update after rebuttal\nThe authors have sufficiently addressed all my concerns and included more analyses to show the strength of their method. I have accordingly increased my score as a result.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer GA1s,\n\nThank you for your meticulous review and valuable suggestions on our work. We appreciate your recognition of our work's strengths, e.g., novelty, sound and valid experiments. \n\nWe provide responses to address the comments as follows: \n\n- **Q1: Universal applicability and reliance on CLIP.**\n- **A1:** Thank you for pointing this out. We agree that the applicability and effectiveness of pretrained CLIP may vary across domains, particularly in some highly specialized fields.\nHowever, we note that this concern is not unique to our work but is shared by many state-of-the-art works that leverage pretrained VLM such as CLIP. \nSince our work focuses on dynamic data selection in general-purpose vision domains, fully addressing CLIP’s applicability across all domains and tasks is, in our opinion, beyond the scope of this work but of interest for our future work.\nSome feasible solutions may include leveraging fine-tuned VLM tailored to the target domain, or incorporating domain-specific adapters or alignment modules to improve semantic consistency estimation in more specialized scenarios. \n\n    We will add this discussion to the main paper on page 8 before the Conclusion section.\n\n- **Q2: Extreme noise conditions.**\n- **A2:** Thank you for the suggestion. To assess the robustness of our method under extreme noise conditions, we conducted experiments on Tiny-ImageNet with high label, input noise, and both (with 50% and 70% noise ratio).\n\n    As shown in Table C-1, while the overall performance naturally degrades under severe noise, our approach consistently achieves significantly higher accuracy, e.g., over 10% accuracy improvement in the presence of input noise. Moreover, we further analyze the average proportion of noisy samples retained during training (Table C-2). It shows that the proportion of noisy samples in our selected datasets remains considerably low even in high-noise conditions. These results underscore the benefits of our approach: even in highly noisy environments, the multimodal semantic consistency used for sample selection helps retain informative, high-quality data and maintain model robustness.\n\n**Table C-1**: Comparison of accuracy with Tiny-ImageNet in high-noise conditions. \n|Noise Ratio (%)|50||70||\n|-|-|-|-|-|\n|Selection Ratio (%)|20|30|20|30|\n|**Label Noise**||||\n|Random*|25.5|26.1|17.6|18.0|\n|Ours|**27.6**|**30.7**|**19.2**|**21.8**|\n|**Input Noise**||||\n|Random*|28.5|28.1|18.6|19.0|\n|Ours|**38.3**|**40.9**|**35.6**|**39.4**|\n|**Label\\&Input Noise**||||\n|Random*|24.1|25.2|14.2|15.9|\n|Ours|**26.1**|**28.1**|**16.2**|**18.5**|\n\n**Table C-2**: Further analysis of the label noise proportion (%). We report the average introduced noise ratio in the selected datasets through the entire training process.\n|Noise Ratio (%)|50||70||\n|-|-|-|-|-|\n|Selection Ratio (%)|20|30|20|30|\n|Random|20.3|30.5|20.1|29.9|\n|Ours|**3.1**|**5.7**|**4.0**|**6.1**|\n\n\n- **Q3: More suggested related works.**\n- **A3:** We will include the suggested works in Sec. 2.2 of the revised version. Specifically, \n\n     - Sec 2.2 line 152: add references \"*Beyond these, generative data augmentation (Moreno-Barea et al., 2020) ...*\" and \"*Recent studies also emphasize representation consistency (Atienza, 2022) and address distribution gaps between clean and augmented data (He et al., 2019) ...*\".\n\n\n- **Q4: Minor Typos.**\n- **A4:** We have corrected all the mentioned issues and thoroughly reviewed the entire manuscript to ensure clarity and correctness in the final version.\n\n\n- **Q5: Explanation of $p_{sel}$.**\n- **A5:** As defined in Eq. (3) of Sec. 3.4, $p_{sel}$ is computed as the product of $p_\\rho$, which reflects the density distribution, and $p_{con}$, which captures semantic consistency.\nSpecifically, $p_\\rho$ assigns higher values to low-density samples, promoting coverage of underrepresented regions in the feature space.\n$p_{con}$ prioritizes samples with strong semantic alignment, ensuring their informativeness and relevance\nBy combining the two, $p_{sel}$ encourages samples that are both structurally important and semantically meaningful. \n\n    We appreciate your suggestion and will revise Sec. 3.4 to include this clarification. \n\n\n- **Q6: Sensitivity to the choice of pre-trained model.**\n- **A6:** Insightful question. To assess the sensitivity of our method to the choice of pre-trained models, we conduct experiments replacing CLIP with another pretrained multimodal model LanguageBind (LB)[a], which focuses on aligning diverse modalities (e.g., video, audio, infrared, and depth) into a shared language space.\nWhile CLIP is highly optimized for image-text alignment estimation, our framework still achieves consistent performance when using LB, with only a marginal drop (< 0.7%) in Table C-3.\n\n**Table C-3**: Sensitivity analysis on the pretrained model. \n|Selection Ratio (%)|30|50|70|\n|-|-|-|-|\n|Random*|41.5|42.8|43.1|\n|CLIP|44.9|47.0|49.4|\n|LB|44.2|46.5|48.7|\n\n[a] Zhu, Bin, et al. Languagebind, ICLR'24.", "claims_and_evidence": "Most of the claims made by the authors are supported by clear and convincing evidence. From the results, integrating dynamic data selection with targeted augmentation seems to improve efficiency and generalization. One claim that needs a bit more validation though is the universal applicability of the method to all domains or tasks. The reliance on a pre-trained CLIP model for semantic consistency might not transfer well to domains where such models are less effective. It may also be useful to study the benefits and limits of the method under extreme noise conditions (both label and input noise) which might affect the augmenter part of the framework.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem at hand. The authors use the CIFAR-10, CIFAR-100, Tiny-ImageNet, and also the ImageNet-Hard, ImageNet-A, ImageNet-R, and ImageNet-O datasets. They use the ResNet-18 and ResNet-50 models along with more advanced architectures such as ViT-B, ViT-L, and Swin-T to show the strength of their method. The selection ratio and accuracy is reported across all of these experiments.", "theoretical_claims": "There are no major theoretical claims or proofs in this paper.", "experimental_designs_or_analyses": "The experimental design and analyses seem sound and valid. The authors include extensive experiments to show evidence that their method outperforms several baselines. They include results showing that models trained across different selection ratios achieve comparable performance to models trained on the full dataset and the proposed method has one of the lowest computational costs. They also report results showing that their method works on several datasets and models, including more advanced architectures. They show the method's robustness to noise and perform an ablation study on the density distribution, consistency distribution, and augmenter to show that all three are necessary for the best results.\n\nThe authors mention that they leverage a pre-trained CLIP model to embed images and text into a shared multimodal space for semantic alignment assessment. They mention that they use lightweight adapters to adapt embeddings to target domains, however, if there are target domains where CLIP-like models or adapters are less effective, this method may not work too well since this is a major part of the framework.", "relation_to_broader_scientific_literature": "The key contributions of this work are in combining the existing ideas of data selection and data augmentation to pick the best data to augment. Previous data selection methods are more static and might not find low-density samples. Previous data augmentation methods are not as targeted and might not fill in sparse gaps in the training distribution. Combining these two areas to both identify underrepresented samples while selecting ones that are best for augmentation fills a gap in the literature while reducing training costs and enhancing robustness to noise.", "essential_references_not_discussed": "It may be useful to mention generative methods of data augmentation:\n\nMoreno-Barea, F.J., Jerez, J.M. and Franco, L., 2020. Improving classification accuracy using data augmentation on small data sets. Expert Systems with Applications, 161, p.113696.\n\nSome more recent works that talk about improving generalization by agreement and the gap between clean and augmented data:\n\nAtienza, R., 2022. Improving model generalization by agreement of learned representations from data augmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 372-381).\n\nHe, Z., Xie, L., Chen, X., Zhang, Y., Wang, Y. and Tian, Q., 2019. Data augmentation revisited: Rethinking the distribution gap between clean and augmented data. arXiv preprint arXiv:1909.09148.", "other_strengths_and_weaknesses": "Strengths:\n- Novel combination of the existing ideas of data selection and data augmentation to come up with data selection for data augmentation.\n- Extensive experiments across various models, datasets, and selection ratios.\n- Robustness to noise.\n- Low computational costs.\n\nWeaknesses:\n- The use of CLIP-like models in the framework which might not transfer to all tasks and domains.\n- Might be good to study the method's limits under extreme noise conditions (label and input noise).\n- A few typos (mentioned below).", "other_comments_or_suggestions": "Minor typos:\n- 012: \"lossless performances\" -> \"lossless performance\"\n- 042: \"reinforces the model learning\" -> \"reinforces model learning\"\n- 088: \"riak\" -> \"risk\"\n- 101: unclear \"balances the distribution and the sample distribution and importance in selection\"\n- 104: \"Optimization-based methods formulates\" -> \"Optimization-based methods formulate\"\n- 645: \"occlution\" -> \"occlusion\"", "questions_for_authors": "1. How exactly is $p_{sel}$ understood as the joint distribution that combines both density and consistency distributions? The definition and explanation need more clarity - it is defined but never mentioned again.\n2. Have you tested the sensitivity of your approach to the choice of the pre-trained model or explored its applicability in domains where CLIP or the available pre-trained models/adapters might not perform well? More details on the pre-trained models and adapters would be very beneficial.\n\nAddressing these questions sufficiently would improve the strength of this method significantly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6dmsPg6kP", "reviewer": "Reviewer_3S6N", "summary": "Paper centers around the idea of combining dynamic data selection and augmentation in order to increase both quality and diversity of data. The proposed method is applicable specifically to multimodal data. The proposed methods selects augmentation candidates that are both low density and do not represent noisy outliers. The later selection criteria is fulfilled by focusing on the examples for which the samples' and labels' representations align (cos similarity of embeddings) according to a CLIP model fine-tuned on the original (pre-selection&augmentation) data. Authors demonstrate the effectiveness and efficiency of the proposed method and its components in various comparisons to various baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 3S6N,\n\nThank you for your careful review and constructive suggestions on our work.  We appreciate your recognition of our work's strengths, e.g, novelty, well-structured experiments.\n\nWe provide responses to address the comments as follows: \n\n- **Q1: More results in Table 1.**\n- **A1:** Thank you for your suggestions. We have extended our evaluation to include more results in Tab. 1 (as Tab. 4 shows results on Tiny-ImageNet).\n\n    As shown in Tab. D-1, while both baselines benefit from augmentation, our method consistently achieves higher accuracy. This indicates that the performance gains are not solely due to augmentation but stem from our method's ability to identify samples suitable for augmentation. Thus, by unifying dynamic data selection and augmentation, our framework achieves both training acceleration and enhanced performance.\n\n**Table D-1:** Additional results for Tab. 1.\n||C-10|||C-100|||\n|-|-|-|-|-|-|-|\n|Selection ratio (%)|30|50|70|30|50|70|\n|Random*|93.3| 94.8| 95.2|76.8|77.9| 78.5|\n|Infobatch|94.8|95.3| 95.9|77.1|78.5 |78.7|\n|Ours|**94.9**|**95.5**|**96.0**|**77.6**|**78.9**|**79.5**|\n\n\n- **Q2: Clarification on the fine-tuning process.**\n- **A2:** We would like to clarify that the fine-tuning process does **NOT** use the selection\\&augmentation methods. The CLIP backbone is kept frozen; we only train the lightweight adapter from scratch.\n\n- **Q3: Performance of the fine-tuned model.**\n- **A3:** Good question. First, in Tab. D-2, we show the accuracy of CLIP before and after fine-tuning.\nIt shows that the fine-tuned CLIP achieves notable improvements, indicating enhanced dataset-specific alignment ability.\nSecond, we employ the pretrained CLIP without fine-tuning in our method. In Tab. D-3, while the vanilla CLIP achieves high performance, the fine-tuned model further improves the performance.\n\n**Table D-2:** Performance of CLIP in classification accuracy.\n||C-10|C-100|\n|-|-|-|\n|zero-shot|89.8|65.1|\n|fine-tuned|**94.8**|**76.8**|\n\n**Table D-3:** Performance of CLIP with Tiny-ImageNet.\n|Selection ratio (%)|30|50|70|\n|-|-|-|-|\n|w/o fine-tuning| 40.2|45.7|48.5|\n|Ours| **44.9**|**47.0**|**49.4**|\n\n- **Q4: Clarification on the cost in Tab. 2.**\n- **A4:** Yes, the cost of fine-tuning is **included** in the total costs reported in Tab. 2 to ensure a fair comparison with other methods.\nWe freeze the CLIP backbone and only fine-tune a lightweight adapter (a simple linear layer constituting only 0.04% of CLIP ViT-B/32’s parameters) with minimal training iterations. Thus, the fine-tuning process can be completed efficiently.\n\n- **Q5: More results for Tab.3.**\n- **A5:** Thank you for the suggestion. \nAs shown in Tab. D-4, our method maintains superior performance across both settings and selection ratios. Notably, Infobatch performs worse than the random baseline in some cases in noisy conditions. This is because it prioritizes high-loss samples, which are often noisy or corrupted under noisy scenarios. Applying augmentation to noisy samples further exacerbates their negative impact as more misleading training signals are introduced, ultimately degrading performance.\n\n    This highlights that our performance gain is not solely due to augmentation, but rather due to selecting the semantic-representative samples for training.\n\n**Table D-4:** Additional results for Table 3.\n||Noisy||Corrupted||\n|-|-|-|-|-|\n||20|30|20|30|\n|Random*|34.8|37.9|36.0|39.1|\n|Infobatch|34.9|37.1|35.1|38.1|\n|Ours|**35.9**|**39.6**|**39.1**|**42.0**|\n\n- **Q6: Clarification on the selection ratio.**\n- **A6:** We would like to clarify that, following common practice in data augmentation and selection research, our method trains models **only on the augmented data**. In each epoch, we generate **exactly one augmented sample per selected data point**, and only these augmented samples are used in both forward and backward passes. Thus, the selection ratio is exactly equal to the proportion of the data used for training.\n- **Q7: Clarification of multimodal focus in the Introduction.**\n- **A7:** Thank you for the suggestion. We will revise the intro part, clarifying the multimodal characteristic and focus of our framework.\n- **Q8: Typo.**\n- **A8:** We have corrected the mentioned typo and thoroughly reviewed the manuscript to ensure clarity and correctness.\n- **Q9: Applicability to LLM training.**\n- **A9:** Applying our method to other domains, such as data selection for LLM training, is a promising direction for future research. Since our current work focuses on the image and image-text domains, extending this framework to LLM training requires adapting the definitions and estimation for both semantic consistency and sample density in a purely textual feature space. Feasible solutions may involve leveraging pre-trained language models (e.g., BERT, GPT) to assess semantic alignment and correctness, and using task-specific metrics such as token-level uncertainty or embedding-space sparsity to approximate density distributions.", "claims_and_evidence": "The central claim of the paper is: selecting samples according to the proposed methodology improves model generalization with reduced training cost. The claim seem to be justified overall. Yet, I still have some doubts and questions regarding the experiments presented in the paper, see following sections.", "methods_and_evaluation_criteria": "Yes, they make sense overall.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "I checked the experimental design and analysis. I have the following questions and I apologize if I have overlooked something and the answers are already in the manuscript:\n- Is the only method that includes augmentation in Table 1 the methods presented in this paper? I would like to see at least the result of Random + augmentation (and optimally InfoBatch + augmentation) on all datasets (Table 4 only shows it for Tiny-ImageNet)\n- In the presented experiments, did authors fine-tune a pre-trained CLIP model with the selection&augmentation methods appleid? Or was training from scratch used? \n- iiuc authors have to fine-tune a CLIP model that is used for semantic alignment estimation on the data prior to filtering. It would be interesting to include the performance of this model in the results.\n- is the cost of fine-tuning CLIP included in the results presented in Table 2?\n- for Table 3, I would be interested in seeing the performance of top 2 methods with augmentation (i.e. in Table 3 also the only method using augmentation is the one presented in this work?)\n- when talking about selection ratio, e.g. Figure 3 x-axis, Table 4 etc., the ratio does not include the augmented examples, doe it? i.e. the numb er of examples the model is trained on is actually larger than e.g. 20% of the full dataset for selection ratio = 20%? What is the augmentation ratio (i.e. how many new examples are generated per selected example in the presented experiments?)", "relation_to_broader_scientific_literature": "While the title and the introductions are formulated rather generally, the paper's focus is very much on multimodal image-text data. Hence, the focus of the related works section is on multimodal or image only domains. I suggest authors make it more clear already in the intro that the focus of this work is on multimodal data specifically.", "essential_references_not_discussed": "Given that the focus is image & multimodal image-text domains, the references are discussed sufficiently well to the best of my knowledge.", "other_strengths_and_weaknesses": "I would like to see the experimental section as the main strength of this paper. The experiments are telling a nice story and are well designed in order to justify the main claims of the paper. However, given the various questions I had while reading this section (see experimental design section), it is also the main weakness of this paper at this point.", "other_comments_or_suggestions": "- typo in line 088 second column \"riak\" -> \"risk\"\n-  can authors discuss how applcable this method would be to other domains like e.g. data selection for LLM training?", "questions_for_authors": "See \"experimental design\" section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework which combines dynamic data selection and data augmentation to improve training efficiency without sacrificing generalization. By selecting low-density, semantically consistent samples for augmentation, the method achieves strong empirical performance across several benchmarks and architectures. 

Three reviewers recommended acceptance, as they found the motivation clear, the integration of ideas effective, and empirical performance solid, and ablations well-designed. However, one reviewer raised a concern about whether robustness to noise adequately supports generalization claims. The authors addressed this with additional evaluations on ImageNet-A/O/R/Hard and a cross-domain transfer task, which helped mitigate the concern from the reviewer, but was not strong enough to update the score. Yet, there was a consensus among the reviewers to accept the paper. One remaining limitation is that the method trains only on augmented samples, which may limit its applicability to generative models where preserving the original data distribution is more important.
Decision: Accept (poster)