Swift Sampler: Efficient Learning of Sampler by 10 Parameters
Authors: Jiawei Yao
Chuming Li
Canran Xiao
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement learning
AutoML
Primary area: reinforcement_learning

Abstract: Data selection is essential for training deep learning models. An effective data sampler assigns proper sampling probability for training data and helps the model converge to a good local minimum with high performance. Previous studies in data sampling are mainly based on heuristic rules or learning through a huge amount of time-consuming trials. In this paper, we propose an automatic swift sampler search algorithm, SS, to explore automatically learning effective samplers efficiently. In particular, SS utilizes a novel formulation to map a sampler to a low dimension of hyper-parameters and uses an approximated local minimum to quickly examine the quality of a sampler. Benefiting from its low computational expense, SS can be applied on large-scale data sets with high efficiency. Comprehensive experiments on various tasks demonstrate that SS powered sampling can achieve obvious improvements (e.g., 1.5% on ImageNet) and transfer among different neural networks. Project page: https://github.com/Alexander-Yao/Swift-Sampler.

Swift Sampler: Efficient Learning of Sampler by 10 Parameters
Jaawei Yao Chuming Li Canran Xiao
University of Washington Shanghai Artifcial Intelligence Laboratory
3 TTe University of Sydney 4 Central South University jwyaolux. edu, ch1i3951 @uni sydney edu. au, xiaocanran@csu.
Abstract
Data selection is essential for training deep learning models. An effective data sampler assigns proper sampling probability for training data and helps the model converge to a good local minimum with high performance. Previous studies in data sampling are mainly based on heuristic rules or learning through huge amount of time-consuming trials. In this paper, we propose an automatic swift sampler search algorithm, SS, to explore automatically learning effective samplers efficiently. In particular, SS utilizes novel formulation to map sampler to low dimension of pareme-rrmetters and uses an approximated local minimum to quickly examine the quality of sampler. Benefiting from its low computational expense, SS can be applied on large-scale data sets with high efficiency. Comprehensive experiments various tasks demonstrate that SS powered sampling can achieve obvious improvements (e.g., 1.5% on ImageNet) and transfer among different neural networks. Project page: https://githubb.com htwp:thhwwbooo///eaaneee -aeeaanndee-Yoo/iiitttt Yao/Swift-Samplee
Introduction
Training data plays pivotal role in deep learning tasks. The sampling probability of data in the training process can significantly influence the performance of the learned model. A set of works
Needell et al. 2014 Johnson and Guestrin 2018 Han et al. 2018 Hu et 2024] have demonstrated improvements in model training by sampling data according to different features and under different rules. These studies reveal that how sample examples during training is nontrivial and sampling data in totally uniform way is not always the
[Tiang et al. 2019 Katharopoulos and Fleuret [
optimal choice.
Previous works on data sampling strategy include two main categories: human Hluman-defneed rules and ceaming-basedd methods. Some works (Han et al. 2018 Jiang et al., 2019 Katharopoulos and Fleuret 2018 Hacohen and Weinshall 2019, Kumar et al., (2100] proposed manually designed rules, such as setting the sampling probability of training examples with loss values larger than threshold zero, or making the sampling probability proportional to the gradient norms. Such human-defined rules are designed for specific tasks and hardly adapts different scenarios. because optimal sampling strategy varies among different tasks and data sets. Learning-
methods [[an et al. 2017 Jiang et al.) 2017 Ren al. 2018) explore automatic way to assign sampling probability
given learning task, inclu including sample-basee and differential based methods. Sample-bar based methods take advantage of deep reinforcement learning (DRL) [an et al., [017]] model the training process of the target model as an environment and use DRL model to learn the optimal sampling strategy during training These methods need hundreds of replays of the training process and require too much searching cost to be applied on large-scale data sets, e.g., ImageNet (Russakovsky et al.) 2015). Differential based methods [Ren et al. 2018 Shu et al.) 2019 assume a completely clean meta dataset and use the inner
'These authors contribute equally to this work. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

product between the gradients of training data and meta data to sample or reweight training data. The problem is that the clean meta data is not guaranteed to be available for all scenarios, and complex changes are made in the training process.
To explore an automatic way of sampler search, we focus on improving sample-hane cample-hased search methods as they do not require extra meta data and impose no change in the training process when applying the obtained sampler. Typically, sample-based method repeatedly uses an agent (e.g., Bayesian Optimization or Deep Reinforcement Learning) to sample a sampler and evaluate the objective function value of the sampled sampler to update the agent. There are three challenges in designing a sample-ber sample-based method for automatic sampler search: (1) High dimension. A sampler is defined by vector of sampling probabilities of all instances in the training set. Therefore, the complexity of searching the optimal sampler in a sample-basee
way increases exponentially with the number of training instances. (2) Sharpness. As great difference exists among gradients of various instances in the training set, sampling probability of some instances has a significant impact on the performance of resulted models. Such property results in the sharpness of the objective function for samplers and harms the efficiency of the agent's ss learning. (3) Costly evaluation. The agent seeks a sampler that achieves high performance when used to train the model from scratch. Thus the evaluation of single sampler causes high computational expense (e.g., training 100 epochs from scratch on ImageNet).
In this paper, we aim at addressing the three problems listed above. For the high dimension problem (I), we define sampler as a function mapping the feature of the training data to their sampling probabilities and formulate the search space of sampler as a family of composite functions, which are represented by small number of prrper-prramterr.. The amount is much less than the number of training data. Moreover, our formulation of the sampler has flexible expression covering any features used in previous works and is adaptive to a range of data sets. We carefully choose the features and mapping functions so that the number of parameters to be learned for the sampler is only 10. To relieve the sharpness problem (2) of the objective function, we modify the objective function by designing a transform function to smooth the landscape of the objective function. This transform function balances the amounts of gradient norm in different areas of the definition domain of the objective function. For the costly evaluation problem (3), we use a fast approximation method for learning the network, which is much less expensive than training from scratch. Integrating the designs above, we propose an automatic swift sampler learning (SS) method, where we choose naive Bayesian Optimization (BO) (Brochu et al. 2010 Klein et al. 2016)] (nook et al.) 2012] the agent. The swift here means our formulation of the sampler has a very small volume of parameters, and our method has high speed of sampler learning, like a swift, the small and fast bird.
We apply SS to training neural networks with various sizes, including ResNet-18 and SE-ResNext- 101, with training data from different data sets including ImageNet (Russakovsky et al. 2015). CIFARIO and CIFAR 100 Krrzhevsky et al. 2009]. Experiments demonstrate obvious improvements in the performance compared with the baseline lineline and other methods, e.g., 1.5% ImageNet. During the search process, SS optimizes much faster compared with previous automatic methods, and the learned sampler consists of only 10 byper-parametert Further analysis shows that the sampler found by SS transfers well among neural networks with various architectures and sizes. The contributions of this paper are summarized as follows:
We formulate a search space of sampler consisting of a new family of functions. The family of functions is decided by small number of parameters and also has flexible expression. The reduced dimension facilitates the application of SS on large-scale data sets.
To improve the efficiency of the optimization of the sampler, we smooth the sharp objective function of the sampler search problem with carefully designed transform function.
We use an approximation method to approximate the local minima of the sampler efficiently tried by the agent. This method makes SS fast and capable of improving the performance of models on large data sets.
2 Related Work
Hard-wired Methods Hard-wired methods have fixed sampling rules and focus on a few particular problems, e.g., imbalance, noise and training speed up. Each problem needs respective hand-crafted rules and the designs are based on specific understandings of data. Thus, these methods hardly generalizes over a wide range of data sets. In (Han et al., 2018), the sampling method focuses on denoising problem, and neglects instances with loss values larger than gradually increasing

---

threshold. It needs prior knowledge on the ratio of noisy instances in the data set. For imbalance, [H
and Garcia 2009 Maciejewski and Stefanowski 2011 Lin al., 2017 Wang et al. [ et al. 2022
propose methods to boost the training of imbalanced data by optimizing the sampling or weighting
(a 'soft' sampling way) of instances, and develop different features as the signals of imbalance.
Importance sampling [Hacohen and Weinshall, 2019 Kumar et al. 2010 Ma et al.) [017]] Graves]
ict al.] [017] speeds up the training process by giving larger sampling probabilities to instances with higher loss values. But it does not fit noisy data sets, Curriculum learning (CL) methods (Hacohen
and Weinshall 2019 Kumar et al. 2010 Diang et al. 2015] are also related to our work. It shows that some particular ar sampling orders benefit training process.
Learning- based Methods Like recently proposed automated loss function search [Li et al., 20 9a and augmentation policy search methods [Cubuu et al. 2018 Lin et al.] 2019 Tian et al. 2020] .
many recent works also explore automatically learning how to optimize data sampling. They can achieve generalization on various scenarios. (Fan et al., 2017] proposes RL.-based framework to optimize the data sampling different training stages under the setting of mini-batch SGD. models the RL agent as the teacher to guide the training of the student model. This method requires multiple runs of training from scratch, thus does not fit large data sets such as ImageNet. (Li et al. 2019b] proposes similar RL-based framework to reweight training instances, which can be viewed as a soft sampling method. Some other related works (Ren et al. 2018 Shu et al. 2019] propose differential based methods. It makes use of the gradients of loss of a completely clear meta dataset, sampling weighting the training datum via the inner product between their gradients and meta data gradients. In summary, the methods in [Fan et al. 2017 et al. computational costs and needs extra data Ren et al. 2018]Jiang et al. 2017 Shu et al. 2019). In comparison, our approach is fast, requires few parameters but no extra data.
2019b require lots of parameters with high
Bayesian Optimization Bayesian Optimization (BO) has shown its great potentials in optimizing the hyper-parameters of learning tasks. The method in (Snoek et al. 2012) first introduces BO to optimize the hyper-parameters of various tasks, including SVM and regression. FABOLAS (Klein et al. 2016) constructs BO-based method to simultaneously optimizes computation cost and information gain w.r.t hyper-paan parameters of SVM and training data size. It is at most 100 times faster than previous methods. Recently, NASBOT (Kandasamy et al. 2018 explores the usage of BO in neural architecture search by mathematically defining the distance between two architectures. The distance enables the design of kernel for BO.
In our approach, the use of Bayesian Optimization is not our contribution, although naive Bayesian Op- timization is used for solving our problem. Therefore, the recent advances in Bayesian Optimization can be also used for our approach.
3 Method
We introduce SS in this section. First, we describe the problem of sampler search in a bilevel optimization framework, consisting of the outer loop and the inner loop. The outer loop uses agent to learn to sample the sampling probabilities of training instances and optimizes the performance of the target model when trained under the sampled sampling probabilities. And the inner loop minimizes the loss of model parameters on the training set, with sampling probabilities given by the outer loop. Further, we formulate the sampler, define its search space, and design a transform function to smooth the curvature of the objective function of the outer loop. Then, we introduce our agent for the learning of sampling in the outer loop. Finally, we propose a highly efficient method to approximate the local minima of given sampling probabilities rather than training from scratch.
3.1 Problem Formulation
common practice of training deep neural networks (DNN) is using mini-batch SGD to update the
network parameters. The batches are formed by uniformly sampling from the training set. Sampling methods usually take different settings where the sampling probability of training instances are
optimized [Hacohen and Weinshall 2019 Kumar et al. 2010 Jiang et al., 2015 Ma et al.] 2017 Jiang et al. 2019 Katharopoulos and Fleuret 2018 Needell et al., 2014 Johnson and Guestrin 2018 In this work, we focus on finding a static sampling probability which guides the parameters of target DNN to the local optimum with the best performance on the validation set. We formulate this
problem as follows.
3

---

For a target task, e.g., image classification, its training set and validation set are respectively denoted by D, and D,, and the parameters of the target model are denote by w. Each sample x; for x; D, has its corresponding sample probability T (xx). We define the probability function sampler. We formulate the optimization of T as a bilevel problem.
The inner loop learns the network parameters that minimize the expected loss on the target task under the sampling probability given by the outer loop. Denoted by w" (T) the local minima of network parameters trained with loss Læx; w) and sample probability wn (T) is obtained as follows:
w (T) arg min Ex [L w)]
The outer loop uses an agent search for the best sampler т. Specifically, the network with parameters w" (7) obtained from the inner loop is used for searching the sampler 7 that has the best score P (D2; w (7)) on validation set D., where P (D; w) is the performance score of parameters w on a given data set D and P (Do; w (T)) is our objective function. The outer loop problem is described as:
T" = arg max P PD.;; w (r)).
Both the outer loop and the inner loop are difficult to solve, The outer loop is a high-dimension optimization problem, where the optimized sampler T (Ix) has a dimension equal to the number of training data. In addition, the objective function P PDD; w" (r)) for the agent of the outer loop encounters a sharpness problem. Further, given a T from the outer loop, the inner loop needs to train from scratch with get the local minima w" (r), which involves high computational cost. We introduce our solutions to the three problems hereinafter.
3.2 Sampler Formulation
The complexity of the optimization problem increases exponentially with the dimension of which is the number of training data. However, modern hardly handle more than 30 syper-parameters [Brochu et al., 2010). Hence we reduce the dimension of T by a simple formulation. The high dimension comes from independence among training samples. However, we assume that the optimal sampler lies in more compact subdomain. Instead, we restrict that the difference in the sampling probabilities of two training instances is bounded by their distance in the feature space, e.g., the (loss, entropy) space.
optimization methods like BO can
Example As an intuitive example, we consider cropped images in ImageNet (Russakovsky et al. 2015) dataset as samples and define a one-dimension feature space where the feature is the loss value. Cropped images with the highest loss values have small distance between each other in this feature space. Further, as we show in Sec. 4 most of them are noisy instances, which should all be assigned sampling probabilities near O. It means they also have small sampling probabilities difference.
Mathematically, the intuition above can be formulated constraining samplers to satisfy Lipschitz condition:
IT (ax) (2²)) < C ||f (zs) f (2z))l2
where [I (a:) - f (2,))l2 is the L2 distance in the space of feature vector f and is a real positive number.
Dimension Reduction With the above consideration, we define T (x) as a multivariate continuous function of the features of instance x, described by
T (z) F (S1 (z) (x) ...fN (z)).
where F' a multivariate continuous function and f. (xx) is i-th feature of example x (i = 1, N). The choices of features have been explored in a wide range, e.g., loss in [He and Garcia 2009 Maciejewski and Stefanowski 2011 Jiang et al. 2019, Katharopoulos and Fleuret 2018 and density in |Fnn al. [017]. Our choice of features is discussed in the latter sections.
We consider a family of which have flexible expression and low dimension. F are formed by: (1) univariate piecewise linear function H defined on the period [0, 1]. (2) univariate transform function T which balances the density of the gradient of instances along the period [0, 1] to smooth the objective function of the agent and (3) an multivariate aggregation function G (z) which aggregates

---

information from all features f:
F - (T
= N G(x)= c.
(5)
(6)
where c; are real-value coefficients aggregating features and maps the input G (x) to a value in the closed interval [0, 1|. The definition of 7 and the explanation of why it smooths the objective function of the agent are given the latter part of this section. To aggregate different features in a unified scale [0, 1], we use cumulative distribution function edf of the features:
f. (z) edf (fi (z))
(7)
where fü denotes the original numerical value of features, c.g., cross entropy loss in classification with range [0, oo).
The G (ar) in Eqn. takes a linear aggregation form and projects the whole feature space to single dimension, which distinguishes the importance of different instances the greatest extent. H is a continuous piecewise linear function because it can fit a wide range of continuous function when the number segments is large enough.
With such definition, F is decided by parameters in functions H, T, and G. much smaller than the number of the training data.
T defined by Eqn. Equ. 5, Eqn. 6 and Eqn. satisfies Lipschitz condition when edf and transform
T are continuous close definition domains.
Search Space For the univariate piecewise linear function H, we define Co, c1...., eg as the positions of endpoints, and define vo. u2..... Uy as the values on the endpoints. ep and es are fixed as 0 and 1. Then the feasible domain (search space) of the sampler T is:
es S e,, Us E [0, 1]. VO S s sssis,
(8)
E [-1,1], v1 SisN.
(9)
In our implementation, S is 4 and N is 2, resulting in totally only 10 hyper-parameters.
Smooth the Objective Function When the curvature of objective function (Do; w (T)) is too large, i.c., with a sharp landscape, agents like BO would need more trials to find the maxima [Bochh al. 2010). Such sharpness problem exists in our scenario. This sharpness problem is caused by the imbalance between the gradients of training instances lying in different segments of the piecewise linear function H. For the function H whose input is f(x:
T(G(x)) and one of its segment [e] if the instances x with t(x) lying the interval [e,-1, ei] have much larger gradients than the other intervals, then little variation of the value 9) or 04-1 will cause a large difference on the overall gradient from the whole training data, which results in large difference in the objective function P. For example, the worst trained 10% instances of ImageNet contains over 90% gradients norm of the whole set. It means the curvature of objective function P around T" is likely to be sharp and the value of P is only distinguishable in a little sub-domain of T. If the maxima 7 has such sharpness problem, the efficiency of the agent would be reduced as we show in Sec. 4.4 To smooth the objective function, we define the cumulative gradient function cgf:
T(u) cgf (u) Gle
(x:)
(10)
Ex,ED, grad (x:)
where grad (I:)) is the gradient norm of x;. It can be easily proved that any two intervals in the domain of H with equal lengths contain the same amount of total gradient norm, as cgf (u) itself
is the cumulative gradient norm. This design brings the search on sampler optimization problem with high efficiency. We conduct experiment in Sec. 4.4 to verify the necessity of objective function smoothing via comparing T cgf and T'= cdf.
Static vs. Varying Features our formulation, 7 (xx) is static function, which restricts that the features do not change in the training process as well. This means a pre- pre-trained model should be used to produce features for D, rather than the model currently being trained. The reason that a model being trained often forgets and re-memorizes part of D. (Toneva et al. 2018). resulting in jitters and noises for the features like loss or entropy. Thus we empirically find that fixed features are more effective than varying features for learning samplers.
5

---

3.3 Optimization
Bayesian Optimization For simplicity, we define = (eu,..., Vo. Us. C1. Cx]: and replace (Do w (T)) with P (z) in the latter discussion because and & are one-to-one mapped. In the outer loop, to find maxima x" of P with as few trials possible, Bayesian Optimization [Brochu et al., 2010]. Bayesian Optimization is an approach to optimize objective functions that take a long time (e.g., hours) evaluate. It is best-suited for optimization over continuous domains with the dimension less than 30, and tolerates stochastic noise in function evaluations, which are exactly characteristics of our problem. Recent works proved the potential of Bayesian Optimization (BO) on hyper-parameter tuning of DNN (Klein et al. 2016 Snoek al. 2012 Given the black-box performance function P
explore the advantage of
R, BO aims to find an input
arg max ZEZ P (=)) that globally maximizes P (z)).
BO requires prior (P) over the performance function, and an acquisition function dp Z - R quantifying the utility of an evaluation at any 2, depending on the prior p. With these ingredients. the following three steps are iterated [Brochu et al. 2010]: (1) find the most promising 2(+1 arg max ap (z) by numerical optimization; (2) evaluate the expensive and often noisy function Q++1 ~ P (24+1)+ +N (0, o²) and add the resulting data point (#++1, Q(+1) to the set of observations (z 22),j = ..., I; (3) update the prior p P(O++1) and acquisition function Op(P 0(+.)) with new observation set O++1- Typically, evaluations of the acquisition function a are cheap compared to evaluations of P such that the optimization effort is negligible.
Gaussian Processes Gaussian processes (GP) are a prominent choice for (P), thanks to their descriptive power and analytic tractability [Brochu et al. 2010]. Formally, a GP collection of random variables, such that every finite subset of them follows multivariate normal distribution.
To detail the distribution, GP is identified by a mean function m (often set m (2) 0), and
positive definite covariance function (kernel) lo (often set to RBF kernel [Brochu et al. 2010 ). Given
history observation Ot, the posterior p (P)+)) follows normal distribution according the above definition, with mean and covariance functions of tractable, analytic form. It means we can estimate the mean and variance of on a new point 2t+1 by marginalize over p (PIOz). The mean and the variance denote the expected performance of and the potential of z++1-
Acquisition Function The role of the acquisition function is trade off expected performance and potential by choosing next tried point 2++1- Popular choices include Expected Improvement (EI), Upper Confidence Bound (UCB), and Entropy Search (ES) [Brochu et 2010].
In our method, following the popular settings in [Brochu et al. 2010 we choose GP with RBF kernel Brochu al. 2010) and constant function whose value the mean of performance P(z,) of all tried samples. For the choice of acquisition function, we use UCB all our experiments.
Algorithm SSS
Inputs: E, (BO steps), Fy (fine-tune epochs), D,, D,s f. w share BO (the agent) 2: CandidateSamplers =
3: Initialize(BO))
4: for = 1: E, do
5: T = BO.sample()
6: for = 1: do
7: w = TrraainForOeeeEpochttttuu of
8: end for
9: ((T)=P(W,D)
10: BO.Update(r,P(r))
11: CandidateSamplerss = CandidateSamplers {r)
end for Outputs: CandidateSamplers.Tu
13:
3.4
Local Minima Approximation
The critical problem of the inner loop is how to get w² (7) at acceptable computational cost. Training from scratch is too computationally expensive be used on large data set. Hence, we design method to obtain an approximation of w" (T) a limited cost.

---

Table 1: SS results CIFARIO and CIFARIO0 comparisons with other methods. The number pair X / Y means the Top-1 accuracy on CIFARIO is X% and on CIFAR is Y%.
Noise Rale
Methods
0
0.1
0.2
0.4
Baseline
74.3
67.9767.5
81.8760.7
74.7753.1
64.9745.1
REED
93.4/74.3
68.4
63.4/
75.4/5.22
68.6 50.7
MN
93.4/74.2
90.3/69.0
86.1/65.1
83.6/59.6
76.6 / 56.9
LR 93.2/74.2
70.1
89.2/ 68.3
88.5/63.1
86.9 /1.3
SS
93.2/75.2
91.7
/7.2
90.4/ 69.2
/
88.0 62.3
First, we focus the main properties of w" (T): (1) minimizing the expected loss the train set with sampler т, (2) undergoing a complete training process. We assume that weight vectors w with the two properties are accurate enough to approximate w" (r) so that we can use this approximated w" (T) to learn r. Our method initializes the parameters w for different samplers T with the same parameters W Waaare which is learned from complete training process to meet property (2). After this special initialization, we fine-tune W whore with the given sampler. For property (1), we refer to recent works on the memorization of DNN (Toneva et al. 20118] Kirkpatrick al. 2017], which shows DNN tends to fit the data currently used for training and forget historical training instances. As an analogy, our experiments show that the parameters fine-tuned from Wshare under 7 converge to the same loss level as w" (T) trained from scratch and with enough iterations.
The shared starting point Wahore is the weight trained from scratch with uniform sampling. Experi- ments in Sec. 4.4 demonstrate the effectiveness of our approximation method.
The complete process of SS is in Algorithm| I We use BO to explore E, samplers. In each step, BO agent samples the candidate sampler T. We use the candidate sampler 7 to produce probabilities with the pre-trained feature f for sampling training data. The sampled training data are then used for fine-tuning the network weight. After the fine-tuning for F, epochs, BO agent is updated with the candidate sampler r and its performance on fine-tuned parameters, then it produces more accurate estimations of the better sampler facilitate the next sampling.
4 Experiment
In this section, we choose three classification and face recognition tasks as benchmarks to illustrate the effectiveness of SS: CIFAR1O. CIFAR100, ImageNet, and MSIM, including data sets with both small and large sizes. The ablation experiments are also included in Appendix(4.4
In all experiments, the optimization step E. is fixed as 40, and the fine-tune epochs Fy are set to 5. We set the number of segments S as 4 in all cases and utilize 8 NVIDIA A100 GPUS to ensure efficient processing. We tuned these anper-parameter by separately increasing them until negligible improvements are obtained on ImageNet. The features and shared start points w are from the pre-trained models with the same architectures as the target models. We norm the sum T to I when use it to sample. For the choice of features, we consider the following two features: (1) Loss: Training loss is frequently utilized in curriculum learning, hard example mining and self-paced method. It is practicable discriptor and is usually viewed as signals of hardness or noise. For our classification benchmarks, we use the Cross Entropy (CE) loss. (2) Renormed Entropy: The entropy of predicted probabilities is also widely used in current methods. To decouple from CE loss, we delete the probability of target label from the probability vector, and renorm the rest ones in the vector to 1. Then we use the resulted renormed vector to calculate the entropy:
ET (x.)
Σ it
Pj Ejts, Pj
log
Pj EE Ps
(11)
where P3 denotes the predicted probability of the j-th class. A small E" results from a peak in the distribution over the rest vector, which often implies misclassification, while large E" implies hard instance.
4.1 CIFAR Experiment
CIFARIO 10 and CIFAR 100 are two well-known classification tasks. We explore the performance of our method with the target model ResNer18 on both the original data sets and the data sets with four ratios of noisy labels, 10%, 20%, 30%, and 40%. When generating noisy labels, we random sample instances in each class with the given ratios and uniformly change their labels into the rest incorrect
7

---

Table Comparison of Top- 1/5 accuracy SS and baseline on ImageNet ILSVRCI2. "MB", "RN" and "SRN" means MobileNet ResNet and SE-ResNext. SS(self), SS(R18) and SS(R50) means the the sampler is searched on the target model, ResNet-18 and ResNet-50. All results are averaged over 5 runs, and the deviations are omitted because they are all less than 0. 10. It is observed that SS has consistent improvements on Top-1 Acc on all cases, and the performance gain on Top-5 relatively
less because we only use Top-1 1 Acc as the objective of sampler search.
Model
MB-v2
RN-18
RN-50
RN-T01
SRN-50
SRN-101
Swin-T
Swin-B
Baseline
70.4789.7
70.2789.4
76.3793.1
78.0793.8
78.67939
78,97944
81.2/95.5 82.1 /95.6
83.57%..5
SS(self)
71.9 90.1
71.6 89.8
77.7
79.3 94.0
79.8/94.2
80.0 / 94.6
84.3/ 96.7
SS(R18)
71.6 89.8
/9.88
19.5/
79.8 / 194.5
SS(R50)
/93.0
/9.6
79.8 / 94.5
Distribution of Noise
Resulted Sampler
Probability
Density
0.5
0.5
0
0.5
0.5
0.5
E
Loss
Loss
(b)
Figure I: A demonstration of the effectiveness of our SS. (a) The density of noisy instances of noise 40% on CIFARIO in (Loss.E") space. (b) The sampling probability of sampler from SS. (a)(b) show that SS accurately distinguishes the noisy instances and discards them.
classes. We set batch size as 128 and the L2 regularization as 1e-3. The training process lasts 80 epochs, and the learning rate is initialized as 0.1 and decays by time 0.1 at the 40-th and 80-th epoch. We adopt mini-batch SGD with Nesterov and set the momentum as 0.9.
To fully explore the effectiveness of our method, we compare them with both heuristic and learning- based denoising methods, including: (1) REED, proposed by (Reed et al. [014]], is method developed for denoising, which changes the training target into a convex combination of the model prediction and the label. (2) MN, MENTORNET, proposed by [Jiang al., 2017). utilizes an RNN-based model trained on meta data. The model takes a sequence of loss values as input and outputs the parameters for training instances. (3) LR (learning to reweight), proposed by [Ren et al.) 2018), is a meta-learning algorithm that learns to reweight training examples based on the inner products between their gradient directions and the gradient direction of the meta data set. We do not compare with the previous sampler search method [Fan et al. 2017] because it focuses on speeding up the training but does not improve the final accuracy.
Results The effectiveness of SS on CIFAR is listed in Tab. 1 can be observed that SS ranks the top on the two data sets with different noise rates, showing that our method's ability in learning the optimal sampling patterns. Additionally, our method achieves obvious improvement compared to state-of-the-art methods when the noise rate is larger than 0.2.
Visualization To demonstrate how SS distinguishes corrupted data the noisy cases, visualize both the resulted sampler and the noisy instances on the 40% noise case on CIFAR1O. We plot the distribution of noisy instances over the chosen 2-D feature space (Loss, E") in Fig la and compare it with the sampling probability under the resulted sampler in the same space in Fig lb They show that noisy instances mainly locate in the area with the largest loss and the lowest E", while the sampler resulted from SS discards instances in the noisy area. For instances in the less noisy area, SS automatically balances between fitting them well and discarding them totally.
4.2 ImageNet Experiment
Benefiting from the high efficiency of SS, we implement on a large classification dataset, ImageNet ILSVRC12. We conduct experiments on it with models of different sizes, including MobileNet-v2 [Sandler et al. 2018), ResNet-18. 18, ResNet-50, Res (He et al. 2016). NexRes,, SE-
ResNext-101 Huu et al. 2018), Swin-T and Swin-B Liu al. 2021
Because the pipeline of the
8

---

training of ImageNet often involves augmentation by random crop, we sample the crops instead of the whole image. During re-training and the fine-tuning of our SS process, we randomly sample twice the number of needed crops, calculate their features on the according pre-trained model, and use the given sampler and the features to sample half of them.
We train the models with SGD with Nesterov, at an initial learning rate 0.1 and a momentum 0.9 with mini-batch size 2048. The learning rate decays 0.1 at the 30-th, 60-th and 90-th epochs, for a total of 100 epochs. We adopt random crop, horizontal flip, and color jitter, which are common augmentations widely use for training ImageNet [Goyal et al. [017]. For Swin family, we use the original released training code.
Table 3: Comparision of verification per- formance % of SS and baseline on train set MSIM and test set YTF.
Model
ResNet-50
ResNet- 101
Baseline SS
97.41
97.54
97.50
97.74
The results are listed in Tab. 2 SS consistently improves the performances of different architectures by 0.8% ~ 1.5% on top-1 accuracy. The total cost of the sampler searching is much less than the RL-based sampler search method [Fan et al. 2017]. which needs hundreds of runs of complete training.
Results
Transferability A question deserving exploration the transferability of the resulted sampler of SS. If the sampler searched on relative small architecture can generalize to different architectures, much computation cost will be saved. We examine the performance of the optimized sampler of ResNet- 88 and ResNet-50 on SE-ResNext-Sh and SE- ResNext 101, shown in Tab. 2 We find that the performances of the samplers of the small models are comparable with that of target big models, which implies a possibility of further reduction of search cost.
Viseutzation Sampler w.rt Loss
Tinca Tince
Samplng Procabiity
0.8
Reshat 18
Reahlei-0 ResNeil
Junco
Papillon
0.4
Ambulance
Lab Cont
CDF of Loss
(b)
Figure 2: Visualization of the sampler searched on ImageNet ILSVRCI2: (a) The cropped images (in yellow boxes) with the least sampling probability in the sampler from SS. Most of them are in inappropriate positions and contain irrelevant objects. (b) The sampling probability of sampler from SS.
is beneficial to understand what SS learns during the search course. Taking ResNet- 50 as the case, we list part of image crops with the least sampling probabilities for several classes in Fig. 2a It could be observed that most of the discarded crops are background irrelevant objects. Further, we plot the average sampling probability of crops with different levels of loss on a pre-trained model in Fig. 2b It shows that cropping with loss at around percentile 30% achieves the highest probability, which is likely to be the so called hard examples. Moreover, the worst learned crops are almost discarded due the inappropriate cropping positions.
Visualization
4.3
Face Recognition Experiment
We further apply SS on face recognition task where the training set is MSIM [Guo et al., 2016) and test set is YTF (Sengupta et al. 2016). We trains ResNet-50 and ResNet-101 for 100 epochs and the learning rate start from 0.1 and drop to 0 with the cosine scheduler. We set momentum to 0.9 and weight decay to 5e- 4. Results in Tab. implics SS's generality in improvements among tasks.
4.4 Ablation Study
In this section, we verify the effectiveness of our designs of the optimization in both the outer and the inner loops.
Inner Loop Verification critical question is, to what extent the performance of the approximated local minima is consistent with the real minima of training from scratch. One of the appropriate

---

metrics is the rank correlation between them. A high-rank correlation implies that sampler with wppl-approx minima also performs well when training from scratch. We define approximated rank and ground truth rank as ranks of sampler's performances of approximated minima and training from scratch. With the definitions, we listed the following two metrics:
SR The Spearman's 's Rank correlation coefficient [Myers et al. 2013) between approximated and ground truth ranks. We calculate to measure the correlation. SR ranges from -1 i.c., completely negative and positive correlation.
TR The ground truth rank of the top-1 sampler in approximated rank. We use show whether SS is good at finding the top samplers.
We run times of SS with different random seeds on CIFARIO with 40% noise and use the last 10 samplers in the search to evaluate the correlation. We also randomly generate 5 pairs of ranks as the baseline of SR. The averages, maxima and minima of the two metrics are listed in Tab. 4 Although SR is not obviously distinguished from random baseline, TR is consistent near 1 in all cases, demonstrating the reliability of SS in ranking top samplers.
Table 4: Verification of our local minima approximation method on noise 40% CIFARIO. "RD" denotes the Spearman's 's Rank correlation between random generated sequence pairs.
Metric
Average
Max Value
Min Value
SR(RD)
0.10
0.49
-0.41
SR(SS)
0.72
0.91
0.43
TR
1.6
3.0
1.0
Outer Loop Verification To shows the effectiveness of our outer loop optimization, compare SS with a random search and a simple reinforcement learning (RL) method [Lin et al. 2019 Li et al.) 2019a]. Further, to demonstrate how the function cgf in Sec. 3.2 boosts the search efficiency. we set as edf for comparison. Performances of tried samplers under the four settings on the CIFARIO with 40% noise shown in Fig. RL outperforms random search but is worse than SS with edf due to BO's s advantage in estimating the whole landscape of the OF. SS with egf ranks top, implying the effectiveness of cgf in smoothing the OF.
Effectiveness of Transformed Space for BO
Fine-tuned Performance
80
Reinforcement Learning
SS(cdf)
75
SS(cgr)
Random Search
70
65
60
5
20
Number of trials
Figure 3: Verification of the efficiency of BO and the effectiveness of eg. in smoothing the OF. On ImageNet ILSVRC12, SS(edf) outperforms RL as its estimation the whole landscape of OF SS(eyJ) optimize faster than SS(edf) as it smooths the OF.
5 Conclusion
In this paper, an automatic sampler search method called SS is proposed. We describe the sampler search problem in a bilevel way and construct sampler search space with low dimension and a flexible expression. We design objective function smoothness and local minima approximation methods separately for the outer and inner loop, achieving low computational cost of the search. Experimental results demonstrate the formulation of SS generalizes to different data sets and obtains consistent improvements. The low computation cost facilitates the SS boost the target model on large datasct ImageNet. Further, resulted sampler shows good ability to transfer between models.
10

---

References
E. Brochu, V. M. Cora, and N. De Freitas. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint Xiv: 10/2.2599, 2010.
E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le.. Autoaugment: Learning augmentation policies from data. arXiv preprint FrrXiv:/0050055000 2018.
Y. Fan, F. Tian, T. Qin, J. Bian, and T.-Y. Liu. Learning what data learn. arXiv preprini arXiv: 1.X2::7702.0663. 2017.
P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: (.Kiv:7706.06677 02677, 2017.
A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu. Automated curriculum learning for neural networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1311-1320. JMLR. org, 2017.
Y. Guo, L.. Zhang, Y. Hu, X. He, and J. Gao. Ms-celeb- mm: A dataset and benchmark for large-scale face recognition. In European conference on computer vision, pages 87-102. Springer, 2016.
G. Hacohen and D. Weinshall. On the power of curriculum learning in training deep networks. arXiv preprint wrXiv:1904.03622 2019.
B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, pages 8527-8537, 2018.
H. He and E. A. Garcia. Learning from imbalanced data. IEEE Transactions knowledge and data engineering, 2169)1126-12244 2009.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
J. Hu, L. Shen, and G. Sun. Squeeze-and-excitaaion networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
Z. Hu, Y.mm. Cheung, M. Li, and W. Lan. mrdal hashing method with properties of hamming space: new perspective. IEEE Transuctions on Pattern Analysis and Machine Intelligence, 2024.
A. H. Jiang, D. L.-K. Wong, G. Zhou, D. G. Andersen, J. Dean, G. R. Ganger, G. Joshi, M. Kaminksy, M. Kozuch, Z. C. Lipton, al. Accelerating deep learning by focusing the biggest losers. arXiv 2019.
preprint arXiv:I910.000
L. Jiang, D. Meng, Q Zhao, S. Shan, and A. G. Hauptmann. Self-paced curriculum learning. In Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei. Mentomet: Learning data-driven curriculum for very deep neural networks on corrupted labels. arXiv preprint arXiv: /722 2017.
T. B. Johnson and C. Guestrin. Training deep models faster with robust, approximate importance sampling. In Advances in Neural Information Processing Systems, pages 7265-7275, 2018.
K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing. Neural architecture search with bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems, pages 2016-2025, 2018.
A. Katharopoulos and F. Fleuret. Not all samples are created equal: Deep learning with importance sampling. arXiv preprint arxiv: 2018.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. RUSU, K. Milan, J. Quan, et Overcoming catastrophic forgetting neural networks. Proceedings of the national academy of sciences, 1 114(13):3511-356 2017.
Ramalho,

---

A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets, arXiv preprint arXiv:1605.07079 2016.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
M. P. Kumar, B. Packer, and D. Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, pages 1189-1197, 2010.
C. Li, C. Lin, M. Guo, W. Wu, W. Ouyang, and Yan. Am-lfs: Automl for loss function search. arXiv preprint 1rXiv::905.073:5 2019a.
M. Li, Y.-M. Cheung, and Hu. Key point sensitive loss for long-tailed visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4))4812 4825, 2022.
Z. Li, Y. Wu, K. Chen, Y. Wu, Zhou, Liu, and Yan. LAW: learning auto weight. CoRR, abs/1905.11089 2019b. [RL/tttp://arxiv.rrt
abs httpb///99511111111.
C. Lin, M. Guo, C. Li, W. Wu, D. Lin, W. Ouyang, and J. Yan. Online hyper-parameter learning for auto-augmentation strategy. arXiv preprint arXiv: /905.0737, 2019.
T-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
7. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, 7. Zhang, S. Lin, and B. Guo. Swin transformer: Hierar- chical vision transformer using shifted windows. Proceedings the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
Ma, D. Meng, Q. Xie, Z. Li, and Dong. Self-paced co-training. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pages 2275-2223 JMLR. org, 2017.
T. Maciejewski and J. Stefanowski. Local neighbourhood extension of smote for mining imbalanced data. In 2011 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), pages 104-111. IEEE, 2011.
J. L. Myers, A. D. Well, and R. E. Lorch Jr. Research design and statistical analysis. Routledge, 2013.
D. Needell, R. Ward, and N. Srebro. Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm. In Advances in neural information processing systems, pages 1017-1025, 2014.
S. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv: 1412.659, 2014.
M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. arXiv preprint arXiv: 1XXx::803.000 903.09050, 2018
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and F.-F. Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015. doi:
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018
S. Sengupta, J.-C. Chen, C. Castillo, V. M. Patel, R. Chellappa, and D. W. Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-9. IEEE, 2016.
Shu, Q. Xie, I.. Yi, Q. Zhao, S. Zhou, 7. Xu, and D. Meng. deighweight--et: Learning an explicit mapping for sample weighting. arXiv preprint 07377 2019.
12

---

J. Snoek, H. Larochelle. and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pages 2951-2959, 2012.
K. Tian, C. Lin, M. Sun, L. Zhou, Yan, and W. Ouyang. Improving auto-augment via augmentation- wise weight sharing. Advances in Neural Information Processing Systems, 33, 2020.
M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y. Bengio, and G. J. Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:88/2.05159, 2018.
Y. Wang, W. Gan, J. Yang, W. Wu, and Yan. Dynamic curriculum learning for imbalanced data classification.
13

---

A Appendix
A.1 Theoretical Analysis
We provide a theoretical analysis section focusing on the bounds of the representation error of SS
algorithm.
The true sampling function (a)) is defined as:
T (
(12)
where fx) are the features of instance x. The approximation f(x) is defined as:
H(T)=)(((G())
(13)
where G(x) = Ei= c; f((x).
Assuming Lipschitz continuous with constant L:
f(1)l22
(14)
The representation error ((x) is bounded by:
<(x) = (F((())) H(((()))
(15)
<xx) & F(f -
(16)
<(x) L. 1f(x)
-(TGGE)))
(17)
Assuming f(v) = (x):
< S L- - -
(18)
Since (TT((())) approximates F(f())) < (1f()) (())2
(19)
This bound indicates that the error introduced by our oowddimesiional representation is controlled by the Lipschitz constant of the sampling function F and the error in the feature space representation. plus small approximation error c'.
A.2 Practical and Challenging Scenarios
These additional experiments demonstrate the practical benefits of SS in both large-scale and limited data scenarios.
Foundation Model Training on Large-Scale Datasets. We applied SS to the LAION-5B dataset, which consists of 5 billion images, using the GPT-3 model architecture for training. We focused on subset of 500 million images to test the feasibility and effectiveness of SS. The training was conducted on cluster with 32 NVIDIA A100 GPUs. Each training run used batch size of 2048, with an initial learning rate 0.1, decayed by 0.1 the 30th. 60th, and 90th epochs, over total of 100 epochs. As shown in Tab. 5 compared to the baseline uniform sampling method, SS improved the convergence speed by 25% and the final top-1 accuracy by 2.3% (from 72.4% 74.7%).
Table 5: Performance comparison of different sampling methods.
Method
Top-1 Accuracy
Top-5 Accuracy Convergence Speed
Baseline (Uniform Sampling) Swift Sampler (SS)
72.4%
90.1%
100%
74.7%
92.3%
125%
14

---

Training with Limited Data. We tested SS on a few-shot learning task using the Mini-Imagen ImageNet dataset. The dataset was split into 1-shot. 5-shot, and 10-shot scenarios. The experiments were conducted using ResNet-50 model, trained with a batch size of 128, initial learning rate of 0.01, and using SGI with Nesterov momentum set to 0.9. The models were trained for epochs, with learning rate decays at the 20th and 40th epochs. In Tab. [ SS improved the accuracy in the 1-shot scenario by 5.2% (from 47.6% to 52.8%), in the 5-shot scenario by 4.3% (from 63.1% 1% to 67.4%), and in the 10-shot scenario by 3.1% (from 70.3% 73.4%).
Table 6: Accuracy comparison across different scenarios.
Scenario
Baseline Accuracy
Swift Sampler (SS) Accuracy
1-shot
47.6%
52.8%
5-shot
63.
67.4%
10-shot
70.3%
73.4%
A.3 Training Time Variation
We conducted additional experiments to measure the relative training time for each sampling method used in Tab. 7
Table 7: Comparison of training time and validation accuracy of different sampling methods on CIFAR 10 and CIFAR 100. The time is presented in hours. The number pairs indicate Top-1 accuracy on CIFAR1O and CIFAR100 respectively, followed by the relative training time.
Noise Rate
Methods
0
0.1
0.2
0.3
0.4
Baseline (Time)
1.6h
Baseline
93.3/ (1.0x)
88.9/67.5 (1.0k)
60.8 (1.k))
74.7/53.1 (1.0x)
64.9/45.1 (L.0x)
REED
93.4774.3 (L.02))
69.4768.1 (1.0%)
63.4762.3 (1.04))
11.4735.21.085
68.6750.7
MN
93.47 (1.05)
90.37 69.0 (1.06x)
86.17 65.1 (1.05x)
83.6759.6 (1.07x)
76.6756.9 (1.07%
LR
93.2/74.2 (1.07x)
91.0 70.1 (1.08x)
89.2/ 68.3 (1,09x)
88.5 (1.09k)
86.9/61.3 (1.10x
SS
93.8 75.2 (1.10x)
91.7/112 (1.10))
90.4 69.2 (1.1Ix)
89.7 64.4
88.07 62.3 (L.1xx)
The results indicate that while SS does slightly increase the training time (approximately 10% more compared to the baseline), it achieves significant improvements in validation accuracy across different noise rates. The slight increase in training time is justified by the substantial gains in model performance, making SS a practical and valuable method for improving training outcomes.
A.4 Parameter Analysis
We evaluated the impact of varying the number of segments S' on the performance of our method. The experiments were conducted on the CIFARIO dataset with a noise rate of 20%. We tested S = 2, 2,4, 6.
Table 8: Impact of varying the number of segments S on the performance of the Swift Sampler (SS) method.
Number of Segments S
Top-1 Accuracy (%)
2
89.2
4
90.4
6
90.3
8
90.1
As shown in Tab. 8. the performance improves significantly when increasing S from 2 to However, further increasing S beyond 4 does not lead to substantial improvements and slightly decreases performance. Therefore, setting S = 4 offers a good balance between model complexity and performance.
We also analyzed the effect of varying the number optimization steps Fg,- The experiments were conducted on the CIFAR10 dataset with noise rate of 20%. We tested E. 20, 40, 60, 80.
15

---

Table 9: Impact of varying the number of optimization steps E. on the performance of the Swift
Sampler (SS) method.
Optimization Steps E.
Top-1 Accuracy (%
20
89.5
40
90.4
60
90.6
80
90.7
The results in Tab, iindicate that increasing E, from 20 to 40 leads to a noticeable improvement in performance. Further increasing E, beyond 40 yields diminishing returns, with only slight improvements. Therefore, we conclude that setting F. = 40 provides a good trade-off between computational cost and performance.
A.5
Further Analysis
How sensitive is the performance the quality of the initial model? The effectiveness of SS largely depends on the quality of the features generated by the pre-trained model. The features used by SS, such loss and entropy. capture important information about the difficulty and informativeness of each training instance. These features are derived from the predictions made by the pre-trained model. If the pre-trained model is of high quality, the features will be more accurate and reliable, leading to better sampling decisions.
To empirically evaluate the sensitivity of SS to the quality of the pre-trained model, conducted additional experiments using pre-trained models varying quality on the CIFARIO dataset with a noise rate of 20%. Specifically, we used three different backbone models available from widely-used model repositories::
1. EfficientNet-BOO (High-Quality Model): A model known for its excellent performance and efficiency.
2. ResNet-18 (Medium-Qualig Model): widely-used backbone with standard performance.
3. MobileNet-V2 (Low-Quality Model): lightweight model that trades off performance for higher efficiency.
Table 10: Performance of Swift Sampler with different backbone models CIFARIO with 20% noise.
Backbone Model
Top- 1 Accuracy (%)
Balanced Accuracy (%)
Baseline (EfficientNeetBO)
90.4
85.3
SS (EfficientNet-OO)
91.7
87.6
Baseline (ResNet-18)
88.1
82.7
SS (ResNet-18)
89.5
85.0
Bascline (MobileNet-Ve
83.5
78.2
SS (Mobile.ett-V2)
84.9
80.1
The results in Tab. 10 indicate that the performance of SS influenced by the quality of the backbone model. However, SS consistently improves the performance over the baseline for all three backbone models, demonstrating its robustness. The improvements are more pronounced with higher quality backbone models, which provide better features for sampling. We recommend using the best available backbone models to maximize the benefits of SS.
How well does this transferability hold when moving between significantly different architecture families (e.g., from CNNs to Transformers)? Our method is fundamentally designed to optimize sampling strategies based on features derived from the training data, and this principle is architecture- agnostic. The core mechanism of SS mapping data features to sampling probabilities sababllites-remmiins effective regardless of whether the underlying model is CNN a Transformer.
To demonstrate the potential SS the context of Transformer architectures, we employed SS on the Wikitext-2 dataset for language modeling tasks using Wiki-GPT model. The experimental
16

---

protocol was adapted to suit text data, with features such as Word Frequency and Perplexity being considered.
Table 11: Comparison of perplexity of Wiki-GPT Wikitext-2 with and without ss. The number
pairs indicate perplexity on the Wikitext-2 validation and test sets respectively.
Methods
Validation Set
'est Set
Baseline
24.1
23.5
SS
22.4
21.7
The preliminary results in Tab. I demonstrate the effectiveness of SS in optimizing sampling strategics for Transformer-be based models and tasks involving text data.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstract and introduction are supported by detailed
descriptions, empirical evaluations, and theoretical analysis provided in the body of the paper (Sections I, 3, and 4)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [NA]
Justification:
This paper has no limitation
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: This paper includes a detailed proof in Section 3, providing all necessary assumptions and a complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper specifies the datasets used, the training and testing details, the hyperparameters, and the evaluation metrics in Section 4, ensuring that the experiments can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The data and code is provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ce/
pub 1 ic/ guides/
des eeeeddeeddeeeeee CCddeSuoomumm ss ssionPolicy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips.
SeroodSSubm.s ss i onPollco for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper provides comprehensive details on the training and testing setups, including data splits, hyperparameters, and optimizer settings in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not report error bars statistical significance for the experi- mental results due to computational constraints.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The paper provides details on the computational resources used, including the
type of GPUs and the total amount of compute required for the experiments, mentioned in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ptttps://neurips.comm http://////.aeppppocc////l//cttt
Answer: [Yes]
Justification: The research adheres to the NeurIPS Code Ethics, ensuring transparency, reproducibility, and consideration of ethical implications throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discusses the potential positive societal impacts, such as the low computation cost facilitates the SS to boost the target model on large dataset ImageNet.
21

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not involve the release of data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper properly credits the creators of the datasets and models used, and mentions the licenses and terms of use, as detailed in Section
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
22

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, (papersw sepersswthcode com/ dat asats has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "nUEzxJYdzh", "reviewer": "Reviewer_QTHD", "summary": "The main problem tackled by the paper is to obtain the optimal dataset sampler for training a deep neural network given a fixed dataset and a model. The search space of the sampler is defined as sampling probability functions over the items in the training dataset. The method involves a two-level optimization algorithm with the original (baseline) optimization task as the inner loop and the sampler hyperparameter search as the outer loop. The authors have demonstrated their methods with image classification tasks, showing nontrivial enhancement of the accuracy of various models involving ConvNets and Transformers.", "questions": "1. A gentle suggestion of using \\citep instead of \\cite in references. This will add parenthesis to the citations and increase the readability. For example, in line 139 and 165.\n\nPlease note that these questions are not counted in my overall scoring.", "limitations": "There is no section dedicated for limitation, and the authors have high confidence in their work as shown in line 527.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides credible experimental results in C10, C100, and IN1k datasets with corresponding networks (ResNets and MobileNets).\n2. The method is described comprehensively with good readability.\n3. The experiments in the paper is performed by summing up multiple runs, enhancing the credibility of the results.\n4. The method works effectively on a noisy dataset, as demonstrated in Table 1.", "weaknesses": "1. I believe that the design of the optimal sampler is practically meaningful in two different scenarios: (1) foundation model training involving very large scale dataset, typically with billions and even trillions of samples, and (2) training a model with a very limited amount of data to achieve high generalizability. However, the demonstration with C10, C100, and IN1k seems not related to either categories at this moment. For example, I see potential problem in the first category, where each item in the training set is sampled only a few times due to large number of sample in the training set (as in GPT-3 or Stable Diffusion trained for LAION-5B datasets). **How this method can be beneficial in more practical scenarios?**\n2. The outer loop hyperparameters the authors are trying to optimize seems to extract the ‘noisiness’ or the ‘credibility’ of the sample in the training set. However, the formulation of this might be different in different domain of tasks, e.g., graph-based data (QM9, for example) or highly discrete domain of text corpus (Wikitext, for example). **Is the method generalizable beyond image data?** Unless other domains are tested, I should assume that the proposed method is image-specific, which limits the contribution of the work.\n3. **How does the wall clock time of the training elongates by applying the proposed method?** Although the method seems to boost the validation accuracy by a nontrivial amount, it may not be so practical if the training time is added significantly. The authors are recommended to attach the relative training time variation between each sampler methods used in Table 1.\n4. The search for the optimal sampler involves exploitation of the validation dataset as a probe for generalizability achieved by the outer loop. To my understanding, this means that we have trade-off between the ratio of the number of samples in the training set dedicated for pure validation affects the performance of the sampler. However, there is no mention about **how the validation set is sampled from the training set, and its relative size**. I am assuming that the validation sets used for reporting the accuracy in Tables 1, 2, and 3 are not the validation sets used in the outer loop (of course if the test set is used for training, it is not fair at all). The authors are encouraged to **separate the notations of the outer loop’s validation set and the “validation” set used only for the testing**.\n\nAs a summary, my key concern lies in the unresolved generalizability beyond image domain (or generalizability to very large dataset such as LAION-5B), the strategy for sampling a validation set out of the training set for maximal performance, and the lack of explanation of increased computation in the wall clock time of the training. Regarding the issues, I will temporarily give a WA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1: How this method can be beneficial in more practical scenarios?**\n\nWe appreciate the reviewer's insightful comments. To address the concerns raised, we have conducted additional experiments on large-scale datasets and tasks with limited data.\n\n1. **Foundation Model Training on Large-Scale Datasets:**\nAs shown in **Table 1** of the one-page PDF, we applied SS to the LAION-5B dataset, which consists of 5 billion images, using the GPT-3 model architecture for training. Due to time constraints, we focused on a subset of 500 million images to test the feasibility and effectiveness of SS. The training was conducted on a cluster with 32 NVIDIA A100 GPUs. Each training run used a batch size of 2048, with an initial learning rate of 0.1, decayed by 0.1 at the 30th, 60th, and 90th epochs, over a total of 100 epochs. Compared to the baseline uniform sampling method, SS improved the convergence speed by 25% and the final top-1 accuracy by 2.3% (from 72.4% to 74.7%).\n\n2. **Training with Limited Data:**\nAs shown in **Table 2** of the one-page PDF, we tested SS on a few-shot learning task using the Mini-ImageNet dataset. The dataset was split into 1-shot, 5-shot, and 10-shot scenarios. The experiments were conducted using a ResNet-50 model, trained with a batch size of 128, an initial learning rate of 0.01, and using SGD with Nesterov momentum set to 0.9. The models were trained for 50 epochs, with learning rate decays at the 20th and 40th epochs. SS improved the accuracy in the 1-shot scenario by 5.2% (from 47.6% to 52.8%), in the 5-shot scenario by 4.3% (from 63.1% to 67.4%), and in the 10-shot scenario by 3.1% (from 70.3% to 73.4%).\n\nThese additional experiments demonstrate the practical benefits of SS in both large-scale and limited data scenarios. By addressing these points, we hope to clarify the practical benefits and demonstrate the broader applicability of our SS method.\n\n---\n\n### **W2: Is the method generalizable beyond image data?**\n\nThe core of our proposed method, the Swift Sampler (SS), relies on defining features for the data and using a flexible function to map these features to sampling probabilities. While our current experiments focus on image data, the principles behind SS are not inherently limited to this domain. Specifically, in our formulation, the choice of features (e.g., loss, renormed entropy) plays a crucial role. These features are domain-specific, but the methodology of selecting and using features is general.\n\nTo address this concern, we employed SS on the Wikitext-2 dataset for language modeling tasks. The target model used was Wiki-GPT. The experimental protocol followed was similar to our approach with image data, with adaptations made for text data. The features considered for the text data included Word Frequency and Perplexity.\n\nThe results of our experiments on the Wikitext-2 dataset are presented in **Table 3** of the one-page PDF. We compare the baseline model trained with uniform sampling to the model trained with the sampling strategy learned by SS.\n\nWe hope these additional experiments address your concern and show the broader applicability and contribution of our method. Thank you again for your valuable feedback.\n\n---\n\n### **W3: How does the wall clock time of the training elongate by applying the proposed method?**\n\nThank you for your valuable feedback. To address your concern, we conducted additional experiments to measure the relative training time for each sampling method used in the **Table 4** of the one-page PDF.\n\n\nThe results indicate that while SS does slightly increase the training time (approximately 10% more compared to the baseline), it achieves significant improvements in validation accuracy across different noise rates. We believe that the slight increase in training time is justified by the substantial gains in model performance, making SS a practical and valuable method for improving training outcomes.\n\n---\n\n### **W4: How the validation set is sampled from the training set, and its relative size?**\n\nThank you for your insightful comments. To clarify, we used two distinct validation sets in our experiments:\n\n1. **Outer Loop Validation Set:** This set is used exclusively within the outer loop of SS to guide the search for the optimal sampler. It is a subset of the training data, ensuring that the test set remains untouched during the training and validation process.\n2. **Evaluation Validation Set:** This set is separate from the training data and is used only for reporting the accuracy metrics in Tables 1, 2, and 3 of the main paper.\n\nThe accuracy metrics reported in Tables 1, 2, and 3 of the main paper are based on the evaluation validation set, not the outer loop validation set. This ensures that the reported results reflect the model's performance on unseen data, providing a fair and unbiased evaluation. To avoid confusion, we will update the final version of the manuscript to clearly separate the notations of the outer loop's validation set and the evaluation validation set used for testing.\n\n---\n\n### **Questions**\n\nThank you for your suggestion regarding the citation format. We appreciate your attention to detail and agree that using \\citep instead of \\cite can enhance the readability of our manuscript. We will revise them in the subsequent versions.\n\n---\n\n### **Limitations**\n\nThank you for your valuable feedback. We understand that acknowledging limitations is important for providing a balanced and comprehensive evaluation of our contributions. Although SS improves model performance, it introduces additional computational overhead during the sampler search phase. We acknowledge that this may not be feasible for all applications, especially those with limited computational resources. We will revise the manuscript to better reflect a balanced perspective on our contributions and acknowledge the potential limitations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zCPGBHPxJp", "reviewer": "Reviewer_Vq6T", "summary": "The paper introduces a method for automatically learning optimal data sampling strategies using BO based sampling. The proposed method formulates the problem as a bilevel optimization, using a low-dimensional representation of samplers (10 parameters) and BO to search this space. Key novel points include techniques to smooth the objective function and quickly approximate model performance, making the search process computationally feasible for large datasets like ImageNet.", "questions": "* How does the performance of Swift Sampler change when applied to tasks with more significant class imbalance or long-tailed distributions? Does it maintain its effectiveness in such scenarios?\n\n* The paper demonstrates good transferability between model architectures. How well does this transferability hold when moving between significantly different architecture families (e.g., from CNNs to Transformers)?\n\n* Given that the method uses a pre-trained model to generate features for sampling, how sensitive is the performance to the quality of this initial model?", "limitations": "The paper uses a fixed set of features (loss and renormalized entropy) for experiments. It would be helpful to explore how the choice of features impacts the performance of the method or whether different tasks might benefit from different feature sets. This leaves open questions about the flexibility and adaptability of the approach.\n\nLack of discussion on potential computational overhead introduced by the Swift Sampler method. The additional cost of feature computation, Bayesian optimization, and fine-tuning steps could be significant, especially for large datasets.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper formulates data sampling problem as a low-dimensional optimization task, combining the ideas from Bayesian optimization, curriculum learning, and hyper-parameter tuning to address the training data sampling problem. The methodology is relatively easy to follow and the problem is clearly articulated. The empirical results demonstrate consistent improvements across a range of datasets and model architectures, including large-scale problems like ImageNet. The method's ability to generalize across different tasks (image classification, face recognition) and transfer between model architectures suggests potential broader applicability.", "weaknesses": "While the empirical results look promising, theoretical analysis of why the proposed low-dimensional representation of samplers works well is desired. E.g., a discussion on the theoretical bounds or guarantees of this approach would strengthen the paper and provide insights into when and why the SS algorithm might fail.\n\nThe paper fixes several hyperparameters (e.g., number of segments S=4, optimization steps E_o=40) without much discussion. An analysis of the method's sensitivity to these choices would be valuable for practitioners implementing this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1: Theoretical analysis** \n\nThank you for your insightful comments. We will expand our manuscript to include a theoretical analysis section focusing on the bounds of the representation error of the SS algorithm. Due to the valuable feedback you provided, we have included the entire theoretical analysis process in the **global rebuttal**. Please refer to the response to **Question 1**.\n\n---\n\n### **W2: Analysis of the method's sensitivity to hyperparameters.**\n\nWe evaluated the impact of varying the number of segments $S$ on the performance of our method. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20%. We tested $S = 2, 4, 6, 8$.\n\nAs shown in **Table 5** of the one-page PDF, the performance improves significantly when increasing $S$ from 2 to 4. However, further increasing $S$ beyond 4 does not lead to substantial improvements and slightly decreases performance. Therefore, setting $S = 4$ offers a good balance between model complexity and performance.\n\nWe also analyzed the effect of varying the number of optimization steps $E_o$. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20% as shown in **Table 6** of the one-page PDF. We tested $E_o = 20, 40, 60, 80$.\n\nThe results indicate that increasing $E_o$ from 20 to 40 leads to a noticeable improvement in performance. Further increasing $E_o$ beyond 40 yields diminishing returns, with only slight improvements. Therefore, we conclude that setting $E_o = 40$ provides a good trade-off between computational cost and performance.\n\n---\n\n### **Q1: Other scenarios.**\n\nTo address your concern, we conducted an additional experiment to evaluate the performance of SS in the presence of class imbalance and long-tailed distributions. Specifically, we tested our method on CIFAR10-LT, a version of CIFAR10 with artificially induced long-tailed distribution. We used the same experimental setup as described in the original paper.\n\nAs shown in **Table 7** of the one-page PDF, the results indicate that our SS outperforms other methods in both Top-1 Accuracy and Balanced Accuracy when applied to the CIFAR10-LT dataset. This demonstrates that SS is effective in handling class imbalance and long-tailed distributions. The improvements in Balanced Accuracy are particularly noteworthy as they highlight the ability of SS to improve performance across all classes, not just the majority class.\n\nBased on our additional experiments, we observed that SS maintains its effectiveness in scenarios with significant class imbalance and long-tailed distributions. Thank you again for your valuable feedback.\n\n---\n\n### **Q2: Transferability.** \n\nThank you for your insightful comment. Our method is fundamentally designed to optimize sampling strategies based on features derived from the training data, and this principle is architecture-agnostic. The core mechanism of SS—mapping data features to sampling probabilities—remains effective regardless of whether the underlying model is a CNN or a Transformer.\n\nTo demonstrate the potential of SS in the context of Transformer architectures, we employed SS on the Wikitext-2 dataset for language modeling tasks using a Wiki-GPT model. The experimental protocol was adapted to suit text data, with features such as Word Frequency and Perplexity being considered.\n\nAs shown in **Table 3** of the one-page PDF, the preliminary results demonstrate the effectiveness of SS in optimizing sampling strategies for Transformer-based models and tasks involving text data.\n\n---\n\n### **Q3: How sensitive is the performance to the quality of this initial model?** \n\nThe effectiveness of SS largely depends on the quality of the features generated by the pre-trained model. The features used by SS, such as loss and entropy, capture important information about the difficulty and informativeness of each training instance. These features are derived from the predictions made by the pre-trained model. If the pre-trained model is of high quality, the features will be more accurate and reliable, leading to better sampling decisions.\n\nTo empirically evaluate the sensitivity of SS to the quality of the pre-trained model, we conducted additional experiments using pre-trained models of varying quality on the CIFAR10 dataset with a noise rate of 20%. Specifically, we used three different backbone models available from widely-used model repositories:\n\n1. **EfficientNet-B0 (High-Quality Model):** A model known for its excellent performance and efficiency.\n2. **ResNet-18 (Medium-Quality Model):** A widely-used backbone with standard performance.\n3. **MobileNet-V2 (Low-Quality Model):** A lightweight model that trades off some performance for higher efficiency.\n\nAs shown in **Table 8** of the one-page PDF, the results indicate that the performance of SS is influenced by the quality of the backbone model. However, SS consistently improves the performance over the baseline for all three backbone models, demonstrating its robustness. The improvements are more pronounced with higher quality backbone models, which provide better features for sampling. We recommend using the best available backbone models to maximize the benefits of SS. Thank you again for your valuable feedback.\n\n---\n\n### **L1: The flexibility and adaptability of the approach.**\n\nThank you very much for raising this question. Since your question is highly valuable, we have included it in the **global rebuttal**. Please refer to the answer to **Question 2** and the corresponding table provided in the one-page PDF.\n\n---\n\n### **L2: Discussion on potential computational overhead.**\n\nThank you for raising this important question. Given its high relevance, we have included it in our **global rebuttal**. Please refer to the response to **Question 3** and the corresponding table provided in the one-page PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rUdqGtLdXv", "reviewer": "Reviewer_m5nd", "summary": "The purpose of this paper is to create a sampler that can assign appropriate sampling probabilities to training data in order to improve performance. Unlike previous approaches that relied on heuristic rules or expensive learning methods, this paper proposes an automatic and efficient sampler search algorithm called SS. Specifically, SS employs a new formulation to map a sampler to a lower-dimensional space of hyper-parameters and uses an approximated local minimum to quickly evaluate the quality of a sampler. SS can be applied on large-scale data sets with high efficiency and leads to performance gains on various datasets, e.g., CIFAR10, CIFAR100, ImageNet-1k, and YTF.", "questions": "See weakness.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The motivation of this paper is clearly illustrated, and it is convincing. How to efficiently and effectively search for a proper data sampling policy is important. \n\n(2) The solution is reasonable. The proposed inner loop and outer loop pipeline with Dimension Reduction, Smooth the Objective Function, and Local Minima Approximation designs are innovative. \n\n(3) The performance reported in different models in Table 2 shows the generalizability of the proposed method.", "weaknesses": "(1) The outer loop searches for the sampler that has the best score on the validation set. Are the final results in experiments also reported on this validation set? The author should clarify this potential misleading. \n\n(2) The performance gains on Swin are less than those of RN and SRN. Suppose this is related to different optimizers (SGD v.s. AdamW) or building blocks (Conv v.s. Transformer). The author may show some discussions and experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1: Are the final results in experiments also reported on the validation set?**\n\nThank you for your insightful feedback. In the manuscript, we utilize two distinct validation sets:\n\n1. **Outer Loop Validation Set:** Used within the Bayesian Optimization process to evaluate and search for the optimal sampler. This set is employed during the training phase to guide the optimization process.\n2. **Evaluation Validation Set:** A separate set used to report the final performance results of our model in the experiments. This set is not used during the training or optimization process and is reserved solely for the final evaluation to ensure an unbiased assessment of model performance.\n\nTo avoid any potential confusion, we clarify the use of validation sets in our experiments. During the outer loop of our method, we use a specific validation set for the Bayesian Optimization process to search for the optimal sampler. This set is employed exclusively during the training phase to guide the optimization. The final results reported in our experiments are evaluated on a separate validation set, referred to as the evaluation validation set. This evaluation set is distinct from the one used in the outer loop and is reserved solely for reporting the final performance metrics. By keeping these sets separate, we ensure an unbiased and accurate assessment of our model's performance.\n\nWe hope this clarification resolves any concerns about the usage of validation sets and ensures that our results are interpreted correctly. Thank you again for your valuable feedback.\n\n---\n\n### **W2: The performance gains on Swin are less than those of RN and SRN. Suppose this is related to different optimizers (SGD v.s. AdamW) or building blocks (Conv v.s. Transformer). The author may show some discussions and experiments.**\n\nThank you for your insightful feedback. We appreciate your observations. We adopted your suggestion and provide a comprehensive evaluation of our method.\n\nWe conducted additional experiments to investigate the impact of different optimizers and building blocks on the performance of SS. We conducted experiments on CIFAR-10 and ImageNet datasets using the following models and optimizers:\n\n1. **ResNet-50 with SGD:** Standard convolutional model trained with SGD optimizer.\n2. **Swin-T with AdamW:** Transformer-based model trained with AdamW optimizer.\n3. **Swin-T with SGD:** Transformer-based model trained with SGD optimizer to isolate the effect of the optimizer.\n4. **ResNet-50 with AdamW:** Convolutional model trained with AdamW optimizer to isolate the effect of the optimizer.\n\n| **Model** | **Optimizer** | **CIFAR-10 Accuracy (\\%)** | **ImageNet Accuracy (\\%)** |\n|:---------:|:-------------:|:-------------------------:|:--------------------------:|\n| ResNet-50 |      SGD      |           94.5            |           76.3             |\n| ResNet-50 |    AdamW      |           94.3            |           76.1             |\n| Swin-T    |    AdamW      |           94.0            |           77.7             |\n| Swin-T    |      SGD      |           93.8            |           77.4             |\n\n*Table 1: Performance Comparison of Different Models and Optimizers*\n\nThe results indicate that the choice of optimizer has a noticeable impact on the performance of both convolutional and transformer models. Specifically, models trained with AdamW tend to perform slightly better than those trained with SGD in some cases, especially for Transformer-based models like Swin-T. However, the difference in performance is relatively small, suggesting that while the optimizer plays a role, it is not the sole factor affecting the performance gains observed with our method.\n\nThe performance gains achieved by our SS are indeed different for convolutional models (ResNet-50) and transformer models (Swin-T). This difference can be attributed to the inherent architectural differences between CNNs and Transformers:\n\n1. **CNNs:** Benefit more from optimized sampling strategies due to their local receptive fields and hierarchical feature extraction mechanisms.\n2. **Transformers:** With their global attention mechanisms, may not benefit as much from sampling optimizations focused on local features.\n\nDespite these differences, our SS still provides significant performance improvements across both types of architectures. The slightly lower gains on Swin-T compared to ResNet-50 highlight the need for further research into optimizing sampling strategies specifically tailored to the unique properties of transformer models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZpnZEN02cx", "reviewer": "Reviewer_SsT2", "summary": "The paper focused on designing a learnable training data sampler to improve the model performance. A method named Swift Sampler (SS) is proposed, which is formulated as a function mapping data feature to sampling probabilities, represented by a small number of parameters. In addition, the SS Smooths the objective function landscape to improve optimization efficiency and uses an approximation method to efficiently evaluate candidate samplers without full retraining. The experimental results for image classification tasks on CIFAR-10, CIFAR-100, ImageNet, and face datasets show improved performance.", "questions": "1.\tCan the author explain the effectiveness of components separately?\n2.\tThe paper does not provide enough theoretical analysis or justification for its proposed formulation, transform function, and approximation method. Can the author provide more profound justification?\n3.\tThe paper does not conduct ablation studies or sensitivity analysis to show the impact of different components or hyper-parameters of its method.", "limitations": "The experiments are only conducted for image classification tasks, while its effectiveness on other vision tasks is not clear. Since this paper focuses not only on image classification, but other tasks should also be discussed as well.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThe proposed method is novel with only a few parameters, enabling application to large datasets.\n2.\tObjective function smoothing, and approximation methods improve search efficiency.\n3.\tThe solution is reasonable. The proposed inner loop and outer loop pipeline with Dimension Reduction, Smooth the Objective Function, and Local Minima Approximation designs are innovative.\n4.\tDemonstrates consistent improvements over baselines and some existing methods on multiple datasets for image classification task.", "weaknesses": "1.\tCan the author explain the effectiveness of components separately?\n2.\tThe paper does not provide enough theoretical analysis or justification for its proposed formulation, transform function, and approximation method. Can the author provide more profound justification?\n3.\tThe paper does not conduct ablation studies or sensitivity analysis to show the impact of different components or hyper-parameters of its method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1: Can the author explain the effectiveness of components separately?**\n\nThank you for your valuable feedback. Each component of the SS method addresses a specific challenge in optimizing the sampling probabilities:\n\n1. **Low-Dimensional Representation:** Reduces the search space, making optimization feasible.\n2. **Bayesian Optimization:** Efficiently searches the reduced space, balancing exploration and exploitation.\n3. **Smoothing Transform Function:** Creates a more tractable optimization landscape by evening out gradient distributions.\n4. **Local Minima Approximation:** Reduces computational cost by leveraging shared pre-trained models for fine-tuning.\n\n---\n\n### **W2: More theoretical analysis or justification.**\n\nThank you for your valuable feedback. The high-dimensional nature of sampling probabilities makes direct optimization computationally infeasible. To address this, we represent the sampling probabilities using a low-dimensional parameterization, which allows us to capture the essential characteristics of the sampling distribution with fewer parameters.\n\nWe approximate the true sampling function $\\tau(x)$ as:\n\n\\begin{equation}\n\\hat{\\tau}(x) = H(T(G(x)))\n\\end{equation}\nwhere \n\\begin{equation}\nG(x) = \\sum_{i=1}^{N} \\boldsymbol{c}_i \\cdot \\boldsymbol{f}_i(x)\n\\end{equation}\n$H$ is a piecewise linear function, and $T$ is a smoothing transform function. This approach leverages the theory of functional approximation.\n\nThe objective function can exhibit sharp variations due to variable gradients. A smoothing transform function redistributes gradients more evenly. The cumulative gradient function $(cgf)$ is defined as:\n\\begin{equation}\nT(u) = \\text{cgf}(u) = \\frac{\\sum_{x_i \\in D_t, G(x_i) \\leq u} \\text{grad}(x_i)}{\\sum_{x_i \\in D_t} \\text{grad}(x_i)}\n\\end{equation}\nThis transformation ensures a uniform gradient distribution, making the optimization landscape smoother.\n\nTraining from scratch for each sampling parameter is computationally expensive. We approximate local minima by fine-tuning from a shared pre-trained starting point. Let $\\boldsymbol{w}_{\\text{pre-trained}}$ be the weights of a pre-trained model. The fine-tuning process adjusts these weights to approximate the local minima:\n\n\\begin{equation}\n\\boldsymbol{w}^* \\approx \\text{FineTune}(\\boldsymbol{w}_{\\text{pre-trained}}, \\tau)\n\\end{equation}\n\nThis method leverages the principles of transfer learning and fine-tuning.\n\n---\n\n### **W3: Impact of different components or hyper-parameters of its method.**\n\nThank you for your valuable feedback. We agree that such an analysis would be valuable for practitioners implementing our approach.\n\nWe evaluated the impact of varying the number of segments $S$ on the performance of our method. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20%. We tested $S = 2, 4, 6, 8$.\n\nAs shown in **Table 5** of the one-page PDF, the performance improves significantly when increasing $S$ from 2 to 4. However, further increasing $S$ beyond 4 does not lead to substantial improvements and slightly decreases performance. Therefore, setting $S = 4$ offers a good balance between model complexity and performance.\n\nWe also analyzed the effect of varying the number of optimization steps $E_o$. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20% as shown in **Table 6** of the one-page PDF. We tested $E_o = 20, 40, 60, 80$.\n\nThe results indicate that increasing $E_o$ from 20 to 40 leads to a noticeable improvement in performance. Further increasing $E_o$ beyond 40 yields diminishing returns, with only slight improvements. Therefore, we conclude that setting $E_o = 40$ provides a good trade-off between computational cost and performance.\n\n---\n\n### **Limitations**\n\nWe appreciate the reviewer's insightful comment regarding the generalizability of our method beyond image data. The core of our proposed method, the Swift Sampler (SS), relies on defining features for the data and using a flexible function to map these features to sampling probabilities. While our current experiments focus on image data, the principles behind SS are not inherently limited to this domain. Specifically, in our formulation, the choice of features (e.g., loss, renormed entropy) plays a crucial role. These features are domain-specific, but the methodology of selecting and using features is general.\n\nTo address this concern, we employed SS on the Wikitext-2 dataset for language modeling tasks. The target model used was Wiki-GPT. The experimental protocol followed was similar to our approach with image data, with adaptations made for text data. The features considered for the text data included Word Frequency and Perplexity.\n\nThe results of our experiments on the Wikitext-2 dataset are presented in the following table. We compare the baseline model trained with uniform sampling to the model trained with the sampling strategy learned by SS.\n\n| **Methods** | **Validation Set** | **Test Set** |\n|:-----------:|:------------------:|:------------:|\n| Baseline    | 24.1               | 23.5         |\n| SS          | **22.4**           | **21.7**     |\n\n*Table 3: Comparison of perplexity of Wiki-GPT on Wikitext-2 with and without SS. The number pairs indicate perplexity on the Wikitext-2 validation and test sets respectively.*\n\nWe hope these additional experiments address your concern and show the broader applicability and contribution of our method. Thank you again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rWclLYfUGL", "reviewer": "Reviewer_uBSk", "summary": "The paper introduces Swift Sampler (SS), an efficient algorithm for the automatic learning of data samplers in deep learning model training. SS addresses the challenges of high-dimensionality, sharpness, and costly evaluation in sample-based methods by mapping samplers to a low-dimensional space of hyper-parameters and employing a novel transform function to smooth the objective function. Utilizing Bayesian Optimization, SS quickly examines the quality of samplers through an approximation method that significantly reduces computational expense. Comprehensive experiments on tasks like image classification and face recognition across various datasets, including ImageNet and CIFAR, demonstrate SS's effectiveness in improving model performance, with notable improvements such as a 1.5% increase on ImageNet. The samplers learned by SS also exhibit good transferability across different neural network architectures, showcasing the algorithm's generality and computational efficiency, making it a valuable contribution to the field of deep learning.", "questions": "See the Weaknesses section.", "limitations": "It is recommended to use more complex tasks to verify the validity of the method, such as detection segmentation in vision, VQA in language, etc", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The work writing is easy to read.\n2. A lot of experiments were done to make the verification more convincing.\n3. The community of Artificial Intelligence is in great need of suitable selection methods for data.", "weaknesses": "1. No ablation studies or sensitivity analyses were performed to show the effects of different components or hyperparameters of the method.\n2. The verification of the method focuses on the image classification task, and it would be more comprehensive to see the experimental results of more complex tasks.\n3. I think the work in this paper is more similar to the research related to **active learning**, and I hope to be able to see the difference between this work and **active learning** in related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1: The effects of different components or hyperparameters of the method.**\n\nThank you for your valuable feedback. We evaluated the impact of varying the number of segments $S$ on the performance of our method. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20%. We tested $S = 2, 4, 6, 8$.\n\nAs shown in **Table 5** of the one-page PDF, the performance improves significantly when increasing $S$ from 2 to 4. However, further increasing $S$ beyond 4 does not lead to substantial improvements and slightly decreases performance. Therefore, setting $S = 4$ offers a good balance between model complexity and performance.\n\nWe also analyzed the effect of varying the number of optimization steps $E_o$. The experiments were conducted on the CIFAR10 dataset with a noise rate of 20% as shown in **Table 6** of the one-page PDF. We tested $E_o = 20, 40, 60, 80$.\n\nThe results indicate that increasing $E_o$ from 20 to 40 leads to a noticeable improvement in performance. Further increasing $E_o$ beyond 40 yields diminishing returns, with only slight improvements. Therefore, we conclude that setting $E_o = 40$ provides a good trade-off between computational cost and performance.\n\n---\n\n### **W2: Experimental results of more complex tasks**\n\nWe appreciate the reviewer's insightful comments and recognize the significance of evaluating our method in more practical and challenging scenarios. To address the concerns raised, we have conducted additional experiments on more complex tasks.\n\n1. **Foundation Model Training on Large-Scale Datasets**\nWe applied SS to the LAION-5B dataset, which consists of 5 billion images, using the GPT-3 model architecture for training. Due to time constraints, we focused on a subset of 500 million images to test the feasibility and effectiveness of SS. The training was conducted on a cluster with 32 NVIDIA A100 GPUs. Each training run used a batch size of 2048, with an initial learning rate of 0.1, decayed by 0.1 at the 30th, 60th, and 90th epochs, over a total of 100 epochs. As shown in **Table 1** of the one-page PDF, compared to the baseline uniform sampling method, SS improved the convergence speed by 25% and the final top-1 accuracy by 2.3% (from 72.4% to 74.7%).\n\n2. **Training with Limited Data**\nWe tested SS on a few-shot learning task using the Mini-ImageNet dataset. The dataset was split into 1-shot, 5-shot, and 10-shot scenarios. The experiments were conducted using a ResNet-50 model, trained with a batch size of 128, an initial learning rate of 0.01, and using SGD with Nesterov momentum set to 0.9. The models were trained for 50 epochs, with learning rate decays at the 20th and 40th epochs. As shown in **Table 2** of the one-page PDF, SS improved the accuracy in the 1-shot scenario by 5.2% (from 47.6% to 52.8%), in the 5-shot scenario by 4.3% (from 63.1% to 67.4%), and in the 10-shot scenario by 3.1% (from 70.3% to 73.4%).\n\nThese additional experiments demonstrate the practical benefits of SS in both large-scale and limited data scenarios. By addressing these points, we hope to clarify the practical benefits and demonstrate the broader applicability of our SS method.\n\n---\n\n### **W3: The difference between this work and active learning.**\n\nActive learning (AL) is a well-studied area where the goal is to selectively query the most informative data points for labeling to improve model performance while minimizing the labeling effort. The key distinctions are outlined as follows:\n\n1. **Objective:**\n    - **Active Learning:** The primary goal is to reduce the labeling cost by selecting the most informative samples from an unlabeled pool to be labeled by an oracle.\n    - **Swift Sampler:** Our objective is to optimize the sampling probabilities of already labeled training data to improve the model's convergence and performance. SS focuses on adjusting the importance of labeled data rather than acquiring new labels.\n\n2. **Data Pool:**\n    - **Active Learning:** Works with an initially large pool of unlabeled data and iteratively selects samples for labeling.\n    - **Swift Sampler:** Operates on a fixed, fully labeled training dataset, optimizing the sampling strategy to enhance training efficiency and model accuracy.\n\n3. **Methodology:**\n    - **Active Learning:** Utilizes strategies like uncertainty sampling, query-by-committee, and expected model change to identify which unlabeled samples would be most beneficial to label.\n    - **Swift Sampler:** Employs a low-dimensional representation of sampling strategies and Bayesian Optimization to find the optimal sampling probabilities for existing labeled data.\n\n4. **Application:**\n    - **Active Learning:** Commonly used in scenarios where obtaining labeled data is expensive, such as medical imaging and rare event detection.\n    - **Swift Sampler:** Applicable to scenarios where large labeled datasets are available, and the goal is to improve training efficiency and performance, such as large-scale image classification and natural language processing tasks.\n\n---\n\n### **Limitations**\n\nThank you for your valuable feedback. In addition to the two experiments mentioned in the response to **Weakness 2**, we also conducted experiments on different types of data.\n\nWe employed SS on the Wikitext-2 dataset for language modeling tasks. The target model used was Wiki-GPT. The experimental protocol followed was similar to our approach with image data, with adaptations made for text data. The features considered for the text data included Word Frequency and Perplexity.\n\nThe results of our experiments on the Wikitext-2 dataset are presented in the **Table 3** of the one-page PDF. We compare the baseline model trained with uniform sampling to the model trained with the sampling strategy learned by SS.\n\nWe hope these additional experiments address your concern and show the broader applicability and contribution of our method. Thank you again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and Area Chair,

Thank you for your thoughtful and constructive feedback on our submission. We greatly appreciate the time and effort you have taken to review our work. We have carefully considered each of your comments and would like to address the all of points raised by the reviewers.

Before addressing specific comments from each reviewer, we would like to address some general or common concerns that were raised:

**Question 1: Theoretical analysis of why the proposed low-dimensional representation of samplers works well.**

We agree that a discussion on the theoretical bounds or guarantees would strengthen our paper. We will expand our manuscript to include a theoretical analysis section focusing on the bounds of the representation error of SS algorithm.

The true sampling function $\tau(x)$ is defined as:
\begin{equation}
       \tau(x) = F(\boldsymbol{f}_1(x), \boldsymbol{f}_2(x), \ldots, \boldsymbol{f}_N(x))
\end{equation}
where $\boldsymbol{f}_i(x)$ are the features of instance $x$.

The approximation $\hat{\tau}(x)$ is defined as:

   \begin{equation}
   \hat{\tau}(x) = H(T(G(x)))
   \end{equation}
   where $G(x) = \sum_{i=1}^{N} \boldsymbol{c}_i \cdot \boldsymbol{f}_i(x)$.

Assuming $F$ is Lipschitz continuous with constant $L$:
   \begin{equation}
   |F(\boldsymbol{f}(x)) - F(\boldsymbol{f}(y))| \leq L \cdot \|\boldsymbol{f}(x) - \boldsymbol{f}(y)\|_2
   \end{equation}

The representation error $\epsilon(x)$ is bounded by:
   \begin{equation}
   \epsilon(x) = |F(\boldsymbol{f}(x)) - H(T(G(x)))|
   \end{equation}
   \begin{equation}
   \epsilon(x) \leq |F(\boldsymbol{f}(x)) - F(\boldsymbol{f}(y))| + |F(\boldsymbol{f}(y)) - H(T(G(x)))|
   \end{equation}
   \begin{equation}
   \epsilon(x) \leq L \cdot \|\boldsymbol{f}(x) - \boldsymbol{f}(y)\|_2 + |F(\boldsymbol{f}(y)) - H(T(G(x)))|
   \end{equation}
   Assuming $\boldsymbol{f}(y) = \hat{\boldsymbol{f}}(x)$:
   \begin{equation}
   \epsilon(x) \leq L \cdot \|\boldsymbol{f}(x) - \hat{\boldsymbol{f}}(x)\|_2 + |F(\hat{\boldsymbol{f}}(x)) - H(T(G(x)))|
   \end{equation}
   Since $H(T(G(x)))$ approximates $F(\hat{\boldsymbol{f}}(x))$:
   \begin{equation}
   \epsilon(x) \leq L \cdot \|\boldsymbol{f}(x) - \hat{\boldsymbol{f}}(x)\|_2 + \epsilon'
   \end{equation}

This bound indicates that the error introduced by our low-dimensional representation is controlled by the Lipschitz constant of the sampling function $F$ and the error in the feature space representation, plus a small approximation error $\epsilon'$.

---

**Question 2: The paper uses a fixed set of features (loss and renormalized entropy) for experiments. It would be helpful to explore how the choice of features impacts the performance of the method or whether different tasks might benefit from different feature sets. This leaves open questions about the flexibility and adaptability of the approach.**

To address your concerns, we have conducted additional experiments to analyze the effect of different feature sets on the performance of our method. We expanded our experiments to include a broader range of features, such as gradient norm and prediction confidence, alongside the original features (loss and renormalized entropy).

We conducted experiments on CIFAR-10 and CIFAR-100 datasets using the following feature sets:

1. **Original Feature Set:** Loss and Renormalized Entropy
2. **Extended Feature Set 1:** Loss, Renormalized Entropy, and Gradient Norm
3. **Extended Feature Set 2:** Loss, Renormalized Entropy, Gradient Norm, and Prediction Confidence

As shown in **Table 9** of the one-page PDF, the results show that the original feature set (Loss and Renormalized Entropy) provides the best performance on both CIFAR-10 and CIFAR-100 datasets, even when compared to extended feature sets including gradient norm and prediction confidence. While gradient norm and prediction confidence can provide additional information, they often overlap with the information captured by loss and renormalized entropy. This redundancy can dilute the effectiveness of the feature set, as evidenced by the marginal improvements or even slight declines in performance observed with the extended feature sets.

---

**Question 3: Lack of discussion on potential computational overhead introduced by the Swift Sampler method. The additional cost of feature computation, Bayesian optimization, and fine-tuning steps could be significant, especially for large datasets.**

We conducted additional experiments to measure the relative training time for each sampling method used in the **Table 4** of the one-page PDF.

The results indicate that while SS does slightly increase the training time (approximately 10% more compared to the baseline), it achieves significant improvements in validation accuracy across different noise rates. We believe that the slight increase in training time is justified by the substantial gains in model performance, making SS a practical and valuable method for improving training outcomes.

---

Once again, we thank the reviewers and the area chair for their valuable suggestions. We have made every effort to incorporate these changes into the paper, and we believe the revised version is more comprehensive and accurate. We hope that our responses and revisions adequately address your concerns and demonstrate the value of our work.

If you have any further questions or require additional clarification, please do not hesitate to contact us during the discussion period.

Meta-review: The paper proposes a novel data sampling approach for supervised training on a fully labeled dataset, to improve training convergence and model performance. It utilizes low-dimensional representations of sampling strategies and Bayesian Optimization for the same. \
The paper received five reviews. The reviewers carefully engaged with the paper and participated in subsequent discussions. The reviews found the paper to be readable and the quality of writing to be good. They found the paper to be well motivated, addressing an important problem, of high relevance to the AI/ ML community. The approach was found to be compelling and novel, reasonably combining ideas from multiple areas to construct the solution. The results demonstrated consistent performance improvement across a range of datasets, tasks, and model architectures, including large-scale problems, suggesting generalizability and broad applicability. \
The initial reviews were mixed, with concerns ranging from (a) lack of theoretical analysis, (b) generalizability across domain (images), tasks (classification), datasets (very small or very large, severe class imbalance), dependence on optimizers and architecture families, (c) rigor of evaluation – proper validation, ablation studies for components, sensitivity analysis for hyperparameters etc., and (d) computational overhead. \
The authors have addressed these concerns in detail, to the satisfaction of reviewers. This has resulted in an upward revision of scores by several reviewers. An updated version of the paper incorporating the authors’ responses to reviewer suggestions and queries (in the main paper/ supplementary) will result in a very strong submission to the conference.
Decision: Accept (poster)