Swift Sampler: Efficient Learning of Sampler by 10 Parameters

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:02:06

Abstract: Data selection is essential for training deep learning models. An effective data sampler assigns proper sampling probability for training data and helps the model converge to a good local minimum with high performance. Previous studies in data sampling are mainly based on heuristic rules or learning through a huge amount of time-consuming trials. In this paper, we propose an automatic swift sampler search algorithm, SS, to explore automatically learning effective samplers efficiently. In particular, SS utilizes a novel formulation to map a sampler to a low dimension of hyper-parameters and uses an approximated local minimum to quickly examine the quality of a sampler. Benefiting from its low computational expense, SS can be applied on large-scale data sets with high efficiency. Comprehensive experiments on various tasks demonstrate that SS powered sampling can achieve obvious improvements (e.g., 1.5% on ImageNet) and transfer among different neural networks. Project page: https://github.com/Alexander-Yao/Swift-Sampler.

Main paper: 
 introduction: 
 Training data plays a pivotal role in deep learning tasks. The sampling probability of data in the training process can significantly influence the performance of the learned model. A set of works Jiang et al. (2019) ; Katharopoulos & Fleuret (2018) ; Needell et al. (2014) ; Johnson & Guestrin (2018) ; Han et al. (2018) have demonstrated improvements in model training by sampling data according to different features and under different rules. These studies reveal that how to sample examples during training is nontrivial and sampling data in a totally uniform way is not always the optimal choice. Previous works on data sampling strategy include two main categories: human-defined rules and learning-based methods. Some works Han et al. (2018) ; Jiang et al. (2019) ; Katharopoulos & Fleuret (2018) ; Hacohen & Weinshall (2019) ; Kumar et al. (2010) proposed manually designed rules, such as setting the sampling probability of training examples with loss values larger than a threshold as zero, or making the sampling probability proportional to the gradient norms. Such human-defined rules are designed for specific tasks and hardly adapts to different scenarios, because optimal sampling strategy varies among different tasks and data sets. Learning-based methods Fan et al. (2017) ; Jiang et al. (2017) ; Ren et al. (2018) explore an automatic way to assign sampling probability to a given learning task, including sample-based and differential based methods. Sample-based methods take advantage of deep reinforcement learning (DRL) Fan et al. (2017) to model the training process of the target model as an environment and use a DRL model to learn the optimal sampling strategy during training. These methods need hundreds of replays of the training process and require too much searching cost to be applied on large-scale data sets, e.g., ImageNet Russakovsky et al. (2015) . Differential based methods Ren et al. (2018) ; Shu et al. (2019) assume a completely clean meta dataset and use the inner product between the gradients of training data and meta data to sample or reweight training data. The problem is that the clean meta data is not guaranteed to be available for all scenarios, and complex changes are made in the training process.To explore an automatic way of sampler search, we focus on improving sample-based search methods as they do not require extra meta data and impose no change in the training process when applying the obtained sampler. Typically, a sample-based method repeatedly uses an agent (e.g., Bayesian Optimization or Deep Reinforcement Learning) to sample a sampler and evaluate the objective function value of the sampled sampler to update the agent. There are three challenges in designing a sample-based method for automatic sampler search: (1) High dimension. A sampler is defined by a vector of sampling probabilities of all instances in the training set. Therefore, the complexity of searching the optimal sampler in a sample-based way increases exponentially with the number of training instances. (2) Sharpness. As great difference exists among gradients of various instances in the training set, sampling probability of some instances has a significant impact on the performance of resulted models. Such property results in the sharpness of the objective function for samplers and harms the efficiency of the agent's learning. (3) Costly evaluation. The agent seeks a sampler that achieves high performance when used to train the model from scratch. Thus the evaluation of a single sampler causes high computational expense (e.g., training 100 epochs from scratch on ImageNet).In this paper, we aim at addressing the three problems listed above. For the high dimension problem (1), we define sampler as a function mapping the feature of the training data to their sampling probabilities and formulate the search space of sampler as a family of composite functions, which are represented by a small number of hyper-parameters. The amount is much less than the number of training data. Moreover, our formulation of the sampler has a flexible expression covering any features used in previous works and is adaptive to a range of data sets. We carefully choose the features and mapping functions so that the number of parameters to be learned for the sampler is only 10. To relieve the sharpness problem (2) of the objective function, we modify the objective function by designing a transform function to smooth the landscape of the objective function. This transform function balances the amounts of gradient norm in different areas of the definition domain of the objective function. For the costly evaluation problem (3), we use a fast approximation method for learning the network, which is much less expensive than training from scratch. Integrating the designs above, we propose an automatic swift sampler learning (SS) method, where we choose naive Bayesian Optimization (BO) Brochu et al. (2010) ; Klein et al. (2016) ; Snoek et al. (2012) as the agent. The swift here means our formulation of the sampler has a very small volume of parameters, and our method has a high speed of sampler learning, like a swift, the small and fast bird.We apply SS to training neural networks with various sizes, including ResNet-18 and SE-ResNext-101, with training data from different data sets including ImageNet Russakovsky et al. (2015) , CIFAR10 and CIFAR100 Krizhevsky et al. (2009) . Experiments demonstrate obvious improvements in the performance compared with the baseline and other methods, e.g., 1.5% on ImageNet. During the search process, SS optimizes much faster compared with previous automatic methods, and the learned sampler consists of only 10 hyper-parameters. Further analysis shows that the sampler found by SS transfers well among neural networks with various architectures and sizes. The contributions of this paper are summarized as follows: (1) We formulate a search space of sampler consisting of a new family of functions. The family of functions is decided by a small number of parameters and also has flexible expression. The reduced dimension facilitates the application of SS on large-scale data sets. (2) To improve the efficiency of the optimization of the sampler, we smooth the sharp objective function of the sampler search problem with a carefully designed transform function. (3) We use an approximation method to approximate the local minima of the sampler efficiently tried by the agent. This method makes SS fast and capable of improving the performance of models on large data sets. 
related work: 
 Hard-wired Methods. Hard-wired methods have fixed sampling rules and focus on a few particular problems, e.g., imbalance, noise and training speed up. Each problem needs respective hand-crafted rules and the designs are based on specific understandings of data. Thus, these methods hardly generalizes over a wide range of data sets. In Han et al. (2018) , the sampling method focuses on denoising problem, and neglects instances with loss values larger than a gradually increasing threshold. It needs prior knowledge on the ratio of noisy instances in the data set. For imbalance, Lin et al. (2017) ; Wang et al. propose methods to boost the training of imbalanced data by optimizing the sampling or weighting (a 'soft' sampling way) of instances, and develop different features as the signals of imbalance. Importance sampling Ma et al. (2017) ; Graves et al. (2017) speeds up the training process by giving larger sampling probabilities to instances with higher loss values, which does not fit noisy data sets. Curriculum learning (CL) methods Kumar et al. (2010) ; Jiang et al. (2015) are also related to our work. It shows that some particular sampling orders benefit training process.Learning-based Methods. Like recently proposed automated loss function search Li et al. (2019a) and augmentation policy search methods Lin et al. (2019) ; Tian et al. (2020) , many recent works also explore automatically learning how to optimize data sampling. They can achieve generalization on various scenarios. Fan et al. (2017) proposes a RL-based framework to optimize the data sampling in different training stages under the setting of mini-batch SGD. It models the RL agent as the teacher to guide the training of the student model. This method requires multiple runs of training from scratch, thus does not fit large data sets such as ImageNet. Li et al. (2019b) proposes a similar RL-based framework to reweight training instances, which can be viewed as a soft sampling method. Some other related works Ren et al. (2018) ; Shu et al. (2019) propose differential based methods. It makes use of the gradients of loss of a completely clear meta dataset, sampling or weighting the training datum via the inner product between their gradients and meta data gradients. In summary, the methods in Fan et al. (2017) ; Li et al. (2019b) require lots of parameters with high computational costs and needs extra data Ren et al. (2018) ; Shu et al. (2019) . In comparison, our approach is fast, requires few parameters but no extra data.Bayesian Optimization. Bayesian Optimization (BO) has shown its great potentials in optimizing the hyper-parameters of learning tasks. The method in Snoek et al. (2012) first introduces BO to optimize the hyper-parameters of various tasks, including SVM and regression. FABOLAS Klein et al. (2016) constructs a BO-based method to simultaneously optimizes computation cost and information gain w.r.t hyper-parameters of SVM and training data size. It is at most 100 times faster than previous methods. Recently, NASBOT Kandasamy et al. (2018) explores the usage of BO in neural architecture search by mathematically defining the distance between two architectures. The distance enables the design of kernel for BO. In our approach, the use of Bayesian Optimization is not our contribution, although naive Bayesian Optimization is used for solving our problem. Therefore, the recent advances in Bayesian Optimization can be also used for our approach. 
method: 
 We introduce SS in this section. First, we describe the problem of sampler search in a bilevel optimization framework, consisting of the outer loop and the inner loop. The outer loop uses an agent to learn to sample the sampling probabilities of training instances and optimizes the performance of the target model when trained under the sampled sampling probabilities. And the inner loop minimizes the loss of model parameters on the training set, with sampling probabilities given by the outer loop. Further, we formulate the sampler, define its search space, and design a transform function to smooth the curvature of the objective function of the outer loop. Then, we introduce our agent for the learning of sampling in the outer loop. Finally, we propose a highly efficient method to approximate the local minima of given sampling probabilities rather than training from scratch. (2014) . In this work, we focus on finding a static sampling probability which guides the parameters of target DNN to the local optimum with the best performance on the validation set. We formulate this problem as follows.For a target task, e.g., image classification, its training set and validation set are respectively denoted by D t and D v , and the parameters of the target model are denote by w. Each sample x i for x i ∈ D t has its corresponding sample probability τ (x i ). We define the probability function τ as sampler. We formulate the optimization of τ as a bilevel problem.The inner loop learns the network parameters that minimize the expected loss on the target task under the sampling probability τ given by the outer loop. Denoted by w * (τ ) the local minima of network parameters trained with loss L(x; w) and sample probability τ , w * (τ ) is obtained as follows:wEQUATION)The outer loop uses an agent to search for the best sampler τ . Specifically, the network with parameters w * (τ ) obtained from the inner loop is used for searching the sampler τ that has the best score P (D v ; w * (τ )) on validation set D v , where P (D; w) is the performance score of parameters w on a given data set D and P (D v ; w * (τ )) is our objective function. The outer loop problem is described as:EQUATION)Both the outer loop and the inner loop are difficult to solve. The outer loop is a high-dimension optimization problem, where the optimized sampler τ (x i ) has a dimension equal to the number of training data. In addition, the objective function P (D v ; w * (τ )) for the agent of the outer loop encounters a sharpness problem. Further, given a τ from the outer loop, the inner loop needs to train from scratch with τ to get the local minima w * (τ ), which involves high computational cost. We introduce our solutions to the three problems hereinafter.The complexity of the optimization problem increases exponentially with the dimension of τ , which is the number of training data. However, modern hyper-parameter optimization methods like BO can hardly handle more than 30 hyper-parameters Brochu et al. (2010) . Hence we reduce the dimension of τ by a simple formulation. The high dimension comes from independence among training samples. However, we assume that the optimal sampler lies in a more compact subdomain. Instead, we restrict that the difference in the sampling probabilities of two training instances is bounded by their distance in the feature space, e.g., the (loss, entropy) space.Example. As an intuitive example, we consider cropped images in ImageNet Russakovsky et al. (2015) dataset as samples and define a one-dimension feature space where the feature is the loss value. Cropped images with the highest loss values have small distance between each other in this feature space. Further, as we show in section 4, most of them are noisy instances, which should all be assigned sampling probabilities near 0. It means they also have small sampling probabilities difference.Mathematically, the intuition above can be formulated as constraining samplers to satisfy Lipschitz condition:EQUATIONwhere ∥f (x i ) -f (x j )∥ 2 is the L 2 distance in the space of feature vector f and C is a real positive number.Dimension Reduction With the above consideration, we define τ (x) as a multivariate continuous function of the features of instance x, described byEQUATION)where F is a multivariate continuous function and f i (x) is i-th feature of example x (i = 1, ..., N ). The choices of features have been explored in a wide range, e.g., loss in Jiang et al. (2019) ; Katharopoulos & Fleuret (2018) , and density in Fan et al. (2017) . Our choice of features is discussed in the latter sections.We consider a family of F which have flexible expression and low dimension. F are formed by:(1) a univariate piecewise linear function H defined on the period [0, 1], (2) a univariate transform function T which balances the density of the gradient of instances along the period [0, 1] to smooth the objective function of the agent and (3) an multivariate aggregation function G (x) which aggregates information from all features f :EQUATIONEQUATIONwhere c i are real-value coefficients aggregating features and T maps the input G (x) to a value in the closed interval [0, 1]. The definition of T and the explanation of why it smooths the objective function of the agent are given in the latter part of this section. To aggregate different features in a unified scale [0, 1], we use cumulative distribution function cdf of the features:EQUATIONwhere f o i denotes the original numerical value of features, e.g., cross entropy loss in classification with range [0, ∞).The G (x) in Equ. 6 takes a linear aggregation form and projects the whole feature space to a single dimension, which distinguishes the importance of different instances to the greatest extent. H is a continuous piecewise linear function because it can fit a wide range of continuous function when the number of segments is large enough.With such definition, F is decided by parameters in functions H, T , and G, much smaller than the number of the training data. τ defined by Equ. 4, Equ. 5, Equ. 6, and Equ. 7 satisfies Lipschitz condition when cdf and transform T are continuous in close definition domains.Search Space For the univariate piecewise linear function H, we define e 0 , e 1 , ..., e S as the positions of endpoints, and define v 0 , v 2 , ..., v S as the values on the endpoints. e 0 and e S are fixed as 0 and 1. Then the feasible domain (search space) of the sampler τ is:EQUATIONEQUATION)In our implementation, S is 4 and N is 2, resulting in totally only 10 hyper-parameters.Smooth the Objective Function When the curvature of objective function P (D v ; w * (τ )) is too large, i.e., with a sharp landscape, agents like BO would need more trials to find the maxima Brochu et al. (2010) . Such sharpness problem exists in our scenario. This sharpness problem is caused by the imbalance between the gradients of training instances lying in different segments of the piecewise linear function H. For the function H whose input is t(x) = T (G(x)) and one of its segment [e i-1 , e i ], if the instances x with t(x) lying the interval [e i-1 , e i ] have much larger gradients than the other intervals, then a little variation of the value v i or v i-1 will cause a large difference on the overall gradient from the whole training data, which results in large difference in the objective function P . For example, the worst trained 10% instances of ImageNet contains over 90% gradients norm of the whole set. It means the curvature of objective function P around τ * is likely to be sharp and the value of P is only distinguishable in a little sub-domain of τ . If the maxima τ * has such sharpness problem, the efficiency of the agent would be reduced as we show in Appendix A.1. To smooth the objective function, we define the cumulative gradient function cgf :EQUATIONwhere grad (x i ) is the gradient norm of x i . It can be easily proved that any two intervals in the domain of H with equal lengths contain the same amount of total gradient norm, as cgf (u) itself is the cumulative gradient norm. This design brings the search on sampler optimization problem with high efficiency. We conduct experiment in Appendix A.1 to verify the necessity of objective function smoothing via comparing T = cgf and T = cdf .In our formulation, τ (x) is a static function, which restricts that the features do not change in the training process as well. This means a pre-trained model should be used to produce features for D t rather than the model currently being trained. The reason is that a model being trained often forgets and re-memorizes part of D t Toneva et al. (2018) , resulting in jitters and noises for the features like loss or entropy. Thus we empirically find that fixed features are more effective than varying features for learning samplers.For simplicity, we define z = [e 0 , ..., e S , v 0 , ..., v S , c 1 , ..., c N ], and replace P (D v ; w * (τ )) with P (z) in the latter discussion because τ and z are one-to-one mapped. In the outer loop, to find maxima z * of P with as few trials as possible, we explore the advantage of Bayesian Optimization Brochu et al. (2010) . Bayesian Optimization is an approach to optimize objective functions that take a long time (e.g., hours) to evaluate. It is best-suited for optimization over continuous domains with the dimension less than 30, and tolerates stochastic noise in function evaluations, which are exactly characteristics of our problem. Recent works proved the potential of Bayesian Optimization (BO) on hyper-parameter tuning of DNN Klein et al. (2016) ; Snoek et al. (2012) . Given the black-box performance function P : Z → R, BO aims to find an input z * = arg max z∈Z P (z) that globally maximizes P (z).BO requires a prior p (P ) over the performance function, and an acquisition function a p : Z → R quantifying the utility of an evaluation at any z, depending on the prior p. With these ingredients, the following three steps are iterated Brochu et al. (2010) : (1) find the most promising z t+1 ∈ arg max a p (z) by numerical optimization; (2) evaluate the expensive and often noisy function Q t+1 ∼ P (z t+1 )+N 0, σ 2 and add the resulting data point (z t+1 , Q t+1 ) to the set of observations O t = (z j , Q j ), j = 1, ..., t;(3) update the prior p (P |O t+1 ) and acquisition function a p(P |Ot+1) with new observation set O t+1 . Typically, evaluations of the acquisition function a are cheap compared to evaluations of P such that the optimization effort is negligible.Gaussian Processes Gaussian processes (GP) are a prominent choice for p (P ), thanks to their descriptive power and analytic tractability Brochu et al. (2010) . Formally, a GP is a collection of random variables, such that every finite subset of them follows a multivariate normal distribution.To detail the distribution, a GP is identified by a mean function m (often set to m (z) = 0), and a positive definite covariance function (kernel) k (often set to RBF kernel Brochu et al. (2010) for e = 1 : E f do BO.Update(τ ,P (τ ))11:CandidateSamplers = CandidateSamplers ∪ {τ } 12: end for 13: Outputs: CandidateSamplers.TopThe critical problem of the inner loop is how to get w * (τ ) at an acceptable computational cost. Training from scratch is too computationally expensive to be used on a large data set. Hence, we design a method to obtain an approximation of w * (τ ) at a limited cost.First, we focus on the main properties of w * (τ ): (1) minimizing the expected loss on the train set with sampler τ , (2) undergoing a complete training process. We assume that weight vectors w with the two properties are accurate enough to approximate w * (τ ) so that we can use this approximated w * (τ ) to learn τ . Our method initializes the parameters w for different samplers τ with the same parameters w share which is learned from a complete training process to meet property (2). After this special initialization, we fine-tune w share with the given sampler. For property (1), we refer to recent works on the memorization of DNN Toneva et al. (2018) ; Kirkpatrick et al. (2017) , which shows DNN tends to fit the data currently used for training and forget historical training instances. As an analogy, our experiments show that the parameters fine-tuned from w share under τ converge to the same loss level as w * (τ ) trained from scratch and with enough iterations.The shared starting point w share is the weight trained from scratch with uniform sampling. Experiments in Appendix A.1 demonstrate the effectiveness of our approximation method.The complete process of SS is in Algorithm 1. We use BO to explore E o samplers. In each step, BO agent samples the candidate sampler τ . We use the candidate sampler τ to produce probabilities with the pre-trained feature f for sampling training data. The sampled training data are then used for fine-tuning the network weight. After the fine-tuning for E f epochs, BO agent is updated with the candidate sampler τ and its performance on fine-tuned parameters, then it produces more accurate estimations of the better sampler to facilitate the next sampling. 
experiment: 
 In this section, we choose three classification and face recognition tasks as benchmarks to illustrate the effectiveness of SS: CIFAR10, CIFAR100, ImageNet, and MS1M, including data sets with both small and large sizes. Due to space limitations, we have included the ablation experiments in Appendix A.1.In all experiments, the optimization step E o is fixed as 40, and the fine-tune epochs E f are set to 5. We set the number of segments S as 4 in all cases. We tuned these hyper-parameters by separately increasing them until negligible improvements are obtained on ImageNet. The features and shared start points w share are from the pre-trained models with the same architectures as the target models.We norm the sum τ to 1 when use it to sample. For the choice of features, we consider the following two features: (1) Loss: Training loss is frequently utilized in curriculum learning, hard example mining and self-paced method. It is a practicable discriptor and is usually viewed as signals of hardness or noise. For our classification benchmarks, we use the Cross Entropy (CE) loss.(2) Renormed Entropy: The entropy of predicted probabilities is also widely used in current methods. To decouple it from CE loss, we delete the probability of target label from the probability vector, and renorm the rest ones in the vector to 1. Then we use the resulted renormed vector to calculate the entropy:EQUATION)where p j denotes the predicted probability of the j-th class. A small E r results from a peak in the distribution over the rest vector, which often implies a misclassification, while a large E r implies a hard instance.4.1 CIFAR EXPERIMENT CIFAR10 and CIFAR100 are two well-known classification tasks. We explore the performance of our method with the target model ResNet18 on both the original data sets and the data sets with four ratios of noisy labels, 10%, 20%, 30%, and 40%. When generating noisy labels, we random sample instances in each class with the given ratios and uniformly change their labels into the rest incorrect classes. We set batch size as 128 and the L2 regularization as 1e-3. The training process lasts 80 epochs, and the learning rate is initialized as 0.1 and decays by time 0.1 at the 40-th and 80-th epoch.We adopt mini-batch SGD with Nesterov and set the momentum as 0.9.To fully explore the effectiveness of our method, we compare them with both heuristic and learningbased denoising methods, including: (1) REED Reed et al. (2014) , is a method developed for denoising, which changes the training target into a convex combination of the model prediction and the label. (2) MN, MENTORNET Jiang et al. (2017) , utilizes an RNN-based model trained on meta data. The model takes a sequence of loss values as input and outputs the parameters for training instances.(3) LR (learning to reweight), is a meta-learning algorithm that learns to reweight training examples based on the inner products between their gradient directions and the gradient direction of the meta data set. We do not compare with the previous sampler search method Fan et al. (2017) because it focuses on speeding up the training but does not improve the final accuracy.The effectiveness of SS on CIFAR is listed in Tab. 1. It can be observed that SS ranks the top on the two data sets with different noise rates, showing that our method's ability in learning the optimal sampling patterns. Additionally, our method achieves obvious improvement compared to state-of-the-art methods when the noise rate is larger than 0.2. Visualization To demonstrate how SS distinguishes corrupted data on the noisy cases, we visualize both the resulted sampler and the noisy instances on the 40% noise case on CIFAR10. We plot the distribution of noisy instances over the chosen 2-D feature space (Loss, E r ) in Fig. 1 (a), and compare it with the sampling probability under the resulted sampler in the same space in Fig. 1(b) . They show that noisy instances mainly locate in the area with the largest loss and the lowest E r , while the sampler resulted from SS discards instances in the noisy area. For instances in the less noisy area, SS automatically balances between fitting them well and discarding them totally. We train the models with SGD with Nesterov, at an initial learning rate 0.1 and a momentum 0.9 with mini-batch size 2048. The learning rate decays 0.1 at the 30-th, 60-th and 90-th epochs, for a total of 100 epochs. We adopt random crop, horizontal flip, and color jitter, which are common augmentations widely use for training ImageNet Goyal et al. (2017) .The results are listed in Tab. 2. SS consistently improves the performances of different architectures by 1.1% ∼ 1.5% on top-1 accuracy. The total cost of the sampler searching is much less than the RL-based sampler search method Fan et al. (2017) , which needs hundreds of runs of complete training.Transferability A question deserving exploration is the transferability of the resulted sampler of SS. If the sampler searched on a relative small architecture can generalize to different architectures,A.1 ABLATION STUDY In this section, we verify the effectiveness of our designs of the optimization in the outer, and the inner loops verification can be found in appendices.A critical question is, to what extent the performance of the approximated local minima is consistent with the real minima of training from scratch. One of the appropriate metrics is the rank correlation between them. A high-rank correlation implies that sampler with well-approximated minima also performs well when training from scratch. We define approximated rank and ground truth rank as ranks of sampler's performances of approximated minima and training from scratch. With the definitions, we listed the following two metrics: (1) SR. The Spearman's Rank correlation coefficient Myers et al. (2013) between approximated and ground truth ranks. We calculate it to measure the correlation. SR ranges from -1 to 1, i.e., completely negative and positive correlation. (2) TR. The ground truth rank of the top-1 sampler in approximated rank. We use it to show whether SS is good at finding the top samplers.We run 5 times of SS with different random seeds on CIFAR10 with 40% noise and use the last 10 samplers in the search to evaluate the correlation. We also randomly generate 5 pairs of ranks as the baseline of SR. The averages, maxima and minima of the two metrics are listed in Tab. 4. Although SR is not obviously distinguished from a random baseline, TR is consistent near 1 in all cases, demonstrating the reliability of SS in ranking top samplers.Average On ImageNet ILSVRC12, SS(cdf ) outperforms RL as its estimation of the whole landscape of OF. SS(cgf ) optimize faster than SS(cdf ) as it smooths the OF. 
 
 Appendix:  


Decision: The paper proposes a novel data sampling approach for supervised training on a fully labeled dataset, to improve training convergence and model performance. It utilizes low-dimensional representations of sampling strategies and Bayesian Optimization for the same. \
The paper received five reviews. The reviewers carefully engaged with the paper and participated in subsequent discussions. The reviews found the paper to be readable and the quality of writing to be good. They found the paper to be well motivated, addressing an important problem, of high relevance to the AI/ ML community. The approach was found to be compelling and novel, reasonably combining ideas from multiple areas to construct the solution. The results demonstrated consistent performance improvement across a range of datasets, tasks, and model architectures, including large-scale problems, suggesting generalizability and broad applicability. \
The initial reviews were mixed, with concerns ranging from (a) lack of theoretical analysis, (b) generalizability across domain (images), tasks (classification), datasets (very small or very large, severe class imbalance), dependence on optimizers and architecture families, (c) rigor of evaluation – proper validation, ablation studies for components, sensitivity analysis for hyperparameters etc., and (d) computational overhead. \
The authors have addressed these concerns in detail, to the satisfaction of reviewers. This has resulted in an upward revision of scores by several reviewers. An updated version of the paper incorporating the authors’ responses to reviewer suggestions and queries (in the main paper/ supplementary) will result in a very strong submission to the conference.

Reviews: review: paper_summary: paper_summary: The main problem tackled by the paper is to obtain the optimal dataset sampler for training a deep neural network given a fixed dataset and a model. The search space of the sampler is defined as sampling probability functions over the items in the training dataset. The method involves a two-level optimization algorithm with the original (baseline) optimization task as the inner loop and the sampler hyperparameter search as the outer loop. The authors have demonstrated their methods with image classification tasks, showing nontrivial enhancement of the accuracy of various models involving ConvNets and Transformers.
strength_weakness: strength_weakness: 1. I believe that the design of the optimal sampler is practically meaningful in two different scenarios: (1) foundation model training involving very large scale dataset, typically with billions and even trillions of samples, and (2) training a model with a very limited amount of data to achieve high generalizability. However, the demonstration with C10, C100, and IN1k seems not related to either categories at this moment. For example, I see potential problem in the first category, where each item in the training set is sampled only a few times due to large number of sample in the training set (as in GPT-3 or Stable Diffusion trained for LAION-5B datasets). **How this method can be beneficial in more practical scenarios?**
2. The outer loop hyperparameters the authors are trying to optimize seems to extract the ‘noisiness’ or the ‘credibility’ of the sample in the training set. However, the formulation of this might be different in different domain of tasks, e.g., graph-based data (QM9, for example) or highly discrete domain of text corpus (Wikitext, for example). **Is the method generalizable beyond image data?** Unless other domains are tested, I should assume that the proposed method is image-specific, which limits the contribution of the work.
3. **How does the wall clock time of the training elongates by applying the proposed method?** Although the method seems to boost the validation accuracy by a nontrivial amount, it may not be so practical if the training time is added significantly. The authors are recommended to attach the relative training time variation between each sampler methods used in Table 1.
4. The search for the optimal sampler involves exploitation of the validation dataset as a probe for generalizability achieved by the outer loop. To my understanding, this means that we have trade-off between the ratio of the number of samples in the training set dedicated for pure validation affects the performance of the sampler. However, there is no mention about **how the validation set is sampled from the training set, and its relative size**. I am assuming that the validation sets used for reporting the accuracy in Tables 1, 2, and 3 are not the validation sets used in the outer loop (of course if the test set is used for training, it is not fair at all). The authors are encouraged to **separate the notations of the outer loop’s validation set and the “validation” set used only for the testing**.

As a summary, my key concern lies in the unresolved generalizability beyond image domain (or generalizability to very large dataset such as LAION-5B), the strategy for sampling a validation set out of the training set for maximal performance, and the lack of explanation of increased computation in the wall clock time of the training. Regarding the issues, I will temporarily give a WA.
questions: questions: 1. A gentle suggestion of using \citep instead of \cite in references. This will add parenthesis to the citations and increase the readability. For example, in line 139 and 165.

Please note that these questions are not counted in my overall scoring.
limitations: limitations: There is no section dedicated for limitation, and the authors have high confidence in their work as shown in line 527.
paper_summary: paper_summary: The paper introduces a method for automatically learning optimal data sampling strategies using BO based sampling. The proposed method formulates the problem as a bilevel optimization, using a low-dimensional representation of samplers (10 parameters) and BO to search this space. Key novel points include techniques to smooth the objective function and quickly approximate model performance, making the search process computationally feasible for large datasets like ImageNet.
strength_weakness: strength_weakness: While the empirical results look promising, theoretical analysis of why the proposed low-dimensional representation of samplers works well is desired. E.g., a discussion on the theoretical bounds or guarantees of this approach would strengthen the paper and provide insights into when and why the SS algorithm might fail.

The paper fixes several hyperparameters (e.g., number of segments S=4, optimization steps E_o=40) without much discussion. An analysis of the method's sensitivity to these choices would be valuable for practitioners implementing this approach.
questions: questions: * How does the performance of Swift Sampler change when applied to tasks with more significant class imbalance or long-tailed distributions? Does it maintain its effectiveness in such scenarios?

* The paper demonstrates good transferability between model architectures. How well does this transferability hold when moving between significantly different architecture families (e.g., from CNNs to Transformers)?

* Given that the method uses a pre-trained model to generate features for sampling, how sensitive is the performance to the quality of this initial model?
limitations: limitations: The paper uses a fixed set of features (loss and renormalized entropy) for experiments. It would be helpful to explore how the choice of features impacts the performance of the method or whether different tasks might benefit from different feature sets. This leaves open questions about the flexibility and adaptability of the approach.

Lack of discussion on potential computational overhead introduced by the Swift Sampler method. The additional cost of feature computation, Bayesian optimization, and fine-tuning steps could be significant, especially for large datasets.
paper_summary: paper_summary: The purpose of this paper is to create a sampler that can assign appropriate sampling probabilities to training data in order to improve performance. Unlike previous approaches that relied on heuristic rules or expensive learning methods, this paper proposes an automatic and efficient sampler search algorithm called SS. Specifically, SS employs a new formulation to map a sampler to a lower-dimensional space of hyper-parameters and uses an approximated local minimum to quickly evaluate the quality of a sampler. SS can be applied on large-scale data sets with high efficiency and leads to performance gains on various datasets, e.g., CIFAR10, CIFAR100, ImageNet-1k, and YTF.
strength_weakness: strength_weakness: (1) The outer loop searches for the sampler that has the best score on the validation set. Are the final results in experiments also reported on this validation set? The author should clarify this potential misleading. 

(2) The performance gains on Swin are less than those of RN and SRN. Suppose this is related to different optimizers (SGD v.s. AdamW) or building blocks (Conv v.s. Transformer). The author may show some discussions and experiments.
questions: questions: See weakness.
limitations: limitations: N/A
paper_summary: paper_summary: The paper focused on designing a learnable training data sampler to improve the model performance. A method named Swift Sampler (SS) is proposed, which is formulated as a function mapping data feature to sampling probabilities, represented by a small number of parameters. In addition, the SS Smooths the objective function landscape to improve optimization efficiency and uses an approximation method to efficiently evaluate candidate samplers without full retraining. The experimental results for image classification tasks on CIFAR-10, CIFAR-100, ImageNet, and face datasets show improved performance.
strength_weakness: strength_weakness: 1.	Can the author explain the effectiveness of components separately?
2.	The paper does not provide enough theoretical analysis or justification for its proposed formulation, transform function, and approximation method. Can the author provide more profound justification?
3.	The paper does not conduct ablation studies or sensitivity analysis to show the impact of different components or hyper-parameters of its method.
questions: questions: 1.	Can the author explain the effectiveness of components separately?
2.	The paper does not provide enough theoretical analysis or justification for its proposed formulation, transform function, and approximation method. Can the author provide more profound justification?
3.	The paper does not conduct ablation studies or sensitivity analysis to show the impact of different components or hyper-parameters of its method.
limitations: limitations: The experiments are only conducted for image classification tasks, while its effectiveness on other vision tasks is not clear. Since this paper focuses not only on image classification, but other tasks should also be discussed as well.
paper_summary: paper_summary: The paper introduces Swift Sampler (SS), an efficient algorithm for the automatic learning of data samplers in deep learning model training. SS addresses the challenges of high-dimensionality, sharpness, and costly evaluation in sample-based methods by mapping samplers to a low-dimensional space of hyper-parameters and employing a novel transform function to smooth the objective function. Utilizing Bayesian Optimization, SS quickly examines the quality of samplers through an approximation method that significantly reduces computational expense. Comprehensive experiments on tasks like image classification and face recognition across various datasets, including ImageNet and CIFAR, demonstrate SS's effectiveness in improving model performance, with notable improvements such as a 1.5% increase on ImageNet. The samplers learned by SS also exhibit good transferability across different neural network architectures, showcasing the algorithm's generality and computational efficiency, making it a valuable contribution to the field of deep learning.
strength_weakness: strength_weakness: 1. No ablation studies or sensitivity analyses were performed to show the effects of different components or hyperparameters of the method.
2. The verification of the method focuses on the image classification task, and it would be more comprehensive to see the experimental results of more complex tasks.
3. I think the work in this paper is more similar to the research related to **active learning**, and I hope to be able to see the difference between this work and **active learning** in related work.
questions: questions: See the Weaknesses section.
limitations: limitations: It is recommended to use more complex tasks to verify the validity of the method, such as detection segmentation in vision, VQA in language, etc

Comments: comment: Thank you for your feedback and for considering our responses. We're glad the additional data addressed your concerns, and we appreciate the increased rating. Please let us know if you have any further questions.
Thanks for the author's responses to my questions and the additional numerical results in the pdf. My main concerns are addressed, and happy to increase the rating from 4 to 6.
We are very grateful for your recognition. We will integrate these results into the next version of our paper based on your valuable feedback.
Thank you for acknowledging our detailed rebuttal. We are glad we could address your concerns. Your decision to raise the rating is greatly appreciated and serves as significant encouragement for us. We will also integrate the additional content from our rebuttal into the subsequent version of our work to further improve its quality.
I appreciate the detailed rebuttal provided by the authors, which addresses most of my concerns. I will raise my rating from 5 to 6.
We are very grateful for your recognition. We will incorporate the experimental results and the difference between this work and active learning into the main paper, following your valuable suggestions.
Thanks for the clarification, it solved my problem. I will increase the rating from 4 to 5.
After reviewing the authors' rebuttal, I am pleased to see that they have thoroughly addressed all of my questions and concerns. The additional experiments provided demonstrate that Swift Sampler is a versatile data sampling method applicable to various types of data, including images and text. I particularly appreciate that the authors showed its effectiveness across both large-scale and small-scale datasets. Moreover, the added theoretical analysis further enhances the credibility of this research.

Given the sufficient novelty of the proposed method and the improvements made in response to the review, I am raising my score to 7.
Thank you for your feedback and for raising your score. We appreciate your recognition of our method's potential beyond image tasks. We will incorporate these additional experiments carefully in the final version to demonstrate the broader applicability of our method.
Thank you for the rebuttal and for the effort to demonstrate with additional experiments that help me improve my understanding of the work. I will raise my score as most of my concerns are resolved. Since the authors have decided to claim for the general application of their sampling method beyond images, I believe additional experiments for natural language tasks in their final copy of the work will be really helpful for improving the completeness of this work.