Data Valuation using Reinforcement Learning

1 INTRODUCTION . Data is an essential ingredient in machine learning . Machine learning models are well-known to improve when trained on large-scale and high-quality datasets ( Hestness et al. , 2017 ; Najafabadi et al. , 2015 ) . However , collecting such large-scale and high-quality datasets is costly and challenging . One needs to determine the samples that are most useful for the target task and then label them correctly . Recent work ( Toneva et al. , 2019 ) suggests that not all samples are equally useful for training , particularly in the case of deep neural networks . In some cases , similar or even higher test performance can be obtained by removing a significant portion of training data , i.e . low-quality or noisy data may be harmful ( Ferdowsi et al. , 2013 ; Frenay & Verleysen , 2014 ) . There are also some scenarios where train-test mismatch can not be avoided because the training dataset only exists for a different domain . Different methods ( Ngiam et al. , 2018 ; Zhu et al. , 2019 ) have demonstrated the importance of carefully selecting the most relevant samples to minimize this mismatch . Accurately quantifying the value of data has a great potential for improving model performance for real-world training datasets which commonly contain incorrect labels , and where the input samples differ in relatedness , sample quality , and usefulness for the target task . Instead of treating all data samples equally , lower priority can be assigned for a datum to obtain a higher performance model – for example in the following scenarios : 1 . Incorrect label ( e.g . human labeling errors ) . 2 . Input comes from a different distribution ( e.g . different location or time ) . 3 . Input is noisy or low quality ( e.g . noisy capturing hardware ) . 4 . Usefulness for target task ( label is very common in the training dataset but not as common in the testing dataset ) . In addition to improving performance in such scenarios , data valuation also enables many new use cases . It can suggest better practices for data collection , e.g . what kinds of additional data would the model benefit the most from . For organizations that sell data , it determines the correct value-based pricing of data subsets . Finally , it enables new possibilities for constructing very large-scale training datasets in a much cheaper way , e.g . by searching the Internet using the labels and filtering away less valuable data . How does one evaluate the value of a single datum ? This is a crucial and challenging question . It is straightforward to address at the full dataset granularity : one could naively train a model on the entire dataset and use its prediction performance as the value . However , evaluating the value of each datum is far more difficult , especially for complex models such as deep neural networks that are trained on large-scale datasets . In this paper , we propose a meta learning-based data valuation method which we name Data Valuation using Reinforcement Learning ( DVRL ) . Our method integrates data valuation with the training of the target task predictor model . DVRL determines a reward by quantifying the performance on a small validation set , and uses it as a reinforcement signal to learn the likelihood of each datum being using in training of the predictor model . In a wide range of use cases , including domain adaptation , corrupted sample discovery and robust learning , we demonstrate significant improvements compared to permutation-based strategies ( such as Leave-one-out and Influence Function ( Koh & Liang , 2017 ) ) and game theory-based strategies ( such as Data Shapley ( Ghorbani & Zou , 2019 ) ) . The main contributions can be summarized as follows : 1 . We propose a novel meta learning framework for data valuation that is jointly optimized with the target task predictor model . 2 . We demonstrate multiple use cases of the proposed data valuation framework and show DVRL significantly outperforms competing methods on many image , tabular and language datasets . 3 . Unlike previous methods , DVRL is scalable to large datasets and complex models , and its computational complexity is not directly dependent on the size of the training set . 2 RELATED WORK . Data valuation : A commonly-used method for data valuation is leave-one-out ( LOO ) . It quantifies the performance difference when a specific sample is removed and assigns it as that sample ’ s data value . The computational cost is a major concern for LOO – it scales linearly with the number of training samples which means its cost becomes prohibitively high for large-scale datasets and complex models . In addition , there are fundamental limitations in the approximation . For example , if there are two exactly equivalent samples , LOO underestimates the value of that sample even though that sample may be very important . The method of Influence Function ( Koh & Liang , 2017 ) was proposed to approximate LOO in a computationally-efficient manner . It uses the gradient of the loss function with small perturbations to estimate the data value . In order to compute the gradient , Hessian values are needed ; however these are prohibitively expensive to compute for neural networks . Approximations for Hessian computations are possible , although they generally result in performance limitations . From data quality assessment perspective , the method of Influence Function inherits the major limitations of LOO . Data Shapley ( Ghorbani & Zou , 2019 ) is another relevant work . Shapley values are motivated by game theory ( Shapley , 1953 ) and are commonly used in feature attribution problems such as relating predictions to input features ( Lundberg & Lee , 2017 ) . For Data Shapley , the prediction performance of all possible subsets is considered and the marginal performance improvement is used as the data value . The computational complexity for computing the exact Shapley value is exponential with the number of samples . Therefore , Monte Carlo sampling and gradient-based estimation are used to approximate them . However , even with these approximations , the computational complexity still remains high ( indeed much higher than LOO ) due to re-training for each test combination . In addition , the approximations may result in fundamental limitations in data valuation performance – e.g . with Monte Carlo approximation , the ratio of tested combinations compared to all possible combinations decreases exponentially . Moreover , in all the aforementioned methods data valuation is decoupled from predictor model training , which limits the performance due to lack of joint optimization . Meta learning-based adaptive learning : There are relevant studies that utilize meta learning for adaptive weight assignment while training for various use cases such as robust learning , domain adaptation , and corrupted sample discovery . ChoiceNet ( Choi et al. , 2018 ) explicitly models output distributions and uses the correlations of the output to improve robustness . Xue et al . ( 2019 ) estimates uncertainty of predictions to identify the corrupted samples . Li et al . ( 2019 ) combines meta learning with standard stochastic gradient update with generated synthetic noise for robust learning . Shen & Sanghavi ( 2019 ) alternates the processes of selecting the samples with low loss and model training to improve robustness . Shu et al . ( 2019 ) uses neural networks to model the relationship between current loss and the corresponding sample weights , and utilizes a meta-learning framework for robust weight assignment . Köhler et al . ( 2019 ) estimates the uncertainty to discover the noisy labeled data and relabels mislabeled samples to improve the prediction performance of the predictor model . Gold Loss Correction ( Hendrycks et al. , 2018 ) uses a clean validation set to recover the label corruption matrix to re-train the predictor model with corrected labels . Learning to Reweight ( Ren et al. , 2018 ) proposes a single gradient descent step guided with validation set performance to reweight the training batch . Domain Adaptive Transfer Learning ( Ngiam et al. , 2018 ) introduces importance weights ( based on the prior label distribution match ) to scale the training samples for transfer learning . MentorNet ( Jiang et al. , 2018 ) proposes a curriculum learning framework that learns the order of mini-batch for training of the corresponding predictor model . Our method , DVRL , differs from the aforementioned as it directly models the value of the data using learnable neural networks ( which we refer to as a data value estimator ) . To train the data value estimator , we use reinforcement learning with a sampling process . DVRL is model-agnostic and even applicable to non-differentiable target objectives . Learning is jointly performed for the data value estimator and the corresponding predictor model , yielding superior results in all of the use cases we consider . 3 PROPOSED METHOD . Framework : Let us denote the training dataset asD = { ( xi , yi ) } Ni=1 ∼ P where xi ∈ X is a feature vector in the d-dimensional feature space X , e.g . Rd , and yi ∈ Y is a corresponding label in the label space Y , e.g . ∆ [ 0 , 1 ] c for classification where c is the number of classes and ∆ is the simplex . We consider a disjoint testing dataset Dt = { ( xtj , ytj ) } Mj=1 ∼ Pt where the target distribution Pt does not need to be the same with the training distribution P . We assume an availability of a ( often small1 ) validation dataset Dv = { ( xvk , yvk ) } Lk=1 ∼ Pt that comes from the target distribution Pt . The DVRL method ( overview in Fig . 1 ) consists of two learnable functions : ( 1 ) the target task predictor model fθ , ( 2 ) data value estimator model hφ . The predictor model fθ : X → Y is trained to minimize a certain weighted loss function Lf ( e.g . Mean Squared Error ( MSE ) for regression or cross entropy for classification ) on training set D : fθ = arg min f̂∈F 1 N N∑ i=1 hφ ( xi , yi ) · Lf ( f̂ ( xi ) , yi ) . ( 1 ) fθ can be any trainable function with parameters θ , such as a neural network . The data value estimator model hφ : X · Y → [ 0 , 1 ] , on the other hand , is optimized to output weights that determine 1We provide empirical results that how small the validation set can be in Section 4.5. the distribution of selection likelihood of the samples to train the predictor model fθ . We formulate the corresponding optimization problem as : min hφ E ( xv , yv ) ∼P t [ Lh ( fθ ( xv ) , yv ) ] s.t . fθ = arg minf̂∈F E ( x , y ) ∼P [ hφ ( x , y ) · Lf ( f̂ ( x ) , y ) ] ( 2 ) where hφ ( x , y ) represents value of the training sample ( x , y ) . The data value estimator is also a trainable function , such as a neural network . Similar to Lf , we use MSE or cross entropy for Lh . Training : To encourage exploration based on uncertainty , we model training sample selection stochastically . Let w = hφ ( x , y ) denote the probability that ( x , y ) is used to train the predictor model fθ . hφ ( D ) = { hφ ( xi , yi ) } Ni=1 is the probability distribution for inclusion of each training sample . s ∈ { 0 , 1 } N is a binary vector that represents the selected samples . If si = 1/0 , ( xi , yi ) is selected/not selected for training the predictor model . πφ ( D , s ) = ∏N i=1 [ hφ ( xi , yi ) si · ( 1 − hφ ( xi , yi ) ) 1−si ] is the probability that certain selection vector s is selected based on hφ ( D ) . We assign the outputs of the data value estimator model , w = hφ ( x , y ) , as the data values . We can use the data values to rank the dataset samples ( e.g . to determine a subset of the training dataset ) and to do sample-adaptive training ( e.g . for domain adaptation ) . The predictor model can be trained using standard stochastic gradient descent because it is differentiable with respect to the input . However , gradient descent-based optimization can not be used for the data value estimator because the sampling process is non-differentiable . There are multiple ways to handle the non-differentiable optimization bottleneck , such as Gumbel-softmax ( Jang et al. , 2017 ) or stochastic back-propagation ( Rezende et al. , 2014 ) . In this paper , we consider reinforcement learning instead , which directly encourages exploration of the policy towards the optimal solution of Eq . ( 2 ) . We use the REINFORCE algorithm ( Williams , 1992 ) to optimize the policy gradients , with the rewards obtained from a small validation set that approximates performance on the target task . For the loss function l̂ ( φ ) : l̂ ( φ ) = E ( xv , yv ) ∼P t [ Es∼πφ ( D , · ) [ Lh ( fθ ( xv ) , yv ) ] ] = ∫ P t ( xv ) [ ∑ s∈ [ 0,1 ] N πφ ( D , s ) · [ Lh ( fθ ( xv ) , yv ) ] ] dxv , we directly compute the gradient∇φ l̂ ( φ ) as : ∇φ l̂ ( φ ) = ∫ P t ( xv ) [ ∑ s∈ [ 0,1 ] N ∇φπφ ( D , s ) · [ Lh ( fθ ( xv ) , yv ) ] ] dxv = ∫ P t ( xv ) [ ∑ s∈ [ 0,1 ] N ∇φ log ( πφ ( D , s ) ) · πφ ( D , s ) · [ Lh ( fθ ( xv ) , yv ) ] ] dxv = E ( xv , yv ) ∼P t [ Es∼πφ ( D , · ) [ Lh ( fθ ( xv ) , yv ) ] ∇φ log ( πφ ( D , s ) ) ] , where∇φ log ( πφ ( D , s ) ) is ∇φ log ( πφ ( D , s ) ) = ∇φ N∑ i=1 log [ hφ ( xi , yi ) si · ( 1− hφ ( xi , yi ) ) 1−si ] = N∑ i=1 si∇φ log [ hφ ( xi , yi ) ] + ( 1− si ) ∇φ log [ ( 1− hφ ( xi , yi ) ) ] . To improve the stability of the training , we use the moving average of the previous loss ( δ ) , with a window size ( T ) , as the baseline for the current loss . The pseudo-code is shown in Algorithm 1 . Computational complexity : DVRL models the mapping between an input and its value with a learnable function . The training time of DVRL is not directly proportional to the dataset size , but rather dominated by the required number of iterations and per-iteration complexity in Algorithm 1 . One way to minimize the computational overhead is to use pre-trained models to initialize the Algorithm 1 Pseudo-code of DVRL training 1 : Inputs : Learning rates α , β > 0 , mini-batch size Bp , Bs > 0 , inner iteration count NI > 0 , moving average window T > 0 , training dataset D , validation dataset Dv = { ( xvk , yvk ) } Lk=1 2 : Initialize parameters θ , φ , moving average δ = 0 3 : while until convergence do 4 : Sample a mini-batch from the entire training dataset : DB = ( xj , yj ) Bsj=1 ∼ D 5 : for j = 1 , ... , Bs do 6 : Calculate selection probabilities : wj = hφ ( xj , yj ) 7 : Sample selection vector : sj ∼ Ber ( wj ) 8 : for t = 1 , ... , NI do 9 : Sample a mini-batch ( x̃m , ỹm , s̃m ) Bp m=1 ∼ ( xj , yj , sj ) Bs j=1 10 : Update the predictor model network parameters θ θ ← θ − α 1 Bp Bp∑ m=1 s̃m · ∇θLf ( fθ ( x̃m ) , ỹm ) ) 11 : Update the data value estimator model network parameters φ φ← φ− β [ 1 L L∑ k=1 [ Lh ( fθ ( xvk ) , yvk ) ] − δ ] ∇φ log πφ ( DB , ( s1 , ... , sBs ) ) 12 : Update the moving average baseline ( δ ) : δ ← T−1T δ + 1 LT ∑L k=1 [ Lh ( fθ ( xvk ) , yvk ) ] predictor networks at each iteration . Unlike alternative methods like Data Shapley , we demonstrate the scalability of DVRL to large-scale datasets such as CIFAR-100 , and complex models such as ResNet-32 ( He et al. , 2016 ) and WideResNet-28-10 ( Zagoruyko & Komodakis , 2016 ) . Instead of being exponential in terms of the dataset size , the training time overhead DVRL is only twice of conventional training . Please see Appendix D for further analysis on learning dynamics of DVRL and Appendix B for additional computational complexity discussions .

Summary: This paper proposes a meta learning approach based on data valuation for reinforcement learning tasks. The core idea is to train a second network (the data value estimator) in conjunction to a regular predictor network. The predictor is then trained with samples chosen via the data value estimation. The authors motivate this construction with the goal to filter out unreliable and corrupted data.