Patent Description:
Machine learning models receive an input and generate an output, e. g, a predicted output, based on the received input. Machine learning models are trained on data. However, quantifying the value of data is a fundamental problem in machine learning. Machine learning models are generally improved when trained on large-scale and high-quality datasets. However, collecting such large-scale and high-quality datasets can be costly and challenging. Moreover, there is the additional complexity of determining the samples in a large-scale dataset that are most useful for training and labeling accordingly. Real-world training datasets commonly contain incorrect labels, or the input samples differ in relatedness, sample quality, or usefulness for the target task.

Accurately quantifying the value of data improves model performance for training datasets. Instead of treating all data samples equally, lower priority may be assigned for a datum to obtain a higher-performance model when the value of the datum is low. Typically, quantifying data valuation performance requires individually removing samples to calculate performance loss and then assigning the loss as that sample's data. However, these methods scale linearly with the number of training samples, making it cost prohibitive for large-scale datasets and complex models. Besides building insights about the problem, data valuation has diverse use-cases, such as in domain adaptation, corrupted sample discovery, and robust learning.

"Deep Learning from Noisy Image Labels with Quality Embedding", arxiv. org, XP080833724 discloses a quality embedding model, which explicitly introduces a quality variable to represent the trustworthiness of noisy labels. The key idea is to identify the mismatch between the latent and noisy labels by embedding the quality variables into different subspaces, which effectively minimizes the noise effect. At the same time, the high-quality labels is still able to be applied for training. An additive layer aggregates the prior predictions and noisy labels as the posterior to train the classifier.

Training deep neural networks to be highly accurate in predictions generally requires large quantities of training data. However, collecting large-scale and high-quality real world datasets is costly and challenging. Additionally, accurately training neural networks can take a significant amount of time and computational overhead. Accurately quantifying the value of training data has the significant potential of improving model performance for real-word training datasets which often contain incorrect labels or differ in quality and usefulness. Rather than treating all data samples in a training dataset equally, lower priority can be assigned to samples with lower quality to obtain a higher performance model In addition to improving performance, data valuation may help develop better practices for data collection. However, historically data valuation has been limited by computational costs, as the methods scale linearly with the number of training samples in the dataset.

Implementations herein are directed toward data valuation using reinforcement learning (DVRL), which is a meta learning framework to adaptively learn data values jointly with the training of a predictor model. A data value estimator function, modeled by a deep neural network, outputs a likelihood a training sample will be used in training of the predictor model. Training of the data value estimator is based on a reinforcement signal using a reward directly obtained from performance on a target task With a small validation set, DVRL can provide computationally efficient and high quality ranking of data values for training datasets that save both time and outperform other methods. The DVRL can be used in various applications across multiple types of datasets.

Referring to <FIG>, in some implementations, an example system <NUM> includes a processing system <NUM>. The processing system <NUM> may be a single computer, multiple computers, or a distributed system (e g. , a cloud environment) having fixed or scalable/elastic computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM> (e.g., memory hardware). The processing system <NUM> executes a meta learning framework <NUM> (also referred to herein as a DVLR framework or just DVLR). The DVLR framework <NUM> obtains a set of training samples <NUM>. Each training sample includes training data and a label for the training data. The label includes annotations or other indications of a correct result for a prediction based on the training data. In contrast, unlabeled training samples only include the training data without the corresponding label.

For example, the training samples <NUM> may include tabular datasets, audio datasets (e.g., for transcription or speech recognition, etc.), image datasets (e g. , for object detection or classification, etc.), and/or text datasets (e.g., for natural language classification, text translation, etc.). The set of training samples <NUM> may be stored in the processing system <NUM> (e.g., within memory hardware <NUM>) or received, via a network or other communication channel, from another entity The data value estimator <NUM> may select training samples <NUM> from the set of training samples <NUM> in batches (i.e., a select or random portion of the set of training samples <NUM>). In some examples, the data value estimator <NUM> samples a batch of training samples <NUM> (i.e., a different batch for each iteration of the training).

The DVLR framework <NUM> includes a data value estimator model <NUM> (e.g., a machine learning model). In some implementations, the data value estimator model <NUM> is a neural network. The data value estimator model <NUM>, for each training sample <NUM> in the batch of training samples <NUM>, determines a selection probability <NUM> based on estimator parameter values <NUM> of the data value estimator model <NUM>. The selection probability <NUM> represents a prediction of how valuable each training sample <NUM> in the batch of the training samples <NUM> will be to the predictor model <NUM>. In some examples, the data value estimator model <NUM> determines the values of the input training samples <NUM> by quantifying the relevancy of the input training samples <NUM> to a predictor model <NUM>.

The DVLR framework <NUM> includes a sampler <NUM>. The sampler <NUM> receives, as an input, the selection probabilities <NUM> determined by the data value estimator model <NUM> for each training sample <NUM> in the batch. The sampler <NUM> selects, based on the selection probabilities <NUM> of each training sample <NUM>, a subset of training samples <NUM> to provide to the predictor model <NUM>. As discussed in more detail below, the sampler <NUM> may discard, based on the selection probabilities <NUM>, the remaining training samples <NUM> in the batch of training samples <NUM>. In some implementations, the selection probabilities <NUM> provided as input to the sampler <NUM> are based on a multinomial distribution.

The predictor model <NUM> (e. g , a machine learning model) receives the subset of training samples <NUM> sampled by the sampler <NUM>. The predictor model <NUM> determines performance measurements <NUM> based on the subset of training samples <NUM> sampled from the batch of input training samples <NUM> selected for the current training iteration. The predictor model <NUM> is trained only with the subset of training samples <NUM> sampled by the sampler <NUM> That is, in some implementations, the predictor model <NUM> is not trained on the training samples <NUM> that are not selected or sampled by the sampler <NUM>.

The predictor model <NUM> includes model parameter values <NUM> that control the prediction capabilities of the predictor model <NUM>. The predictor model <NUM> makes predictions <NUM> based on the input training samples <NUM>. A performance evaluator <NUM> receives the predictions <NUM> and determines, based on the predictions <NUM> and the training sample <NUM> (i.e., the label associated with the training sample <NUM>) performance measurements <NUM> (e.g., an accuracy of the prediction <NUM>). In some implementations, the performance measurements <NUM> includes loss data (e g. , cross-entropy loss data). In these implementations, the DVLR framework <NUM> determines a reinforcement signal based on the loss data. Optionally, the DVLR framework <NUM> may generate a reward value <NUM> (<FIG>) based on the performance measurements <NUM>.

The DVLR framework <NUM> adjusts and/or updates the model parameter values <NUM> of the predictor model <NUM> and the estimator parameter values <NUM> of the data value estimator model <NUM> based on the performance measurements <NUM> During each training iteration of a plurality of training iterations, the DVLR <NUM> may use a feedback loop <NUM> (e.g., back-propagation) to adjust the model parameter values <NUM> of the predictor model <NUM> based on the performance measurements <NUM> of the training iteration. The DVLR <NUM> may adjust, using the performance measurements <NUM> of the training iteration, the estimator parameter values <NUM> of the data value estimator model <NUM> based on the same or a different feedback loop <NUM>. In some implementations, the DVLR framework <NUM> updates the estimator parameter values <NUM> of the data value estimator model <NUM> by updating layer parameter values of a neural network of the data value estimator <NUM>.

Referring now to <FIG>, a schematic view <NUM> includes the DVLR <NUM> with a reinforcement signal <NUM> and the feedback loop <NUM>. The performance measurements <NUM> may include loss data. The DVRL framework <NUM> may determine the loss data <NUM> using a loss function based on the subset of training samples <NUM> input to the predictor model <NUM>. In some examples, the DVRL framework <NUM> trains the predictor model <NUM> using a stochastic gradient descent optimization algorithm with a loss function (e.g., mean squared error (MSE) for regression or cross entropy for classification). When the performance evaluator <NUM> determines the loss data <NUM> based on the loss function, the DVLR <NUM> updates the model parameter values <NUM> of the predictor model <NUM> with the performance measurements <NUM> (e.g., loss data <NUM>) using the feedback loop <NUM>.

After the DVRL framework <NUM> determines the loss data <NUM> for the training iteration, the DVLR <NUM> may generate a reinforcement signal <NUM>. In some implementations, the DVRL framework <NUM> updates the estimator parameter values <NUM> of the data value estimator model <NUM> based on the reinforcement signal <NUM>. The reinforcement signal <NUM> may also include reward data <NUM>. The performance evaluator <NUM> may determine the reward data <NUM> by quantifying the performance measurements <NUM>. For example, when the performance measurements <NUM> indicate low loss data <NUM> (i.e., minimal error or an accurate prediction) from the subset of training samples <NUM> received by the predictor model <NUM>, the reward data <NUM> may reinforce the estimator parameters values <NUM> of the data value estimator model <NUM>. Conversely, when the performance measurements <NUM> indicate high loss data <NUM> (i.e., high error) from the subset of training samples <NUM> received by the predictor model <NUM>, the reward data <NUM> may indicate that the estimator parameter values <NUM> of the data value estimator model <NUM> need further updating.

In some implementations, the performance evaluator <NUM> calculates reward data <NUM> based on historical loss data. For example, the performance evaluator <NUM> determines, using a moving average calculator <NUM>, a moving average of loss data based on N-most recent training iterations of the predictor model <NUM>. In other words, for each training iteration, the moving average calculator <NUM> may obtain the loss data <NUM> and determine the difference between the current training iteration loss data <NUM> and the average of the N-most recent training iterations of loss data. The DVLR <NUM> may generate a reward value <NUM> based on the moving average of loss data determined by the moving average calculator <NUM>. The reward value <NUM> may be based on the difference between the current training iteration loss data <NUM> and the average of the N-most recent training iterations of loss data. In some implementations, the DVRL framework <NUM> adds the reward value <NUM> to the reward data <NUM> of the reinforcement signal <NUM>. In other implementations, the DVRL framework <NUM> merely uses the reward value <NUM> to influence the reward data <NUM> by increasing or decreasing the reward data <NUM> of the reinforcement signal <NUM>.

Referring now to <FIG>, a schematic view <NUM> includes the DVLR <NUM> selecting the subset of training samples <NUM>. In some implementations, the DVLR <NUM> selects the training samples <NUM> in the batch of training samples <NUM> for the subset of training samples <NUM> by determining a selection value <NUM> for each training sample <NUM>. The selection value <NUM> may indicate selection or no selection for the corresponding training sample <NUM>. The sampler <NUM>, after the data value estimator model <NUM> generates the selection probabilities <NUM> for each of the training samples <NUM> in the batch of training samples <NUM>, determines the corresponding selection value <NUM> indicating either selection <NUM> or no selection <NUM> Optionally, the selection probabilities <NUM> generated by the data value estimator model <NUM> conform to a multinomial distribution. The sampler <NUM> obtains the distribution of selection probabilities <NUM> and corresponding training samples <NUM> of the batch of training samples <NUM> and determines the selection values <NUM> by determining a likelihood that each training sample <NUM> in the batch of training samples <NUM> will train the predictor model <NUM>.

When the sampler <NUM> determines that the selection value <NUM> of the training sample <NUM> indicates selection <NUM>, the sampler <NUM> adds the training sample <NUM> to the subset of training samples <NUM>. Conversely, when the sampler <NUM> determines that the selection value of the training sample <NUM> indicates no selection <NUM>, the sampler <NUM> may discard the training sample <NUM> (e.g., to discarded training samples <NUM>). In some implementations, the DVLR framework <NUM> returns the discarded training samples <NUM> back to the set of training samples <NUM> for future training iterations. In other implementations, the DVRL framework <NUM> isolates the discarded training samples <NUM> (i.e., removed from the set of training samples <NUM>) to prevent inclusion in future training iterations.

Referring now to <FIG>, in some implementations, the DVLR <NUM> implements an algorithm <NUM> to train the data value estimator <NUM> and the predictor model <NUM>. Here, the DVLR <NUM> accepts the set of training samples <NUM> (i.e., D), and initializes the estimator parameter values of the data value estimator model <NUM>, the model parameter values of the predictor model <NUM>, and resets the moving average loss in the moving average loss calculator <NUM>. The DVLR <NUM>, for each training iteration, until convergence, samples a batch of training samples <NUM> (i e. , mini-batch B) from the set of training samples <NUM> and updates the estimator parameter values <NUM> of the data value estimator model <NUM> and the model parameter values <NUM> of the predictor model <NUM> Using the algorithm <NUM>, for each training sample <NUM> (i.e.,j) in the batch of training samples <NUM>, the data value estimator model <NUM> calculates selection probabilities <NUM> and samples, using the sampler <NUM> selection values <NUM>. The DVLR <NUM>, for each training iteration (i.e., t), samples the batch of training samples <NUM>, with respective selection probabilities <NUM> and selection values <NUM> indicating selection <NUM> and determines the performance measurements <NUM> (i.e., loss data). At the next step, the DVLR <NUM> updates the model parameter values <NUM> of the predictor model <NUM> based on the performance measurements <NUM> for the training iteration. The DVLR <NUM> next updates the estimator parameter values <NUM> of the data value estimator model <NUM> based on the performance measurements <NUM> for the training iteration including the moving average loss from the moving average loss calculator <NUM>. At the final step, the DVLR updates the moving average loss in the moving average loss calculator <NUM>.

<FIG> is a flowchart of an exemplary arrangement of operations for a method <NUM> for data valuation using reinforcement learning. The method <NUM>, at operation <NUM>, includes obtaining, at data processing hardware <NUM>, a set of training samples <NUM>. At operation <NUM>, during each of a plurality of training iterations, the method <NUM> includes, for each training sample <NUM> in a batch of training samples <NUM>, determining, by the data processing hardware <NUM>, using a data value estimator <NUM>, a selection probability <NUM> for the training sample <NUM> based on estimator parameter values of the data value estimator <NUM>.

The method <NUM> includes, at operation <NUM>, selecting, by the data processing hardware <NUM>, based on the selection probabilities <NUM> of each training sample <NUM>, a subset of training samples <NUM> from the batch of training samples <NUM>. At operation <NUM>, the method <NUM> includes determining, by the data processing hardware <NUM>, using a predictor model <NUM> with the subset of training samples <NUM>, performance measurements <NUM>. The method <NUM> also includes, at operation <NUM>, adjusting, by the data processing hardware <NUM>, model parameter values <NUM> of the predictor model <NUM> based on the performance measurements <NUM>. At operation <NUM>, the method includes updating, by the data processing hardware <NUM>, the estimator parameter values <NUM> of the data value estimator <NUM> based on the performance measurements <NUM>.

The non-transitory memory <NUM> may be physical devices used to store programs (e. g, sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device <NUM>.

The high speed controller <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low speed controller <NUM> manages lower bandwidth-intensive operations Such allocation of duties is exemplary only In some implementations, the high-speed controller <NUM> is coupled to the memory <NUM>, the display <NUM> (e g. , through a graphics processor or accelerator), and to the high-speed expansion ports <NUM>, which may accept various expansion cards (not shown). In some implementations, the low-speed controller <NUM> is coupled to the storage device <NUM> and a low-speed expansion port <NUM> The low-speed expansion port <NUM>, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e. g, internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

Other kinds of devices can be used to provide interaction with a user as well, for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

Claim 1:
A method (<NUM>) for valuating training samples (<NUM>), the method (<NUM>) comprising:
obtaining, at data processing hardware (<NUM>), a set of training samples (<NUM>); and
during each of a plurality of training iterations:
sampling, by the data processing hardware (<NUM>), a batch of training samples (<NUM>) from the set of training samples (<NUM>);
for each training sample (<NUM>) in the batch of training samples (<NUM>), determining, by the data processing hardware (<NUM>), using a data value estimator (<NUM>), a selection probability (<NUM>) for the training sample (<NUM>) based on estimator parameter values (<NUM>) of the data value estimator (<NUM>), wherein the selection probability (<NUM>) for each training sample (<NUM>) is determined based on a priority value assigned to each training sample (<NUM>) based on sample quality of each training sample (<NUM>);
selecting, by the data processing hardware (<NUM>), based on the selection probabilities (<NUM>) of each training sample (<NUM>), a subset of training samples (<NUM>) from the batch of training samples (<NUM>);
determining, by the data processing hardware (<NUM>), using a predictor model (<NUM>) with the subset of training samples (<NUM>), performance measurements (<NUM>), wherein determining the performance measurements (<NUM>) using the predictor model (<NUM>) comprises determining loss data (<NUM>) by a loss function;
adjusting, by the data processing hardware (<NUM>), model parameter values (<NUM>) of the predictor model (<NUM>) based on the performance measurements (<NUM>); and
updating, by the data processing hardware (<NUM>), the estimator parameter values (<NUM>) of the data value estimator (<NUM>) based on the performance measurements (<NUM>), wherein updating the estimator parameter values (<NUM>) of the data value estimator (<NUM>) based on the performance measurements (<NUM>) comprises:
determining, from the loss data (<NUM>), a reinforcement signal (<NUM>); and
updating estimator parameter values (<NUM>) of the data value estimator (<NUM>) based on the reinforcement signal (<NUM>).