Patent Description:
Training deep neural nets usually requires large-scale labeled data. However, acquiring clean labels for large-scale datasets is very challenging and expensive to achieve in practice, especially in data domains where the labelling cost is high, such as healthcare. Deep neural nets also have high capacity of memorization. Although many training techniques attempt to regularize neural nets and prevent noisy label invasion, when noisy labels become prominent, a neural net inevitably fits into noisy labeled data.

Typically, a small trusted training dataset is usually feasible to acquire. A practically realistic setting is to increase the size of the training data in a cheap and untrusted way (e.g., crowd-sourcing, web search, cheap labeling practices, etc.), based on the given small trusted set. If this setting can demonstrate clear benefits, it could significantly change machine learning practices. However, to increase the size of the training data, many methods still need a substantial amount of trusted data to make the neural nets generalize well. A naive usage of small trusted dataset can thus cause rapid overfilling and eventually leads to negative effects.

[<NUM>] "Deep Learning from Noisy Image Labels with Quality Embedding", arxiv. org, XP080833724 discloses a quality embedding model, which explicitly introduces a quality variable to represent the trustworthiness of noisy labels. The key idea is to identify the mismatch between the latent and noisy labels by embedding the quality variables into different subspaces, which effectively minimizes the noise effect. At the same time, the high-quality labels is still able to be applied for training. An additive layer aggregates the prior predictions and noisy labels as the posterior to train the classifier.

One aspect of the disclosure provides a method for robust training of a model in the presence of label noise.

The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

Training modern deep neural networks to be highly accurate generally requires vast quantities of labeled training data. However, the process of obtaining high quality labeled training data (e.g., via human annotation) is often both challenging and expensive. Because training data with noisy (i. e, inaccurate labels) is often much cheaper to acquire, methods for training neural networks from datasets with noisy labels (e.g., loosely-controlled procedures, crowd-sourcing, web search, text extraction, etc) is an active area of research. However, because many deep neural networks have high capacity for memorization, noisy labels may become prominent and cause overfitting.

Conventional techniques primarily consider a setting where the entire training dataset is acquired using the same labeling technique. However, it is often advantageous to supplement the primary training set with a smaller dataset that contains highly trusted and clean labels. The smaller dataset may help the model demonstrate high robustness even when the primary training set is extremely noisy.

Implementations herein are directed toward a model trainer provides robust neural network training with noisy labels The model trainer implements three primary strategies: isolation, escalation, and guidance (IEG). The model trainer first isolates noisy and cleanly labeled training data by reweighing training samples to prevent mislabeled data from misleading the neural network training. The model trainer next escalates supervision from mislabeled data via pseudo labels to take advantage of information within the mislabeled data. Finally, the model trainer guides the training using a small trusted training dataset with strong regularization to prevent overfitting.

Thus, the model trainer implements meta learning based re-weighting and re-labelling objectives to simultaneously learn to weight the per-datum importance and progressively escalate supervised losses of training data using pseudo labels as replacements to given labels. The model trainer uses a label estimation objective to serve as an initialization of the meta re-labeling and to escalate supervision from mislabeled data. An unsupervised regularization objective enhances label estimation and improves overall representation learning.

Referring to <FIG>, in some implementations, an example system <NUM> includes a processing system <NUM>. The processing system <NUM> may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having fixed or scalable / elastic computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM> (e.g., memory hardware). The processing system <NUM> executes a model trainer <NUM>. The model trainer <NUM> trains a target model <NUM> (e.g., a deep neural network (DNN)) to make predictions based on input data For example, the model trainer <NUM> trains a convolutional neural network (CNN). The model trainer <NUM> trains the target model <NUM> on a set of labeled training samples <NUM>, <NUM>. A labeled training sample includes both training data and a label for the training data. The label includes annotations or other indications of the correct result for the target model <NUM>. In contrast, unlabeled training samples only include the training data without the corresponding label.

For example, labeled data for a model that is trained to transcribe audio data includes the audio data as well as a corresponding transcription of the audio data. Unlabeled data for the same target model <NUM> would include the audio data without the transcription. With labeled data, the target model <NUM> may make a prediction based on a training sample and then compare the prediction to the label serving as a ground-truth to determine how accurate the prediction was Thus, each labeled training sample <NUM> includes both training data <NUM> and an associated given label <NUM>.

The labeled training samples <NUM> may be representative of whatever data the target model <NUM> requires to make its predictions. For example, the training data <NUM> may include frames of image data (e.g., for object detection, classification, etc.), frames of audio data (e.g., for transcription, speech recognition, etc.), and/or text (e.g., for natural language classification, etc.). Each training sample <NUM> of the set of training samples <NUM>, in some implementations, are images and the given labels <NUM> are text descriptors of the images. The labeled training samples <NUM> may be stored on the processing system <NUM> (e.g., within memory hardware <NUM>) or received, via a network or other communication channel, from another entity The model trainer <NUM> may select labeled training samples <NUM> from the set of training samples <NUM> in batches (i e. , a different batch for each iteration of the training).

The model trainer <NUM> includes a pseudo label generator <NUM>. During each training iteration of a plurality of training iterations, and for each training sample <NUM> in the set of labeled training samples <NUM>, the pseudo label generator <NUM> generates a pseudo label 116P for the corresponding labeled training sample <NUM>. The pseudo label 116P represents a relabeling of the training sample <NUM><NUM> with a pseudo label 116P generated by the pseudo label generator <NUM>.

Referring now to <FIG>, in some implementations, the pseudo label generator <NUM> includes a sample augmenter <NUM> and a sample average calculator <NUM>. The sample augmenter <NUM>, when the pseudo label generator <NUM> generates the pseudo label 116P for the training sample <NUM>, generates a plurality of augmented training samples 112A, 112Aa-n based on the labeled training sample <NUM>. The sample augmenter <NUM> generates the augmented training samples 112A by introducing different changes to the input training sample <NUM> for each augmented training sample 112A. For example, the sample augmenter <NUM> increases or decreases values by a predetermined or random amount to generate an augmented training sample 112A from the labeled training sample <NUM>. As another example, when the labeled training sample <NUM> includes a frame of image data, the sample augmenter <NUM> may rotate the image, flip the image, crop the image, etc. The sample augmenter <NUM> may use any other conventional means of augmenting or perturbing the data as well.

In order to add labels to the augmented training samples 112A, the pseudo label generator <NUM>, in some examples, uses the target model <NUM> (i.e., a machine learning model) to generate a predicted label <NUM>, 220a-n for each of the augmented training samples 112A. The sample average calculator <NUM> may average each predicted label <NUM> generated by the target model <NUM> for each of the augmented training samples 112A to generate the pseudo label 116P for the input label training sample <NUM>. That is, in some implementations, the pseudo label generator <NUM>, for a given labeled training sample <NUM>, generates a plurality of augmented training samples 112A, generates a predicted label <NUM> for each of the augmented training samples 112A, and averages the predicted labels <NUM> for each generated augmented training sample 112A to generate the pseudo label 116P for the corresponding labeled training sample <NUM>.

Referring back to <FIG>, the model trainer <NUM> also includes a weight estimator <NUM>. The weight estimator <NUM>, for each training sample <NUM> in the set of training samples <NUM> during each training iteration estimates a weight <NUM> of the training sample <NUM>. The weight <NUM> of the training sample <NUM> indicates an accuracy of the given label <NUM> of the labeled training sample <NUM>. For example, a higher weight indicates a greater probability of an accurate given label <NUM>. Thus, the weight estimator <NUM> determines a likelihood that a labeled training sample <NUM> is mislabeled.

In some examples, the weight estimator <NUM> determines the weight <NUM> based on predictions made by the target model <NUM> from labeled training samples <NUM> and trusted training samples 112T from a set of trusted training samples 112T. The model trainer <NUM> assumes trusted labels 116T of the trusted samples 112T are of high-quality and/or are clean. That is, the trusted labels 116T are accurate The model trainer <NUM> may treat the weight <NUM> as a learnable parameter by determining an optimal weight <NUM> for each labeled training samples <NUM> such that the trained target model <NUM> obtains the best performance on the set of trusted training samples 112T.

Because it may be computationally expensive to determine the weight <NUM> (as each update step requires training the target model <NUM> until convergence), optionally, the weight estimator <NUM> estimates the weight <NUM> by determining an online approximation of an optimal weight <NUM> of the labeled training sample <NUM>. The online approximation may include using stochastic gradient descent optimization. In some implementations, the optimal weight <NUM> minimizes a training loss of the target model <NUM>. That is, the optimal weight <NUM> is a weight that results in the lowest training loss of the target model <NUM>. The model trainer <NUM> may optimize the weight <NUM> based on back-propagation with second-order derivatives.

A sample partitioner <NUM> receives each training sample <NUM> and the associated weight <NUM> and the associated pseudo label 116P. The sample partitioner <NUM> includes a weight threshold <NUM>. For each labeled training sample <NUM>, the sample partitioner <NUM> determines whether the weight <NUM> of labeled training sample <NUM> satisfies the weight threshold <NUM> For example, the sample partitioner <NUM> determines whether the weight <NUM> exceeds the weight threshold <NUM>.

When the weight <NUM> of the labeled training sample <NUM> satisfies the weight threshold <NUM>, the sample partitioner <NUM> adds the training sample <NUM> to a set of cleanly labeled training samples 112C. The cleanly labeled training samples 112C include the training data <NUM> and clean labels 116C (i.e., given labels <NUM> determined clean by the sample partitioner <NUM>). When the weight <NUM> of the labeled training sample <NUM> fails to satisfy the weight threshold <NUM>, the sample partitioner <NUM> adds the labeled training sample <NUM> to a set of mislabeled training samples <NUM>. Thus, the likely mislabeled training samples <NUM> are isolated from the likely cleanly labeled training samples <NUM> to escalate supervision from mislabeled data.

When noise ratio is high (i.e., many of the labeled training samples <NUM> are noisy), the meta optimization-based reweighing and relabeling by the model trainer effectively prevents misleading optimization (i.e., most labeled training samples <NUM> will have zero or close to zero weights <NUM>). However, the mislabeled training samples <NUM> may still provide valuable training data. Thus, to avoid potentially discarding a significant amount of data, the mislabeled training samples <NUM> include the training data <NUM> and, instead of the given label <NUM>, the associated pseudo label 116P. That is, for mislabeled training samples <NUM>, the pseudo label 116P is substituted for the given label <NUM>.

In some examples, the model trainer <NUM> trains the target model <NUM> with the set of cleanly labeled training samples 112C using corresponding given labels <NUM> and the set of mislabeled training samples <NUM> using corresponding pseudo labels 116P. The target model <NUM> may be incrementally trained using any number of training iterations that repeat some or all of the steps described above.

Referring now to <FIG>, in some implementations, the model trainer <NUM> includes a convex combination generator <NUM>. The convex combination generator <NUM> obtains the set of trusted training samples 112T that includes training data <NUM> and associated trusted labels 116T. The convex combination generator <NUM> generates convex combinations <NUM> for training the target model <NUM>. In some examples, the convex combination generator <NUM> applies a pairwise MixUp to the set of trusted training samples 112T and the set of labeled training samples <NUM> The MixUp regularization allows the model trainer <NUM> to leverage the trusted information from the trusted training samples 112T without fear of overfilling. The MixUp regularization constructs extra supervision losses using the training samples <NUM>, 112T in the form of convex combinations and a MixUp factor.

In some examples, the model trainer <NUM> includes a loss calculator <NUM>. The loss calculator <NUM> determines a first loss <NUM>, 322a based on the cleanly labeled set of training samples 112C using corresponding given labels <NUM>. The loss calculator <NUM> may determine a second loss 322b based on the mislabeled set of training samples <NUM> using the corresponding pseudo labels 116P. The loss calculator <NUM> may determine a third loss 322c based on the convex combinations 310a of the set of trusted training samples 112T and a fourth loss 322d based on the convex combinations 310b of the set of labeled training samples <NUM> In some implementations, the loss calculator <NUM> determines a fifth loss 322e based on a Kullback-Leibler (KL) divergence between the given labels <NUM> of the set of labeled training samples <NUM> and the pseudo labels 116P of the set of labeled training samples <NUM>. The KL-divergence loss 322e sharpens the generation of pseudo labels 116P by reducing controversy of the augmented training samples 112A. This is because ideal pseudo labels 116P should be as close to accurate labels as possible. When the predictions for the augmented training samples 112A are controversial to each other (e. g, small changes in the training data <NUM> lead to large changes in the prediction), the contribution from the pseudo label 116P does not encourage the target model <NUM> to be discriminative. Thus, the KL-divergence loss 322e helps enforce consistency of the pseudo labels 116P.

The loss calculator <NUM> may determine a total loss <NUM> based on one or more of the first loss 322a, the second loss 322b, the third loss 322c, the fourth loss 322d, and the fifth loss 322e. In some examples, one or more of the losses 322a-e (i.e., the third loss 322c and the fourth loss 322d) are softmax cross-entropy losses. Based on the total loss <NUM>, the loss calculator <NUM> updates model parameters <NUM> of the target model <NUM>. The loss calculator may apply a one-step stochastic gradient based on the total loss <NUM> to determine the updated model parameters <NUM>.

Referring now to <FIG>, in some implementations, the model trainer <NUM> implements an algorithm <NUM> to train the target model <NUM>. Here, the model trainer accepts as input the labeled training samples <NUM> (i.e., Du) and the trusted training samples 112T (i.e., Dp). The model trainer <NUM>, for each training iteration (i.e., time step t), updates the model parameters <NUM> of the target model <NUM> Using the algorithm <NUM>, the model trainer <NUM> trains the target model <NUM> by generating the augmented training samples 112A at step <NUM> and estimating or generating the pseudo labels 116P at step <NUM>. At step <NUM>, the model trainer <NUM> determines the optimal weight <NUM> and/or updates the weight estimator <NUM> (ie. At step <NUM>, the model trainer <NUM> splits the set of labeled training samples <NUM> into the set of cleanly labeled training samples 112C and the set of mislabeled training samples <NUM>. At step <NUM>, the model trainer computes the MixUp convex combinations <NUM>. At step <NUM>, the model trainer <NUM> determines the total loss <NUM> and at step <NUM>, conducts a one-step stochastic gradient to obtain updated model parameters <NUM> for the next training iteration In some examples, the model trainer <NUM> determines an exact momentum update using a momentum value during the one-step stochastic gradient optimization.

<FIG> is a flowchart of an exemplary arrangement of operations for a method <NUM> for robust training in the presence of label noise. The method <NUM>, at operation <NUM>, includes obtaining, at data processing hardware <NUM>, a set of labeled training samples <NUM>. Each labeled training sample <NUM> is associated with a given label <NUM>. At operation <NUM>, during each of a plurality of training iterations, the method <NUM> includes, for each labeled training sample <NUM> in the set of labeled training samples <NUM>, generating, by the data processing hardware <NUM>, a pseudo label 116P for the labeled training sample <NUM>. At operation <NUM>, the method <NUM> includes, estimating, by the data processing hardware <NUM>, a weight <NUM> of the labeled training sample <NUM> indicative of an accuracy of the given label <NUM>.

The method <NUM> includes, at operation <NUM>, determining, by the data processing hardware <NUM>, whether the weight <NUM> of the labeled training sample <NUM> satisfies a weight threshold <NUM> When the weight <NUM> of the labeled training sample <NUM> satisfies the weight threshold <NUM>, the method <NUM> includes, at operation <NUM>, adding, by the data processing hardware <NUM>, the labeled training sample <NUM> to a set of cleanly labeled training samples 112C. At operation <NUM>, the method <NUM> includes, when the weight <NUM> of the labeled training sample <NUM> fails to satisfy the weight threshold <NUM>, adding, by the data processing hardware <NUM>, the labeled training sample <NUM> to a set of mislabeled training samples <NUM>. At operation <NUM>, the method <NUM> includes training, by the data processing hardware <NUM>, a machine learning model <NUM> with the set of cleanly labeled training samples 112C using corresponding given labels <NUM> and the set of mislabeled training samples <NUM> using corresponding pseudo labels 116P.

Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e g. , as a server bank, a group of blade servers, or a multi-processor system).

Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) % electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs) Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

The high speed controller <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low speed controller <NUM> manages lower bandwidth-intensive operations Such allocation of duties is exemplary only In some implementations, the high-speed controller <NUM> is coupled to the memory <NUM>, the display <NUM> (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports <NUM>, which may accept various expansion cards (not shown).

Various implementations of the systems and techniques described herein can be realized in digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit) Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. However, a computer need not have such devices Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user, for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

Claim 1:
A method (<NUM>) for training a machine learning model (<NUM>), the method (<NUM>) comprising:
obtaining, at data processing hardware (<NUM>), a set of labeled training samples (<NUM>), each labeled training sample (<NUM>) associated with a given label (<NUM>); and
during each of a plurality of training iterations:
for each labeled training sample (<NUM>) in the set of labeled training samples (<NUM>):
generating, by the data processing hardware (<NUM>), a pseudo label (116P) for the labeled training sample (<NUM>), by:
generating a plurality of augmented training samples (112A) based on the labeled training sample (<NUM>);
for each augmented training sample (112A), generating, using the machine learning model (<NUM>), a predicted label (<NUM>); and
averaging each predicted label (<NUM>) generated for each augmented training sample (112A) of the plurality of augmented training samples (112A) to generate the pseudo label (116P) for the corresponding labeled training sample (<NUM>);
estimating, by the data processing hardware (<NUM>), a weight (<NUM>) of the labeled training sample (<NUM>) indicative of an accuracy of the given label (<NUM>);
determining, by the data processing hardware (<NUM>), whether the weight (<NUM>) of the labeled training sample (<NUM>) satisfies a weight threshold (<NUM>);
when the weight (<NUM>) of the labeled training sample (<NUM>) satisfies the weight threshold (<NUM>), adding, by the data processing hardware (<NUM>), the labeled training sample (<NUM>) to a set of cleanly labeled training samples (112C); and
when the weight (<NUM>) of the labeled training sample (<NUM>) fails to satisfy the weight threshold (<NUM>), adding, by the data processing hardware (<NUM>), the labeled training sample (<NUM>) to a set of mislabeled training samples (<NUM>); and
training, by the data processing hardware (<NUM>), the machine learning model (<NUM>) with the set of cleanly labeled training samples (112C) using corresponding given labels (<NUM>) and the set of mislabeled training samples (<NUM>) using corresponding pseudo labels (116P).