Patent Description:
Neural networks are machine learning models that employ one or more layers of models to generate an output, e.g., a classification, for a received input. The output of each hidden layer is used as an input to the next layer, i.e., the next hidden layer or the output layer of the network. Each layer of the network generates an output from the received input in accordance with current values of a respective set of parameters.

A neural network may overfit on training data. Overfitting may be described as the neural network becoming overly confident in view of a particular set of training data. When a neural network is overfitted, it may begin to make poor generalizations with respect to items that are not in the training data. Considering the overfitting problem when training a neural network is known from the following documents: <NPL>; <NPL>; <NPL>; <NPL>.

Aspects of the present disclosure have the technical effect of improving the performance of a trained neural network by reducing overfitting.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. The accuracy of inferences generated by trained a neural network can be improved. In particular, by modifying labels associated with training data used to train the neural network, the neural network can be discouraged from overfitting on the training data, i.e., from becoming overly reliant, and overconfident, in making inferences based on information learned by the neural network during training, thereby improving performance of the neural network on new inputs after training.

The details of these, and other, implementations are set forth in the accompanying drawings and the description below.

This specification describes how a system implemented as computer programs on one or more computers in one or more locations can regularize the training of a neural network by generating a set of regularizing training data. The neural network is configured to receive an input data item and to process the input data item to generate a respective score for each label in a predetermined set of multiple labels.

The system can receive a training data set for training the neural network that includes a set of multiple training items. Each training item in the set of multiple training items may be associated with a respective training label distribution that associates a respective score with each label of the set of labels. Then, the neural network may be iteratively trained by optimizing a specified objective function that takes as input a neural network output generated by the neural network for a neural network input and a target output for the neural network input.

<FIG> is a block diagram of an example of a neural network training system <NUM> for training a neural network <NUM>. The neural network training system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The neural network training system <NUM> includes a neural network <NUM> and a database <NUM> of training data items.

The neural network <NUM> is configured to receive an input data item and to process the input data item to generate a respective score for each label in a predetermined set of multiple labels.

The neural network <NUM> can be configured to receive any kind of digital data input and to generate any kind of score or classification output based on the input. In the claimed invention, if the inputs to the neural network <NUM> are images or features that have been extracted from images, the output generated by the neural network <NUM> for a given image are scores for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category.

As another example, if the inputs to the neural network <NUM> are Internet resources (e.g., web pages), documents, or portions of documents or features extracted from Internet resources, documents, or portions of documents, the output generated by the neural network <NUM> for a given Internet resource, document, or portion of a document may be a score for each of a set of topics, with each score representing an estimated likelihood that the Internet resource, document, or document portion is about the topic.

As another example, if the inputs to the neural network <NUM> are features of a personalized recommendation for a user, e.g., features characterizing the context for the recommendation, e.g., features characterizing previous actions taken by the user, the output generated by the neural network <NUM> may be a score for each of a set of content items, with each score representing an estimated likelihood that the user will respond favorably to being recommended the content item. In some of these examples, the neural network <NUM> is part of a reinforcement learning system that provides content recommendations to users.

As another example, if the input to the neural network <NUM> is text in one language, the output generated by the neural network <NUM> may be a score for each of a set of pieces of text in another language, with each score representing an estimated likelihood that the piece of text in the other language is a proper translation of the input text into the other language.

In the claimed invention, if the input to the neural network <NUM> is features of a spoken utterance, the output generated by the neural network <NUM> are a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript for the spoken utterance.

To allow the neural network <NUM> to generate accurate outputs for received data items, the neural network training system <NUM> trains the neural network <NUM> to adjust the values of the parameters of the neural network <NUM>, e.g., to determine trained values of the parameters from initial values.

In training the neural network <NUM>, the neural network training system <NUM> uses training items from the database <NUM> of labeled training items. The database <NUM> stores a set of multiple training items, with each training item in the set of multiple training items being associated with a respective label. Generally, the label for the training item identifies one or more correct labels for the training item, i.e., the label or labels that should be identified as the label or labels of the training item by the scores generated by the neural network <NUM>. In some implementations, the label data for a given training item is a score distribution that includes a respective score for each label in the set of labels, with the scores reflecting the correct label or labels for the training item. For example, a training data item <NUM> may be associated with a training label 122a.

In particular, the neural network training system <NUM> trains the neural network <NUM> to minimize a loss function <NUM>. Generally, the loss function <NUM> is a function that depends on the (i) network output generated by the neural network <NUM> by processing a given training item and (ii) the label for the training item, i.e., the target output that the neural network <NUM> should have generated by processing the training item.

The neural network training system <NUM> can train the neural network <NUM> to minimize the loss function <NUM> by performing multiple iterations of a conventional neural network training technique on training items from the database <NUM>, e.g., stochastic gradient descent with backpropagation, to iteratively adjust the values of the parameters of the neural network <NUM>.

In order to reduce overfitting and to improve the performance of the trained neural network, the neural network training system <NUM> regularizes the training of the neural network <NUM> by either (i) modifying the label data for the training items prior to using the training items to train the neural network <NUM> or (ii) modifying the loss function <NUM> that is used to train the neural network <NUM>. Modifying the label data is described below with reference to <FIG> while modifying the loss function is described below with reference to <FIG>.

<FIG> is a contextual diagram of an example of a system <NUM> that generates a regularizing set of training data. The system includes a neural network training system <NUM> and a database <NUM> of training data items.

At stage A, a set of training data items stored in the database <NUM> of training data items includes a set of n training images 222A-<NUM>, 222A-<NUM>, 222A-<NUM>, 222A-n, where n is any positive integer. Each image in the set of training images is associated with a label 223A-<NUM>, 223A-<NUM>, 223A-<NUM>, 223A-n that describes a classification associated with each respective training image 222A-<NUM>, 222A-<NUM>, 222A-<NUM>, 222A-n. For example, the training image 222A-<NUM> is labeled as a "cat" 223A-<NUM>, the training image 222A-<NUM> is labeled as a "dog" 223A-<NUM>, the training image 222A-<NUM> is labeled as a "snake" 223A-<NUM>, and the training image 222A-n is labeled as a "bear" 223A-n. For ease of description, the labels 223A-<NUM>, 223A-<NUM>, 223A-<NUM>, 223A-n associated with each training image 222A-<NUM>, 222A-<NUM>, 222A-<NUM>, 222A-n are shown and described as English words. However, when implemented, each label may be a score distribution that identifies the English word.

For example, a predetermined set of labels associated with the set of training data images at stage A may include "cat," "dog," "snake," and "bear. " Accordingly, by way of example, a training image such as training image 222A-<NUM> may have a training label distribution of ". <NUM>" for the set of labels "cat," "dog," "snake," and "bear," respectively.

In some implementations, a training label distribution of ". <NUM>" for the set of labels "cat," "dog," "snake," and "bear," respectively may indicate that the training image 222A-<NUM> is labeled as a "cat" because the highest score of the training label distribution corresponds to the category "cat. " Alternatively, a training label distribution may be a one-hot distribution. In such as distribution, the value assigned to the correct label is a positive value such as "<NUM>" and all other labels are assigned a value such as "<NUM>. " Accordingly, a one-hot training label distribution of "<NUM>," "<NUM>," "<NUM>," "<NUM>" for the set of labels "cat," "dog," "snake," and "bear," respectively, may indicate that the image 222A-<NUM> is classified as a cat.

In some implementations, a neural network that is trained using the training data provided at stage A may be prone to overfitting. In such instances, a neural network may begin to process training data accurately, and become overconfident. However, when a neural network is overfitted, it may begin to make poor generalizations with respect to images that are not in the training data. For instance, the neural network may begin to classify images as cats after it has been trained that are not cats even though the neural network accurately classified an entire set of labeled training data items. Aspects of the present disclosure seek to reduce overfitting by using a regularizing training data set to train a neural network. A regularizing training data set may be generated by modifying one or more labels associated with training data items in a training data set used to train a neural network.

Generally, the regularizing training data set may be generated by introducing a predetermined amount of noise into the labels of a training data set. For instance, the neural network training system <NUM> may process <NUM> a training data set at stage A and determine whether or not to modify a label 223A-<NUM>, 223A-<NUM>, 223A-<NUM>, 223A-n associated with a particular training data item such as training image 222A-<NUM>, 222A-<NUM>, 222A-<NUM>, 222A-n. In some implementations, the neural network training system <NUM> may randomly determine whether or not to modify a label 223A-<NUM>, 223A-<NUM>, 223A-<NUM>, 223A-n associated with a particular training data item such as a training image 222A-<NUM>, 222A-<NUM>, 222A-<NUM>, 222A-n. Alternatively, the neural network training system <NUM> may determine to modify a label 223A-<NUM>, 223A-<NUM>, 223A-<NUM>, 223A-n associated with a particular training data item such as a training image 222A-<NUM>, 222A-<NUM>, 222A-<NUM>, 222A-n a predetermined probability of the time. For example, the neural network training system <NUM> may determine to modify a label 223A-<NUM>, 223A-<NUM>, 223A-<NUM>, 223A-n associated with a particular training data item such as a training image 222A-<NUM>, 222A-<NUM>, 222A-<NUM>, 222A-n <NUM> percent of the time, <NUM> percent of the time, <NUM> percent of the time, or the like.

In some implementations, modifying the label 223A-<NUM>, 223A-<NUM>, 223A-<NUM>, 223A-n associated with a particular training data item such as a training image 222A-<NUM>, 222A-<NUM>, 222A-<NUM>, 222A-n may include changing the label 223A-<NUM>, 223A-<NUM>, 223A-<NUM>, 223A-n associated with a particular training data item such as a training image 222A-<NUM>, 222A-<NUM>, 222A-<NUM>, 222A-n from a correct label to an incorrect label. Modifying the label may include, for example, changing the distribution of scores in a training data item's training label distribution from a distribution of scores representing a correct label to a distribution of scores representing an incorrect label. Alternatively, modifying the label may include, for example, changing the distribution of scores in a training data item's training label distribution to reduce the label's identification with a particular class. For example, changing the distribution of scores in a training data item's training label distribution may include, for example, reducing the highest score in a training label distribution by a predetermined amount. Similarly, changing the distribution of scores in a training data item's training label distribution may include, for example, increasing one or more of the lowest scores in the training label distribution by a predetermined amount. The resulting training data set may be referred to as a regularizing data set.

With reference to the example of <FIG>, the neural network training system <NUM> may obtain the set of training images shown at stage A. The neural network training system <NUM> may process <NUM> the set of training images shown at stage A, and determine whether or not to modify the label 223A-<NUM>, 223A-<NUM>, 223A-n associated with each respective training image 222A-<NUM>, 222A-<NUM>, 222A-<NUM>, 222A-n. In the example of <FIG>, the neural network training system <NUM> may randomly determine to modify the label associated with the training image 222A-<NUM> from the label "dog" to the label "snake" 223B-<NUM>. Though the training image 222A-<NUM> is correctly labeled as a "dog," the neural network training system <NUM> modifies the label 223A-<NUM> so that the image 222A-<NUM>, 222B-<NUM> is now incorrectly labeled as a "snake" 223B-<NUM> at stage B. This modification results in a regularizing set of training images shown at stage B.

A neural network may then be trained using the regularizing training images shown at stage B. Training a neural network using the regularizing training images shown at stage B helps the neural network to become less reliant on the training data and produce better generalizations in view of processed training data.

In the claimed invention, a label modification process referred to as label smoothing is employed. Assume, for example, a distribution over labels u(k), independent of the training example x, and a smoothing parameter ε. For a training example with correct label y, the neural network training system <NUM> can replace the label distribution q(k|x) = δk,y, where δk,y is the Dirac delta which equals <NUM> for k = y and <NUM> when k is not equal to y, with: <MAT>.

Thus, the new label distribution q' is a mixture of the original ground-truth distribution q(k|x) and fixed distribution u(k), with weights <NUM>- ε and ε, respectively. This can be seen as the distribution of the label k obtained by first setting k to the ground-truth label k = y, and then using neural network training system <NUM> to, with probability ε, replace k with a sample drawn from the fixed distribution u(k).

In some implementations, a uniform distribution may be used by assigning u(k) = <NUM>/K, where K is the number of labels, to achieve label smoothing regularization, so that <MAT>.

The example of <FIG> provided an example of a single label of a single training data item that was randomly modified. However, the present disclosure need not be so limited. In some implementations, multiple labels may be modified. Moreover, in some implementations, some, or all, of the labels may be modified via smoothing using the label smoothing process described above to create a regularizing set of training data.

<FIG> is a flowchart of a process <NUM> for generating a regularizing set of training data. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network training system, e.g., the neural network training system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

At stage <NUM>, the system obtains a set of training data from a database of training data items. The set of training data may include multiple training data items. The training data items may include an item of content or a set of one or more features that were extracted from the item of content. Each training data item in the set of training data items is associated with a respective label from a predetermined set of multiple labels. The label may include a training label distribution that includes a score for the training image for each label in a predetermined set of labels associated with a set of training images.

At stage <NUM>, the system determines whether to modify the training data to generate regularizing training data that regularizes the training of the neural network. For each training data item in the set of training data items, the system determines whether or not to modify the label associated with the training item. Determining whether or not to modify the label associated with the training item may include, for example, randomly determining to modify a label associated with a particular training data item. Alternatively, the system may determine, with a predetermined probability, to modify a label associated with a particular training data item. For example, the system may determine to modify a label associated with a particular training data item such as a training image <NUM> percent of the time, <NUM> percent of the time, <NUM> percent of the time, or the like.

At stage <NUM>, when the system determines to modify the label associated with the training item, the system modifies the label associated with the training data item by changing the label associated with the training item to a different label that is selected from the predetermined set of labels. In some implementations, modifying the label associated with a particular training data item may include changing the label associated with a particular training data item from a correct label to an incorrect label. Changing the label may include, for example, changing the distribution of scores in a training data item's training label distribution from a distribution of scores representing a correct label to a distribution of scores representing an incorrect label. Alternatively, modifying the label may include, for example, changing the distribution of scores in a training data item's training label distribution to reduce the label's identification with a particular class. For example, changing the distribution of scores in a training data item's training label distribution may include, for example, reducing the highest score in a training label distribution by a predetermined amount. Similarly, for example, changing the distribution of scores in a training data item's training label distribution may include, for example, increasing one or more of the lowest scores in the training label distribution by a predetermined amount. The resulting training data set may be referred to as a regularizing training data set.

At stage <NUM>, the system trains a neural network using the regularizing training data set. Such training may be performed as described with reference to <FIG>. However, instead of using the database <NUM> of training data items described with reference to <FIG>, the training of stage <NUM> would train a neural network using the regularizing training data set generated using the process of stages <NUM>, <NUM>, and <NUM> respectively.

<FIG> is a flowchart of a process <NUM> for generating a regularizing set of training data according to the claimed invention.

For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network training system, e.g., the neural network training system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

At stage <NUM>, the system obtains a set of training data from a database of training data items. Each training data item in the set of training data items is associated with an initial target label distribution.

At stage <NUM>, the system obtains data identifying a smoothing label distribution. The smoothing label distribution may include a smoothing score for each label in the smoothing label distribution. In some implementations, the smoothing label distribution may be a uniform distribution that assigns the same smoothing score to each label of the smoothing label distribution as described above. Alternatively, in other implementations, the smoothing label distribution may be a non-uniform distribution that includes one or more smoothing scores that are capable of being different from one or more other smoothing scores in the same smoothing label distribution.

At stage <NUM>, the system modifies the training data to generate regularizing training data that can regularize the training of the neural network. Modifying the training data includes, for each training data item in the set of training data items obtained at stage <NUM>, combining the initial training distribution with the smoothing label distribution to generate a modified target label distribution. Combining the initial training distribution with the smoothing label distribution may include, for example, calculating a weighted sum of the initial target label distribution and the smoothing label distribution. The resulting training data set may be referred to as a regularizing training data set.

At stage <NUM>, the system trains a neural network using the regularizing training data set that includes training data with labels that have been modified using the smoothing label distribution as described at stage <NUM>. Such training may be performed as described with reference to <FIG>. However, instead of using the database <NUM> of training data items described with reference to <FIG>, the training of stage <NUM> would train a neural network using the regularizing training data set generated using the process of stages <NUM>, <NUM>, and <NUM> respectively.

<FIG> is a flowchart of a process <NUM> for minimizing a loss function having a regularizing error term. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a system, e.g., the neural network training system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

At stage <NUM>, the system receives a request to train a neural network to optimize a loss function that includes a first error term. In one example, the first error term may be a cross-entropy loss. The cross-entropy loss may be defined as: <MAT> where p(k) is the probability of a label k from a set of K labels as reflected by the label scores generated by the neural network for a particular training item and q(k) is the ground-truth value of each label k from the set of K labels.

As opposed to modifying label data associated with one or more training data labels, the system may alternatively seek to achieve label-smoothing regularization by modifying the cross-entropy loss function as follows: <MAT>.

Accordingly, the label-smoothing regularization described above is equivalent to replacing a single cross-entropy loss H(q,p) with a pair of losses that include a first error term in the form of cross entropy loss H(q,p) and a second error term H(u,p). The second error term H(u,p) is referred to as a regularizing error term that penalizes the neural network based on the error calculated between a set of scores generated by the neural network and a smoothing distribution that includes a respective smoothing score for each of the labels in the set. For example, the smoothing distribution may be a uniform distribution u that assigns the same score for each label in the set of labels associated with the smoothing distribution.

At stage <NUM>, the system trains a neural network to minimize the regularizing loss function that includes the first error term and the regularizing error term. Such training may be performed as described with reference to <FIG>. However, the loss function <NUM> of system <NUM> would be replaced by the regularizing loss function.

Embodiments of the subject matter, the functional operations and the processes described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible nonvolatile program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD-ROM and DVD-ROM disks.

In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's user device in response to requests received from the web browser.

Particular embodiments of the subject matter have been described.

Claim 1:
A computer-implemented method of regularizing the training of a neural network, wherein the neural network is configured to receive an input data item and to process the input data item to generate an output comprising a respective score for each label in a predetermined set of multiple labels, wherein either:
the input data item corresponds to an image or features that have been extracted from an image, the set of multiple labels corresponds to a set of object categories, and the output generated by the neural network for the input data item are scores for each of the set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category, or
the input data item corresponds to features of a spoken utterance, the set of multiple labels corresponds to a set of pieces of text, and the output generated by the neural network for the input data item are scores for each of the set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript for the spoken utterance,
the method of regularizing the training of the neural network comprising:
obtaining (<NUM>) a set of training data that includes a plurality of training items, wherein each training item is associated with an initial target label distribution that assigns a respective target score to each label in the predetermined set of multiple labels; and
modifying (<NUM>) the training data to generate regularizing training data that regularizes the training of the neural network, comprising:
for each training item, generating a modified target label distribution by combining the initial target label distribution with a smoothing label distribution, wherein the smoothing label distribution is independent of the training item; and
training (<NUM>) the neural network on the regularizing training data.