Patent ID: 12190223

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG.1shows an example reinforcement learning system100. The reinforcement learning system100is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The reinforcement learning system100selects actions104to be performed by an agent102interacting with an environment106at each of multiple time steps. In order for the agent102to interact with the environment106, the system100receives an observation108characterizing the current state of the environment106and selects an action104to be performed by the agent102in response to the received observation108.

In some implementations, the environment106is a simulated environment and the agent102is implemented as one or more computer programs interacting with the simulated environment by navigating through the simulated environment. For example, the simulated environment may be a video game and the agent102may be a simulated user playing the video game by navigating through the simulated environment. As another example, the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent102is a simulated vehicle navigating through the motion simulation environment. In these cases, the actions104may be control inputs to control the simulated user or simulated vehicle and the observations can include images of the simulated environment, features characterizing the state of the simulated user or simulated vehicle, or both.

In some other implementations, the environment106is a real-world environment and the agent102is a mechanical agent interacting with the real-world environment. For example, the agent102may be an autonomous or semi-autonomous vehicle navigating through the environment. In these cases, the actions104may be control inputs to control the robot or the autonomous vehicle and the observations can include images captured by sensors of the mechanical agent, other sensor data captured by other sensors of the mechanical agent.

At each time step during the interaction of the agent102with the environment106, the system100uses an action selection policy neural network112to select the action to be performed by the agent102in response to the observation108at the time step.

In particular, the action selection policy neural network112is configured to receive an input including an observation108and to process the input in accordance with a set of parameters of the action selection policy neural network, referred to in this specification as policy parameters, to generate a network output that the system100uses to determine an action104to be performed by the agent102at the time step. The network output defines a score distribution over possible actions that can be performed by the agent to interact with the environment in response to the observation.

In some cases, the set of possible actions that can be performed by the agent is finite, i.e., the system selects from a pre-determined finite, discrete set of actions at each time step. In these cases, i.e., when the set of possible actions that can be performed is finite, the network output is a score distribution over the set of possible actions.

In other cases, the set of possible actions is not finite, e.g., actions are represented by points in a continuous action space. For example, the actions may be torques applied to various joints of a mechanical agent that can be any real value in some predetermined range. In cases where the space of possible actions is continuous, the network output can be parameters, e.g., mean and covariance parameters, of a distribution over the possible actions.

In order for the action selection policy neural network112to be effectively used to select actions to be performed by the agent, a training engine118trains the action selection policy neural network112to generate policy outputs that maximize the expected cumulative reward received by the system100, e.g., a long-term time-discounted sum of rewards received by the system100, as a result of interactions by the agent with the environment.

In particular, during the training, the system100stores trajectories generated as a result of interactions of the agent with the environment in a replay memory150. Each trajectory in the replay memory150includes respective action selection data at each of multiple time steps. For example, each trajectory in the replay memory150may include the same fixed number of time steps. The trajectory data at a given time step identifies: (i) an observation characterizing a state of the environment, (ii) an action performed by the agent in response to the observation, (iii) a reward received in response to the agent performing the action, and (iv) the score distribution for the actions in the set of actions that was used in determining which action to perform in response to the observation. This distribution will be referred to in this specification as an action score distribution.

In particular, the system100uses the action selection policy neural network112to select the actions that are performed by the agent to generate the trajectories that are stored in the replay memory150. However, because the values of the policy parameters change during the training, it may not be possible for the system100to re-generate the same score distribution that was previously generated by the action selection policy neural network112for a given observation at the time at which the system100samples the trajectory from the replay memory150. In order for the system100to track the scores that were generated for previously performed actions, the system100stores the action score distribution (or parameters defining the action score distribution) in the replay memory150.

To train the action selection policy neural network112, the training engine118samples trajectories from the replay memory150and trains the action selection policy neural network112on the sampled trajectories using an off-policy actor-critic reinforcement learning technique. As part of the actor-critic technique, the training engine118trains the action selection policy neural network112jointly with a critic neural network130.

Generally, the critic neural network130is a neural network having a set of parameters, referred to in this specification as critic parameters. When the set of actions is finite, the critic neural network is configured to receive an observation and to process the observation to generate an output that defines a respective Q value for each action in the set of actions. The Q value for a given action is an estimate, according to the current values of the critic parameters, of the cumulative reward that would be received by the system100if the agent performed the given action in response to the observation.

When the set of actions is finite, the critic neural network130and the action selection policy neural network112can share some parameters. For example, the critic neural network130and the action selection policy neural network112can be a single neural network with two heads, one that outputs the Q values for the actions and another that outputs the score distribution. This neural network may include, for example, a convolutional neural network and/or a recurrent neural network.

Training the action selection policy neural network112(and the critic neural network130) when the set of actions is finite is described in more detail below with reference toFIGS.2and3.

When the set of actions is not finite, for example in a continuous action space, the critic neural network130can be a stochastic dueling neural network. A stochastic dueling neural network is a neural network that generates deterministic value estimates for states characterized by input observations while also generating stochastic Q values. In particular, the stochastic dueling neural network is configured to receive an input observation, an action of interest, and n actions sampled from a probability distribution over the set of possible actions. For example, the n actions can be sampled from a probability distribution defined by the output of the action selection policy neural network112.

The stochastic dueling neural network is configured to process the observation to generate the deterministic value estimate for the state characterized by the observation. The value estimate is an estimate of the cumulative reward received starting from the environment being in the state characterized by the observation if actions to be performed by the agent are selected using the action selection policy neural network112.

The stochastic dueling neural network is also configured to process the observation, the action of interest, and the n actions sampled from the probability distribution to generate a respective advantage estimate for each of the actions. An advantage estimate for a given action is a relative measure of importance of the action, i.e., how much cumulative reward will be received if the action is preformed relative to how much cumulative reward will be received if other actions are performed.

The stochastic dueling neural network then combines the value of the state, and the advantage estimates to generate the stochastic Q value for the action of interest. The Q value is stochastic because during different processing iterations, the n actions that are sampled may be different, resulting in different Q values being generated for the same action of interest. In particular, the Q value for the action of interest at in response to an observation xtsatisfies:

Q~θv′⁡(xt,at)=V⁡(xt)+A⁡(xt,at)-1n⁢Σi=1n⁢A⁡(xt,ui),
where V(xt) is the value estimate for the state characterized by the observation, A(xt, at) is the advantage estimate for the action of interest, and A(xt, ui) is the advantage estimate for the i-th sampled action.

Training the action selection policy neural network112(and the critic neural network130) when the set of actions is not finite, i.e., the space of possible actions is continuous, is described in more detail below with reference toFIGS.4and5.

In some implementations, to assist in the training of the action selection policy neural network112, the system100maintains an average neural network170. The average neural network170is a network that generates the same kind of output and has the same architecture as the action selection policy neural network112, but has different parameter values from those of the action selection policy neural network112during the training of the action selection policy neural network112. In particular, during the training, the system100updates the values of the parameters of the average network so that they represent a moving average of the policy parameters. The parameters of the average neural network will be referred to in this specification as “average network parameters.” Using the average network during training will be described below with reference toFIGS.2-6.

FIG.2is a flow diagram of an example process200for training the action selection policy neural network when the set of possible actions to be performed by the agent is finite. For convenience, the process200will be described as being performed by a system including one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system100ofFIG.1, appropriately programmed in accordance with this specification, can perform the process200.

The system samples a trajectory from the replay memory (step202). As described above, the trajectory includes respective action selection data at each of a plurality of time steps.

For each time step in the trajectory starting from the last time step, the system processes the observation at the time step using the action selection policy neural network in accordance with current values of the policy parameters to generate a main score distribution for the time step and processes the observation at the time step using the critic neural network to generate a set of Q values for the actions for the time step (step204).

If the system is using an average neural network for the training, the system also processes the observation at the time step using the average neural network in accordance with current values of the average network parameters to generate an auxiliary score distribution for the time step.

The system determines a respective action selection parameter update and a respective critic parameter update for each of the time steps using the main score distributions for the time steps, the sets of Q values for the time steps and, if generated, the auxiliary score distributions for the time steps (step206). Determining these updates for a given time step is described in more detail below with reference toFIG.3.

The system updates the current values of the policy parameters and the current values of the critic parameters using the action selection parameter updates and the critic parameter updates for the time steps (208), i.e., by adding each update to the corresponding set of parameters.

If the system is using an average neural network, the system updates the current values of the average network parameters using the updated values of the policy parameters to determine updated values of the average network parameters. In particular, the updated values of the average network parameters may be a weighted sum of the current values of the average network parameters and the updated values of the updated values of the policy parameters. A weight for combining these values can be selected to control the influence of the average neural network on the update of the policy parameters and hence provide a “soft” update.

The system can repeatedly perform the process200for different trajectories sampled from the replay memory to train the action selection policy neural network, i.e., to determine trained values of the policy parameters from initial values of the policy parameters.

FIG.3is a flow diagram of an example process300for determining an action selection parameter update and a critic parameter update for a time step when the set of possible actions to be performed by the agent is finite. For convenience, the process300will be described as being performed by a system including one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system100ofFIG.1, appropriately programmed in accordance with this specification, can perform the process300.

The system performs the process300for each time step in the trajectory, starting from the last time step in the trajectory and continuing until the first time step in the trajectory.

The system determines an importance weight for the time step (step302) from the score for the action performed at the time step in the main score distribution for the time step and the action selection score for the performed action in the action selection distribution for the time step, i.e., the distribution identified in the action selection data for the time step.

In particular, the system determines the importance weight by dividing the score for action performed at the time step in the main score distribution by the action selection score in the action selection distribution.

The system also determines a truncated importance weight for the time step as the minimum of a positive constant, e.g., one, and the importance weight.

The system determines a value estimate for the time step (step304) from the Q values and the main score distribution for the time step. In particular, the system determines a weighted sum of the Q values with each Q value being weighted by the score for the corresponding action in the main score distribution.

The system determines a Retrace estimate for the time step by multiplying a current Retrace estimate by a discount factor and adding the reward for the time step to the product (step306). The current Retrace estimate is the Retrace estimate after the preceding iteration of the process300. For the first iteration of the process, i.e., for the last time step in the trajectory, the system sets the current Retrace estimate to zero. Generally, the Retrace estimate is an estimate of a target Q value for the time step, i.e., an estimate of the Q value that should have been generated by the critic network by processing the observation at the time step.

The system determines a first gradient for the time step that is based on the performed action at the time step (step308). In particular, the first gradient term satisfies:
min{c,pi}∇φθ′(xi)logf(ai|φθ′(xi))Qret−Vi),
where piis the truncated importance weight for the time step, the gradient is taken with respect to φθ′(xi), which is the output of the action policy network that defines the main score distribution, f is the main score distribution score for the time step, aiis the action performed at the time step, Qretis the Retrace estimate for the time step, and Viis the value estimate for the time step.

The system determines a second gradient for the time step that is based on all of the actions in the set of actions (step310). Generally, the second gradient term can be viewed as a correction term for the first gradient term. In particular, the second gradient satisfies:

∑a⁢min⁢{0,1-cpi⁡(a)}⁢f⁡(a⁢|⁢φθ′⁡(xi))⁢∇φθ′⁡(xi)⁢log⁢⁢f⁡(a⁢|⁢φθ′⁡(xi))⁢(Qθv′⁡(xi,ai)-Vi),
where the sum is a sum over all of the actions a in the finite set of actions, pi(a) is the importance weight for the action a, i.e., the ratio between the score for the action a in the main score distribution and the score for the action a in the action score distribution, f(a|φθ′(xi)) is the score for the action a in the main score distribution, and

Qθv′⁡(xi,ai)
is the Q value for the performed action.

The system determines a main gradient for the time step from the first gradient and the second gradient (step312), e.g., by adding the first gradient term and the second gradient term.

In implementations where the system uses an average network, the system determines an auxiliary gradient of a measure of difference between the auxiliary score distribution for the time step and the main score distribution for the time step (step314). In particular, the auxiliary gradient can be equal to the gradient with respect to the main distribution, i.e., the output of the action policy neural network, of the KL divergence between (i) the auxiliary score distribution and (ii) the main score distribution.

The system determines a final gradient from the main gradient and the auxiliary gradient (step316). In particular, the system determines, from the main gradient and the auxiliary gradient, a scaling factor for the main gradient and then applies the scaling factor to the main gradient to generate the final gradient. For example, the scaling factors can satisfy:

s=max⁢{0,κT⁢g-δk22}⁢k,
where k is the auxiliary gradient, g is the main gradient, and δ is a constant value. The system can then apply the scaling factor by subtracting the scaling factor from the main gradient.

The system determines an update to the current values of the policy parameters from the final gradient (step318).

In particular, the system can backpropagate the final gradient through the action selection policy neural network to determine a gradient with respect to the parameters of the action selection policy neural network that is used to update the current values of the policy parameters.

In implementations where the system does not use an average neural network, the system can use the main gradient as the final gradient or can scale the main gradient using some other technique, e.g., a trust region technique that limits the per-step changes to the parameters. An example of such a technique is the Trust Region Policy Optimization (TRPO) technique described in Schulman, et al, J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization, in ICML, 2015.

The system determines an update to the current values of the critic parameters that uses the Retrace estimate as the target Q value (step320). In particular, the update to the critic parameters can satisfy:

∇θv′⁢(Qret-Qθv′⁡(xi,ai))2
where θ′vare the critic parameters.

The system updates the Retrace estimate before the estimate is used in the next iteration of the process300(step322). For example, the updated Retrace estimate can satisfy:

p¯i⁡(Qret-Qθv′⁡(xi,ai))+Vi,
wherepiis the truncated importance weight.

The system then uses the updated Retrace estimate as the current estimate for the next iteration of the process300.

FIG.4is a flow diagram of an example process400for training the action selection policy neural network when the set of possible actions to be performed by the agent is not finite, i.e., when the actions are selected from a continuous action space. For convenience, the process400will be described as being performed by a system including one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system100ofFIG.1, appropriately programmed in accordance with this specification, can perform the process400.

The system samples a trajectory from the replay memory (step402). As described above, the trajectory includes respective action selection data at each of a plurality of time steps.

For each time step in the trajectory, the system processes the observation at the time step using the action selection policy neural network in accordance with current values of the policy parameters to generate parameters of a main score distribution for the time step (step404). If the system is using an average neural network for the training, the system also processes the observation at the time step using the average neural network in accordance with current values of the average network parameters to generate parameters of an auxiliary score distribution for the time step.

For each time step in the trajectory, the system processes the observation at the time step and the performed action at the time step using the critic neural network (which in this case is a stochastic dueling network) to generate a stochastic Q value for the performed action and a value estimate for the time step (step406).

The system determines a respective action selection parameter update and a respective critic parameter update for each of the time steps using the main score distributions for the time steps, the sets of Q values for the time steps and, if generated, the auxiliary score distributions for the time steps (step408). Determining these updates for a given time step is described in more detail below with reference toFIG.5.

The system updates the current values of the policy parameters and the current values of the critic parameters using the action selection parameter updates and the critic parameter updates for the time steps (step410), i.e., by adding each update to the corresponding set of parameters.

If the system is using an average neural network, the system updates the current values of the average network parameters using the updated values of the policy parameters to determine updated values of the average network parameters. In particular, the updated values of the average network parameters are a weighted sum of the current values of the average network parameters and the updated values of the policy parameters.

The system can repeatedly perform the process400for different trajectories sampled from the replay memory to train the action selection policy neural network, i.e., to determine trained values of the policy parameters from initial values of the policy parameters.

FIG.5is a flow diagram of an example process500for determining an action selection parameter update and a critic parameter update for a time step when the set of possible actions to be performed by the agent is not finite. For convenience, the process500will be described as being performed by a system including one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system100ofFIG.1, appropriately programmed in accordance with this specification, can perform the process500.

The system performs the process500for each time step in the trajectory, starting from the last time step in the trajectory and continuing until the first time step in the trajectory.

The system samples an alternate action from the main score distribution (step502). That is, the system samples an alternate action from the continuous action space such that the likelihood that any given action is sampled is defined by the score for the action in the main score distribution.

The system determines an importance weight for the time step and an alternate importance weight for the time step (step504). In particular, the system determines the importance weight by dividing (i) the score for action performed at the time step, i.e., the action identified in the action selection data for the time step, in the main score distribution for the time step by (ii) the action selection score in the action selection distribution for the time step and determines the alternate importance weight by dividing (i) the score for alternate action in the main score distribution for the time step by (ii) the score for the alternate action in the action selection distribution for the time step.

The system determines a Retrace estimate for the time step by multiplying a current Retrace estimate by a discount factor and adding the reward for the time step to the product (step506). The current Retrace estimate is the Retrace estimate after the preceding iteration of the process500.

The system determines an off-policy corrections (OPC) estimate by multiplying a current OPC estimate by the discount factor and adding the reward for the time step to the product (step508). Like the current Retrace estimate, the current OPC estimate is the OPC estimate after the preceding iteration of the process500.

For the first iteration of the process, i.e., for the last time step in the trajectory, the system sets both the current Retrace estimate and the current OPC estimate to zero.

The system determines a first gradient term for the time step that is based on the performed action at the time step (step510). In particular, the first gradient term satisfies:
min{c,pi}∇φθ′(xi)logf(ai|φθ′(xi))(Qopc−Vi),
where c is a constant value, piis the importance weight for the time step, f is the main score distribution for the time step, aiis the performed action at the time step, φθ′(xi) are the parameters that are output by the action selection policy network (i.e., the parameters that define the main score distribution for the time step), Qopcis the OPC estimate for the time step and Viis the value estimate for the time step.

The system determines a second gradient term for the time step that is based on the alternate action sampled for the time step (512). In particular, the second gradient term satisfies:

min⁢{0,1-cpi′}⁢(Q~θv′⁡(xi,⁢ai′)-Vi)⁢∇φθ′⁡(xi)⁢log⁢⁢f⁡(ai′⁢|⁢φθ′⁡(xi)),
where p′iis the alternate importance weight for the time step, and

Q˜θv′⁡(xi,ai′)
is the Q value for the alternate action a′iin response the observation at the time step as generated by the stochastic dueling network.

The system determines a main gradient for the time step from the first gradient and the second gradient (step514), e.g., by adding the first gradient term and the second gradient term.

In implementations where the system uses an average network, the system determines an auxiliary gradient of a measure of difference between the auxiliary score distribution for the time step and the main score distribution for the time step (step516). In particular, the auxiliary gradient can be equal to the gradient with respect to the parameters of the main distribution of the KL divergence between (i) the auxiliary score distribution and (ii) the main score distribution.

The system determines a final gradient from the main gradient and the auxiliary gradient (step518). In particular, the system determines, from the main gradient and the auxiliary gradient, a scaling factor for the main gradient and then applies the scaling factor to the main gradient to generate the final gradient. For example, the scaling factor s can satisfy:

s=max⁢{0,κT⁢g-δk22}⁢k,
where k is the auxiliary gradient, g is the main gradient, and δ is a constant value. The system can then apply the scaling factor by subtracting the scaling factor from the main gradient.

The system determines an update to the current values of the policy parameters from the final gradient (step520).

In particular, the system can backpropagate the final gradient through the action selection policy neural network to determine a gradient with respect to the parameters of the action selection policy neural network that is used to update the current values of the policy parameters.

In implementations where the system does not use an average neural network, the system can use the main gradient as the final gradient or can scale the main gradient using some other technique, e.g., a trust region technique that limits the per-step changes to the parameters. An example of such a technique is the Trust Region Policy Optimization (TRPO) technique described in Schulman, et al, J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In ICML, 2015.

The system determines an update to the current values of the critic parameters (step522). In particular, the update to the critic parameters can be a sum of an update of a gradient that moves the critic parameters in the direction of a target Q value and another gradient that moves the critic parameters in the direction of a target value estimate. For example, the update can satisfy:

(Qret-Q˜θv′⁡(xi,ai))⁢∇θv′⁢Q˜θv′⁡(xi,ai)+min⁢{1,pi}⁢(Qret-Q˜θv′⁡(xi,ai))⁢∇θv′⁢Vθv′⁡(xi),
where θ′vare the critic parameters.

The system updates the Retrace estimate and the OPC estimate before the estimates are used in the next iteration of the process500(step524). For example, the updated Retrace estimate can satisfy:

ci⁡(Qret-Q˜θv′⁡(xi,ai))+Vθv′⁡(xi),
where ciis equal to

min⁢{1,(pi)1d}
and d is a constant, e.g., the dimensionality of the action space. The optional exponent of 1/d can lead to faster learning. The updated OPC estimate can satisfy:

(Qopc-Q˜θv′⁡(xi,ai))+Vθv′⁡(xi).

The system then uses the updated Retrace and OPC estimates as the current estimates for the next iteration of the process500.

The above description has described the average network as being used only to adjust the gradients of the action selection policy neural network before the gradients are used to update the parameters of the action selection policy network. However, an average neural network can be used to improve the training of any neural network (referred to as a “main” neural network) by scaling the gradients of the main neural network before they are used to update in updated the values of the parameters of the main neural network (referred to as the “main parameters”).

For example, the main neural network can be the action selection policy neural network described above or can be a different neural network that receives a different kind of network input and generates a different kind of network output.

Generally, the system can use the average neural network during the training of the main neural network by adjusting gradients based on a measure of difference between the output of the main neural network and the output of the average neural network.

The main neural network can be configured to receive any kind of digital data input and to generate any kind of score, classification, or regression output based on the input.

For example, if the inputs to the main neural network are images or features that have been extracted from images, the output generated by the main neural network for a given image may be scores for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category.

As another example, if the inputs to the main neural network are Internet resources (e.g., web pages), documents, or portions of documents or features extracted from Internet resources, documents, or portions of documents, the output generated by the main neural network for a given Internet resource, document, or portion of a document may be a score for each of a set of topics, with each score representing an estimated likelihood that the Internet resource, document, or document portion is about the topic.

As another example, if the inputs to the main neural network are features of an impression context for a particular advertisement, the output generated by the main neural network may be a score that represents an estimated likelihood that the particular advertisement will be clicked on.

As another example, if the inputs to the main neural network are features of a personalized recommendation for a user, e.g., features characterizing the context for the recommendation, e.g., features characterizing previous actions taken by the user, the output generated by the main neural network may be a score for each of a set of content items, with each score representing an estimated likelihood that the user will respond favorably to being recommended the content item.

As another example, if the input to the main neural network is a sequence of text in one language, the output generated by the main neural network may be a score for each of a set of pieces of text in another language, with each score representing an estimated likelihood that the piece of text in the other language is a proper translation of the input text into the other language.

As another example, if the input to the main neural network is a sequence representing a spoken utterance, the output generated by the main neural network may be a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript for the utterance.

As above, whenever an average neural network is used in training of a main neural network, the average neural network has the same architecture as the main neural network but different parameter values from those of the main neural network at any given point during the training. In particular, during the training, the current values of the average network parameters represent a running average of values of the main network parameters.

FIG.6is a flow diagram of an example process600for training a main neural network using an average neural network. For convenience, the process600will be described as being performed by a system including one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system100ofFIG.1, appropriately programmed in accordance with this specification, can perform the process600.

The system receives a batch of training data for use in training the main neural network to optimize an objective function (step602). Generally, the training data includes multiple training examples and specifies, for each training example, a target output that should be generated by the main neural network for the training example. The objective function is generally a function of, for a given training example, the output generated by the main neural network for the training example and the target output for the training example. For example, the objective function can be a cross-entropy loss function or a maximum likelihood objective function.

For each of the plurality of training examples in the batch, the system processes the training example using the main neural network and in accordance with current values of the main network parameters the training example to generate a main network output for the training example (step604).

For each of the plurality of training examples in the batch, the system determines a main gradient of the objective function from the main network output (step606) using conventional machine learning training techniques.

For each of the plurality of training examples in the batch, the system processes the training example using the average neural network and in accordance with current values of the average network parameters to generate an average network output for the training example (step608).

For each of the plurality of training examples in the batch, the system determines an auxiliary gradient of a measure of a difference between the average network output and the main network output (step610). In particular, the auxiliary gradient is a gradient with respect to the main network output and the measure of the difference is a Kullback-Leibler divergence between (i) the average network output and (ii) main network output.

For each of the plurality of training examples in the batch, the system determines a final gradient from the main gradient and the auxiliary gradient (step612). In particular, the system determines, from the auxiliary gradient and the main gradient, a scaling factor for the main gradient and applies the scaling factor to the main gradient to determine the final gradient. For example, the scaling factor s can satisfy:

s=max⁢{0,κT⁢g-δk22}⁢k,
where k is the auxiliary gradient, g is the main gradient, and δ is a constant value. The system can apply the scaling factor to the main gradient by subtracting the scaling factor from the main gradient.

For each of the plurality of training examples in the batch, the system determines an update to the current values of the main network parameters from the final gradient (step614). For example, the system can backpropagate the final gradient through the main neural network to determine an update for each of the main network parameters. Depending on the training technique being used, the system can either directly use the backpropagated gradients as the updates or can modify the backpropagated gradients to generate the update, e.g., by applying a learning rate to the backpropagated gradients.

The system updates the current values of the main network parameters using the updates for the training examples in the batch to determine updated values of the main network parameters, e.g., by adding the updates to the current values of the main network parameters (step616).

The system updates the current values of the average network parameters using the updated values of the main network parameters to determine updated values of the average network parameters (step618). In particular, the updated values of the average network parameters are a weighted sum of the current values of the average network parameters and the updated values of the main network parameters.

The system can repeat the process600multiple times on multiple different batches of training data to train the main neural network, i.e., to determine trained values of the main network parameters from initial values of the parameters.

This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.