Continuous control with deep reinforcement learning

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

BACKGROUND

This specification relates to selecting actions to be performed by a reinforcement learning agent.

Reinforcement learning agents interact with an environment by receiving an observation that characterizes the current state of the environment, and in response, performing an action. Some reinforcement learning agents use neural networks to select the action to be performed in response to receiving any given observation.

SUMMARY

This specification describes technologies that relate to reinforcement learning.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. A reinforcement learning system can effectively and directly learn an effective action selection policy for an agent in high-dimensional, continuous action spaces, i.e., by training an actor neural network as described in this specification. In particular, by training the actor neural network as described in this specification, the reinforcement learning system can effectively learn an effective action selection policy even for tasks that require fine control of actions and when the action space is intractable for discretizing and then exploring effectively. Additionally, the reinforcement learning system can learn an effective policy both from observations that are low-dimensional observations and from observations that are high-dimensional pixel inputs.

DETAILED DESCRIPTION

This specification generally describes a reinforcement learning system that selects actions to be performed by a reinforcement learning agent interacting with an environment. In order to interact with the environment, the agent receives data characterizing the current state of the environment and performs an action from a continuous action space in response to the received data. Data characterizing a state of the environment will be referred to in this specification as an observation.

In some implementations, the environment is a simulated environment and the agent is implemented as one or more computer programs interacting with the simulated environment. For example, the simulated environment may be a video game and the agent may be a simulated user playing the video game. As another example, the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent is a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be points in a space of possible control inputs to control the simulated user or simulated vehicle.

In some other implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment. For example, the agent may be a robot interacting with the environment to accomplish a specific task. As another example, the agent may be an autonomous or semi-autonomous vehicle navigating through the environment. In these implementations, the actions may be points in a space of possible control inputs to control the robot or the autonomous vehicle.

In some cases, the observations characterize states of the environment using low-dimensional feature vectors that characterize the state of the environment. In these cases, values of different dimensions of the low-dimensional feature vectors may have varying ranges.

In some other cases, the observations characterize states of the environment using high-dimensional pixel inputs from one or more images that characterize the state of the environment, e.g., images of the simulated environment or images captured by sensors of the mechanical agent as it interacts with the real-world environment.

FIG. 1shows an example reinforcement learning system100. The reinforcement learning system100is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The reinforcement learning system100selects actions to be performed by a reinforcement learning agent102interacting with an environment104. That is, the reinforcement learning system100receives observations, with each observation characterizing a respective state of the environment104, and, in response to each observation, selects an action from a continuous action space to be performed by the reinforcement learning agent102in response to the observation.

In particular, the reinforcement learning system100selects actions using an actor neural network110. The actor neural network110is a neural network that is configured to receive an observation and to process the observation to map the observation to a next action, i.e., to a point in the continuous action space that defines an action that should be performed by the agent in response to the observation.

To allow the agent102to effectively interact with the environment, the reinforcement learning system100trains the actor neural network110to determine trained values of the parameters of the actor neural network110.

Once the actor neural network110has been trained, the reinforcement learning system100can effectively use the actor neural network110to select actions to be performed by the agent104. In particular, when an observation is received, the reinforcement learning system100can process the observation using the actor neural network110to map the observation to a new action in accordance with the trained values of the parameters of the actor neural network110and then direct the agent102to perform the new action in response to the observation, i.e., by sending instructions to the agent102that cause the agent to perform the new action.

To assist in the training of the actor neural network110, the reinforcement learning system100maintains training components120that include a replay memory130, a critic neural network140, a target actor neural network150, and a target critic neural network160.

The replay memory130stores experience tuples generated as a consequence of the interaction of the agent102with the environment104for use in training the actor neural network110.

In particular, each experience tuple in the replay memory includes a training observation that characterizes a training state of the environment, an action performed by the agent102in response to the training observation, a training reward received by the system100in response to the agent102performing the action, and a next observation characterizing a next state of the environment, i.e., the state that the environment transitioned into after the agent performed the action.

The reinforcement learning system100generates the experience tuples from the interactions of the agent102with the environment104during the training of the actor neural network110. An example process for generating an experience tuple during training is described in more detail below with reference toFIG. 2.

The critic neural network140is a neural network that is configured to receive as input an action and an observation and to process the action and the observation to generate a neural network output. As will be described in more detail below, during the training, the reinforcement learning system100adjusts the values of the parameters of the critic neural network140and uses the critic neural network140in updating the values of the parameters of the actor neural network110.

In some implementations, the critic neural network140, the actor neural network110, or both include one or more batch normalization layers in order to minimize covariance shift during training. Batch normalization layers are described in more detail in Ioffe, Sergey and Szegedy, Christian.Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.

The target actor neural network150is a neural network that is the same as, i.e., has the same neural network architecture as, the actor neural network110, but that has possibly different parameter values from those of the actor neural network110.

Similarly, the target critic neural network160is a neural network that is the same as the critic neural network130but that has possibly different parameter values from the critic neural network130.

To train the neural network using the training components120, the reinforcement learning system100repeatedly selects minibatches of experience tuples from the replay memory130. Each minibatch of experience tuples includes a predetermined number, e.g., a predetermined number of randomly selected experience tuples.

For each experience tuple in a given selected minibatch, the reinforcement learning system100uses the critic neural network140, the target actor neural network150, and the target critic neural network160to determine updates for the current values of the parameters of the actor neural network110and the current values of the parameters of the critic neural network150and then adjusts the current values of the parameters of the actor neural network110and the current values of the parameters of the critic neural network150using the updates. Generating these updates and adjusting the current values of the parameters of the critic neural network140and the actor neural network110will be described in more detail below with reference toFIG. 3.

During the training, the reinforcement learning system100also periodically updates the values of the parameters of the target critic neural network160and the values of the parameters of the target actor neural network150so that the values slowly track the changes to the values of the parameters of the critic neural network140and the values of the parameters of the actor neural network110, respectively.

Once a minibatch of experience tuples has been used in training, the reinforcement learning system100can remove the experience tuples in the minibatch from the replay memory120.

Generally, during the training, the reinforcement learning system100generates experience tuples and adds the generated tuples to the replay memory120independently of, i.e., asynchronously from, sampling experience tuples from the replay memory120and adjusting the parameters of the actor neural network110.

FIG. 2is a flow diagram of an example process200for adding an experience tuple to a replay memory. For convenience, the process200will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process200.

The system receives a current observation characterizing the current state of the environment (step202).

The system processes the observation using an actor neural network in accordance with current values of the parameters of the actor neural network (step204). As described above, the actor neural network is configured to map the current observation to a next action, i.e., a point in the continuous action space, in accordance with the current values of the parameters.

The system selects an action to be performed by the agent using the next action (step206).

In some implementations, the system selects the next action as the action to be performed by the agent.

In some other implementations, to encourage exploration of the action space during training, the system samples from a noise process to obtain a noise factor and then adjusts the next action by the noise factor to generate the action to be performed by the agent.

The noise process used to obtain the noise factor can be chosen to suit the environment. For example, for some environments, the noise process may be an Ornstein-Uhlenbeck process to generate temporally correlated exploration. Ornstein-Uhlenbeck processes are described in more detail in George E Uhlenbeck and Leonard S Ornstein. “On the theory of the Brownian motion”. In: Physical review 36.5 (1930), p. 823.

The system receives a reward and a next observation (step208). The next observation characterizes the next state of the environment, i.e., the state that the environment transitioned into as a result of the agent performing the selected action, and the reward is a numeric value that is received by the system from the environment as a result of the agent performing the selected action.

The system generates an experience tuple that includes the current observation, the selected action, the reward, and the next observation and stores the generated experience tuple in a replay memory for use in training the actor neural network (step210).

FIG. 3is a flow diagram of an example process300for determining an update to the current values of the parameters of the actor neural network. For convenience, the process300will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process300.

The system receives an experience tuple (step302). The experience tuple is one of the experience tuples in a minibatch of experience tuples sampled from the replay memory by the system.

The experience tuple includes a training observation characterizing a training state of the environment, a training action from the continuous space of actions performed by the agent in response to receiving the training observation, a training reward received by the agent for performing the training action, and a next training observation characterizing a next training state of the environment.

The system processes the training observation and the selected action in the experience tuple using the critic neural network to determine a neural network output for the experience tuple in accordance with current values of the parameters of the critic neural network (step304).

The system determines a target neural network output for the experience tuple from the training reward in the experience tuple and the next training observation in the experience tuple (step306). Generally, the system determines the target neural network output using the target actor neural network and the target critic neural network. Determining the target neural network output is described in more detail below with reference toFIG. 4.

The system determines an update for the current values of the parameters of the critic neural network using an error between the target neural network output for the experience tuple and the neural network output that was generated by the critic neural network for the experience tuple (step308). That is, the system can determine an update to the current values of the parameters that reduces the error using conventional machine learning training techniques, e.g., by performing an iteration of gradient descent with backpropagation. As will be clear from the description ofFIG. 4, by updating the current values of the parameters in this manner, the system trains the critic neural network to generate neural network outputs that represent time-discounted total future rewards that will be received in response the agent performing a given action in response to a given observation.

The system determines an update for the current values of the parameters of the actor neural network using the critic neural network (step310).

In particular, to determine the update, the system processes the training observation in the tuple using the actor neural network in accordance with the current values of the parameters to generate a next action for the training observation.

The system then determines a parameter update for the current values of the actor neural network that is dependent on, i.e., is the product of or is a different combination of, (i) the gradient of the critic neural network with respect to the next action taken at the training observation—next action input pair and in accordance with the current values of the parameters of the critic neural network and (ii) the gradient of the actor neural network with respect to the parameters of the actor neural network taken at the training observation and in accordance with current values of the parameters of the actor neural network. The system can determine gradient (i) and gradient (ii) by backpropogating the respective gradients through the respective networks.

Generally, the system performs the process300for each experience tuple in a given minibatch to determine, for each tuple, an update for the parameters of the critic neural network and an update for the parameters of the actor neural network. Once the updates for each tuple in the minibatch have been determined, the system updates the current values of the parameters of the actor neural network and the current values of the parameters of the critic neural network using the updates for the tuples in the minibatch. For example, for each network, the system can add each update to the current values of the parameters of the network to update those values.

Once updated values of the parameters of actor neural network and the critic neural network have been determined, the system updates the current values of the target critic neural network parameters and the target actor neural network parameters so that the values slowly track the changes to the values of the parameters of the critic neural network and the value of the parameters of the actor neural network, respectively. In particular, the system constrains the values of the target critic neural network parameters and the target actor neural network parameters to change slowly during the training in order to improve the stability of the training process.

For example, the updated values of one of the target networks may be a linear interpolation between the updated values of the corresponding actor or critic network and the current values of the target network, with the current values of the target network being weighted more heavily in the interpolation.

By repeatedly performing the process300on multiple different minibatches of experience tuples, the system can train the actor neural network to determine trained values of the parameters of the actor neural network and to allow the actor neural network to effectively be used to select actions to be performed by the agent in interacting with the environment.

FIG. 4is a flow diagram of an example process400for determining a target neural network output for an experience tuple. For convenience, the process400will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process400.

The system processes the next training observation in the experience tuple using a target actor neural network in accordance with current values of the parameters of the target actor neural network to generate a predicted next action (step402). As described above, the target actor neural network is identical to the actor neural network, but with possibly different parameter values.

The system processes the next training observation and the predicted next action using a target critic neural network in accordance with current values of the parameters of the target critic neural network to generate a predicted next neural network output (step404). As described above, the target critic neural network is identical to the critic neural network, but with possibly different parameter values.

The system determines the target neural network for the experience tuple from the training reward and the predicted neural network output for the experience tuple (step406). In particular, the system multiplies the predicted neural network output by a predetermined time discount factor and then sums the resulting product and the training reward to generate the target neural network output for the experience tuple.