Patent Description:
In a reinforcement learning system, an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

Some reinforcement learning systems select the action to be performed by the agent in response to receiving a given observation in accordance with an output of a neural network.

Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from a previous time step in computing an output at a current time step. An example of a recurrent neural network is a long short term (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more cells that each include an input gate, a forget gate, and an output gate that allow the cell to store previous states for the cell, e.g., for use in generating a current activation or to be provided to other components of the LSTM neural network.

<NPL>, describes an extension to the Generative Adversarial Imitation Learning method that can infer the latent structure of human decision-making in an unsupervised way. A component is introduced which maximizes the mutual information between latent structure and trajectories, similar to InfoGAN (Chen et ai. , <NUM>), resulting in a policy where low-level actions can be controlled through high-level latent variables.

<NPL>, describes model-free imitation learning algorithms for finding a parameterized stochastic policy that performs at least as well as an expert policy on an unknown cost function, based on sample trajectories from the expert.

<NPL>, describes an extension to Generative Adversarial Imitation Learning method that can infer the latent structure of human decision-making in an unsupervised manner.

This specification describes how a system implemented as computer programs on one or more computers in one or more locations can adjust the parameters of a neural network used to select actions to be performed by an agent interacting with an environment in response to received observations. This is generally referred to as "training" a neural network.

Implementations described herein utilize a combination of variational auto encoding and reinforcement learning to train the system to imitate the behavior of a training set of trajectories.

In a reinforcement learning system data may be output for selecting actions to perform, under control of the system. In order for the agent to interact with the environment, the system receives data characterizing the current state xt of the environment ε at time t and selects an action at to be performed by the agent in response to the received data according to its policy π. A policy π is a mapping from states to actions. In return, the agent receives a scalar reward rt. The return <MAT> is the total accumulated reward from time step t with discount factor γk ∈ (<NUM>,<NUM>]. The goal of the agent is to maximize the expected return from each state. Data characterizing a state of the environment will be referred to in this specification as an observation.

In accordance with the claimed invention, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment. For example, the agent may be a robot interacting with the environment to accomplish a specific task. As another example, the agent may be an autonomous or semi-autonomous vehicle navigating through the environment. In these implementations, the actions may control inputs to control the robot or the autonomous vehicle.

According to an aspect of the present disclosure there is provided a method for training a policy generator neural network according to claim <NUM>.

In general, one innovative aspect of the subject matter described in this specification can be embodied in a method for training a neural network used to select actions to be performed by an agent interacting with an environment. The method comprises obtaining data identifying a set of trajectories, each trajectory comprising a set of observations characterizing a set of states of the environment and corresponding actions performed by another agent in response to the states and obtaining data identifying an encoder that maps the observations onto embeddings for use in determining a set of imitation trajectories. The method further comprises determining, for each trajectory, a corresponding embedding by applying the encoder to the trajectory, determining a set of imitation trajectories by applying a policy defined by the neural network to the embedding for each trajectory, and adjusting parameters of the neural network based on the set of trajectories, the set of imitation trajectories and the embeddings.

The set of imitation trajectories may be trajectories comprising state action pairs that aim to copy the set of (training) trajectories. Each embedding can comprise a set of latent variables that can be decoded to determine a set of imitation trajectories. Once the parameters for the neural network have been adjusted (once the neural network has been trained) the neural network can imitate behavior that is observed in the set of (training) trajectories.

By adjusting the parameters of the neural network based on embeddings (latent variables) determined via an encoder, the resulting neural network is better able to imitate the behavior of the set of trajectories in a robust manner over a wider range of behaviors. As a wider range of behaviors are modelled by the neural network, a smaller number of training trajectories are required to train the neural network. Accordingly, this method allows for one-shot learning. Furthermore, this method allows for re-use in compositional controllers.

The methods described herein provide improved training compared to, for instance, behavioral cloning. Behavioral cloning suffers from inefficiencies stemming from its sequential nature and an inability to correct errors effectively without the training data set demonstrating appropriate correcting behaviors. In contrast, by training the neural network using an encoder that has been trained on the training trajectories, the methods described herein are better able to learn multiple behaviors robustly from small training datasets. Accordingly, the methods described herein are more efficient and effective at training neural networks.

Adjusting parameters of the neural network uses values output from a discriminator that have been conditioned using the embeddings. Conditioning the discriminator values using the latent variables results in the neural network becoming more robust and exhibiting a greater diversity of modelled behaviors. More specifically, conditioning the discriminator values also allows for the generation of a variety of reward functions, each of them tailored to imitating a different trajectory. The increased diversity of the reward functions provides a more stable means for training the neural network, as the method will not collapse into one particular mode. This allows for a greater diversity in the behaviors that are modelled.

Adjusting the parameters of the neural network comprises determining a set of parameters that improves the return from a reward function, the reward function being based on a value output from the discriminator. Accordingly, the neural network may be trained via reinforcement learning using a reward function that is based on the discriminator (that is, a variety of reward functions that are dependent on the discriminator values for the corresponding trajectories). As the discriminator has been conditioned using the latent variables, the reward function is also dependent on the latent variables that have been encoded from the set of trajectories. This leads to increased robustness of the neural network. The parameters may be determined via a stochastic gradient ascent or descent process. More specifically, the parameters may be determined via a trust region policy optimization process.

More specifically, the reward function may be:<MAT> wherein:.

The method may further comprise updating a set of discriminator parameters based on the embeddings. This allows the method to be iteratively repeated to further improve the neural network.

The method may comprise iteratively: updating the parameters of the neural network based on the discriminator; updating the discriminator parameters based on the set of trajectories, the set of imitation trajectories and the embeddings; and updating the embeddings and imitation trajectories using the updated neural network, until an end condition is met. The end condition may be a maximum number of iterations or maximum amount of time allocated for training the neural network. The method may further comprise, in response to the end condition being met, updating the parameters of the neural network based on the updated discriminator and outputting the parameters of the neural network.

Updating the set of discriminator parameters may utilize a gradient ascent method. More specifically, updating the set of discriminator parameters may comprise implementing: <MAT> wherein:.

Accordingly, the method may comprise minimizing the above function with respect to θ and maximizing the above function with respect to ψ.

Updating the set of discriminator parameters may utilize a gradient ascent method with gradient: <MAT> wherein:.

By updating the discriminant parameters via the above method, the updated discriminator may be utilized to determine improved neural network parameters.

Obtaining the encoder may comprise training a variational auto encoder based on the set of trajectories, wherein the encoder forms part of the variational auto encoder. Accordingly, whilst a pre-trained encoder may be utilized, the method may also include training the encoder based on a training set of trajectories. This may be achieved by training a variational auto encoder. Variational auto encoders generally include an encoder for producing a set of latent variables from a set of training trajectories, and the decoder for decoding the latent variables to produce imitation trajectories.

The variational auto encoder may further comprise a state decoder for decoding the embeddings to produce imitation states and an action decoder for decoding the embeddings to produce imitation actions. The imitation states and imitation actions combine as state action pairs to form imitation trajectories.

The action decoder may be a multilayer perceptron and the state decoder may be an autoregressive neural network, such as a wavenet.

The policy may be based on the action decoder. This allows the training of the neural network to be bootstrapped on the back of the action decoder that has already been trained on the trajectories. Initially, the policy may incorporate weights taken from the action decoder. Having said this, taking weights directly from the action decoder can lead to poor performance initially and destroy behavior present in the action decoder due to noise injected into the policy.

Advantageously the policy πθ may be: <MAT> wherein:.

This provides improved performance and helps avoid issues caused by noise.

Weights of the action decoder may be kept constant after the action decoder has been determined. By freezing the weights of the action decoder, deterioration of the action decoder can be prevented.

The encoder may be a bi-directional long short term memory encoder.

In general, one innovative aspect of the subject matter described in this specification can be embodied in a system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the operations of the respective method of any one of the methods described herein.

In general, one innovative aspect of the subject matter described in this specification can be embodied in one or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of the methods described herein.

Once the neural network has been trained, it may be used to determine actions in response to input states. This may be used to control an agent such as a robot, an autonomous vehicle, or a computer avatar. Whilst the implementations described herein discuss determining actions that correspond to specific input states, interpolated actions may also be generated. Interpolated actions may be based on an interpolated state (a state formed by interpolating two input states) or an interpolated embedding (an embedding formed by interpolating between two embeddings of two corresponding states).

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. The methods can be used to more efficiently and effectively train a neural network. For example by utilizing an encoder to train the neural network, the resulting neural network is better able to imitate the behavior of a smaller number of training trajectories in a robust manner over a wider range of behaviors. As a smaller number of training trajectories is required, the neural network can learn more quickly from observed actions, whilst also avoiding errors usually associated when small training sets are used. Accordingly, the resulting neural network is more robust and displays an increased diversity in behavior. Utilizing a smaller set of training trajectories means that a smaller number of computations is required, therefore the methods described herein display improved computational efficiency.

This specification generally describes a reinforcement learning system implemented as computer programs on one or more computers in one or more locations that selects actions to be performed by a reinforcement learning agent interacting with an environment by using a neural network. This specification also describes how such a system can adjust the parameters of the neural network.

The system has an advantage that an agent such as a robot, or autonomous or semi-autonomous vehicle can improve its interaction with a real-world environment. It can enable for example the accomplishment of a specific task or improvement of navigation through or interaction with the real-world environment.

Some implementations of the system address the problem of assigning credit for an outcome to a sequence of decisions which led to the outcome. More particularly they aim to improve the estimation of the value of a state given a subsequent sequence of rewards, and hence improve the speed of learning and final performance level achieved. They also reduce the need for hyperparameter fine tuning, and hence are better able to operate across a range of different problem domains.

In accordance with the claimed invention, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment. For example, the agent may be a robot interacting with the environment to accomplish a specific task. As another example, the agent may be an autonomous or semi-autonomous vehicle navigating through the environment. In these cases, the observation can be data captured by one or more sensors of the mechanical agent as it interacts with the environment, e.g., a camera, a LIDAR sensor, a temperature sensor, and so on.

Continuous control via deep reinforcement learning has made much progress in the last few years with several impressive demonstrations of how sophisticated motor skills can be learned from scratch or from demonstrations in simulation and, to some extent, on real robots.

Yet, the flexibility and agility of animals remains unmatched. One hallmark of biological motor control is that animals are able to recruit a large variety of different movements as required by the circumstances. Imagine a football player in action: she will run forward or backwards, at different speeds, perform quick turns, dribble the ball, feint the goal keeper and finally kick the ball into the goal. Building versatile embodied agents, both in the form of real robots and in the form of animated avatars, capable of a wide and diverse set of behaviors is one of the long-standing challenges of AI.

Behavioral cloning (BC) is a training method in which the actions of an agent mimicked. Given a set of demonstration trajectories (τi)i where the i-th trajectory of state-action pairs is <MAT>, behavioral cloning seeks apply Maximum Likelihood to imitate the actions. In the ith trajectory, {τi}i:.

When demonstration data is abundant, BC can be effective; however, without an abundance of data, BC can often fail. The inefficiencies of BC stem from the sequential nature of the problem. When using BC, even the slightest errors in mimicking the demonstration behavior can quickly accumulate as the policy is unrolled. A good policy should correct for the mistakes made previously. For BC to learn good corrective policies, there have to be enough corresponding behaviors in the demonstrations. Unfortunately, corrective behaviors are often rare in demonstration trajectories, thus making the learning of good corrective policies difficult.

From a learning perspective the goal of endowing an agent with a diverse set of behaviors therefore poses several challenges as it often requires the acquisition of the behaviors in the first place. The methods described herein seek to overcome this problem.

The starting point is the assumption that a moderate number of demonstrations of a variety of different behaviors is available in the form of state-action sequences, or simply sequences of states. The goal is to learn a control policy that can be conditioned on a behavior embedding vector and, when conditioned appropriately, reproduce any behavior from the original set, and, at least to some extent, interpolate between them.

By training the system based on embeddings (latent variables) determined via an encoder, the resulting system is better able to imitate the behavior of the set of trajectories in a robust manner over a wider range of behaviors. As a wider range of behaviors are modelled by the neural network, a smaller number of training trajectories are required to train the neural network, therefore providing a more efficient training method. Furthermore, this method allows for one-shot learning.

In addition, instead of pre-defining the behavior embedding space, some implementations described herein allow this behavior to emerge by training a control policy jointly with the encoder that maps a demonstration trajectory onto an embedding vector. The policy is then trained to approximately reproduce the trajectory. Besides being a vehicle for learning a suitable embedding space the encoder can subsequently serve to perform one-shot imitation of a given test trajectory.

<FIG> shows an example reinforcement learning system <NUM>. The reinforcement learning system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The reinforcement learning system <NUM> selects actions to be performed by a reinforcement learning agent <NUM> interacting with an environment <NUM>. That is, the reinforcement learning system <NUM> receives observations, with each observation characterizing a respective state of the environment <NUM>, and, in response to each observation, selects an action from an action space to be performed by the reinforcement learning agent <NUM> in response to the observation. The reinforcement learning system <NUM> then instructs or otherwise causes the agent <NUM> to perform the selected action.

After the agent <NUM> performs a selected action, the environment <NUM> transitions to a new state and the system <NUM> receives another observation characterizing the next state of the environment <NUM> and a reward. The reward can be a numeric value that is received by the system <NUM> or the agent <NUM> from the environment <NUM> as a result of the agent <NUM> performing the selected action. That is, the reward received by the system <NUM> generally varies depending on the result of the transition of states caused by the agent <NUM> performing the selected action. For example, a transition into a state that is closer to completing the task being performed by the agent <NUM> may result in a higher reward being received by the system <NUM> than a transition into a state that is farther from completing the task being performed by the agent <NUM>.

In particular, to select an action, the reinforcement learning system <NUM> includes a neural network <NUM> and an encoder <NUM>. The encoder <NUM> generates an embedding for each received action and provides each embedding to the neural network <NUM>. Each embedding describes the corresponding action via a set of latent variables. Generally, the neural network <NUM> is a neural network that is configured to receive an embedding of an observation and to process the embedding to generate an output that defines the action that should be performed by the agent in response to the observation.

In some implementations, the neural network <NUM> is a neural network that receives an embedded observation and an action and outputs a probability that represents a probability that the action is the one that maximizes the chances of the agent completing the task.

In some implementations, the neural network <NUM> is a neural network that receives an embedded observation and generates an output that defines a probability distribution over possible actions, with the probability for each action being the probability that the action is the one that maximizes the chances of the agent completing the task.

In some other implementations, the neural network <NUM> is a neural network that is configured to receive an embedding of an observation and an action performed by the agent in response to the observation, i.e., an observation-action pair, and to generate a Q-value for the observation-action pair that represents an estimated return resulting from the agent performing the action in response the observation in the observation-action pair. The neural network <NUM> can repeatedly perform the process, e.g. by repeatedly generating Q-values for observation-action pairs. The system <NUM> can then use the generated Q-values to determine an action for the agent to perform in response to a given observation.

To allow the agent <NUM> to effectively interact with the environment, the reinforcement learning system <NUM> jointly trains the neural network <NUM> and the encoder <NUM> to determine trained values of the parameters of the neural network <NUM> and the trained encoder <NUM>.

After the agent <NUM> has performed an action in response to a given observation and a reward has been received by the system <NUM> as a result of the agent performing the action, the system trains the neural network <NUM> based on the observation and reward.

Training the reinforcement learning system <NUM> is described in more detail below with reference to <FIG>. Training the encoder <NUM> is described in more detail below with reference to <FIG>. Training the neural network <NUM> is described in more detail below with reference to <FIG>.

<FIG> shows a flow diagram of an example process for training a reinforcement learning system to select actions to be performed by an agent interacting with an environment. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The goal of the training is to learn a single policy that is capable of mimicking a diverse set of behaviors, even when there is not enough data for traditional methods to work well. To this end, a two-stage approach is introduced. First an encoder is trained based on a set of input trajectories. Then the neural network is trained via reinforcement learning using encodings generated by the trained encoder.

The method therefore starts by obtaining a set of trajectories <NUM>. The trajectories are training or demonstration trajectories exhibiting behavior to be imitated. Each trajectory comprises data identifying (i) a first observation characterizing a first state of the environment and (ii) a first action performed by the agent in response to the first observation. In some implementations, e.g., in implementations where the neural network is being trained using an off-policy algorithm, the system can obtain the data from a memory that stores state-action pairs generated from the agent interacting with the environment. In other implementations, e.g., in implementations where the neural network is being trained using an on-policy algorithm, the obtained data includes data that has been generated as a result of a most-recent interaction of the agent with the environment.

Next, the system trains the encoder based on the trajectories <NUM>. In one implementation, a variational autoencoder (VAE) is utilized comprising a bi-directional long short term memory (LSTM) encoder for the demonstration trajectories and two decoders: a multilayer perceptron (MLP) for the actions and a Wavenet to predict the next state. The system is configured to pass the trajectories through the encoder to determine a distribution over embeddings z of the demonstration trajectories, then decode the trajectories to obtain imitation trajectories, and then train the system to improve the encoder and decoder performance. This supervised stage is essentially like behavioral cloning (BC) in terms of the objective being optimized, but architecturally includes an encoder which outputs stochastic embeddings to improve diversity. This shall be discussed in more detail below with reference to <FIG>.

Next, the system trains the neural network via reinforcement learning using embedded trajectories <NUM>. That is, the trained encoder is used to determine embeddings of each trajectory (embedded trajectories) and the neural network is trained using the embedded trajectories. While the first stage is fully supervised, the second stage is about tuning the model via reinforcement learning to increase robustness. This shall be discussed in more detail with reference to <FIG>.

Whilst the implementation of <FIG> includes the training of the encoder, it should be noted that the training methods described herein would equally work by training the neural network based on embeddings generated using a pre-trained encoder. Accordingly, it is not essential for the reinforcement learning system <NUM> to train the encoder, as the encoder may be trained by an external system, i.e. a pretrained encoder may be provided to the reinforcement learning system <NUM> (e.g. loaded into memory) in advance.

Conventional BC without a demonstration trajectory encoder, while simple, has a number of shortcomings. It is difficult for the estimated policy to mimic the expert under minor environmental deviations. For example, suppose the expert was driving a car in the middle of the lane. If the agent trained with BC encounters itself outside the middle of the lane, it will with high probability leave the road altogether; a rather undesirable situation. In addition, there is no obvious way to harness the policies learned with conventional BC within hierarchical controllers.

To overcome this problem, an encoder can be used to encode the demonstration trajectory to form embeddings upon which the BC policy depends. This approach facilitates transfer and one-shot learning.

In the present implementation, to better regularize the latent space, stochastic variational autoencoders (VAEs) having a distribution q(z|x<NUM>:T) are utilized. The encoder maps a demonstration trajectory to a vector. Given this vector, both the state and action trajectories can be decoded, as shown in <FIG>. To achieve this, the system minimizes the following loss function, <IMG>(α, ω, φ; τi): <MAT> where:.

<FIG> shows a state encoder and a state and action decoder according to an implementation.

The state encoder network q takes the form of a bi-directional long short term memory (LSTM) neural network. The encoder takes a set of states and generates a corresponding set of embedded states (embeddings). The encoder has two layers.

To produce the final encoding, the average of all the outputs of the second layer of the bi-directional LSTM is determined before a final linear transformation is applied to generate the mean and standard deviation of a Gaussian representing the encoding. The system then takes a sample from this Gaussian as the encoding ε.

During training the encoding is input into a state decoder and an action decoder to determine imitation states and imitation actions. These are then used to train the encoder, as discussed above.

The action decoder is a multi-layer perceptron (MLP), which takes both the state and the encoding as inputs and produces the parameters of a Gaussian.

The state decoder is shown on the right hand side of <FIG>. The state decoder is similar to a conditional Wavenet. The conditioning is produced by the concatenation of the state xt and the encoding before being passed into an MLP. The remainder of the network is similar to the standard conditional Wavenet architecture. A Wavenet is a type of autoregressive convolutional neural network. Instead of Softmax outputs units, a mixture of Gaussians is used as the output of the Wavenet. Wavenets are described in A. van den Oord, S. Dieleman, H. Simonyan, O. Vinyals, A. Kalchbrenner, A. Senior, and K. Kavukcuoglu. "WaveNet: A generative model for raw audio".

The outputs of the encoder and decoders are then used in the training to find the parameters that minimize the above loss function <IMG>(α, ω, φ; τi).

Once trained, the parameters of the encoder can be stored for future in training the neural network <NUM>.

It should be noted that whilst the above implementation discusses the use of a bi-directional long short term memory (LSTM) neural network, alternative forms of encoder may be used. In addition, whilst the above implementation discusses the use of a conditional Wavenet, alternative forms of state decoder may be used. Furthermore, whilst the above implementation discusses the use of a multi-layer perceptron, alternative forms of action decoder may be used.

As discussed above, BC performs poorly without a large set of demonstrations. Even with a demonstration trajectory encoder, as in the present case, BC can result in policies that make irrecoverable failures.

To solve this problem the implementations described herein include a second stage of policy refinement with reinforcement learning, which leads to significant improvements in robustness.

To this end, the implementations described herein adapt concepts used in Generative Adversarial Imitation Learning (GAIL).

GAIL is a method that can avoid the pitfalls of BC by interacting with the environment. Specifically, GAIL constructs a reward function using Generative Adversarial Networks (GANs) to measure the similarity between the policy generated trajectories and the expert trajectories.

GANs are generative models that use two networks: a generator G and a discriminator D. The generator tries to generate samples that are indistinguishable from real data. The job of the discriminator is to tell apart the data and the samples, predicting <NUM> with a high probability if the sample real and <NUM> otherwise. More precisely, GAN optimizes the following objective function: <MAT>.

GAIL is an imitation learning version of GAN that seeks to imitate expert trajectories. GAIL adopts the following objective function: <MAT>
where πE denotes the expert policy that generated the demonstration trajectories and πθ denotes the policy to be trained. To avoid differentiating through the system dynamics, policy gradient algorithms, instead of backpropagation, are used to train the policy by maximizing the discounted sum rewards: <MAT> wherein:.

Maximizing this reward, which may differ from the expert reward, drives πθ to expertlike regions of the state-action space. In practice, trust region policy optimization (TRPO) is used to stabilize the learning process.

Whilst GAIL can overcome some issues regarding BC, it has been found to be inadequate for training the system described herein. The GAIL optimizer based on policy gradients is mode seeking. It is therefore difficult to recover a diverse set of behaviors using this approach. This problem is further exacerbated by the mode collapse problem of GANs.

To solve this problem, a new approach is proposed that is capable of imitating diverse behaviors via reinforcement learning. The implementation utilized herein conditions the discriminator on encodings generated by the pre-trained encoder. Specifically, the discriminator is trained by optimizing the following objective: <MAT> wherein:.

Since the discriminator is conditional, the reward function <MAT> is now also conditional: <MAT>.

The conditioning therefore allows the generation of set of customized reward functions, each customized reward function being tailored to imitating a different trajectory. The policy gradient algorithm, though mode seeking, will not cause collapse into one particular mode due to the diversity of reward functions.

Since the system already has an action decoder from supervised training, it can be used to bootstrap the learning by RL. One possible route is to initialize the weights of the policy network to be the same as those of the action decoder. Before the policy reaches good performance, however, the noise injected into the policy for exploration (assuming that a stochastic policy gradient is used to train the policy) can lead to poor performance initially and destroy the behavior already present in the action decoder. Instead, a new policy is chosen to be: <MAT> where:.

To prevent the deterioration of the action decoder, its weights are frozen during training. That is, the weights of the action decoder are kept constant as the neural network is trained.

For policy optimization, trust region policy optimization may be adopted.

<FIG> shows a flow diagram of an example process for training a neural network using embedded trajectories. This process can be considered equivalent to step <NUM> in <FIG>.

The process begins, as discussed with regard to <FIG>, with the receipt of a set of trajectories and a trained encoder.

Then, for each trajectory, a corresponding embedding is determined <NUM>. This is achieved by applying the encoder to the trajectory to obtain an embedded trajectory.

Then, policy is applied to the embedded trajectories to obtain corresponding imitation trajectories <NUM>. That is, for each embedded trajectory, the embedded trajectory is input into the neural network, which applies the policy and outputs a corresponding imitation trajectory. If this is the first iteration of the method, then the policy is initiated as discussed above; otherwise, the previously updated policy is applied.

The policy parameters are then updated based on reward functions that are conditioned on the embeddings <NUM>. As discussed, the policy may be updated using trust region policy optimization (TRPO). This aims to determine a set of policy parameters that improve the return from the reward function. The reward function is conditioned on the discriminator that, in turn is conditioned on the embeddings, so that a customized reward function is applied for each embedding (for each trajectory). As discussed above, the reward function is: <MAT> wherein:.

For every trajectory, a different reward function is used, and for every state action pair within the trajectory, a different reward is determined using the corresponding reward function.

The discriminator is the updated using a gradient ascent method based on the imitation trajectories output by the neural network <NUM>. The discriminator is also conditioned on the embeddings. The discriminator is updated by adjusting the parameters of the discriminator neural network using backpropagation of the gradient using a gradient ascent or descent method.

In the present case, the gradient is: <MAT> wherein:.

Once the discriminator has been updated, the system determines whether the end of the training has been reached <NUM>. The end is reached when an end criterion has been satisfied. This might be, for instance, a predefined number of iterations of training or a predefined time for training.

If the end has not been reached, the method loops back to repeat steps <NUM>-<NUM> using the updated discriminator parameters and updated policy parameters. The updated policy is utilized in step <NUM> and the updated discriminator is applied in the reward functions used in step <NUM>.

The method therefore repeatedly updates the policy and discriminator parameters, iteratively improving on them until the end criterion is satisfied.

Once the end has been reached, the method outputs the policy parameters <NUM>. This output may be to memory, either local or otherwise, or via communication to another device or system. The output policy parameters may then be utilized as a trained model for imitating the behaviors indicated by the input training trajectories.

Algorithm <NUM> shows an example process for training a neural network using embedded trajectories.

The algorithm first receives a set of demonstration trajectories and a pre-trained encoder (e.g. trained during step <NUM> or input to the system).

The algorithm then, for each trajectory, determines an embedding and then runs the policy on the embedding to determine a corresponding imitation trajectory. This repeats until an embedding and an imitation trajectory has been determined for all input trajectories.

Then the policy parameters are updated via TRPO using rewards determined from the reward function conditioned on the embeddings and the discriminator parameters are updated with the gradient.

The method repeats until a maximum number of iterations or a maximum time has been reached.

The implementations described herein provide a means for training a neural network to imitate diverse sets of behaviors using fewer training trajectories. This means that the neural network can be trained more efficiently. Furthermore, if a large number of trajectories are used then the neural network can imitate the training behaviors more effectively.

The training methods described herein have been tested to quantify their advantages. After training, it has been found that the trained model is more capable of reproducing most training and test policies.

In addition, To assist better generalization, it would be beneficial for the encoder to encode the trajectories in a semantically meaningful way. To test whether this is indeed the case, two random training trajectories were compared and their embedding vectors were obtained using the encoder. A series of convex combinations of these embedding vectors interpolating from one to the other were produced. The action decoder was conditioned on each of these intermediary points and executed in the environment. It was shown that interpolating in the latent space indeed corresponds to interpolation in the physical dimensions. This highlights the semantic meaningfulness of the discovered latent space.

In light of the above, it can be seen that the use of the encoder provides an effective means of acquiring and compressing a broad range of diverse behaviors into a suitable representation that makes them more effective when training a neural network. By conditioning the reward function used in reinforcement learning on the embeddings, the neural network is trained more effectively and efficiently to imitate a more diverse range of behaviors.

Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. The computer storage medium is not, however, a propagated signal.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of the invention, which is defined by the claims.

It should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claim 1:
A computer-implemented method for training a policy generator neural network to select actions to be performed by a mechanical agent interacting with a real-world environment, the method comprising:
obtaining data identifying a set of one or more trajectories (<NUM>), each trajectory comprising a set of observations characterizing a set of states of the environment and corresponding actions performed by another agent in response to the states, as state action pairs;
obtaining data identifying an encoder of a trained variational autoencoder neural network, wherein the encoder is configured to process trajectories to determine probability distributions over embeddings of the trajectories;
for each trajectory:
passing the trajectory through the encoder to determine a probability distribution over embeddings;
determining an embedding for the trajectory by sampling from the probability distribution encoded for the trajectory by the encoder (<NUM>);
determining an imitation trajectory (<NUM>) by applying a policy defined by the policy generator neural network to the embedding for the trajectory; and
determining a return from the imitation trajectory using a reward function that is conditioned on the embedding, wherein the reward function makes use of a discriminator configured to discriminate between the imitation trajectory and the trajectory, wherein the discriminator is conditioned using the embedding; and
train the policy generator neural network using the reward function that is conditioned on the embedding, by adjusting parameters of the policy generator neural network based on the reward function that is conditioned on the embedding, to improve the return from the reward function.