[Papers]Recurrent Model-Based Reinforcement Learning
By: Ting Qiao
Tags: Reinforcement Learning, Recurrent Neural Network

Why Model-Free Reinforcement Learning Failed.

Model-Free Reinforcement Learning

Model-Free Reinforcement Learning(MFRL) become popular again after the combination of convolutional neural network and Q learning algorithm. The idea behind MFRL is using the Bellman equation to model the temporal relation between current and future.

However, the data efficiency of MFRL is a huge concern. There are two reasons that are causing such inefficiency. The first reason is the Bellman equation tends to overestimate the Q values of the future. The agent becomes an optimistic, happy chap. In other words, the estimation of future Q values is inaccurate.

The second reason for inefficiency is that the signal provided by the Bellman equations is ‘weak’. The agent needs to learn to assign different values to the state-action pair (Q-value) first. After a good estimation of Q-value is learned, the actor can learn a good strategy. The reward information is passed through value estimation and then passed to the actor.

Model-Based Reinforcement Learning

The Model-Based Reinforcement Learning(MBRL) is a good complement to the MFRL. The idea of MBRL is learning the state transition(World Model) first. And then use the state predictions to form the loss function(Difference between Predicted and Target). The actor is trained by directly propagate(red lines) the loss through the world model(world model parameters are not changed, actor parameters are updated).

The world model needs to take state and action pair as input, and the output the next state. It is a duplicate of the environment(Blue Line). The most important is that it can propagate the gradient, i.e differentiable.

RNN as the model

Speaking of model temporal relationships, the Recurrent Neural Network family are inevitable. RNNs adds one extra channel to model temporal relationships. For example, with data size [Batch, Data_Dimension], the RNNs needs the data becomes [Batch, Timesteps, Data_Dimension].

Generally speaking, the development of RNNs includes vanilla RNN, Gate Recurrent Network, and LSTM. Furthermore, the Attention Neural Network and Transformer were used to replace RNNs. There are already enough papers and blogs to cover all of them.

The RNNs are candidates for the world model. The RNNs fulfils the two criteria we proposed. It can take the state-action as input and predict the next state. And the RNNs are differentiable.

CNN and RNN

In the field of model-based reinforcement learning, CNN is often used to reduce the dimensionality of high dimensional inputs like images. It consists of the autoencoder in the system, where latent variables can represent the whole data.

On the other hand, the RNN is used as an inferencing tool to predict the future latent variables. In other words, CNN and RNN are perpendicular with each other.

Papers

There are few papers using RNN as a world model. I will do a brief summary and comparison here.

Action-Conditional Video Prediction using Deep Networks in Atari Games(2015)

Area: Atari Games Reward: Not handled. The goal is not reaching a higher reward. Actor: DQN, Replaced Environment with world model. The RNN is LSTM. It is used to model the latent variables.

4. The unfolded RNN structure is like:

5. The image feature vector and actions are combined via multiplicative interactions. I.e h = W * feature * action + b

World Model(2018)

Area: Car Racing, VisDoom Dimension Reduction: Variational AutoEncoder Reward: Dense Timestep, higher rewards if the agent sustains longer. Only the Controller Model has access to the reward information from the environment. The design is similar to the previous one. The output of the RNN is different. The way to predict the next latent variable for the decoder is different. It is a probability model this time.

4. Simple controller: a = W*[h z] + b

6. The whole system:

7. The training of Actor: CovarianceMatrix Adaptation Evolution Strategy (CMA-ES) maximize the cumulative of the reward.

8. Propose the idea of hallucination.

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control(2018)

Area: Robotic manipulator control. Study different reward functions.

2. Goal Specification: Sparse. Study the generalization.

Designated Pixels: Distances to goal pixel positions

Goal Images: Registration to goal images

Image classifier: Success Classifier

3. Actor: Visual MPC

4. Model: Recurrent Neural Network, ConvLSTM layers.

5. Image Warping.

6. RNN is used to generate the flow field.

7. The images are considered to be hidden states by using ConvLSTM.

8.

9. ConvLSTM

MODEL-BASED REINFORCEMENT LEARNING FOR ATARI (2020)

Area: Atari Games Reward: Dense, Learned with world model. Actor: PPO integrated like Dyna(Providing more data to train with). Initialized with the World Model. Model: CNN + RNN

5. Key Point: Stochastic Latent Variable Discretization.

6. RNN: The discrete latent variable prediction.

7. The network on the top of the graph is called the auxiliary network. Here, it is used to learning to transfer the image into a discrete latent variable. On the other hand, the lower part to the graph (Autoencoder) can preserve the spatial information of the image(Without Fully connected layer.).