Deep reinforcement learning:case study with standard RL testing domains

Reinforcement Learning (RL) is a type of Machine Learning algorithms and it enables the agent to determine the ideal behavior from its own experience. From robot control to autonomous navigation, Reinforcement Learning algorithms have been applied to address increasing difficult problems. In recent studies, a number of papers have shown great success of RL in the field of production control, finance, scheduling, communication and auto vehicle control. However, in most cases, the performance of these algorithms heavily rely on the quality of the handcrafted features. This drawback limits the application scope of traditional Reinforcement Learning algorithms, since some problems have high dimensional state space and are difficult to hand engineered. For instance, it is a long standing challenge for traditional RL algorithms to process high dimensional sensory data like vision and voice. In 2006, the Deep Learning (DL) algorithms were established and have been further developed in recent years. The Convolutional Neural Network is one of the Deep Learning models that could extract high dimensional features direct from the raw pixels, and have been successfully applied in computer vision. It is nature for us to think whether the traditional Reinforcement Learning algorithms could benefit from it. In 2015, Google proposed a novel algorithm called Deep Q network (DQN) that could learn behavior direct from raw pixels of images. As we known, there are stability issues when we use a non linear approximator function, such as a neural network, in a RL algorithm. There are three major reason: first, the inputs are correlated, but a neural network normally requires Independent Identically Distributed (IID) data; second, the policy may oscillate with a neural net; third, the Q learning gradients can be large and unstable when we back propagate the network. By experience replay, freezing the target Q network and reward normalization, the DQN algorithm address these problem and can learn how to behave in diverse environments with no adjustment of the architecture and the corresponding parameters. With experiments in Atari 2600 games, they offer evidence that the DQN algorithm surpasses all previous RL algorithms and achieve a level comparable to a professional human game tester across a set of 49 games. In this thesis, we apply the Deep Q network on standard RL testing domains, such as Grid World, Mountain Car Problem, and Inverted Pendulum Problem. In contrast with Atari games, these domains have low dimensional state space and have been well addressed by other RL algorithms. We would like to see the performance of DQN in theses domains compare to other RL algorithms like traditional Q learning and SARSA. To conduct the experiments, we implement a new agent for traditional RL testing domains which could return the frame, the reward value, and the terminal signal at each time step. Form the results, we demonstrate the DQN algorithm could learn the policy in these environments with the same architecture. It is worth noting that the DQN shows no better result compared to other RL algorithms in these simple domains, since it requires more time resource for complex computing.