Summarise this for me:

Learning to control agents directly from high-dimensional sensory inputs like vision and speech is
one of the long-standing challenges of reinforcement learning (RL). Most successful RL applica-
tions that operate on these domains have relied on hand-crafted features combined with linear value
functions or policy representations. Clearly, the performance of such systems heavily relies on the
quality of the feature representation.
Recent advances in deep learning have made it possible to extract high-level features from raw sen-
sory data, leading to breakthroughs in computer vision [11, 22, 16] and speech recognition [6, 7].
These methods utilise a range of neural network architectures, including convolutional networks,
multilayer perceptrons, restricted Boltzmann machines and recurrent neural networks, and have ex-
ploited both supervised and unsupervised learning. It seems natural to ask whether similar tech-
niques could also be beneficial for RL with sensory data.
However reinforcement learning presents several challenges from a deep learning perspective.
Firstly, most successful deep learning applications to date have required large amounts of hand-
labelled training data. RL algorithms, on the other hand, must be able to learn from a scalar reward
signal that is frequently sparse, noisy and delayed. The delay between actions and resulting rewards,
which can be thousands of timesteps long, seems particularly daunting when compared to the direct
association between inputs and targets found in supervised learning. Another issue is that most deep
learning algorithms assume the data samples to be independent, while in reinforcement learning one
typically encounters sequences of highly correlated states. Furthermore, in RL the data distribu-
tion changes as the algorithm learns new behaviours, which can be problematic for deep learning
methods that assume a fixed underlying distribution.
This paper demonstrates that a convolutional neural network can overcome these challenges to learn
successful control policies from raw video data in complex RL environments. The network is
trained with a variant of the Q-learning [26] algorithm, with stochastic gradient descent to update
the weights. To alleviate the problems of correlated data and non-stationary distributions, we use
1 an experience replay mechanism [13] which randomly samples previous transitions, and thereby
smooths the training distribution over many past behaviors.
We apply our approach to a range of Atari 2600 games implemented in The Arcade Learning Envi-
ronment (ALE) [3]. Atari 2600 is a challenging RL testbed that presents agents with a high dimen-
sional visual input (210 × 160 RGB video at 60Hz) and a diverse and interesting set of tasks that
were designed to be difficult for humans players. Our goal is to create a single neural network agent
that is able to successfully learn to play as many of the games as possible. The network was not pro-
vided with any game-specific information or hand-designed visual features, and was not privy to the
internal state of the emulator; it learned from nothing but the video input, the reward and terminal
signals, and the set of possible actions—just as a human player would. Furthermore the network ar-
chitecture and all hyperparameters used for training were kept constant across the games. So far the
network has outperformed all previous RL algorithms on six of the seven games we have attempted
and surpassed an expert human player on three of them. Figure 1 provides sample screenshots from
five of the games used for training.