Summarise this for me:

Recent breakthroughs in computer vision and speech recognition have relied on efficiently training
deep neural networks on very large training sets. The most successful approaches are trained directly
from the raw inputs, using lightweight updates based on stochastic gradient descent. By feeding
sufficient data into deep neural networks, it is often possible to learn better representations than
handcrafted features [11]. These successes motivate our approach to reinforcement learning. Our
goal is to connect a reinforcement learning algorithm to a deep neural network which operates
directly on RGB images and efficiently process training data by using stochastic gradient updates.
Tesauro’s TD-Gammon architecture provides a starting point for such an approach. This architec-
ture updates the parameters of a network that estimates the value function, directly from on-policy
samples of experience, st, at, rt, st+1, at+1, drawn from the algorithm’s interactions with the envi-
ronment (or by self-play, in the case of backgammon). Since this approach was able to outperform
the best human backgammon players 20 years ago, it is natural to wonder whether two decades of
hardware improvements, coupled with modern deep neural network architectures and scalable RL
algorithms might produce significant progress.
In contrast to TD-Gammon and similar online approaches, we utilize a technique known as expe-
rience replay [13] where we store the agent’s experiences at each time-step, et = (st, at, rt, st+1)
in a data-set D = e1, ..., eN , pooled over many episodes into a replay memory. During the inner
loop of the algorithm, we apply Q-learning updates, or minibatch updates, to samples of experience,
e ∼ D, drawn at random from the pool of stored samples. After performing experience replay,
the agent selects and executes an action according to an -greedy policy. Since using histories of
arbitrary length as inputs to a neural network can be difficult, our Q-function instead works on fixed
length representation of histories produced by a function φ. The full algorithm, which we call deep
Q-learning, is presented in Algorithm 1.
This approach has several advantages over standard online Q-learning [23]. First, each step of
experience is potentially used in many weight updates, which allows for greater data efficiency.
4
Algorithm 1 Deep Q-learning with Experience Replay
Initialize replay memory D to capacity N
Initialize action-value function Q with random weights
for episode = 1, M do
Initialise sequence s1 = {x1} and preprocessed sequenced φ1 = φ(s1)
for t = 1, T do
With probability  select a random action at
otherwise select at = maxa Q∗(φ(st), a; θ)
Execute action at in emulator and observe reward rt and image xt+1
Set st+1 = st, at, xt+1 and preprocess φt+1 = φ(st+1)
Store transition (φt, at, rt, φt+1) in D
Sample random minibatch of transitions (φj , aj , rj , φj+1) from D
Set yj =
{ rj for terminal φj+1
rj + γ maxa′ Q(φj+1, a′; θ) for non-terminal φj+1
Perform a gradient descent step on (yj − Q(φj , aj ; θ))2 according to equation 3
end for
end for
Second, learning directly from consecutive samples is inefficient, due to the strong correlations
between the samples; randomizing the samples breaks these correlations and therefore reduces the
variance of the updates. Third, when learning on-policy the current parameters determine the next
data sample that the parameters are trained on. For example, if the maximizing action is to move left
then the training samples will be dominated by samples from the left-hand side; if the maximizing
action then switches to the right then the training distribution will also switch. It is easy to see how
unwanted feedback loops may arise and the parameters could get stuck in a poor local minimum, or
even diverge catastrophically [25]. By using experience replay the behavior distribution is averaged
over many of its previous states, smoothing out learning and avoiding oscillations or divergence in
the parameters. Note that when learning by experience replay, it is necessary to learn off-policy
(because our current parameters are different to those used to generate the sample), which motivates
the choice of Q-learning.
In practice, our algorithm only stores the last N experience tuples in the replay memory, and samples
uniformly at random from D when performing updates. This approach is in some respects limited
since the memory buffer does not differentiate important transitions and always overwrites with
recent transitions due to the finite memory size N . Similarly, the uniform sampling gives equal
importance to all transitions in the replay memory. A more sophisticated sampling strategy might
emphasize transitions from which we can learn the most, similar to prioritized sweeping.