Summarise this for me:

In supervised learning, one can easily track the performance of a model during training by evaluating
it on the training and validation sets. In reinforcement learning, however, accurately evaluating the
progress of an agent during training can be challenging. Since our evaluation metric, as suggested
by [3], is the total reward the agent collects in an episode or game averaged over a number of
games, we periodically compute it during training. The average total reward metric tends to be very
noisy because small changes to the weights of a policy can lead to large changes in the distribution of
states the policy visits . The leftmost two plots in figure 2 show how the average total reward evolves
during training on the games Seaquest and Breakout. Both averaged reward plots are indeed quite
noisy, giving one the impression that the learning algorithm is not making steady progress. Another,
more stable, metric is the policy’s estimated action-value function Q, which provides an estimate of
how much discounted reward the agent can obtain by following its policy from any given state. We
collect a fixed set of states by running a random policy before training starts and track the average
of the maximum2 predicted Q for these states. The two rightmost plots in figure 2 show that average
predicted Q increases much more smoothly than the average total reward obtained by the agent and
plotting the same metrics on the other five games produces similarly smooth curves. In addition
to seeing relatively smooth improvement to predicted Q during training we did not experience any
divergence issues in any of our experiments. This suggests that, despite lacking any theoretical
convergence guarantees, our method is able to train large neural networks using a reinforcement
learning signal and stochastic gradient descent in a stable manner.

We compare our results with the best performing methods from the RL literature [3, 4]. The method
labeled Sarsa used the Sarsa algorithm to learn linear policies on several different feature sets hand-
engineered for the Atari task and we report the score for the best performing feature set [3]. Con-
tingency used the same basic approach as Sarsa but augmented the feature sets with a learned
representation of the parts of the screen that are under the agent’s control [4]. Note that both of these
methods incorporate significant prior knowledge about the visual problem by using background sub-
traction and treating each of the 128 colors as a separate channel. Since many of the Atari games use
one distinct color for each type of object, treating each color as a separate channel can be similar to
producing a separate binary map encoding the presence of each object type. In contrast, our agents
only receive the raw RGB screenshots as input and must learn to detect objects on their own.
In addition to the learned agents, we also report scores for an expert human game player and a policy
that selects actions uniformly at random. The human performance is the median reward achieved
after around two hours of playing each game. Note that our reported human scores are much higher
than the ones in Bellemare et al. [3]. For the learned methods, we follow the evaluation strategy used
in Bellemare et al. [3, 5] and report the average score obtained by running an -greedy policy with
 = 0.05 for a fixed number of steps. The first five rows of table 1 show the per-game average scores
on all games. Our approach (labeled DQN) outperforms the other learning methods by a substantial
margin on all seven games despite incorporating almost no prior knowledge about the inputs.
We also include a comparison to the evolutionary policy search approach from [8] in the last three
rows of table 1. We report two sets of results for this method. The HNeat Best score reflects the
results obtained by using a hand-engineered object detector algorithm that outputs the locations and
types of objects on the Atari screen. The HNeat Pixel score is obtained by using the special 8 color
channel representation of the Atari emulator that represents an object label map at each channel.
This method relies heavily on finding a deterministic sequence of states that represents a successful
exploit. It is unlikely that strategies learnt in this way will generalize to random perturbations;
therefore the algorithm was only evaluated on the highest scoring single episode. In contrast, our
algorithm is evaluated on -greedy control sequences, and must therefore generalize across a wide
variety of possible situations. Nevertheless, we show that on all the games, except Space Invaders,
not only our max evaluation results (row 8), but also our average results (row 4) achieve better
performance.
Finally, we show that our method achieves better performance than an expert human player on
Breakout, Enduro and Pong and it achieves close to human performance on Beam Rider. The games
Q*bert, Seaquest, Space Invaders, on which we are far from human performance, are more chal-
lenging because they require the network to find a strategy that extends over long time scales.