Title: GENERALIZATION AND REGULARIZATION IN DQN

Abstract: Deep reinforcement learning (RL) algorithms have shown an impressive ability to learn complex control policies in high-dimensional environments. However, despite the ever-increasing performance on popular benchmarks like the Arcade Learning Environment (ALE), policies learned by deep RL algorithms can struggle to generalize when evaluated in remarkably similar environments. These results are unexpected given the fact that, in supervised learning, deep neural networks often learn robust features that generalize across tasks. In this paper, we study the generalization capabilities of DQN in order to aid in understanding this mismatch between generalization in deep RL and supervised learning methods. We provide evidence suggesting that DQN overspecializes to the domain it is trained on. We then comprehensively evaluate the impact of traditional methods of regularization from supervised learning, ℓ 2 and dropout, and of reusing learned representations to improve the generalization capabilities of DQN. We perform this study using different game modes of Atari 2600 games, a recently introduced modification for the ALE which supports slight variations of the Atari 2600 games used for benchmarking in the field. Despite regularization being largely underutilized in deep RL, we show that it can, in fact, help DQN learn more general features. These features can then be reused and fine-tuned on similar tasks, considerably improving the sample efficiency of DQN.

1. INTRODUCTION:

Recently, reinforcement learning (RL) has proven very successful on complex high-dimensional problems, in large part due to the increase in computational power and to the use of deep neural networks for function approximation (e.g., Mnih et al., 2015; Silver et al., 2016) . Despite the generality of the proposed solutions, applying these algorithms to slightly different environments generally requires agents to learn the new task from scratch. Practitioners often realize that the learned policies rarely generalize to other domains, even when they are remarkably similar, and that the learned representations are seldom reusable.

Deep neural networks, though, are lauded for their generalization capabilities (e.g., LeCun et al., 1998) . Some communities heavily rely on reusing representations learned by neural networks. In computer vision, classification and segmentation algorithms are rarely trained from scratch; instead they are initialized with pre-trained models from larger datasets like ImageNet (e.g., Razavian et al., 2014; Long et al., 2015) . The field of natural language processing has also seen successes in reusing and refining weights from certain layers of neural networks using pre-trained word embeddings, with more recent techniques able to reuse all weights of the network (e.g., Howard & Ruder, 2018) .

In light of the successes of traditional supervised learning methods, the current lack of generalization or reusable knowledge (e.g., policies, representation) acquired by current deep RL algorithms is somewhat surprising. In this paper we investigate whether the representation learned by deep RL methods can be generalized, or at the very least reused and refined on small variations to the task at hand. First, we evaluate the generalization capabilities of DQN (Mnih et al., 2015) . We further explore whether the experience gained by the supervised learning community to improve generalization and to avoid overfitting could be used in deep RL. We employ conventional supervised learning techniques, albeit largely unexplored in deep RL, such as fine-tuning (i.e., reusing and refining the representation) and regularization. We show that a learned representation trained with regularization allows us to learn more general features capable of being reused and fine-tuned. Besides improving the generalization capabilities of the learned policies this fine-tuning procedure has the potential to greatly improve sample efficiency on settings in which an agent might face multiple variations of the same task. Finally, the results we present here also can be seen as paving a way towards novel curriculum learning approaches for deep RL.

We perform our experiments using different game modes and difficulties of Atari 2600 games, a newly introduced feature of the Arcade Learning Environment (ALE; Bellemare et al., 2013) . These game modes allow agents to be trained in one environment while being evaluated in a slightly different environment that still captures key concepts of the original environment (e.g., game sprites, agent goals, dynamics). This use of game modes is itself a novel approach for measuring our progress toward a longstanding goal of agents that can learn to be generally competent and generalize across tasks (Bellemare et al., 2013; Machado et al., 2018; Nichol et al., 2018) . This paper also introduces the first baselines for the different modes of Atari 2600 games.

2. BACKGROUND:

2.1 REINFORCEMENT LEARNING Reinforcement learning (RL) is a problem where an agent interacts with an environment with the goal of maximizing some form of cumulative long term reward. RL problems are often modeled as a Markov decision process (MDP), defined by a 5-tuple ⟨S, A, p, r, γ⟩. At a discrete time step t the agent observes the current state S t ∈ S and chooses an action A t ∈ A to probabilistically transition to the next state S t+1 ∈ S according to the transition dynamics function p(s

′ | s, a) . = P (S t+1 = s ′ | S t = s , A t = a).

The agent receives a reward signal R t+1 according to the reward function r : S × A → R. The agents goal is to learn a policy π : S × A defined as the conditional probability of taking action a in state s written as π(a | s). The learning agent refines its policy with the objective of maximizing the expected return, that is, the cumulative discounted reward incurred from time t, defined by Dayan, 1992 ) is a traditional approach to learning an optimal policy from samples obtained from interactions with the environment. It is used to learn an optimal state-action value function via a bootstrapped iterative method. For a given policy π we define the state-action value function as the expected return conditioned on a state and action

G t . = ∑ ∞ k=0 γ k R t+k+1 where γ ∈ [0, 1) is the discount factor. Q-learning (Watkins &

q π (s, a) . = Eπ [ G t |S t = s, A t = a ]

. The agent iteratively updates the state-action value function based on samples from the environment using the update rule

Q(S t , A t ) ← Q(S t , A t ) + α [ R t+1 + γ max a ′ ∈A Q(S t+1 , a ′ ) -Q(S t , A t ) ]

where t denotes the current timestep and α the step size. Generally, due to the exploding size of the state space in many real-world problems, it is intractable to learn a state-action pairing for the entire MDP, with researchers and practitioners often resorting to learning an approximate to q π . DQN approximates the state-action value function such that q π (s, a) ≈ Q(s, a; θ), where θ denotes the weights of a neural network. The network takes as input some encoding of the current state S t and outputs |A| scalars corresponding to the state-action values for that given state. DQN is trained to minimize

L DQN = E St, At, Rt+1, St+1 ∼ U (•) [( R t+1 + max a ′ ∈A Q(S t+1 , a ′ ; θ -) -Q(S t , A t ; θ) ) 2 ]

where (S t , A t , R t+1 , S t+1 ) are uniformly sampled from U (•), the experience replay buffer filled with experience collected by the agent. The weights θ -of a duplicate network are updated less frequently for stability purposes.

2.2. SUPERVISED LEARNING:

In the supervised learning problem we are given a dataset of examples represented by a matrix X ∈ R m×n with m training examples of dimension n, and a vector y ∈ R 1×m denoting the output target y i for each training example X i . We want to learn a function which maps each training example X i to its predicted output label ŷi . The goal is to learn a robust model that accurately predicts y i from X i while also being able to generalize to unseen training examples. In this paper we focus on using a neural network parameterized by the weights θ to learn the function f such that ŷi = f (X i ; θ). We typically train these models by minimizing

min θ λ 2 ∥θ∥ 2 2 + 1 m m ∑ i=1 L(y i , ŷi ) = min θ λ 2 ∥θ∥ 2 2 + 1 m m ∑ i=1 L(y i , f (X i ; θ))

where L is a differentiable loss function which outputs a scalar determining the quality of the prediction (e.g., squared error loss). The first term is a form of regularization, i.e., ℓ 2 regularization, which encourages generalization. ℓ 2 regularization imposes a penalty on large weight vectors with λ being the weighted importance of the regularization term.

Another popular regularization technique is dropout (Srivastava et al., 2014) . When using dropout, during forward propagation each neural unit has a chance of being set to zero according to a Bernoulli distribution with probability p ∈ [0, 1], referred to as the dropout rate. Dropout discourages the network from relying on a small number of neurons to make a prediction, making it hard for the network to memorize the dataset.

Prior to training, the network parameters are usually initialized through a stochastic process (e.g., Xavier initialization; Glorot & Bengio, 2010) . We can also initialize the network using pre-trained weights from a different task. If we reuse one or more pre-trained layers we say the weights encoded by those layers will be fine-tuned during training (e.g., Razavian et al., 2014; Long et al., 2015) .

3. THE ALE AS A PLATFORM FOR EVALUATING GENERALIZATION:

The Arcade Learning Environment (ALE) is a platform used to evaluate agents across dozens of Atari 2600 games (Bellemare et al., 2013) . It has become one of the standard evaluation platforms in the field and has led to a number of exciting algorithmic advances (e.g., Mnih et al., 2015) . The ALE poses the problem of general competency by having agents use the same learning algorithm to perform well in as many games as possible, while learning without using game specific knowledge.

Learning to play multiple games with the same agent, or learning to play a game faster by leveraging knowledge acquired in a different game is much harder, with fewer successes being known (e.g., Rusu et al., 2016; Kirkpatrick et al., 2016; Parisotto et al., 2016; Schwarz et al., 2018; Espeholt et al., 2018) .

In this paper, we use the different modes and difficulties of Atari 2600 games to evaluate a neural network's ability to generalize in high-dimensional state spaces. Game modes, originally native to the Atari console, were recently added in the ALE (Machado et al., 2018) . They give us modifications of the default environment dynamics and state space, often modifying sprites, velocities, and partial observability. These modes pose a tractable way to investigate generalization of RL agents in a high-dimensional environment. Instead of requiring an agent to play multiple games that are visually very different or even non-analogous, it requires agents to play games that are visually very similar and that can be played with policies that are very similar, at least from a human perspective.

We use 13 flavours (combinations of a mode and a difficulty) obtained from 4 games: FREEWAY, HERO, BREAKOUT, and SPACE INVADERS. In FREEWAY, the different modes vary the speed and number of vehicles, while different difficulties change how the player is penalized for running into a vehicle. In HERO, subsequent modes start the player off at increasingly harder levels of the game. The mode we use in BREAKOUT makes the bricks partially observable. The used modes in SPACE INVADERS allow for oscillating shield barriers, increasing the width of the player sprite, and partially observable aliens. Full explanations of specific games, their modes, and their difficulties can be found in Appendix A. Figure 1 provides screenshots showing side by side comparisons of some of the modes explored in this paper. When reading the analyses of this paper it is important to keep in mind how remarkably similar these modes are.

4. GENERALIZATION OF THE LEARNED POLICIES AND OVERFITTING:

In order to test the generalization capabilities of DQN we first evaluate whether a policy learned in one flavour can perform well in a different flavour. As afformentioned, different modes and difficulties of a single game look very similar. If the representation encodes a robust policy we might expect it to be able to generalize to slight variations of the underlying reward signal, game dynamics, or observations. Evaluating the learned policy in a similar but different flavour can be seen as evaluating generalization in RL, similar to cross-validation in supervised learning.

To evaluate DQN's ability to generalize across flavours we evaluate the learned ϵ-greedy policy on a new flavour after being trained for 50M frames in the default flavour, m0d0 (mode 0, difficulty 0). We measure the cumulative reward averaged over 100 episodes in the new flavour, adhering to the evaluation protocol suggested by Machado et al. (2018) . The results are summarized in Table 1 .

Baseline results where the agent is trained from scratch for 50M frames in the flavour we use for evaluation are summarized in the baseline column. Theoretically, this baseline can be seen as an upper bound on the performance DQN can achieve in that flavour, as it represents the agent's performance when evaluated in the same flavour it was trained on. Full baseline results with the agent's performance after different number of frames can be found in Appendix B.

We can see in the results that the policies learned by DQN do not generalize well to different flavours, even when the flavours are remarkably similar. For example, in FREEWAY, a high-level policy applicable to all flavours is to go up while avoiding cars. Perhaps surprisingly, this does not seem to be what DQN learns. For example, the default flavour m0d0 and m4d0 have exactly the same sprites on the screen, the only difference is that in m4d0 some cars accelerate and decelerate over time. The close to optimal policy learned in m0d0 is only able to score 15.8 points when evaluated on m4d0, which is approximately half of what the policy learned from scratch in that flavour achieves (29.9 points). The learned policy when evaluated on flavours that differ more from m0d0 perform even worse.

As previously mentioned, the different modes of HERO can be seen as giving the agent a curriculum or a natural progression. Interestingly, the agent trained in the default mode for 50M frames can progress to at least level 3 and sometimes level 4. Mode 1 starts the agent off at level 5, and performance in this mode suffers greatly during evaluation. There are very few game mechanics added to level 5, indicating that perhaps the agent is memorizing trajectories instead of learning a robust policy capable of solving each level.

The results in some flavours suggest that the agent is overfitting to the flavour it is trained on. We tested this hypothesis by periodically evaluating the policy being learned in each of the other flavours of that game. This process involved taking checkpoints of the network at every 500, 000 frames and evaluating the ϵ-greedy policy in the prescribed flavour for 100 episodes, again further averaged over five runs. The obtained results in FREEWAY, the most pronounced game in which we see this overfitting trend, are depicted in Figure 2 . Learning curves for all flavours can be found in Appendix C.

In FREEWAY, while we see the policy's performance flattening out in m4d0, we do see the traditional bell-shaped curve associated to overfitting in the other modes. At first, improvements in the original policy do correspond to improvements in the performance of that policy in other domains. With time, it seems that it starts to refine its policy for the specific flavour it is being trained on, overfitting to that flavour. With other game flavours being significantly more complex in their dynamics and gameplay, we do not observe this prominent bell-shaped curve though. For example, in BREAKOUT, we actually observe a monotonic increase in performance throughout the evaluation process. Frames before evaluation Table 1 : Direct policy evaluation. Each game was initially trained in the default mode for 50M frames then evaluated in each listed game flavour. Reported numbers are the average over 5 runs. Standard deviation is reported between parentheses.

Figure 2 : Performance of an agent that was trained in the default mode of FREEWAY and evaluated at every 500, 000 frames in each corresponding mode. Results are averaged over five seeds. The y-axis is log scaled.

In conclusion, when looking at Table 1 , it seems that the policies learned by DQN struggle to generalize to even small variations encountered in game flavours. This lack of generalization is surprising, and results as seen in FREEWAY exhibit a troubling notion of overfitting. Based on these results we aim to evaluate whether deep RL could benefit from established methods from supervised learning promoting generalization and reducing overfitting.

5. REGULARIZATION IN DEEP RL:

In order to evaluate the hypothesis that the observed lack of generalization is due to overfitting, we revisit some popular regularization methods from the supervised learning literature. The two forms of regularization we test are dropout and ℓ 2 regularization.

First we want to understand the effect of regularization on evaluating the learned policy in a different flavour. We do so by applying dropout to the first four layers of the network during training, that is, the three convolutional layers and the first fully connected layer. We simultaneously apply ℓ 2 regularization on all weights in the network based on preliminary experiments that showed an additive effect when combining dropout and ℓ 2 regularization. This confirms, for example, Srivastava et al.'s (2014) result that these methods provide benefit in tandem.

We follow the same evaluation scheme described when evaluating the unregularized policy to different flavours. We evaluate the policy learned after 50M frames of the default mode of each game. A grid search was performed on FREEWAY to find reasonable hyperparameters for the dropout rate p ∈ {0.05, 0.1, 0.2, 0.3, 0.4, 0.5} and the weighted regularization parameter λ ∈ {10 -2 , 10 -3 , 10 -4 }. These parameters were then used for each subsequent flavour. Notably, significantly smaller dropout values were required compared to heuristics used in supervised learning, although this could be due to the small size of the network in question. We ended up choosing λ = 10 -4 , p = 0.05 for the first three convolutional layers, and p = 0.1 for the first fully connected layer. We contrast these results with the results presented in the previous section. This evaluation protocol allows us to directly evaluate the effect of regularization on the learned policy's ability to generalize. A baseline agent trained from scratch for 50M frames in each flavour is also provided. The results are presented in Table 2 with the evaluation learning curves being available in the Appendix.

When using regularization during training we sometimes observe a performance hit in the default flavour. Dropout generally requires increased training iterations to reach the same level of performance sans-dropout. Suprisingly, we did not observe this performance hit in all games. Nevertheless, maximal performance in one flavour is not our goal. We are interested in the setting where one may be willing to take lower performance on one task in order to obtain higher performance, or adaptability, on future tasks. Nevertheless, full baseline results using regularization in the default flavour can also be found in In most flavours, evaluating the policy trained with regularization does not negatively impact performance when compared to the performance of the policy trained without regularization. In some flavours we even see an increase in performance. Interestingly, when using regularization the agent in FREEWAY improves for all flavours and even learns a policy capable of outperforming the baseline learned from scratch in two of the three flavours. Moreover, in FREEWAY we now observe increasing performance during evaluation throughout most of the learning procedure as depicted in Figure 3 . These results seem to confirm the notion of overfitting observed in Figure 2 .

Despite slight improvements from these techniques, regularization by itself does not seem sufficient to enable policies to generalize across flavours. As shown in the next section, perhaps the real benefit of regularization in deep RL comes from the ability to learn more general features. These features may lead to a more adaptable representation which can be reused and subsequently fine-tuned on other flavours, which is often the case in supervised learning.

6. VALUE FUNCTION FINE-TUNING:

We hypothesize that the benefit of regularizing deep RL algorithms may not come from improvements during evaluation, but instead in having a good parameter initialization that can be adapted to new tasks that are similar. We evaluate this hypothesis using two common practices in machine learning. First, we the use the weights trained with regularization as the initialization for the entire network. We subsequently fine-tune all weights in the network. This is similar to what is performed in computer vision with supervised classification methods (e.g., Razavian et al., 2014) . Secondly, we evaluate reusing and fine-tuning only early layers of the network. This has been shown to improve generalization in some settings (e.g., Yosinski et al., 2014) , and is sometimes used in natural language processing (e.g., Mou et al., 2016; Howard & Ruder, 2018) .

When fine-tuning the entire network, we take the weights of the network trained in the default flavour for 50M frames and use them to initialize the network commencing training in the new flavour for 50M frames. We perform this set of experiments twice. Once for the weights trained without regularization, and again for the weights trained with regularization, as described in the previous section. Each run is averaged over five seeds. For comparison we provide a baseline trained from scratch for 50M and 100M frames in each flavour. Directly comparing the performance obtained after fine-tuning to the performance after 50M frames (SCRATCH) shows the benefit of re-using a representation learned in a different task instead of randomly initializing the network. Comparing the performance obtained after fine-tuning to the performance of 100M frames (SCRATCH) lets us take into consideration the whole learning process. The results are presented in Table 3 .

Fine-tuning from an unregularized representation yields conflicting conclusions. Although in FREE-WAY we obtained positive fine-tuning results, we note that rewards are so sparse in m1d0 and m1d1 that this initialization is likely to be simply acting as a form of optimistic initialization, biasing the agent to go up. The agent observes rewards more often, therefore, it learns quicker about the new flavour. However, the agent is still unable to reach the maximum score in these flavours. ). An agent is trained with dropout + ℓ 2 regularization in the default flavour of each game for 50M frames, then DQN's parameters θ were used to initialize the fine-tuning procedure on each new flavour for 50M frames. The baseline agent is trained from scratch up to 100M frames. Standard deviation reported between parenthesis.

The results of fine-tuning the regularized representation are more exciting. In FREEWAY we observe the highest scores on m1d0 and m1d1 throughout the whole paper. In HERO we vastly outperform fine-tuning from an unregularized representation. In SPACE INVADERS we obtain higher scores across the board on average when comparing to the same amount of experience. These results suggest that reusing a regularized representation in deep RL might allow us to learn more general features which can be more successfully fine-tuned.

Moreover, initializing the network with a regularized representation has a big impact on the agent's performance when compared to initializing the network randomly. These results are impressive when we consider the potential regularization has in reducing the sample complexity of deep RL algorithms. Such an observation also holds when we take the total number of frames seen between two flavours into consideration. When directly comparing one row of REGULARIZED FINE-TUNING to SCRATCH we are comparing two algorithms that observed 100M frames. However, to generate two rows of SCRATCH we used 200M frames while two rows of REGULARIZED FINE-TUNING used 150M frames (50M from scratch + 50M in each row). The distinction becomes bigger and bigger as more tasks are taken into consideration.

We further investigate which layers may encode general features able to be fine-tuned. Inspiration was taken from other studies that have shown that neural networks can re-learn co-adaptations when their final layers are randomly initialized, sometimes improving generalization (Yosinski et al., 2014) . We conjectured DQN may benefit from re-learning the co-adaptations between early layers comprising general features and the randomly initialized layers which ultimately assign state-action values. We hypothesized that it might be beneficial to re-learn the final layers from scratch since state-action values are ultimately conditioned on the flavour at hand. Therefore, we also evaluated whether fine-tuning only the convolutional layers, or the convolutional layers and the first fully connected layer was more effective than fine-tuning the whole network. Suprisingly, this does not seem to be the case. The performance obtained when the whole network is fine-tuned (Table 3 ) is consistently better than when it is not (Table 4 ). We speculate that this might not be the case on more dissimilar tasks.

7. DISCUSSION AND CONCLUSION:

Many studies have tried to explain generalization of deep neural networks in supervised learning settings (e.g., Zhang et al., 2018; Dinh et al., 2017) . Analyzing generalization and overfitting in deep RL has its own issues on top of the challenges posed in the supervised learning case. Actually, generalization in RL can be seen in different ways. We can talk about generalization in RL in terms of conditioned sub-goals within an environment (e.g., Andrychowicz et al., 2017; Sutton, 1995) , learning multiple tasks at once (e.g., Teh et al., 2017; Parisotto et al., 2016) , or sequential task learning as in a continual learning setting (e.g., Schwarz et al., 2018; Kirkpatrick et al., 2016) .

In this paper we evaluated generalization in terms of small variations of high-dimensional control tasks. This provides a candid evaluation method to study how well features and policies learned by The empirical evaluation presented in this paper has shown that traditional DQN seems to generalize poorly even between very similar high-dimensional control tasks. Given this lack of generality we investigated how dropout and ℓ 2 regularization can be used to improve generalization in deep RL.

Other forms of regularization in RL that have been explored in the past are sticky-actions, random initial states, entropy regularization (Zhang et al., 2018) , and procedural generation of environments (Justesen et al., 2018) . More related to our work, regularization in the form of weight constraints has been applied in the continual learning setting in order to reduce the catastrophic forgetting exhibited by fine-tuning on many sequential tasks (Kirkpatrick et al., 2016; Schwarz et al., 2018) . Similar weight constraint methods have been explored in multitask learning (Teh et al., 2017) .

Evaluation practices in RL often focuses on training and evaluating agents on exactly the same task. Consequently, regularization has traditionally been underutilized in deep RL. With a renewed emphasis on generalization in RL, regularization applied to the representation learning problem can be a feasible method to improving generalization on closely related tasks. Our results suggest that dropout and ℓ 2 regularization seem to be able to learn more general purpose features which can be adapted to similar problems. Although other communities relying on deep neural networks have shown similar successes, this is of particular importance for the deep RL community which struggles with sample efficiency (Henderson et al., 2018) . This work is also related to recent metalearning procedures like MAML (Finn et al., 2017) which aim to find a parameter initialization that can be quickly adapted to new tasks. In fact, some of the results here can also be seen under the light of curriculum learning. The regularization techniques we've evaluated here seem to be effective in leveraging situations where an easier task is presented first, sometimes leading to unseen performance levels (e.g., FREEWAY).

Finally, we believe it would be extremely beneficial for the field if we were able to develop algorithms that can generalize across tasks. Ultimately we want agents that can keep learning as they interact with the world in a continual learning fashion. The ability to generalize is essential. Throughout this paper we often avoided the expression transfer learning because we believe that succeeding in slightly different environments should be actually seen as a problem of generalization.

Our results suggested that regularizing and fine-tuning representations in deep RL might be a viable approach towards improving sample efficiency and generalization on multiple tasks. It is particularly interesting that fine-tuning a regularized network was the most successful approach because this might also be applicable in the continual learning settings where the environment changes without the agent being told so, and re-initializing layers of a network is obviously not an option. In this setting, the work from Kirkpatrick et al. (2016), and Schwarz et al. (2018) might be a great starting point as they provide a more thorough discussion of generalization in continual learning. In FREEWAY a chicken must cross a road containing multiple lanes of moving traffic within a prespecified time limit. In all modes of FREEWAY, the gets rewarded for reaching the top of the screen and is subsequently teleported to the bottom of the screen. If the chicken collides with a vehicle in difficulty 0 it gets bumped down one lane of traffic, alternatively, in difficulty 1 the chicken gets teleported to its starting position on the bottom of the screen. Mode 1 changes some vehicle sprites to include buses, adds more vehicles to some lanes, and increases the velocity of all vehicles. In HERO you control a character who must navigate a maze in order to save a trapped miner within a cave system. The agent scores points for any forward progression such as clearing an obstacle or killing an enemy. Once the miner is rescued, the level is terminated and you continue to the next level with a different maze. Some levels have partially observable rooms, more enemies, and more difficult obstacles to traverse. Past the default mode, each subsequent mode starts off at increasingly harder levels denoted by a level number increasing by multiples of 5. The default mode starts you off at level 1, mode 1 starts at level 5, and so on. In BREAKOUT you control a paddle which can move horizontally along the bottom of the screen. At the beginning of the game, or on loss of life a ball is set into motion and can bounce off the paddle and collide with bricks at the top of the screen. The objective of the game is to break all the bricks without having the ball fall below your paddles horizontal plane. Subsequently, mode 12 of breakout hides the bricks from the player until the ball collides with the bricks in which case the bricks flash for a brief moment before disappearing again. 

B BASELINE RESULTS:

In all experiments performed in this paper we utilize the neural network architecture used by Mnih et al. (2015) . That is, a convolutional neural network with three convolutional layers and two fully connected layers. A visualization of this network can be found in Figure 8 . Hyperparametes are generally kept consistent with Machado et al. ( 2018). Below we provide a table of the key hyperparameters used in the baseline experiments.

NEURAL NETWORK ARCHITECTURE 

L N + d x m g G L 6 X x R m A k M C Z 6 m g Q M u G Q U x 1 k C o 5 P q v m A 6 J J B R 0 Z h U d g r 1 4 8 l / o n N V t q 2 6 3 z q u N 6 y K O M j p E R 6 i G b H S B G u g W N V E b U T R B T + g F v

R q P x r P x Z r z P W 0 t G M b O P f s n 4 + A Z q x J U A < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c q v 7 6 k u 

S k K e v F H N m x X R k O h k D f / U = " > A A A C A H i c b Z D L S s N A F I Y n 9 V b r L e r C h Z v B I l S Q k o i g 4 K b o x m W L 9 g J N C J P J p B 0 6 u T B z I p T Q j a / i x o U i b n 0 M d 7 6 N 0 z Y L r f 4 w 8 P G f c z h z f j 8 V X I F l f R m l p e W V

F + w C q q h Q 0 z M / n S C h W c R i o I I o 1 b e t F N y c S O B U s E n F y R R L C R 2 R A e t r j E n E l J v P D p j g Y + 0 E O E y k f j H g m f t z I i e R U u P I 1 5 0 R g a F a r E 3 N / 2 r 9 D M J L N + d x m g G L 6 X x R m A k M C Z 6 m g Q M u G Q U x 1 k C o 5 P q v m A 6 J J B R 0 Z h U d g r 1 4 8 l / o n N V t q 2 6 3 z q u N 6 y K O M j p E R 6 i G b H S B G u g W N V E b U T R B T + g F v

R q P x r P x Z r z P W 0 t G M b O P f s n 4 + A Z q x J U A < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c q v 7 6 k u et al., 2018) . We allow the agent access to all 18 primitive actions in the ALE, we do not utilize the reduced action set nor the lives signal.

S k K e v F H N m x X R k O h k D f / U = " > A A A C A H i c b Z D L S s N A F I Y n 9 V b r L e r C h Z v B I l S Q k o i g 4 K b o x m W L 9 g J N C J P J p B 0 6 u T B z I p T Q j a / i x o U i b n 0 M d 7 6 N 0 z Y L r f 4 w 8 P G f c z h z f j 8 V X I F l f R m l p e W V

F + w C q q h Q 0 z M / n S C h W c R i o I I o 1 b e t F N y c S O B U s E n F y R R L C R 2 R A e t r j E n E l J v P D p j g Y + 0 E O E y k f j H g m f t z I i e R U u P I 1 5 0 R g a F a r E 3 N / 2 r 9 D M J L N + d x m g G L 6 X x R m A k M C Z 6 m g Q M u G Q U x 1 k C o 5 P q v m A 6 J J B R 0 Z h U d g r 1 4 8 l / o n N V t q 2 6 3 z q u N 6 y K O M j p E R 6 i G b H S B G u g W N V E b U T R B T + g F v R q P x r P x Z r z P W 0 t G M b O P f s n 4 + A Z q x J U A < / l a t

Furthermore, as a crude measure for environment complexity, we measure the best greedy action an agent could take in a game flavour. Simply put, we iterate through every action in A, executing this action ϵ-greeidly, with ϵ = 0.01, at every time step for 100 episodes. These results were then averaged over runs with the standard deviations between runs reported in parenthesis. 

GAME:



C POLICY EVALUATION LEARNING CURVES:

We provide learning curves for evaluating a policy learned in the default flavour (m0d0) to each subsequent flavour of that game. Each subplot are the results of evaluating the policy from a representation trained with and without regularization.

EVALUATION:

Checkpoint of the network weights θ were taken during training every 500, 000 frames, up to 50M frames in total. Each checkpoint was then evaluated in the target mode for 100 episodes averaged over five runs. Hyperparameters are kept consistent with the baseline experiments in Appendix B. 

:



Figures and Tables:

Figure 1: Each column shows variation between two selected flavours of each game. From left to right: FREEWAY, HERO, BREAKOUT, and SPACE INVADERS.

Mode 4 is almost identical to Mode 1; the only difference being vehicles can oscillate between two speeds

INVADERS you control a spaceship which can move horizontally along the bottom of the screen. There is a grid of aliens which are above you and the objective the game is to shoot-out all aliens. You are afforded some protection from the alien bullets with three barriers just above the spaceship. Difficulty 1 of space invaders widens your spaceships sprite making it harder to doge enemy bullets. Mode 1 of SPACE INVADERS causes the shields above you to oscillate horizontally. Mode 9 of SPACE INVADERS is similar to Mode 12 of BREAKOUT where the aliens are partially observable until struck with the players bullet.

t , •; ✓) < l a t e x i t s h a 1 _ b a s e 6 4 = " c q v 7 6 k u S k K e v F H N m x X R k O h k D f / U = " > A A A C A H i c b Z D L S s N A F I Y n 9 V b r L e r C h Z v B I l S Q k o i g 4 K b o x m W L 9 g J N C J P J p B 0 6 u T B z I p T Q j a / i x o U i b n 0 M d 7 6 N 0 z Y L r f 4 w 8 P G f c z h z f j 8 V X I F l f R m l p e W V 1 b X y e m V j c 2 t 7 x 9 z d 6 6 g k k 5 S 1 a S I S 2 f O J Y o L H r A 0 c B O u l k p H I F 6 z r j 2 6 m 9 e 4 D k 4 o n 8 T 2M U + Z G Z B D z k F M C 2 v L M g 1 b t z o N T 7 N A g g S t H A w w Z k B P P r F p 1 a y b 8 F + w C q q h Q 0 z M / n S C h W c R i o I I o 1 b e t F N y c S O B U s E n F y R R L C R 2 R A e t r j E n E l J v P D p j g Y + 0 E O E y k f j H g m f t z I i e R U u P I 1 5 0 R g a F a r E 3 N / 2 r 9 D M J L N + d x m g G L 6 X x R m A k M C Z 6 m g Q M u G Q U x 1 k C o 5 P q v m A 6 J J B R 0 Z h U d g r 1 4 8 l / o n N V t q 2 6 3 z q u N 6 y K O M j p E R 6 i G b H S B G u g W N V E b U T R B T + g F vR q P x r P x Z r z P W 0 t G M b O P f s n 4 + A Z q x J U A < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c q v 7 6 k uS k K e v F H N m x X R k O h k D f / U = " > A A A C A H i c b Z D L S s N A F I Y n 9 V b r L e r C h Z v B I l S Q k o i g 4 K b o x m W L 9 g J N C J P J p B 0 6 u T B z I p T Q j a / i x o U i b n 0 M d 7 6 N 0 z Y L r f 4 w 8 P G f c z h z f j 8 V X I F l f R m l p e W V1 b X y e m V j c 2 t 7 x 9 z d 6 6 g k k 5 S 1 a S I S 2 f O J Y o L H r A 0 c B O u l k p H I F 6 z r j 2 6 m 9 e 4 D k 4 o n 8 T 2 M U + Z G Z B D z k F M C 2 v L M g 1 b t z o N T 7 N A g g S t H A w w Z k B P P r F p 1 a y b 8 F + w C q q h Q 0 z M / n S C h W c R i o I I o 1 b e t F N y c S O B U s E n F y R R L C R 2 R A e t r j E n E l J v P D p j g Y + 0 E O E y k f j H g m f t z I i e R U u P I 1 5 0 R g a F a r E 3 N / 2 r 9 D M J

1 b X y e m V j c 2 t 7 x 9 z d 6 6 g k k 5 S 1 a S I S 2 f O J Y o L H r A 0 c B O u l k p H I F 6 z r j 2 6 m 9 e 4 D k 4 o n 8 T 2 M U + Z G Z B D z k F M C 2 v L M g 1 b t z o N T 7 N A g g S t H A w w Z k B P P r F p 1 a y b 8

1 b X y e m V j c 2 t 7 x 9 z d 6 6 g k k 5 S 1 a S I S 2 f O J Y o L H r A 0 c B O u l k p H I F 6 z r j 2 6 m 9 e 4 D k 4 o n 8 T 2 M U + Z G Z B D z k F M C 2 v L M g 1 b t z o N T 7 N A g g S t H A w w Z k B P P r F p 1 a y b 8

Figure9: Performance curves for policy evaluation results. The x-axis is the number of frames before we evaluated the ϵ-greedy policy from the default flavour on the target flavour. The y-axis is the cumulative reward the agent incurred.

Table7in the Appendix. Policy evaluation using regularization. Each game was initially trained in the default mode for 50M frames with dropout and ℓ2 regularization then evaluated on each listed flavour. Reported numbers are the average over 5 runs. Standard deviation is reported between parentheses.

Table content: <table><tr><td colspan="2">GAME VARIANT</td><td>EVAL. WITH REGULARIZATION</td><td>REGULARIZATION EVAL. WITHOUT</td><td>LEARN SCRATCH</td><td/><td colspan="2">20 25 30 Freeway Policy Evaluation w/ Regularization</td></tr><tr><td/><td>m1d0</td><td>5.8 (3.5)</td><td>0.2 (0.2)</td><td>4.8 (9.3)</td><td/><td>15</td></tr><tr><td>FREEWAY HERO BREAKOUT</td><td>m1d1 m4d0 m1d0 m2d0 m12d0 m1d0</td><td>4.4 (2.3) 20.6 (0.7) 116.8 (76.0) 30.0 (36.7) 31.0 (8.6) 456.0 (221.4)</td><td>0.1 (0.1) 15.8 (1.0) 82.1 (89.3) 33.9 (38.7) 43.4 (11.1) 258.9 (88.3)</td><td>0.0 (0.0) 29.9 (0.7) 1425.2 (1755.1) 326.1 (130.4) 67.6 (32.4) 753.6 (31.6)</td><td>Cumulative Reward</td><td>10 5</td></tr><tr><td>SPACE INVADERS</td><td>m1d1</td><td>146.0 (84.5)</td><td>140.4 (61.4)</td><td>698.5 (31.3)</td><td/><td/><td>m4d0 dropout+ 2 m1d0 dropout+ 2</td></tr><tr><td/><td>m9d0</td><td>290.0 (257.8)</td><td>179.0 (75.1)</td><td>518.0 (16.7)</td><td/><td>0</td><td>m1d1 dropout+ 2</td></tr><tr><td/><td/><td/><td/><td/><td/><td>10M</td><td>20M</td><td>30M</td><td>40M</td><td>50M</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td>Frames before evaluation</td></tr><tr><td/><td/><td/><td/><td/><td colspan="3">Figure 3: Performance of an agent that</td></tr><tr><td/><td/><td/><td/><td/><td colspan="3">was evaluated every 500, 000 frames af-</td></tr><tr><td/><td/><td/><td/><td/><td colspan="3">ter being trained in the default flavour of</td></tr><tr><td/><td/><td/><td/><td/><td colspan="3">FREEWAY with dropout and ℓ2 regular-</td></tr><tr><td/><td/><td/><td/><td/><td colspan="3">ization. Results are averaged over five</td></tr><tr><td/><td/><td/><td/><td/><td colspan="3">seeds. The y-axis is log scaled.</td></tr></table>

Experiments fine-tuning the entire network with and without regularization (dropout + ℓ 2

Table content: <table/>

Experiments fine-tuning early layers of the network trained with regularization. An agent is trained with dropout + ℓ 2 regularization in the default flavour of each game for 50M frames, then DQN's parameters θ were used to initialize the corresponding layers to be further fine-tuned on each new flavour. Remaining layers were randomly initialized. Compared against fine-tuning the entire network from Table3. Standard deviation reported between parenthesis.deep neural networks in RL problems can generalize. The approach of studying generalization with respect to the representation learning problem intersects nicely with the aforementioned problems in RL where generalization is key.

Table content: <table/>

Baselines using vanilla DQN for all tested game variants.

Table content: <table><tr><td/><td>VARIANT</td><td/><td>10M</td><td/><td>50M</td><td>100M</td><td>BEST ACTION</td></tr><tr><td>FREEWAY</td><td>m0d0 m1d0 m1d1</td><td colspan="2">3.0 (1.0) 0.0 (0.1) 0.0 (0.0)</td><td colspan="2">31.4 (0.2) 4.8 (9.3) 0.0 (0.0)</td><td>32.1 (0.1) 7.5 (11.5) 2.5 (7.3)</td><td>23.0 (1.4) 5.0 (1.5) 4.2 (1.3)</td></tr><tr><td/><td>m4d0</td><td colspan="2">4.4 (1.4)</td><td colspan="2">29.9 (0.7)</td><td>32.8 (0.2)</td><td>7.5 (2.8)</td></tr><tr><td>HERO</td><td>m0d0 m1d0 m2d0</td><td colspan="5">3187.8 (78.3) 9034.4 (1610.9) 13961.0 (181.9) 326.9 (40.3) 1425.2 (1755.1) 5026.8 (2174.6) 116.3 (11.0) 326.1 (130.4) 323.5 (76.4)</td><td>150.0 (0.0) 75.8 (7.5) 12.0 (27.5)</td></tr><tr><td>BREAKOUT</td><td>m0d0 m12d0</td><td colspan="2">17.5 (2.0) 17.7 (1.3)</td><td colspan="2">72.5 (7.7) 67.6 (32.4)</td><td>73.4 (13.5) 55.2 (37.2)</td><td>2.3 (1.3) 1.8 (1.1)</td></tr><tr><td>SPACE INVADERS</td><td>m0d0 m1d0 m1d1</td><td colspan="2">250.3 (16.2) 203.6 (24.3) 193.6 (11.0)</td><td colspan="2">698.8 (32.2) 753.6 (31.6) 698.5 (31.3)</td><td>927.1 (85.3) 979.7 (39.8) 906.9 (56.5)</td><td>243.6 (95.9) 192.6 (65.7) 180.9 (101.9)</td></tr><tr><td/><td>m9d0</td><td colspan="2">173.0 (17.8)</td><td colspan="2">518.0 (16.7)</td><td>567.7 (40.1)</td><td>174.6 (65.9)</td></tr><tr><td/><td colspan="2">GAME VARIANT</td><td>10M</td><td/><td>50M</td><td>100M</td><td>BEST ACTION</td></tr><tr><td colspan="2">FREEWAY</td><td>m0d0</td><td colspan="2">4.6 (5.0)</td><td>25.9 (0.6)</td><td>29.0 (0.8)</td><td>23.0 (1.4)</td></tr><tr><td colspan="2">HERO</td><td colspan="5">m0d0 2466.5 (630.8) 6505.9 (1843.0) 12446.9 (397.4) 150.0 (0.0)</td></tr><tr><td colspan="2">BREAKOUT</td><td>m0d0</td><td colspan="2">6.1 (2.7)</td><td>34.1 (1.8)</td><td>66.4 (3.6)</td><td>2.3 (1.3)</td></tr><tr><td colspan="3">SPACE INVADERS m0d0</td><td colspan="2">214.6 (13.8)</td><td>623.1 (16.3)</td><td>617.4 (29.6)</td><td>243.6 (95.9)</td></tr></table>

Baselines using dropout + ℓ 2 regularization for each default flavour.

Table content: <table><tr><td/><td/><td/><td>BASELINE</td><td/><td colspan="3">BASELINE W/ REGULARIZATION</td></tr><tr><td colspan="2">GAME VARIANT</td><td>10M</td><td>50M</td><td>100M</td><td>10M</td><td>50M</td><td>100M</td></tr><tr><td>FREEWAY</td><td>m0d0</td><td>3.0 (1.0)</td><td>31.4 (0.2)</td><td>32.1 (0.1)</td><td>4.6 (5.0)</td><td>25.9 (0.6)</td><td>29.0 (0.8)</td></tr><tr><td>HERO</td><td colspan="7">m0d0 3187.8 (78.3) 9034.4 (1610.9) 13961.0 (181.9) 2466.5 (630.8) 6505.9 (1843.0) 12446.9 (397.4)</td></tr><tr><td>BREAKOUT</td><td>m0d0</td><td>17.5 (2.0)</td><td>72.5 (7.7)</td><td>73.4 (13.5)</td><td>6.1 (2.7)</td><td>34.1 (1.8)</td><td>66.4 (3.6)</td></tr><tr><td colspan="2">SPACE INVADERS m0d0</td><td>250.3 (16.2)</td><td>698.8 (32.2)</td><td>927.1 (85.3)</td><td>214.6 (13.8)</td><td>623.1 (16.3)</td><td>617.4 (29.6)</td></tr></table>

Comparison of baseline results with and without regularization in the default flavour. The baseline agent with regularization was trained with dropout and ℓ 2 regularization.

Table content: <table/>

Peer review (iclr2019):
 I totally disagree with the authors that any of their observations are surprising. Indeed the fact that an RL agent does not generalizes to small modifications of the task (either visual or in the dynamics) is well known. If the agent should generalize though is a different question. And I do not mean this in the sense that it is an undesirable property but rather if it is outside of what “learning one task” means. Particularly I feel this is a very pessimistic view of RL and potentially not even in-line with what happens in supervised learning. I think one mantra of deep learning (and deep RL needs to obey by it) is that one should test in the same setting as training for things to truly work. For supervised, there is a distribution of data, and the test set are samples from the same distribution. However the testing scenario used here is slightly different. During training, if I do not see car accelerating, I think it makes no sense to expect to generalize to a new game that has this property as it is out-of-distribution. Of course it would be ideal if it could do that. And to clarify, while for us some of these extensions seem very similar and minimal changes, hence it should generalize to rather than transfer to, this is just the effect of imposing our own biases on the learning process. Deep Nets do not learn like we do, and in their universe they have never seen a car accelerating -- it makes sense that it might not to be able to generalize to it. Again, I’m not arguing that we don’t want this, but rather if we should expect it as part of what the system should normally generalize to. To that end I think this paper enters in that unresolved dispute of what generalization should be versus what is transfer. At what point do we have truly a new task vs a sample from the same task. I don’t think there is an answer. Going back to the observations in this work. I think the fact that the environment is not stochastic reinforces this overfitting (as in the extreme you end up with a policy that just repeats the optimal sequence of actions). I think in this particular case I can see how finetuning to a variation of the task fails. However true stochasticity in the environment (e.g. having a distribution of variations) like is done in Distral paper (where each episode is a different layout) can behave as a regularizer that will mitigate a bit the overfitting. That is to say that I believe the observed behaviour will be less pronounced in complex stochastic setting. Nevertheless the paper seems to highlight an important observation (and back it up with empirical evidence), namely we should use more regularization like L2 or otherwise in practice. Which is mostly absent from publications. And I think this on its own is valuable.