Generalization and Regularization in DQN

1 INTRODUCTION . Recently , reinforcement learning ( RL ) has proven very successful on complex high-dimensional problems , in large part due to the increase in computational power and to the use of deep neural networks for function approximation ( e.g. , Mnih et al. , 2015 ; Silver et al. , 2016 ) . Despite the generality of the proposed solutions , applying these algorithms to slightly different environments generally requires agents to learn the new task from scratch . Practitioners often realize that the learned policies rarely generalize to other domains , even when they are remarkably similar , and that the learned representations are seldom reusable . Deep neural networks , though , are lauded for their generalization capabilities ( e.g. , LeCun et al. , 1998 ) . Some communities heavily rely on reusing representations learned by neural networks . In computer vision , classification and segmentation algorithms are rarely trained from scratch ; instead they are initialized with pre-trained models from larger datasets like ImageNet ( e.g. , Razavian et al. , 2014 ; Long et al. , 2015 ) . The field of natural language processing has also seen successes in reusing and refining weights from certain layers of neural networks using pre-trained word embeddings , with more recent techniques able to reuse all weights of the network ( e.g. , Howard & Ruder , 2018 ) . In light of the successes of traditional supervised learning methods , the current lack of generalization or reusable knowledge ( e.g. , policies , representation ) acquired by current deep RL algorithms is somewhat surprising . In this paper we investigate whether the representation learned by deep RL methods can be generalized , or at the very least reused and refined on small variations to the task at hand . First , we evaluate the generalization capabilities of DQN ( Mnih et al. , 2015 ) . We further explore whether the experience gained by the supervised learning community to improve generalization and to avoid overfitting could be used in deep RL . We employ conventional supervised learning techniques , albeit largely unexplored in deep RL , such as fine-tuning ( i.e. , reusing and refining the representation ) and regularization . We show that a learned representation trained with regularization allows us to learn more general features capable of being reused and fine-tuned . Besides improving the generalization capabilities of the learned policies this fine-tuning procedure has the potential to greatly improve sample efficiency on settings in which an agent might face multiple variations of the same task . Finally , the results we present here also can be seen as paving a way towards novel curriculum learning approaches for deep RL . We perform our experiments using different game modes and difficulties of Atari 2600 games , a newly introduced feature of the Arcade Learning Environment ( ALE ; Bellemare et al. , 2013 ) . These game modes allow agents to be trained in one environment while being evaluated in a slightly different environment that still captures key concepts of the original environment ( e.g. , game sprites , agent goals , dynamics ) . This use of game modes is itself a novel approach for measuring our progress toward a longstanding goal of agents that can learn to be generally competent and generalize across tasks ( Bellemare et al. , 2013 ; Machado et al. , 2018 ; Nichol et al. , 2018 ) . This paper also introduces the first baselines for the different modes of Atari 2600 games . 2 BACKGROUND . 2.1 REINFORCEMENT LEARNING . Reinforcement learning ( RL ) is a problem where an agent interacts with an environment with the goal of maximizing some form of cumulative long term reward . RL problems are often modeled as a Markov decision process ( MDP ) , defined by a 5-tuple ⟨S , A , p , r , γ⟩ . At a discrete time step t the agent observes the current state St ∈ S and chooses an action At ∈ A to probabilistically transition to the next state St+1 ∈ S according to the transition dynamics function p ( s′ | s , a ) . = P ( St+1 = s ′ |St = s , At = a ) . The agent receives a reward signal Rt+1 according to the reward function r : S × A → R. The agents goal is to learn a policy π : S × A defined as the conditional probability of taking action a in state s written as π ( a | s ) . The learning agent refines its policy with the objective of maximizing the expected return , that is , the cumulative discounted reward incurred from time t , defined by Gt . = ∑∞ k=0 γ kRt+k+1 where γ ∈ [ 0 , 1 ) is the discount factor . Q-learning ( Watkins & Dayan , 1992 ) is a traditional approach to learning an optimal policy from samples obtained from interactions with the environment . It is used to learn an optimal state-action value function via a bootstrapped iterative method . For a given policy π we define the state-action value function as the expected return conditioned on a state and action qπ ( s , a ) . = Eπ [ Gt|St = s , At = a ] . The agent iteratively updates the state-action value function based on samples from the environment using the update rule Q ( St , At ) ← Q ( St , At ) + α [ Rt+1 + γmax a′∈A Q ( St+1 , a ′ ) −Q ( St , At ) ] where t denotes the current timestep and α the step size . Generally , due to the exploding size of the state space in many real-world problems , it is intractable to learn a state-action pairing for the entire MDP , with researchers and practitioners often resorting to learning an approximate to qπ . DQN approximates the state-action value function such that qπ ( s , a ) ≈ Q ( s , a ; θ ) , where θ denotes the weights of a neural network . The network takes as input some encoding of the current state St and outputs |A| scalars corresponding to the state-action values for that given state . DQN is trained to minimize LDQN = E St , At , Rt+1 , St+1 ∼U ( · ) [ ( Rt+1 +max a′∈A Q ( St+1 , a ′ ; θ− ) −Q ( St , At ; θ ) ) 2 ] where ( St , At , Rt+1 , St+1 ) are uniformly sampled from U ( · ) , the experience replay buffer filled with experience collected by the agent . The weights θ− of a duplicate network are updated less frequently for stability purposes . 2.2 SUPERVISED LEARNING . In the supervised learning problem we are given a dataset of examples represented by a matrix X ∈ Rm×n with m training examples of dimension n , and a vector y ∈ R1×m denoting the output target yi for each training example Xi . We want to learn a function which maps each training example Xi to its predicted output label ŷi . The goal is to learn a robust model that accurately predicts yi from Xi while also being able to generalize to unseen training examples . In this paper we focus on using a neural network parameterized by the weights θ to learn the function f such that ŷi = f ( Xi ; θ ) . We typically train these models by minimizing min θ λ 2 ∥θ∥22 + 1 m m∑ i=1 L ( yi , ŷi ) = min θ λ 2 ∥θ∥22 + 1 m m∑ i=1 L ( yi , f ( Xi ; θ ) ) where L is a differentiable loss function which outputs a scalar determining the quality of the prediction ( e.g. , squared error loss ) . The first term is a form of regularization , i.e. , ℓ2 regularization , which encourages generalization . ℓ2 regularization imposes a penalty on large weight vectors with λ being the weighted importance of the regularization term . Another popular regularization technique is dropout ( Srivastava et al. , 2014 ) . When using dropout , during forward propagation each neural unit has a chance of being set to zero according to a Bernoulli distribution with probability p ∈ [ 0 , 1 ] , referred to as the dropout rate . Dropout discourages the network from relying on a small number of neurons to make a prediction , making it hard for the network to memorize the dataset . Prior to training , the network parameters are usually initialized through a stochastic process ( e.g. , Xavier initialization ; Glorot & Bengio , 2010 ) . We can also initialize the network using pre-trained weights from a different task . If we reuse one or more pre-trained layers we say the weights encoded by those layers will be fine-tuned during training ( e.g. , Razavian et al. , 2014 ; Long et al. , 2015 ) . 3 THE ALE AS A PLATFORM FOR EVALUATING GENERALIZATION . The Arcade Learning Environment ( ALE ) is a platform used to evaluate agents across dozens of Atari 2600 games ( Bellemare et al. , 2013 ) . It has become one of the standard evaluation platforms in the field and has led to a number of exciting algorithmic advances ( e.g. , Mnih et al. , 2015 ) . The ALE poses the problem of general competency by having agents use the same learning algorithm to perform well in as many games as possible , while learning without using game specific knowledge . Learning to play multiple games with the same agent , or learning to play a game faster by leveraging knowledge acquired in a different game is much harder , with fewer successes being known ( e.g. , Rusu et al. , 2016 ; Kirkpatrick et al. , 2016 ; Parisotto et al. , 2016 ; Schwarz et al. , 2018 ; Espeholt et al. , 2018 ) . In this paper , we use the different modes and difficulties of Atari 2600 games to evaluate a neural network ’ s ability to generalize in high-dimensional state spaces . Game modes , originally native to the Atari console , were recently added in the ALE ( Machado et al. , 2018 ) . They give us modifications of the default environment dynamics and state space , often modifying sprites , velocities , and partial observability . These modes pose a tractable way to investigate generalization of RL agents in a high-dimensional environment . Instead of requiring an agent to play multiple games that are visually very different or even non-analogous , it requires agents to play games that are visually very similar and that can be played with policies that are very similar , at least from a human perspective . We use 13 flavours ( combinations of a mode and a difficulty ) obtained from 4 games : FREEWAY , HERO , BREAKOUT , and SPACE INVADERS . In FREEWAY , the different modes vary the speed and number of vehicles , while different difficulties change how the player is penalized for running into a vehicle . In HERO , subsequent modes start the player off at increasingly harder levels of the game . The mode we use in BREAKOUT makes the bricks partially observable . The used modes in SPACE INVADERS allow for oscillating shield barriers , increasing the width of the player sprite , and partially observable aliens . Full explanations of specific games , their modes , and their difficulties can be found in Appendix A . Figure 1 provides screenshots showing side by side comparisons of some of the modes explored in this paper . When reading the analyses of this paper it is important to keep in mind how remarkably similar these modes are .

Summary: This paper focuses on a "generalization" in deep Q-network (DQN). Specifically, they showed that when features (parameters of DQN) are trained in one environment (default flavour/mode) and then used as an initialization for the same model but for a slightly different environment ( i.e. still captures key concepts of the original environment ) can boost the performance of the model in the new flavour/environment. More importantly, the performance boost is significant when DQN's parameters which are used to initialize the model for the new environment were trained using dropout and L2 regularization in the default flavour/mode.  For the experiments, 4 games: FREEWAY, HERO, BREAKOUT, and SPACE INVADERS which have 13 flavours (combinations of a mode and a difficulty) are used.