Observational Overfitting in Reinforcement Learning

1 INTRODUCTION . Generalization for RL has recently grown to be an important topic for agents to perform well in unseen environments . Complication arises when the dynamics of the environments entangle with the observation , which is often a high-dimensional projection of the true latent state . One particular framework , which we denote by zero-shot supervised framework ( Zhang et al. , 2018a ; c ; Nichol et al. , 2018 ; Justesen et al. , 2018 ) and is used to study RL generalization , is to treat it analogous to a classical supervised learning ( SL ) problem – i.e . assume there exists a distribution of MDP ’ s , train jointly on a finite “ training set ” sampled from this distribution , and check expected performance on the entire distribution , with the fixed trained policy . In this framework , there is a spectrum of analysis , ranging from almost purely theoretical analysis ( Wang et al. , 2019 ; Asadi et al. , 2018 ) to full empirical results on diverse environments ( Zhang et al. , 2018c ; Packer et al. , 2018 ) . However , there is a lack of analysis in the middle of this spectrum . On the theoretical side , previous work do not provide analysis for the case when the underlying MDP is relatively complex and requires the policy to be a non-linear function approximator such as a neural network . On the empirical side , there is no common ground between recently proposed empirical benchmarks . This is partially caused by multiple confounding factors for RL generalization that can be hard to identify and separate . For instance , an agent can overfit to the MDP dynamics of the training set , such as for control in Mujoco ( Pinto et al. , 2017 ; Rajeswaran et al. , 2017 ) . In other cases , an RNN-based policy can overfit to maze-like tasks in exploration ( Zhang et al. , 2018c ) , or even exploit determinism and avoid using observations ( Bellemare et al. , 2012 ; Machado et al. , 2018 ) . Furthermore , various hyperparameters such as the batch-size in SGD ( Smith et al. , 2018 ) , choice of optimizer ( Kingma & Ba , 2014 ) , discount factor γ ( Jiang et al. , 2015 ) and regularizations such as entropy ( Ahmed et al. , 2018 ) and weight norms ( Cobbe et al. , 2018 ) can also affect generalization . ∗Work partially performed as an OpenAI Fellow . †Work performed during the Google AI Residency Program . http : //g.co/airesidency Due to these confounding factors , it can be unclear what parts of the MDP or policy are actually contributing to overfitting or generalization in a principled manner , especially in empirical studies with newly proposed benchmarks . In order to isolate these factors , we study one broad factor affecting generalization that is most correlated with themes in SL , specifically observational overfitting , where an agent overfits due to properties of the observation which are irrelevant to the latent dynamics of the MDP family . To study this factor , we fix a single underlying MDP ’ s dynamics and generate a distribution of MDP ’ s by only modifying the observational outputs . Our contributions in this paper are the following : 1 . We discuss realistic instances where observational overfitting may occur and its difference from other confounding factors , and design a parametric theoretical framework to induce observational overfitting that can be applied to any underlying MDP . 2 . We study observational overfitting with linear quadratic regulators ( LQR ) in a synthetic environment and neural networks such as multi-layer perceptrons ( MLPs ) and convolutions in classic Gym environments . A primary novel result we demonstrate for all cases is that implicit regularization occurs in this setting in RL . We further test the implicit regularization hypothesis on the benchmark CoinRun from using MLPs , even when the underlying MDP dynamics are changing per level . 3 . In the Appendix , we expand upon previous experiments by including full training curve and hyperparamters . We also provide an extensive analysis of the convex one-step LQR case under the observational overfitting regime , showing that under Gaussian initialization of the policy and using gradient descent on the training cost , a generalization gap must necessarily exist . The structure of this paper is outlined as follows : Section 2 discusses the motivation behind this work and the synthetic construction to abstract certain observation effects . Section 3 demonstrates numerous experiments using this synthetic construction that suggest implicit regularization is at work . Finally , Section 3.4 tests the implicit regularization hypothesis on CoinRun , as well as ablates various ImageNet architectures and margin metrics in the Appendix . 2 MOTIVATION AND RELATED WORK . We start by showing an example of observational overfitting in Figure 1 . This example highlights the issues surrounding MDP ’ s with rich , textured observations - specifically , the agent can use any features that are correlated with progress , even those which may not generalize across levels . This is an important issue for vision-based policies , as many times it is not obvious what part of the observation causes an agent to act or generalize . Currently most architectures used in model-free RL are simple ( with fewer than one million parameters ) compared to the much larger and more complex ImageNet architectures used for classification . This is due to the fact that most RL environments studied either have relatively simple and highly structured images ( e.g . Atari ) compared to real world images , or conveniently do not directly force the agent to observe highly detailed images . For instance in large scale RL such as DOTA2 ( OpenAI , 2018 ) or Starcraft 2 ( Vinyals et al. , 2017 ) , the agent observations are internal minimaps pertaining to object xy-locations , rather than human-rendered observations . 2.1 WHAT HAPPENS IN OBSERVATION SPACE ? . Several artificial benchmarks ( Zhang et al. , 2018b ; Gamrian & Goldberg , 2019 ) have been proposed before to portray this notion of overfitting , where an agent must deal with a changing background - however , a key difference in our work is that we explicitly require the “ background ” to be correlated with the progress rather than loosely correlated ( e.g . through determinism between the background and the game avatar ) or not at all . This makes a more explicit connection to causal inference ( Arjovsky et al. , 2019 ; Heinze-Deml & Meinshausen , 2019 ; Heinze-Deml et al. , 2019 ) where spurious correlations between ungeneralizable features and progress may make training easy , but are detrimental to test performance because they induce false attributions . Previously , many works interpret the decision-making of an agent through saliency and other network visualizations ( Greydanus et al. , 2018 ; Such et al. , 2018 ) on common benchmarks such as Atari . Other recent works such as ( Igl et al. , 2019 ) analyze the interactions between noise-injecting explicit regularizations and the information bottleneck . However , our work is motivated by learning theoretic frameworks to capture this phenomena , as there is vast literature on understanding the generalization properties of SL classifiers ( Vapnik & Chervonenkis , 1971 ; McAllester , 1999 ; Bartlett & Mendelson , 2002 ) and in particular neural networks ( Neyshabur et al. , 2015b ; Dziugaite & Roy , 2017 ; Neyshabur et al. , 2017 ; Bartlett et al. , 2017 ; Arora et al. , 2018c ) . For an RL policy with high-dimensional observations , we hypothesize its overfitting can come from more theoretically principled reasons , as opposed to purely good inductive biases on game images . As an example of what may happen in high dimensional observation space , consider linear least squares regression task where given the set X ∈ Rm×d and Y ∈ Rm , we want to find w ∈ Rd that minimizes ` X , Y ( w ) = ‖Y −Xw‖2 where m is the number of samples and d is the input dimension . We know that if X > X is full rank ( hence d ≤ m ) , ` X , Y ( . ) has a unique global minimum w∗ = ( X > X ) −1X > Y . On the other hand if X > X is not full rank ( eg . when m < d ) , then there are many global minima w∗ such that Y = Xw∗ 1 . Luckily , if we use any gradient based optimization to minimize the loss and initialize with w = 0 , the solution will only span column spaces of X and converges to minimum ` 2 norm solution among all global minima due to implicit regularization ( Gunasekar et al. , 2017 ) . Thus a high dimensional observation space with a low dimensional state space can induce multiple solutions , some of which are not generalizable to other functions or MDP ’ s but one could hope that implicit regularization would help avoiding this issue . We analyze this case in further detail for the convex one-step LQR case in Section 3.1 and Appendix A.4.3 . 2.2 NOTATION . In the zero-shot framework for RL generalization , we assume there exists a distributionD over MDP ’ s M for which there exists a fixed policy πopt that can achieve maximal return on expectation over MDP ’ s generated from the distribution . An appropriate finite training set M̂train = { M1 , . . . , Mn } can then be created by repeatedly randomly samplingM∼ D. Thus for a MDPM and any policy π , expected episodic reward is defined as RM ( π ) . In many empirical cases , the support of the distribution D is made by parametrized MDP ’ s where some process , given a parameter θ , creates a mapping θ →Mθ ( e.g . through procedural generation ) , and thus we may simplify notation and instead define a distribution Θ that induces D , which implies a set of samples Θ̂train = { θ1 , . . . , θn } also induces a M̂train = { M1 , . . . , Mn } , and we may redefine reward as RMθ ( π ) = Rθ ( π ) . 1Given any X with full rank X > X , it is possible to create many global minima by projecting the data onto high dimensions using a semi-orthogonal matrix Z ∈ Rd×d ′ where d′ > m ≥ d and ZZ > = Id . Therefore , we the loss ` XZ , Y ( w ) = ‖Y −XZw‖2 will have many global optima w∗ with Y = XZw∗ . As a simplified model of the observational problem from Sonic , we can construct a mapping θ →Mθ by first fixing a base MDPM = ( S , A , r , T ) , which corresponds to state space , action space , reward , and transition . The only effect of θ is to introduce an additional observation function φθ : S → O , where the agent receives input from the high dimensional observation space O rather than from the state space S. Thus , for our setting , θ actually parameterizes a POMDP family which can be thought of as simply a combination of a base MDP M and an observational function φθ , hence Mθ = ( M , φθ ) . Let Θ̂train = { θ1 , . . . , θn } be a set of n i.i.d . samples from Θ , and suppose we train π to optimize reward against { Mθ : θ ∼ Θ̂train } . The objective JΘ̂ ( π ) = 1 |Θ̂train| ∑ θi∈Θ̂train Rθi ( π ) is the average reward over this empirical sample . We want to generalize to the distribution Θ , which can be expressed as the average episode reward R over the full distribution , i.e . JΘ ( π ) = Eθ∼Θ [ Rθ ( π ) ] . Thus we define the generalization gap as JΘ̂ ( π ) − JΘ ( π ) .

Summary: The paper proposes a method for measuring generalization error in reinforcement learning (RL). The paper proposes to disentangle the observations into the features that relevant and not-relevant for the policy and then perturb the non-relevant features while keeping the relevant features constant. If the agent has learned to generalize, then perturbing the non-relevant features should not change the RL score.