Disentangling Generalization in Reinforcement Learning
Authors: Alex Lewandowski
Dale Schuurmans
Jun Luo
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement learning
generalization

Abstract:   Generalization in Reinforcement Learning (RL) is usually measured according to
  concepts from supervised learning. Unlike a supervised learning model however,
  an RL agent must generalize across states, actions and observations from
  limited reward-based feedback. We propose to measure an RL agent's capacity to
  generalize by evaluating it in a contextual decision process that combines a
  tabular environment with observations from a supervised learning dataset. The
  resulting environment, while simple, necessitates function approximation for
  state abstraction and provides ground-truth labels for optimal policies and
  value functions. The ground truth labels provided by our environment enable us
  to characterize generalization in RL across different axes: state-space,
  observation-space and action-space. Putting this method to work, we combine
  the MNIST dataset with various gridworld environments to rigorously evaluate
  generalization of DQN and QR-DQN in state, observation and action spaces for
  both online and offline learning. Contrary to previous reports about common
  regularization methods, we find that dropout does not improve observation
  generalization. We find, however, that dropout improves action generalization.
  Our results also corroborate recent findings that QR-DQN is able to generalize
  to new observations better than DQN in the offline setting. This success does
  not extend to state generalization, where DQN is able to generalize better
  than QR-DQN. These findings demonstrate the need for careful consideration
  of generalization in RL, and we hope that this line of research will continue
  to shed light on generalization claims in the literature.


Under review as a conference paper at ICLR 2022
DISENTANGLING GENERALIZATION IN REINFORCEMENT LEARNING
A Anonymous authors Paper under double-blind review
ABSTRACT
Generalization in Reinforcement Learning (RL) is usually mcasured accord- ing to concepts from supervised learning. Unlike a supervised learning model however, RL agent must generalize across states, actions and observations from limited reward-based feedback. We propose to measure an RL agent 's capacity to generalize by evaluating it in contextual decision process that combines tabular environment with observations from a supervised learn- ing dataset. The resulting environment, while simple, necessitates function approximation for state abstraction and provides ground-truth labels for optimal policies and value functions. The ground truth labels provided by our environment enable us to characterize generalization in RL across different axes: state-space, observation-space and action-space. Putting this method to work, we combine the MNIST dataset with various gridworld environments to rigorously evaluate generalization of DQN and QR-DQN in state, observation and action spaces for both online and offline learning- Contrary to previous reports about common regularization methods, we find that dropout does not improve observation generalization. We find, however, that dropout improves action generalization. Our results also corroborate recent findings that QR-DQN is able generalize new ob- servations better than DQN in the offline setting. This success does not extend state generalization, where DQN is able to generalize better than QR-DQN. These findings demonstrate the need for careful consideration of generalization RL, and we hope that this line of research will continue to shed light generalization claims in the literature.
1 GENERALIZATION IN REINFORCEMENT LEARNING
A Reinforcement Learning (RL) agent perpetually finds itself in novel states of enviromment. To act intelligently, the agent must generalize its previous experience to new situations. Function approximation helps distill this previous experience into the agent's learnable parameters, which allows previous experience to be leveraged with new state inputs (Boyan) and Moore 1994 Sutton 1995). While there growing literature of new methods for improving general liaaraaiiiiinniic of deep RL algorithms, principled and quantitative methods for evaluating generalization remain lacking. This is due in part the complexity of the MDP problem formulation and the difficulty of disentangling generalization from the performance of RL algorithms in terms of achieving higher expected cumulative reward, i.e. return.
One common notion of generalization in RL evaluates an agent's capabilities by checking can achieve similar performance in an environment that is similar to, but not exactly the same as, the environment in which it was trained (Whiteson et al. 2011p. This can be accomplished through randomization, iee. by randomizing the parameters underlying the environment, such as wind velocity in helicopter hovering the mass of objects in simulator, thereby changing the transition dynamics (Pong et al., 2017). Generalization in this sense draws parallels to supervised learning, where classifiers are often trained a fixed dataset, and evaluated on a separate testing dataset under the L.LD, assumption. The agent is said to generalize well if the difference between training error and testing error is small.

---

Under review as a conference paper at ICLR 2022
While these supervised learning concepts are relevant to RL, there are two problems with taking a cross-environmen approach to evaluating RL generalization. First, this paradigm does not disentangle the various aspects of generalization that are required for an RL agent to succeed in its task, whether it. be the value estimates or the policy, and whether they are robust to variations in states, observations performance criterion should be measured when we discuss generalization. In RL, we have function approximators for many quantities: state-transition, reward, state-value, action- value and policy. While gencralization of state-value is similar to regression, gencralization with quantities related to action, such as policy or oetion-value, do not have supervisod learning analoguesand henee T pe and actiooryallees bave as maauy. outputs as actions, and only the action laken by the agent is updated, which necessitate separale consideration.
actions. This is the question of "what"
Second, the paradigm follows practice in supervised learning to impose a strict separation of test and train environments. This practice effectively focuses transfer performance across environments but fails to evaluate the agent's 's ability to generalize within a single environment. Le. it fails to answer the question of "how" to measure generalization under a performance criterion. Unlike supervised learning, the agent's ' state distribution changes during learning because the policy changes. Even where have randomly generated training and testing sets of environments, the environments' complex dynamics do not admit ground-truth labels to determine optimal actions or values for comparison and evaluation. These environments only allow an agent's generalization capabilities to be measured in terms of Monte- Carlo rollouts of the learned policy, leaving us restricted to the single performance criterion of return and preventing us from specifically mcasuring important muanced differences in the agent's 's ability to generalize. In addition, because the upper bound on return is unknown to the researcher, informed judgments about the quality of the policy is very difficult to make.
Within the RT-oriented literature on generalization, there are two distinct categories of research. The first proposes environments and methodologies for measuring RL gen- eralization. Examples of this approach are ProcGen CoinRun (Cobbe et ab. 2020 ?)
[Cobbe et al. 2020 2019), sandomized-rewaarr CartPole (Zhang et al.] 2018ah, the Grid- World maze (Zhang et al., 2018b), observation projection (?) SSoog ct alj 2020) and the hierarchy of state generalization (Witty et al., 2018). These efforts are aimed at the second issuc raised above, i.c. "how" generalization should be measured in RL. Un- derlying these approaches is specifying how to split the environment. into testing and training scenarios. Early work by Zhang el al. (2018a) proposes using separate seeds. This, however, does not ensure that the states encountered by the agent are truly separate. Other works, such as those by (Cobbe-et al. (200)); Elsayed et al. (2020)
Cobbe ct al. (220 2019): Elsaved et al. (020), procedurally generate separate environ- ments for testing and training. When generalization is measured on truly separate testing and training environments, we are able to determine whether an agent's policy is generalizing from one enviromment to another. Again, this formulation does not allow us to study generalization within a single environment, nor does it allow us to measure generalization of the value functions. The second category proposes investigates RL methods that improve generalization. These include regularization experiments in Atari (Faebrothher
let al. 2018) and continuous control (?) [in et al.) 2021), ) contrastive similarity embeddings
[Agarwal et al. 2021) and bisimulation metrics (?) W et al. 2021h. b. There is also the
hypothesis that better, and hence more generalizable, representations arise from auxiliary tasks Jaderberg et al.) (017),, which is also suspected to be the reason for the success of distributional RL (2Bellemare et al. (017)) [Dabucy et al. 2018 Bellemare ct al 2017)
and has recently been investigated in the offline setting (?). Agarwal et al. 2020
Previous work elosest te-ours---ee- ours is using Contextual Decision Processes (CDPs) as To understand generalization in RL, we use the Contextual Decision Process
Process(CDPP) (CDP) framework, which is a problem class for theoretical analysis of RL algorithms that use function approxi- mation (Du et al. 2019: Jiang et al.) 2017 Dann et al.] 2018). The CDP problem formulation renders the states unobservable, but allows the agent to view observations that contain enough information to recover the state. This formulation enables the study of function approxima- tion as applied to RL and has significant implications for RL in general. For example, it helps to Du et al.) 2019 show that there exist algorithms with exponentially more efficient explo-
2

---

Under review as a conference paper at ICLR 2022
ration than Q-learning Du et designed to take advantage of function approximation and its generalization abilities, rather than naively extending tabular algorithms with function approximation. However, no existing work has leveraged CDPs empirically, despite that it. connects supervised learning with RL.
2019 a result suggesting that RL algorithms should be
The MNIST gridworld 2D MNIST maze environment used by ? Lee et al. (2019) may be considered a CDP, simple CDP Chewwhere where the observations are deterministie and eguivalent to state, but is not recognized as such. The work by 2 Song al. (2020] proposed projecting the state of simple control environment and varying this projection between training and testing sets. These two works have similar goals to ours. However, the first work does not investigate generalization and neither makes use of the ground-truth values to probe general- ization rigorously, suchh in label label- corruption experiment by (Zhang et aba 2017 Hhal-we will extend to RT. Our work is uniquely analogous to genera generalization in supervised learning. simultaneously answering "how" generalization should be measured in RL and "what" should be measured. In answering theses two quesstoos we disentangle generalization across three axes: axcss_states, states, cobseryatiosss and actions.. Finally, this discntangled perspective. shows how. different generalization mechanisms benefit. the different axes of generalization.
we
2 A PROBLEM FORMULATION FOR EVALUATING GENERALIZATION IN RL
To make generalization in RL concrete, we first make clear the Markov De- cision Process (MDP) problem formulation that underpins task specification in RL ( LLLttinore and Spepesvári 2020 White 2017 Lattimore and Szepesvári (020). An No: 7), where A denotes the action space, 8 is the state space, r S x A + I is the reward function that maps a state and an action to a reward, T: S A - [0. 1] the state transition function, Ho is the initial state distribution and E [0, 1] is the discount factor. A policy that interacts with the MDP is defined by a mapping from states to a distribution over actions π S - A(A), which includes deterministic policies that concentrate the probability mass on single action. The discounted state visitation distribution for following the policy n,, starting in state 8o and taking action ag, is defined as d do (s) (1 7) > Pr (st = 8 80: 0o, ax+1 TT, 8141 T(s,, at)) (Agarwal et al. 2019). II action dg is not given, then we define d=, (8) = Bag-~ Bao-m[uda (+)]. For any given policy, we define the state-value and action- value functions as expectations over the discounted state visitation distribution
MDP M is delined by the tuple (8, No:
(1) We denote the optimal policy that maximizes the expected cumulative reward (i.e. return) by x²" arg max, Ey-wo [s)]. Lastly, we use the shorthand v (*)s)= = (s) and q" (s) qa- (s,a) for the value of the optimal policy (Sutton and Barto 2018
Un (8)
1 1-7 Bag ((S,,a)), 0= 1-7 Egg aa al |r(s', aa))
2.1 THE CURRENT PARADICM: GENERALIZATION ACROSS MDPs
Studying generalization in RL is challenging in part due to the MDP problem formulation. As an agent interacts and learns in an environment, which is assumed to be an MDP, the agent's policy changes and it contimuously visits new states. Due to the changing policy, it is not possible to forbid an agent from entering a state that a practitioner would like to use later for testing. To evaluate RL agents' ability to generalize, one can instead use generalized environment E - =O.p) that provides a distribution P over a collection of environments θ { (Whiteson et al. 2011). It is then the responsibility of the environment designer to ensure that there is similarity among the environments M; such that generalization is possible. Yet, the environments must also be different enough to avoid inadvertentl training on experience that is later tested. The state and action spaces, for example, should not change between the environments. Lastly, one can think of the distribution over environments p distribution over the distribution of start states μ4 for cach Mi.
'We use the continuing formulation in Equation I for ease of reading. The equation still holds in the episodic setting by replacing the factor of I- by the expected length of the episode [T] (Bojun] 2020). In our experiments, use the empirical distribution.
3

---

Under review as a conference paper at ICLR 2022
Even with generalized environments, such as procedurally generated games (Cobbe et al. 2020), we are unable to probe the details of an agent's generalization ability during learning. By measuring only the Monte-Carle return of the learned policy an unseen environment instance, we are restricting the evaluation criterion to on-policy control in this new environ- ment instance. Even if we accept that this is the performance criterion that matters, we can only evaluate the relative performance of the policy because we typically do not know the optimal policy, Lastly, this evaluation protocol cannot evaluate the agent's 's ability to generalize in a single environment. Harnessing generalization within a single environment as important as transfer to new environments, and was the original goal for combining reinforcement learning with function approximation (Sulton 1995
Boyan and Moore 1994
Tesauro 1995
Murphy 2005).
2.2
PROPOSED PARADIGM: EVALUATING GENERALIZATION USING CDPs
The Contextual Decision Process (CDP) adds an observation space (also referred to as context) O to the MDP, and renders the state-space unobservable to the agent (Jiang et al. 2017p. Obscrvations are provided to the agent by an emission function Φ: O x S I that samples observation E O al a stale E S. The reward and stale transitions are still defined as functions of the unobservable states 8 E S, and the agent responsible for learning how to map the observations to states. We write the set of observations associated with a specific state s as O, = fo : o (o) = s}, for some decoding function o 6-1, In this work, we assume that the observation space is disjoint: each state a subset of the observations O2 C C where O, O. = 0 for 8 8 s'. The disjoint property ensures the existence of the decoding function d=1 that maps each observation to single state. Recent literature introduced the term "Block MDP" to refer to CDP with the disjoint property (Du et al. 2019). The experiments in this paper only use Block MDPs, but this is to focus on the generalization problem without introducing partial observability.
S is associated with
While CDPs have been the subject of theoretical study, no work has investigated its use as an evaluation platform for studying generalization. To accomplish this, we outline how a CDP can be combined with a supervised learning datasct allow for generalization across observations. To generate observations, we choose to use images from a supervised learning dataset. Given finite state MDP, we use a K-way classification dataset {(r,,,:) y:)) with x; E Idd and labels y: E {1,.... , KJ. If the total number of classes K is equal to the number of states |8] in the MDP, then each state can be uniquely identified with a class label, yielding the observation sets O. {z;; y = s}. If the states have structure, such as if each state is an xy-coordinate in gridworld environment =
concatenate then the emission function the images corresponding to each component, O, {[x,, 91 s(1), ./ 2(2) J. Using the provided training and testing split of supervised learning datasets, one can then use training data during learning and testing data during evaluation. With this formulation, generalization to new observations can be studied within a single environment. In the next section, we discuss how to extend this idea to states and actions.
CHARACTERIZING GENERALIZATION IN RL
With the CDP problem formulation, we can now reexamine the differences between gener- alization in supervised learning and RL. In supervised learning, measuring generalization requires a training distribution Peraiuy testing distribution Diest and performance criterion g which is low if the function approximator ff is performant. The training, and testing set. same population distribution, but we maintain and sample from soparate training www.eest.teeeeeeeeeeeeeeeeeeeeee sets and denote themyby whe.eeeeroor own oww.dwstribuuueeeoeoo distribution, We say that function approximator I exhibits good generalizatio if the generalization gap,
come from the same
GsL(S) = =[s, Drest) - I(ʃ, Dirain)
is small. When the training error can be minimized to zero, in supervised learning with over- peee-arameterized neural networks, generalization can be characterized by the testing performance alone. As discussed in Section 2.1 generalization in RL is often measured by
4

---

Under review as a conference paper at ICLR 2022
the difference in value achieved by policy over distribution of training instances Ptrain and testing instances Plest from a generalized environment E,
(RI.-MD((TT) = B Eepperaco [s)] - Ea-penal
(3)
where the expectation and the value are both estimated using Monte-Carlo methods. Note that a higher Ur is more performant, and so we flip the order in which test and train appear in the gap so that G 2 O. While Equations 2 and seem otherwise similar, they have a number of differences. First, the performance criterion in RL has no ground-truth label for comparison. An RL agent allempls to maximize the value, but the true optimal value function v" is unknown in most environments used to study generalization. A randomly initialized function will exhibit similarly poor performance during testing and training and hence the generalization gap will be small. In supervised learning, we can detect that, although the generalization gap is small, the error is high (or performance is low, in the case of value or accuracy). Second, there is difference in the distributions that are used in the expectation. A supervised learning model Iearns to produce certain outputs, conditioned on certain inputs. The model is then tested on held out inputs. An RL agent evaluated in the current paradigm produces actions conditioned on states. However, it is then evaluated on entire environments where it chooses actions and produces its own next states as input. Lastly, there is a potential mismatch in the performance criterion. The value of the policy is not. the explicit objective being optimized by most reinforcement learning algorithms. This is not an issue in supervised learning because the optimization objectives are calibrated surrogates (Chen et al. 2019).
The two issues that we explicitly address is the discrepancy in the measuring distributions and a lack of ground-truth labels. These two issues make operationalizing the supervised learning notion of generalization difficult in RL. Using the CDP formulation from Section 2 we are able to alleviate these two issues. First, we are able to sample from any distribution over states, observations and actions, meaning that we do not have to explicitly rollout the policy for evaluation. Second, we can use dynamic programming with the unobserved latent states to recover the optimal value and policy, and use those as comparison the learned value and policy. One question remains: what performance criterion should we use to test an agent's 's ability to generalize?
Re-examining the current. paradigm for measuring generalization in RL, we see how the CDP approach can help remedy the compromises made when learning in complex environments. An RL agent's 's task is to maximize return and hence, most RL research reports the returns that an agent receives from interacting with the environment. Due to the the lack of ground- truth labels, the performance critcrion g is limited to the Monte-Carlo estimate of e*. While Monto-Carlo returns do demonstrate policy performance, it does not probe the agent in its estimates of the actual values being learned, and is only one part of the generalization puzzle. Given the optimal policy and action-values an agent can be evaluated with more acuity than Monte-Care
returns. One possibility is to measure the accuracy of the policy, where accuracy is given by the agreement of the learned policy with the true optimal action. Another possibility is to measure the mean squared error between the learned action values and the true optimal action values. While the optimal policy is defined as a function of state, action-values can be evaluated was not taken in a particular state, we can evaluate the agent's 's estimate of that particular action-value. Thus, we consider three types of generalization: state, observation, and action generalization. To case the notation, we define the joint training distribution over state, (s)our (ols) (a, with
arbitrary actions. This means that even if an action
observation and action, starting at state 8o as Pi..t a) ()), the testing distribution defined similarly. The second policy n' is often the same as æ, but this second degree of freedom will be needed for action generalization. Using this shorthand, we define the generalization gap of a function approximator ʃ with respect to performance criterion g as,
GRL-CDP(f) = B (o a') O's',o'''' - -B (e al) of lo(s' ,', a', f)].
(4)
2 As described in Section
2.1 P is a distribution over the distribution of start states for each
environment.
5

---

Under review as a conference paper at ICLR 2022
In the next two subsections, we will investigate each component of this joint distribution and its connection to generalization across each of the three axes: state, observation and action. We will also show examples of performance criteria g that can be minimizedmeasuueed during traannin,,,n other than the scaled negative reward g(s', o d', π) - ,'), that may be of interest when studying generalization in RL using CDPs. Other criteria, are necessary because the scaled negative reward, which defines value under stationary state distribution, is trajectory dependent and conflates generalization across states, observations and actions.
3.1 OBSERVATION AND STATE-SPACE GENERALIZATION
Observation-space generalization is the most commonly explored notion of generalization in RI. (sce, i.e. ASnng et al.] 2020 ) Given an observation o, sampled from the test distribution Otest that the agent has not seen before, the agent is evaluated
quantities related to that observation. In model-free algorithms that learn action-values parameterized by 0, for example, an agent may be evaluated on the mean squared error of its value of the greedy policy, Imse (o, s, a, 0) = (o(o; (s))?. We focus on action- untlon--nooniiionnll quantities to separate observation-space generalization from action-space generalization that we will discuss in the next section. Another example of action unconditional performance criterion is the accuracy of the greody policy induced by q parameterized by 0, Jace (0, s, a, 0) I (arg max, q(0, a; 0) T" (s)) where I is the indicator function.
There is often no distinction between observations ions and states in the current RL literature because, except in toy problems, the underlying MDP (and hence, the true Markovian state) is not known. However, there are important differences. The state transition and reward are functions of the state, hence generalization across states must necessarily generalize to different state-transition and reward functions. For online algorithms, mcasuring state generalization not straightforward because the agent does not put weight on states outside of its state visitation distribution d=, (s). Expecting an agent to generalize to an arbitrary state outside of this distribution similar to the zero-shot generalization problem, and would require meta-learning approach. More reasonable would be to expect the agent to generalize to states near the distribution di, (s). One way this can be achieved is if the observation of (tear(11s) of state that has not been encountered by the agent s' ~ deg.test shares some structure with observations approximator f can generalize to this new state via its observation space gencralization. It remains to construct an environment with naturally unrcachable states, for which the observation structure permits generalization, such as in certain gridworld setups (Witty el al. 2018
(tr(- of seen states d= test Then, the function
3.2 ACTION-SPACE GENERALIZATION
Generalization to out-of-distribution actions is less explored notion compared to observation or state-space generalization. To evaluate action-space generalization, the agent soutd should be evaluated on quantities related to actions that were not taken that have low probability under the current policy. In model-free algorithms that learn action-values parameterized by 0 for example, the agent can be evaluated on its estimate of the action-value for an action a not ever taken by the agent in state s, and hence not taken in any of it's observations E O., (oct(s, 0, f) (q(o, f) 9 (s. One way this can be formalized is by setting T' to put all of its support on actions outside of the support of π. This setting is particularly important for offline learning, where the л would be the behavior policy and nt is the larget policy we are trying to learn.
Control is one of RL's main differences from supervised learning, wherein the agent collects the data that it uses to learn. We argue that action-space generalization is more important than observation or state-space generalization because policies in RL tend to place non-zero probability on all actions for exploration purposes. If the agent chooses to explore when faced in an unknown situation, it must leverage its ability to generalize in action-space. It is thus surprising that action-space generalization has seen limited investigation. The work of Chandak et al. (2019): Dulac- Arnol et al. (205)) finds that learned action representations
6

---

Under review as a conference paper at ICLR 2022
Training Accuracy- Agent (Chrikdor, Office)
Trsinin Accusssy Boy (Creridrr Oflind
Troiking Accuracy Drgpout (Cherstor, Ofline)
02
250
2:01
1000
1800
132
Accurscy Gup- Agei (Cretila, Oilinn!
Accerncy Cap
(Crenderr OBline]
Aouurary Gep Drapout (Corida,, Ofllie)
01g
8
300
PPC
300
1086
2003
7300
30.0 530
150
1r0m
Naxbe Grudiess Shiee
Nonbor Gradict
Naxber Stipe
Figure 1: Observation generalization experiment after offline learning with uncorrupted labels the corridor MNIST CDP. Top: Train accuracy. Bottom: Difference bet ween testing and training accuracy. Righ-Right: best agent, regularization and dropout rate respectively.
Training Accunsry - Agent (Camidar, Ouline!
Trining Acturayy Bog (Curraler, Online)
Taining Jocuracy Dropout (Cunrióu: Online)
Naaber Spbokk
Naxber Eoloddo
Frrder of Ejtoda
Accumay Cap Agent (Curridor, Online!
Acrurny Cap- (Cherider Orllnne)
Accursay Gap Dropout (Corriirr Online))
04
so 020
Aucune Go
100
118
100
150
352
83
2.30
Figure 2: Observation generalization experiment after online learning with uncorrupted labels in the corridor MNIST CDP. Top: Train accuracy. Bottom: Difference between testing and training accuracy. Left-Right: best agent, regularization and dropout rate respectively.
are able to improve the policy's performance in environments with large discrete action spaces. While the authors claim that the learned representation improves performance, there no specific study of this finding. Furthermore, no work has explicitly measured an agent's ability to generalize to an unseen action.
4 EXPERIMENTS
All of our experiments are conducted in a contextual decision process, where the observations are derived from the MNIST dataset (LeCun et al., 2010) and the underlying MDP is
either a corridor environment or a four rooms gridworld (Suton et al. 1999). See the Appendix for details on ench environment, including state, action and the edeeriemeelttal protocol, as well as the enviromments.
speees-
While these two environments are relatively simple, they provide thorough testbed for probing different aspects of generalizat ion for an RL agent. From the agent's perspective, the problem of mapping an observation to state alone is challenging. Early in training, each observation is unique and so the agent does not know if it is making progress until reaches the goal. Even after reaching the goal, it is not immediately clear how many unique states were encountered in the trajectory. The agent must do this for each state, and leurn that the learn the mapping from observation to action-value, and the fact, that action-values in these-at at different observations are the same if the underlying state is the same. Note that the optimal policy in the corridor environment is the classification policy. However, the classification policy must be learned from sparse reward signal that occurs only at the end of the episode. Taken together, the agent must generalize across different states, observations and actions from limited experience.
end
7

---

Under review as a conference paper at ICLR 2022
We focus on off-po algorit namely DQN Mnih et al 2013 and Quantile-Regressee DQN (Dabney et al. 2018 A because their ability to learn from arbitrary data necessitates generalization, Our goal for each experiment is to rigorously evaluate generaliza- tion across states, observations and actions in different RL training regimes, and with different generalization mechanisms. We een statsttatee.... observations saa actions can all be partitioned into training, validation and testing sets - in addition to observation generalization, we alse investigate in our environment sctup..... We identify. throe mechanisms that have b mltpmmelmmed..mm.mmmmmm to improve obseryation Borooogggggegannnn generalization, in RL: RI:LQRRDQL QR-DQN Agarwal et al. 20201. ...as.
do this because the observations
..aas well as 122 regulaaiizaaio aoaadd dropout (I (Larebrou her et al. 2018 Lg el al 2019 Cobbe el al. 2019
We begin by rigorously evaluating these previous claims of observalion generalization We ion then investigate the elfects of these generalizal ion mechanisms
action generalization in the in the four rooms environment, We feeus-on eff poliey algerithms, namely DQN (Muil et anj 2013)] end Quantile Regression DQN (?), because their ability to learn from arbitrary data necessitates goneralizationals introduce corrupted variants of these environments, where the class labels in the datasct are shuffled, wow. wheree - you sepeeve mome lo we.test..u the agent's ability. wtw.://eeeemmmmmm..ee memorize jt'ss........ S.raaiiiiin input. We evaluate both olf policy model free algerithiss all combinations of RL algorithm and generalization mechanism after po
corridor environment and on state-generalizatinn
either online and offiine training. In the online setting, the agent will interact with the environment while updating its policy. truth quantities pertaining to the- environment offline setting, the agent will learn from a logged set of experience in replay buffer without any direct interaction with the environment during learning. We also introduce corrupted variants of these environments, where the eluss lobels in the dataset are shufiled to test the-agenn's ability te memorize it's training imput. Fer all experiment it non corrupted environments, we experiment with L2 regularization and dropout, as they are common regularization techniiues in machine learning and some have reported that these techniques improve generaiizon........ionn....
the-agent- will be periodieally evaluated against tend-eest-ee observations. In the
4.1 OBSERVATION GENERALIZATION
To measure observation generalization, we train an agent using observations from the training set and evaluate on held out test-set. We only show the results for the corridor environment here, but have additional experiments on four rooms in the Appendix. Referring to the offline learning experiments depicted in Figure D we see that QR-DQN is able to generalize better than DQN, as shown in its lower accuracy generalization gap. We also find that small amounts of regularization indeed lowers the generalization gap, whereas dropout does not improve observation generalization. In the online learning experiments, shown in Figure 2 QR-DQN and DQN both generalize to new observations similarly. Interestingly, the generalization gap is overall larger in the online setting. We also find that setting the regularization strength to 0.1 now inhibits learning. Small amounts of regularization still improves generalization to new observations however. Lastly, we see that dropout hurts learning in the offline setting.
Drawing inspiration from the work of Zhang al. 22017 we also run an experiment where the labels are randomized to see if DQN and QR-DQV have similar capacity to memorize the data. Referring to Figure 3. we find that DQN and QR-DQN (with a various nunber of quantile heads) can easily memorize all the training experience in corridor. In the online setting, larger QR-DQN agents begin to struggle to fit the training data. This difficulty also extends to both the offline and online setting in four rooms, where the observations are higher dimensional.
4.2 ACTION GENERALIZATION
For action generalization, we focus on the corridor environment with offline learning from a behaviour policy that systematically avoids certain action in each state. We then measure the action value estimate for the state-action pair that is never seen in the data and compare it to the true action-value. Referring to Figure 4 we find QR-DQN is better able to generalize
8

---

Under review as a conference paper at ICLR 2022
Training Accuracy Agent (Corridor, Offline)
Training Accuracy Agent (Corridor, Online)
1.0
ORDONGO GRNNNI
ORTIINO ORIMPNS
0.8
Accurang
Dgg
Accuracy
P
0.6
0.6
0.4
0.4
0.2
0.2
2500
5000
7300
10000
100
200
300
500
600
Numaber of Gradienl Stepe
Numbor Expioodes
Training Accuracy Agent (Four Rooms, Offline)
Training Accuracy Agent (FourRooms, Online)
1.0
ORDOOROD QRIORIN
ORDgNNO
0.8
Accurecy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
2500
3000
7300
10000
200
300
500
600
Numaber of Gradient Steps
Numbor Epinodee
Figure 3: Corruption experiments where all MNIST labels are randomly reassigned to test whether an agent can memorize its experience. Top Left: Corridor Offline. Top right: Corridor online. Bottom Left: Four rooms offline. Bottom right: Four rooms online
Ttaining MSD
Agent (Comtáor, Offiize)
MSK Reg (Curiitor, Offinn)
Traiting MSE 500
Deopout (Corrióo Otfime)
245E
250
500
1231
100e
120
1000
250
5530
7000
1000
Suge
MSB
(Coeridoe, Ocfline
MSE por Cap Boy (Corridor, Otiline)
(Coridoe,
MATL
SESE
130
1000
530
120
Maaber of (Zuthel Sage
Nastor Gredit Stge
Neabec Gradinn Stgge
Figure 4: Action generalization with offline learning from policy that never takes action a = $ - in state s > Top: Average Training MSE for action-value q(s, - 1). Bottom: Difference between testing and training MSE. Left-Right: best. agent, regularization and dropout rate respectively.
9

---

Under review as a conference paper at ICLR 2022
Well Toikiing Accurcy- Ageni [RorrBocuus Officee)
Wall Training Boonno Oillin)
Wall Tralning Accurecy Dogoou (Homrooom, Offine
2:01
108
533
Wall Accurnry Gup- Ageii (SarRRoons Offins)
Accunacy Gap Bog (Panffoums, Ofline)
Wall Ausuracy Cap Drpouii (FourRecnss Offine)
Org
(.00
000
000
6000
0000
-000
306
200)
2031
1086
200
5000
3003
52.m
2031
2003
Trom
Nanker Gradicce Sapa
Stage
Figure 5: State generalization experiment after offline learning with uncorrupted labels in the four rooms gridworld CDP. Top: Train accuracy. Bottom: Difference between Lesting and training accuracy on wall states. Left-Right: best agent, regularization and dropout. rate respectively.
Wall Training Assurcyy - Agai (Porroomm, Onlline)
Wall Triinin Arcuney Dripont (FonrRooms Ouline
2
05
04
03
10E Apet (Fourfiooms, Oninc)
00
100
100
200
Will Accuncy
Wall Anrurary Gop. Reg (Rouriooms,, Onlito)
Wall Aonurazy Gep Dropont (Poorlooess, Online)
6:10
Cap
-000
100
250
200
116
200
Stadon Rass
Figure 6: State generalization experiment after online learning with uncorrupted labels in the four rooms gridworld CDP. Top: Train accuracy. Bottom: Difference between testing and training accuracy wall states. Ligh-Right: best agent, regularization and dropout rate respectively.
to unseen actions. We also find that, contrasting to observation generalization, dropout now improves action gencralization whereas regularization does not.
4.3 STATE GENERALIZATION
We use the four rooms environment to evaluate agent's ability to generalize to new states. The interior walls of the four rooms environment share either an x or coordinate with states that the agent has encountered, and hence the student can leverage generalization between obscrvations to these new states, We evaluate the accuracy of the policy in these interior wall states. Referring Figure 5 and Figure for offline and online training respectively, see similar trends. In both cases, DQN is able to generalize to the new states better than QR-DQN. We also find that, despite lower generalization gap in some setting, both regularization and dropout huri training accuracy. This means that, although the training performance is more predictive of its testing performance, it's overall testing performance is lower.
4.4 DISCUSSION
Our results have validated previous findings..mamellll namely QR-DQN's superiority in the offline regime Agarwal et al. 2020) a. the benefit of small amounts of L2 regularization Farebrother al 2018
1 [gl et all 2019 Cobbe et al. 2019 We also discover the nuanced benefit of dropout, which provides regularization in the outputs (actiontttooi (action-values) and show that. it benefits getionngeneralizationn Our findings also show that QR-DON's success in
10

---

Under review as a conference paper at ICLR 2022
observation gen.rraizzairiiiinnnnn wwweeoootttteeeooeoo not translateee to success in state generalization, where the mean mean estimate of DQN is superior to guantile regression.
Our experiments require privileged information about the MDP and the ability to do dynamic programming onn underlying states,,arg cannot be conducted on conventional euvironments, These insights. however, however,_rquuny require our environment sctup and ground truth
- quantities, but do not pose any restriction on the RL algorithu, The validation of previous lindings suggest that the contextual decision process environment studied in this paper, are -nd - - reflective of conventional envirouments, such as starii and CoinRun. The genera generalization iization challenge posed by environments is further evidenced by stthee significant performance differences between training and testing generalization gap plats, bottom row of Figures 1:2,000 acroscoseeo. +-crosss all experiments, By first demonstrating that our experimental setup confirms many_previoon previous findings, we are led to believe that our new findings would generalize to conventional environments, as well.
CONCLUSION
Generalization in RL is multi-faceted issue that deserves rigorous empirical investigation. To that end, we have presented a problem formulation using contextual decision processes and supervisd learning datasets that provides ground truth labels for optimal policies and values. Using that disentauggg..gencgliaatiio sencralization across stw.sssttttttsttssss obscrvations and actions in single environment, By combining various gridworld environments and the MNIST dataset, investigate aseneralinatienn.eccoo grooovergoouuu-rruul grownd-Lruth Jabels for corepal policies and values through dynamic, programming, and investigate generalization mechanisms across states, observations and actions. We validate previous claims regarding the effectiveness of L2 regularization and QR-DQN's generalization ability in the offline setting, but find that QR- DQN cannot generalize to states well DQN. We also refute claims, such dropout's ability to improve obscrvation generalization, but find that it improves action generalization. These findings demonstrate the need for careful consideration evaluation of generalization in RL, and we hope that this line of research will continue to shed light on generalization claims in the literalure-,, well as new generalization mechanisms.
11

---

Under review as a conference paper at ICLR 2022
REFERENCES
Agarwal, A., Jiang, N., and Kakade, S. M. (2019). Reinforcement learning: algorithms. Preprint.
Theory and
Agarwal, R., Machado, M. C., Castro, P. S., and Bellemare, M. G. (2021). Contrastive behav- ioral similarity embeddings for generalization ion in reinforcement learning. In International Conference on Learning Representations.
Agarwal, R., Schuurmans, D., and Norouzi, M. (2020). An optimistic perspective on offline reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 104-114. PMLR.
Bellemare, M. G., Dabney, W., and Munos, R. (2017). A distributional perspective on reinforcement learning. In Precup, D. and Tch, Y. W., editors, Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Leurning Research, pages 449 458. PMLR.
Bojun, H. (2020). Steady state analysis of episodic reinforcement learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 1835-9345, Curran Associates, Inc.
Boyan, J. A. and Moore, A. W. (1994). Generalization in reinforcement learning: Safcly approximating the value function. In Proceedings of the 7th International Conference on Newral Information Processing Systems, NIPS'94, page 369 376, Cambridge, MA, USA. MIT Press.
Chandak, Y., Theocharous, G., Kostas, J., Jordan, S. M., and Thomas, P. S. (2019). Learning action representations for reinforcement learning. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 941-950. PMLR.
Chen, M., Gummadi, R., Harris, C., and Schuurmans, D. (2019). Surrogate objectives for batch policy optimization one-step decision making. Advances in Neural Information Processing Systems, 32:8827 8837.
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. (2020). Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings PMLR.
of Machine Learning Research, pages
Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. (2019). Quantifying gen- cralization in reinforcement learning. In Chaudhuri, K. and Salakhutdinov. R., editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1282 1289. PMLR.
Dabney, W., Rowland, M., Bellemare, M. G., and Munos, R. (2018). Distributional reinforce- ment learning with quantile regression. In Mcllraith, S. A. and Weinberger, K. Q., editors, Proceedings of the Thirty-Secone AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovotive Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advonces in Artificial Intelligence: (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 2892 2901. AAAI Press
Dann, C., Jiang, N., Krishnamurthy, A., Agarwal, A., Langford, J., and Schapire, R. E. (2018). On oracle-efficient PAC RL with rich observations. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., desa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information. Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pages 1429 1439.
12

---

Under review as a conference paper at ICLR 2022
Du, S. S., Krishnamurthy, A., Jiang, N., Agarwal, A., Dudik, M., and Langford, J. (2019). Provably efficient RL with rich observations via latent state decoding. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1665-1674. PMLR.
Dulac-Arold, G., Evans, R.,, Hasselt, H. v., Sunchag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., and Coppin, B. (2015). Deep reinforcement learning in large discrete action spaces. XXi::112.076%
Elsayed, M., Hassanzadeh, K., Nguyen, N. M., Alban, M., Zhu, X., Graves, D., and Luo, J. (2020). Ultra: A reinforcement learning generalization benchmark for autonomous driving.
Farebrother, J., Machado, M. C., and Bowling, M. (2018). Generalization and regularization in dqn. arXiv:1810.0011
Tgl. M., Ciosek, K., Li, Y., Tschiatschek, S., Zhang, C., Devlin, S., and Hofmann, K. (2019). Generalization in reinforcement learning with selective noise injection and information bottleneck. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Bue F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. (2017). Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nes net.
Jiang, N., Krishnamurthy, A., Agarwal, A., Langford, J., and Schapire, R. E. (2017). Contextual decision processes with low bellman rank are pac-learnable. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1704-1713. PMLR.
Lattimore, T. and Szepesvári, C. (2020). Bundit Algoritlems. Cambridge University Press.
LeCun, Y., Cortes, C., and Burges, C. (2010). Mnist handwritten digit database. ATT Labs |Online/. Available: htlp://yamn.err waam.lecuu.ccm. /endb/ /exdb/mmissl 2.
Lee, S. Y., Choi, S., and Chung, S. (2019). Sample-effciien deep reinforcement learning via
episodic backward update. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' d'Alchó- Buc, F., Fox, E. B., and Garnett, R., editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pages 2110 2119.
Liu, Z., Li, X., Kang, B., and Darrell, T. (2021). Regularization matters in policy optimization an empirical study on continuous control. In International Conference on Learning Representations.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. 131::13322.5000
Murphy, S. A. (2005). A generalization error for g-learning. Journal of Machine Learning Research, 6(37):1073 1097.
Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbecl, P. (2017). Sim-to-real transfer of robotic control with dynamics randomization. arXiv:1710.0655""
Song, X., Jiang, Y., Tu, S., Du, Y., and Neyshabur, B. (2020). Observational overfitting in reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. Reve
13

---

Under review as a conference paper at ICLR 2022
Sutton, R. S. (1995). Generalization in reinforcement learning: Successful examples using sparse coarse coding, In Proceedings of the 8th International Conference on Neural Information Processing Systems, NIPS'95, page 1038-1044, Cambridge, MA, USA. MIT Press.
Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning on introduction. Adaptive computation and machine learning. MIT Press.
Sutton, R. S., Procup, D., and Singh, S. (1999). Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181 211.
Tesauro, G. (1995). Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58-66) 69.
White, M. (2017), Unifying task specification in reinforcement learning. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Mochine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3742-3750. PMLR.
Whiteson, S., Tanner, B., Taylor, M. F.,, and Stone, (2011). Protecting against evaluation overfitting in empirical reinforcement learning. In 2011 IEEE symposium adaptive dynamic programming and reinforcement learning (ADPRL), pages 120 127. IEEE.
Witty, S., Lee, J. K., Tosch, E., Atrey, A., Littman, M., and Jensen, D. (2018). Measuring and characterizing generalization in deep reinforcement. learning. Xrv: 02868.
Zhang, A., Ballas, N., and Pineau, J. (2018a). A dissection of overfitting and generalization in continuous reinforeement learning. orXio:1806. 306.07937.
Zhang, A., McAllister, R. T., Calandra, R., Gal, Y., and Levine, S. (2021). Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Zhang, C., Vinyals, O., Munos, R., and Bengio, S. (2018b). A study on overfitting in deep reinforcement learning. orXiv: orXiv:1804.0680 06893.
14

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1
EXPERIMENT SETTINGS
Code to reproduce experiments is provided in the following an anonymized dropbox link:
https:/ wwwp//ppbbw..oom.//o pj7dvl4fr70lppe// AAAAAssa
Online
Online Corrupted
Oflline
Offline Corrupted
Action Gen
Init. Num Episodes
50
50
50
50
50
Trit. Policy
Random
Random
Random
Random
Custom
Max Episode Length
100
100
10000
10000
10000
Max Num Episodes
300
300
100
100
100
Num Online Episodes
250
500
Num Grad Steps
(per env step)
1 (per env step)
10000
20000
20000
Target Update Freq
128
128
128
128
128
Activalion
Relu-ReLU ReLU
Relt-ReLC
ReLU
Relu 256
ReltReLU
Hidden Size
256
256
256
256
Num Layers
2
2
2
Optimizer
ADAM Adam
ADAM- 128
ADAM-Adamm 128
ADAM ADAM-Adam
Adam
ADAMAdam
Batch Size
128
128
128
Gamma
0.99
0.99
0.99
0.99
0.99
Corruption Rate
0
1.0
0.0
1.0
Table 1: Shared Settings for both environments, but "Action Gen" is limited to the corridor CDP. Each experiment's hyperparameters are also shared between the two models and not swept over. Custom Policy: Random except never take action A = S in state S 1.
A.2 ENVIRONMENT SPECIFICATION
The corridor environment has states and 10 actions corresponding to the possible classifi- cation decisions the policy can make in each state. The agent begins at the right-most state of the corridor, with s s= and observes handwritten digit from the MNIST dataset. The agent must correctly classify the digit to
up to the next state. For all states, the action a = s sends the agent up a state s = s + 1. For a state > 1, the action a = 10 - s sends the agent down state s' = 1. All other actions keep the agent in the same state, but with a new observation. The reward for all transitions are -1 and the episode terminates 10 is taken in 10. The four rooms environment has four actions for
when action each of the cardinal directions and the observation is a concatenated images corresponding to the class labels for each of the and coordinate. If the agent tries it will stay in its current state with a new observation. Lastly, the rewards are initialized randomly (with fixed seed) ensure that the optimal action in each state is unique.
into wall,
0
1
2
3
4
5
6
8
9
T
Figure 7: An illustration the MNIST Corridor environment, Each state is labeled by number, its index, and observations from MNIST are sampled according to this state. The initial state is state 0, and the agent is given a randomly sampled observation from MNIST corresponding to the class of handwritten zeros. There are 10 actions, one for each possible "classification" decision that must be learned through reinforcement learning, The terminal state, should by a yellow square with the letter ends ends the episode after the correct action
Fach
sxpoooe is Laken in state.9.
15

---

Under review as a conference paper at ICLR 2022
00
01
02
03
04
05
06
07
08
10
11
12
13
14
15
16
17
18
20
21
22
23
24
25
26
27
28
30
31
32
33
34
35
36
37
38
40
41
42
43
44
45
4
47
48
50
51
52
53
54
55
56
57
58
60
61
62
63
64
65
66
67
68
70
71
72
73
7
75
7
T
78
80
81
82
83
84
85
86
87
88
Figure 8: An illustration of the MNIST FourRooms enviroument, Each state is labeled by two numbers. The observation given to the agent is a concatenation of random sample from MNIST for each number in the label. For cxample. the the initial state the initial obserration a concatenation of two jmages p MUST corresponding to the class of handwrilten ones. _ Although the label number is the same for the initial state, the observations can be the same or different depending on the stochastic sample. The white states are accessible to the agent but the blue states correspond to wall states which cannot be transitioned into from a white state. For state generalization lizaaiiiiion,, we evaluate the agent wose wall states in the interior that can transition to white states: (14, 34, 41, 43, 45, 47, 54, 74).
[nd so
A.3 AGENT SPECIFICATION
Both online agents
greedy exploration with c 0.1. QR-DQN computes 10 quantiles
and is optimized with respect to the Huber loss, with & = 1.0. DQN is optimized with respect to the squared error. The representational architecture between the agents is the same, and is shown Table
A.4 HYPERPARAMETRS SETTINGS AND EXPERIMENT PROTOCOL
The ablation setings settings for each experiment are detailed the legend. We For every evereeeeeeerrtteeeeeeeeeeee experiment, we also conducted grid scarch over learning rates a (0.005, 0.001, 0.0005, 0.0001}. All experiments are averaged over 30 seeds, with shaded region corresponding to 95% confidence interval of the mean. The best model was seleeted results shown in each figure profile a w.w.ticulargnneerliiaattet generalization mechanism (DQN ys QRDQN. L2 Regularization, (rtpiiaatioondroouutt/.. Each plot selects the best set of hyperparameters based sose
validation accuracy, which acts as a prexy injjeeooy-indeeenddent criterion for policy perfor- mance the training environment instance. This provides an optimistic but fair assessment
16

---

Under review as a conference paper at ICLR 2022
of each generalization mechanism. In the corruption experiments, the best model was selected based on training accuracy.
For both the corridor and four rooms environment, label to ensure that the agent can fit its training experience, but we do not limit the size of the test and validation sets. The regime where the training, data is unbounded is not particularly intoresting because there, would be little difference difference_ between the training and testing distributions. An RL agent, in only a limited sne - momanns
only 10 samples from each class
of episodes, will only see small fraction of the training, observations, with the rest never observed (and functionally, part of the observation test setl.,.. This is unlike a supervised learning. problem, where all the data is provided ut B without the need for interaction To make the analogy, to supervised learning, wereere required to lower the number oraining samples so that the agent, would encounter large proportion of the training observations in a limited number of episodes. This interpolative regime is also the regime of interest for the theoretical study of generalization in neural networks.
We use dynamic programming to calculate the optimal policy and value function for the stales, and choose lo report accuracy instead of Monte-Carlo rollouts. Accuracy provides more acuity in the differences between policies, allowing us to evaluate the accuracy at each state and under any distribution. All performance criteria (training, validation and testing) are averaged over the same state distribution as the replay buffer.
17

---

Under review as a conference paper at ICLR 2022
A.5 OBSERVATION GENERALIZATION IN FOUR ROOMS
Traninn Accuracy Aget [Reurloooms, Otthine)
Training Acturacy Rog [PourRonss, Offine
Itaining Accursyy Drogout (Fouritomms Offiae)
0.2
290
7802
1886
230
200
1240
1000
2530
132
150
1000
9tmp
Agenz (FerrBocnss Offine)
Accuracy Gup Rog, (PouRRatts, Offiind]
Accuracy
Dropout (Founoons, Ocfine)
Jesnny Ogg
Arasaoo DN
296
Dime
2001
2000
132
10000
Noxker of Gootlca Suyee
Nanber al Gredizt Sup
Ntabez of Grodint Strge
Figure 9: Observation generalization experiment after offline learning with uncorrupted labels in the fourrooms MNIST CDP. Top: Test accuracy for best agent, regularization configuration and droprate configuration. Boltom: training accuracy for each of best agent, regularization configuration and droprate.
Triniing Accuracy Agent FourBomms, Online)
Training Acturury Rag (FrurRoona, Onlind)
Triting Accuncy- Dropoui (ForRosmms Online)
0.2
100
124
53
100
2.0
Sglaske
Nacher Rainnka
Accursayy Cap- Agnnt
Online))
Accuracy Gap Bog (Puirlloome, Onitne)
Cap Daignn (Faurioome,, Online)
Aoneso De
&
Nanibr of Tpircde
Narkon Riiiabe
of Toloob
Figure 10: Observation generalization experiment after online learning with uncorrupted labels in the fourrooms MNIST CDP. Top: Test accuracy for best agent, regularization configuration and droprate configuration. Bottom: training accuracy for each of best agent, regularization configuration and droprate.
18

---

Under review as a conference paper at ICLR 2022
A.6
MONTE- MORLE-CARLO ROLLOUTS IN TRAINING AND TESTING ENVIRONEENTS
MC Reeeen
Agent (Cortóorr Office))
Reg (Domioo,, (ffiee)
MC Beturn
Drepost (Cutstor, Offan))
200
138
7091
1886
2530
200
350
.000
2522
3291
700
1000
Undda Bugy
Reten
Aguzt (Curldor. Office))
MC Relum Rog (Cankte, Offiis))
Return Decpout (Cottion, Offins)
230 Nander of Gradica Sugg
Dine
132
2001
10000
Jusbe ef Cradita Sugee
Ntaber of Gredich Brego
Figure 11: Observation generalization experiment after offiine learning with uncorrupted labels in the corridor MNIST CDP. Top: Train MC return. Bottom: Test MC Return. Left-Right: best agent, regularization and dropout rale respectively.
Agent (Coridoor, Unline)
MC Retura Rog (Corsion, Onlan)
Despout (Cunidar, Online)
Mtstt Botten
Frendr Iotedde
Numbe ligeede
Dkzaber
MC Rotum Agenn (Coridor, Online)
MC Tog Onlon)
MC Raturn Deepont (Curridor Online)
Mcsse Daata Daacer
30
100
130
200
Figure 12: Observation generalization experiment after online learning with uncorrupted labels in the corridor MNIST CDP. Top: Train MC return. Bottom: Test MC Return. Legh-Right: best agent, regularization and dropout rate respectively.
MC Ratun Agezz (FourZooms, Oflim))
Beg (Ranffomm Oilim)
MC Raturn Dropout (Fuzzooums, Offinnd
Par
3.20
4003
1000
1006
380
30000
500
201
1000
1000
Staboe Grdize
Sage
Staater Gradker Stea
(Fanr/oone, Ollinee
3BC Returi Reg (Iomiiooms Ofllon)
MC
Dropont (Fnrroooss, Offline)
Stazs-Dul Batten
Ntrdee Dradent Saye
of Currer Stye
Figure 13: Observation generalization experiment after oflline learning with uncorrupted labels in the fourrooms MNIST CDP. Top: Train MC return. Bottom: Test MC Return. Left-Right best agent, regularization and dropout rate respectively.
19

---

Under review as a conference paper at ICLR 2022
Agern (Fourilooo,
380 Retura Rog (RonRoons Ouline)
Dropont (FomrRooms Online)
Seeta-Cul Boturn
Shanddsh
of Epiede
Nacker Spteade
MC Retur Agens (Kaurooou, Ouline)
(FouRoums Oaline)
MC Return Deoport (Pounoonss, Ouline)
183
3/
Observation generalization experiment after online learning with uncorrupted labels in the fourrooms MNIST CDP. Top: Train MC return. Bottom: Test MC Return. Reft-Right: best agent, regularization and dropout rate respectively.
Figure 14:
Return Agent (Corridor, Offline)
MC Return Agent (Corridor, Online)
Monte- Carto Recurn
-20
UHLLENOI L
-20
UILIGAAR AIUI-ENII
-40
-10
-60
-00
Moste
2500
5000
7500
10000
100
200
300
400
500
600
Number Gradient
Number of Episodes
MC Return
Agent (FourRooms, Offline)
MC Return
Agent (Pour Rooms, Online)
Monte- Carto Return
-50
QRIDONSE
-50
POPNESS DOIN
DOI
-75
-75
-100
-125
-125
10000
Number of Gradient Steps
Number Episodes
Figure 15: Corruption experiments where all MNIST labels are randomly reassigned to test whether an agent can memorize its experience. Top Left: Corridor Offline. Top right: Corridor online. Bottom Left: Four rooms offline. Bottom right: Four rooms online
20


Reviews:
[{"review_id": "vhJ0RzV3s1y", "reviewer": "Reviewer_74nY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has multiple strong points. Authors propose a new perspective to evaluate generalisation in RL agents based not on accumulated rewards as most research usually do now, but through a careful evaluation of the generalisation performance of the different elements of an MDP. From the theoretical point of view, most of the mathematical constructions are drawn from previous works (well referred). Still, this work presents a new empirical formulation to better study generalisation. \nMy main concern is that this paper is not the first pointing to change in this manner the way we evaluate generalisation and, the empirical method prosed here is very limited, in the sense that it requires full knowledge of the state space and transition dynamics, e.g., it uses dynamic programming to recover optimal value and policy. Additionally, the empirical approach used in this work, mapping the states to observations of the MNIST dataset, illustrates that this whole evaluation is carried on relatively small hand-crafted environment. It leaves me doubtful if the results obtained here, e.g., that dropout only helps action generalisation, would really translate to more visually rich and complex domains. Note that authors also state that the RL algorithms were memorising the training data, I wonder what would happen if this was not the case.\n\nMinor mistake/suggestions:\n* End of page 2: “The MNIST gridworld environment used by Lee et al. (2018) may be considered a CDP, but is not recognized as such” why?\n* There where some points of the literature review that became obvious only after reading the whole work, it might be beneficial moving this section to the end\n* Section 2, authors don’t present the bellman equations in standard form but as expectations, it would be beneficial for the reader to anticipate here why this is relevant in this work.\n*Section 2.1 second paragraph, it is not clear to me why rewards cannot be a good way to measure generalisation in the environment, (if you don’t know if your policy is optimal, only way to reduce uncertainty when you don’t have full access to the MDP dynamics is to keep checking new policies to whether one of them produces better reward)\n*Section 4, a figure of the environments showing how they are used with the MNIST dataset, would help here\n* Section 4.1, line 4. Figure 10 should be Figure 1\n* Figure 1, it is not specified which of the two RL algorithms is used while evaluating the regularisation procedures. \n", "summary_of_the_paper": "This paper proposes a new methodology to test the generalisation performance in reinforcement learning. Authors study two  (non state-of-the-art) methods (DQN and QR-DQN) and analyse how L2 regularisation and dropout affect the ability of the algorithms generalising to new observations, states and actions (each of them independently)  ", "summary_of_the_review": "Authors explore a non-standard method to measure generalisation in RL applying it empirically to two deep RL algorithms. Despite of authors multiple points against using rewards as generalisation metric, I am still not convinced that the struggles of RL with respect to supervised learning, which is the absence of the ground truth answer, is tackled by the proposed method. Authors rely on heavy assumptions of the environment, allowing them to have access to the \"ground truth\", i.e., optimal policies and value. This is not the case in most problems about RL generalisation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dSfVQS9gKjx", "reviewer": "Reviewer_1S93", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the theme of the paper, and I indeed agree that designing the right metrics for quantifying generalization in single-task RL are needed and are not quite present in the literature. However, I think the paper falls short of delivering on and convincing the reader that the empirical analysis is useful and generalizable, and the proposed metric is a good one. I will explain why:\n\n1. It is unclear how these observations transfer to other RL problems. Would testing generalization (within a single task) on other domains, e.g., Atari get the same results? Why or why not? For example, if the claim is that dropout doesn't help, does it not help on this task, or does the analysis also support why it shouldn't help on other domains?\n\n2. One good point about converting SL problems to RL problems is that it is easy to measure generalization against a groundtruth SL oracle. With that in mind, the performance of RL agents is worse, so one argument is that everything is generalizing poorly on an absolute scale. Comparing with such metrics should have been more informative, compared to just DQN and QR-DQN, and this is a limitation.\n\n3. In the offline setting, it is known that better algorithms that are more uncertainty-aware and do not commit to a single target value, such as ensemble DQN, random ensemble mixture, QR-DQN, etc work much better compared to the DQN, because they can be more accurate on OOD predictions. Does the action generalization experiment say essentially this? What is the new insight behind this high-level known result?\n\n4. What insights should I take away as to why one algorithm is worse than the other in each of the different settings? Does it indicate that some principle is behind the relative performance differences? A detailed empirical analysis of what about QR-DQN and what about DQN makes them perform differently in these different generalization challenges would shed light on this. Similarly, a rigorous empirical study of why dropout helps or why L2 helps, etc, seems necessary to me to take away from this paper.\n\n5. How does the accuracy translate to metrics I can measure during training, such as some kind of loss, validation error, etc? This is essential in understanding how one should modify algorithms to make them generalize better. \n\n", "summary_of_the_paper": "This paper proposes an approach to measure to quantify generalization properties (state generalization, observation generalization, action generalization) in single-task RL in the context of offline RL. The paper discusses the limitations of several existing approaches for measuring cross-environment generalization, then presents their generic measure of generalization (Equation 4), and evaluates generalization when learning from offline data using a DQN and QR-DQN in a contextual decision process (CDP) problem created out of MNIST classification. The results suggest that dropout is effective for action generalization, not state generalization, and L2 penalty is effective, and QR-DQN can generalize better than DQN in the offline setting, but worse in terms state generalization. ", "summary_of_the_review": "Without presenting any analysis of the **why** question, it seems that the work is not complete in my opinion. Also, the generalizability of these findings to other domains is under question. If these two major concerns can be addressed, especially along the lines of the pointers above, I am happy to revise my score. But for now, I would vote for rejection of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O5Gk_sbD-hp", "reviewer": "Reviewer_iELT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- The paper discussed an important problem in RL. The writting is clear and easy to follow. \n\n- The exprimental results are abundunt and there are enough repeats (30) to ensure statistical significance. The plots are well presented.\n\nMajor Concerns:\n\n- The largest concern is that the experiments are limited to simple grid world envrionments with an unnatural setting to use MNIST images as observation. In particular, the authors wrote in Chap 2.2 \"If the total number of classes K is equal to the number of states |S| in the MDP, then each state can be uniquely identified with a class label\". This assumption restricts the scope of this study to discrete and relatively small state space. Since the results are mostly empirical, it remains unclear how these results generalize to more practical and real-world RL tasks.\n\n- The novelty and significance of technical conritbution is limited. Either the concept of CDP or using performance gap between train and test set as evaluation criteria is not original. The being studied L2 regularization and dropout have also been discussed in Cobbe et al, 2018 (actually should be 2019 as the publication year of ICML).\n\n- Although the experimental results are clear and easy to understand, I expect a more comprehensive empirical investigation, e.g.,  visualizing the learned policy in your envrionments (as in Cobbe et al, 2018), and perform experiments in more tasks. \n\n\nOther issues:\n\n- In the end of Chap 4.1, \"Referring to 14,\", what is 14? Do the authors mean Figure 14?\n- In \"Figure\" 7 (should be a table): Relu -> ReLU, ADAM -> Adam.\n- The spatial interval between Figure 5 and 6 is too narrow. \n- In chap 4, before 4.1, it was written \"some have reported that these techniques improve generalization.\". Are there some references?\n- References are not professional, please check the venue of publications instead of just citing the arXiv version.\n\n", "summary_of_the_paper": "This paper discusses generalization in deep RL. The key contribution of the paper, from my understanding, is that the authors argue that different from generalization in SL, in RL state, observation and action should be considered separately. A measurement (Eq.4) is proposed to evaluate generalization capacity of deep RL within the contectual decision process (CDP) scheme. Experiments were performed on grid world environments with MNIST image as observations and several results were concluded. \n", "summary_of_the_review": "Overall, the paper discussed an significant topic in deep RL and their paradigm for evaluation is clearly presented and empirically investigated. However, the current study restricts itself to simple grid world envrionments with an unnatural setting to use MNIST images as observation. While the main contribution of this paper is empirical, it remains unclear how these experimental results generalize to more practical and real-world RL tasks. Also, the technical novelty and significance is limited. In sum, I consider the current paper fail to meet the acceptance criteria of ICLR and recommend a major revision, probably by performaning more comprehensice investigation on more tasks with preferrably continuous state space.\n\n\n--------- Post Rebuttal -----------\n\nThe authors have addressed most of my concerns, and I increased my score by one level accordingly. However, a core limitation of the current work, \"do the results/insights also apply to more realistic environments with continuous/large state space?\", is not resolved. While I believe this work has its potential, currently I lean toward weak reject.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I0sH_QXrsZq", "reviewer": "Reviewer_NqjA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the ideas in this paper are presented clearly and it is globally well-written. I like the distinction the authors made to distinguish generalization across their three axis is interesting and although they have been looked at separately in the literature, their juxtaposition and evaluation procedure in these different regimes are as far as I know novel. It also enables to measure generalisation in a single task setting and not across MDPs unlike some other works.\n\nOriginally, in statistical learning theory [Vapnik, 1995], the training and test distributions are the same in the definition of the generalisation gap (train error - test error) and the distribution only differs on the test in transfer learning so it would be good if you added a reference for your definition or precise why you are using a different definition. \n\nDo you have some intuitions about why DQN and QR-DQN generalise similarly in the online setting on the observation space? or an hypothesis why DQN generalise better than qr-dqn on the space axis?\n\nI think the empirical measure of an agent generalization capacity would be more convincing and realistic on more complex environment like some from the Atari game for instance. An empirical evaluation on more environment would also strengthen the paper. It would be interesting to also have results for other agents.\n\n“While generalization of state-value is similar to regression, generalization with quantities related to action, such as policy or action-value, do not have supervised learning analogues and hence require separate consideration. “ could you add more justification for this please?\n\nHere are a few minor points that did not affect my rating. Please use the proceedings links in your references instead of the arXiv links.", "summary_of_the_paper": "This paper proposes an empirical evaluation method to measure of the generalization capacity of an RL agent. It relies on CDPs combining a tabular environment with a supervised learning dataset. Generalization is measured across three axis: state space, observation space and action space. The empirical evaluation is led on DQN and QR-DQN on the four room domain and corridor domain combined with the MNIST dataset in the online and offline settings. The authors find that dropout improves action generalization but not observation generalization while regularisation improves observation generalization. They also find that QR-DQN generalise better than DQN in the offline setting on the observation axis and action axis but not on the state space axis.", "summary_of_the_review": "This work provides interesting insights comparing the generalization capacities of DQN and QR-DQN and their evaluation method corroborates some findings in the literature (e.g. about regularization) and some somewhat surprising results on the state generalization capacities of DQN vs QR-DQN. I think the paper would be stronger by providing results for other agents and more complex environments which is why I recommend a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to study the generalization capabilities of RL algorithms using contextual decision processes (CDPs). CDPs allows to study generalization similar to how we are used to studying generalization in supervised learning, and can separate the generalization capabilities of a learned agent wrt observation, state and action space. This proposed measure for generalization is used in an extensive study on grid world domains to evaluate existing algorithms that aim to improve generalization.

**Strengths**
This manuscript is well written and the work is well motivated
A novel perspective and way of measuring generalization of learned agents
An empirical study that compares existing algorithms on how well they generalize in observation, state, action spaces

**Weaknesses**
Some clarity issues existed (missing links to existing literature, experimental details) 
empirical study is (out of necessity) limited to small scale grid worlds
no deeper analysis of the results, why do algorithms perform the way they do from this novel perspective of generalization, which makes it hard to understand how one could choose an algorithm for larger scale settings which don't allow for this type of analysis

**Rebuttal**
The authors updated the paper to improve the parts that were unclear, and had an extensive discussion with reviewers on the intuition of the results and converging on take-aways. Unfortunately, this intuition and take-aways have not been added. 

**Summary**
While I understand the authors wish to not speculate on intuition, I agree with the reviewers that without (experimentally supported) take-aways the provided analysis is incomplete. Understanding why each algorithm achieves the performance they do wrt this novel way of measuring generalization is the only way the proposed method to measure generalization and the evaluation can be used to draw conclusions about more general problem settings. Thus, although this is a very promising direction on an important problem, the manuscript is not ready yet for publication.
Decision: Reject