Contextualize Me – The Case for Context in Reinforcement Learning
Authors: Carolin Benjamins
Theresa Eimer
Frederik Schubert
Aditya Mohan
Sebastian Döhler
André Biedenkapp
Bodo Rosenhahn
Frank Hutter
Marius Lindauer
Venue: Accepted by TMLR ()

Abstract: While Reinforcement Learning ( RL) has made great strides towards solving increasingly
complicated problems, many algorithms are still brittle to even slight environmental changes.
Contextual Reinforcement Learning (cRL) provides a framework to model such changes in
a principled manner, thereby enabling flexible, precise and interpretable task specification
and generation. Our goal is to show how the framework of cRL contributes to improving
zero-shot generalization in RL through meaningful benchmarks and structured reasoning
about generalization tasks. We confirm the insight that optimal behavior in cRL requires
context information, as in other related areas of partial observability. To empirically validate
this in the cRL framework, we provide various context-extended versions of common RL
environments. They are part of the first benchmark library, CARL, designed for generalization
based on cRL extensions of popular benchmarks, which we propose as a testbed to further
study general agents. We show that in the contextual setting, even simple RL environments
become challenging - and that naive solutions are not enough to generalize across complex
context spaces.

Published in Transactions on Machine Learning Research (06/2023)
Contextualize Me - The Case for Context in Reinforcement Learning
Carolin Benjamins* Leibniz University Hannover
c.benjomiinaaal..niii www.://uwwaauiiioooooeee.dddd
Theresa Eimer* Leibniz University Hannover
f. Ceimmer@aa.uiii-aamooeeee
Frederik Schubert Leibniz University Hannover
Aditya Mohan Leibniz University Hannover
a. a.mohan@ai. uni-hunnoenanaaeeeeeeee
Sebastian Döhler
de
Lesbniz University Hannover
André Biedenkapp University of Freibury
biedenka@css. Sondennkaoc..uui-freebuyy
Bodo Rosenhahn
rosenhahn Stnt, wwt..uniihnnnooee.ddee
Leihniz University Hannover
Frank Hutter University of Freibury
fhoBcs.uni-freee uni-freiburg.rcmm de
Marius Lindauer
m. milindauer@ai. hom-hrooeeeddu---ooooooe.eeedddd
Leibniz University Hannover
Reviewed on OpenReview: https: openreview. net/ forum? id- Y420
Abstract
While Reinforcement Learning (RL) has made great strides towards solving increasingly complicated problems, many algorithms are still brittle to even slight environmental changes. Contextual Reinforcement Learning (cRL) provides a framework model such changes in principled manner, thereby enabling flexible, precise and interpretable task specification and generation. Our goal is to show how the framework of cRL contributes to improving zcro-shot generalization in RL through mcaningful benchmarks and structured reasoning about generalization lasks. We confirm the insight that optimal behavior in cRL requires context information, as other related areas partial observability. To empirically validate this in the cRL framework, we provide various context-cxtended versions of common RL environments. They are part of the first benchmark library, CARL, designed for generalization based on cRI extensions of popular benchmarks, which we propose testbed to further study general agents. We show that in the contextual setting, even simple RL environments become challenging and that naive solutions are not enough to generalize across complex context spaces.
Equal Contribution
1

---

Published in Transactions on Machine Learning Research (06/2023)
Largel-tediig
gtayity
(dnfauit) "foint atinttatssEfeess" 6200,
distwnne
"gavitty -9.4, Iriction' a 4..4,
context Joint joint_ stiffness" $290,
actustor attergth
joint_ Finess argular swaping
(b) Variations in target distances
Iriction' 2.3.
Siictionn
Edntact
Contoxt
2
CARLFetchEnv
(a) Example of a contextual extension of Brax' Fetch (Freeman et al., 2021) as part of CARL
(c) Ground friction simulating grass, concrete and ice
Figure 1: CARL allows to configure and modify existing environments through the use of context. This context can be made visible to agents through context features to inform them directly about the current instantiation of the context (see a). Specific instances of CARLFetch with variations in goal distance (see b) and with different ground frictions (grass, concrete and ice, see c).
1 Introduction
Reinforcement Learning (RL) has shown successes in a variety of domains, including (video-)game playing (Silver et al., 2016; Badia et al., 2020), robot manipulation (Tee et al., 2020a; Ploeger et al., 2020), traffic control (Arel et al., 2010), chemistry (Zhou et al., 2017), logistics (Li et al., 2019) and nuclear fusion (Degrave et al., 2022). At the same time, RL has shown little success in real-world deployments that roquire generalization, rather focusing on narrow domains (Bellemare al., 2020; Degrave et al., 2022). We believe this can largely be explained by the fact that modern RL algorithms are not designed with generalization in mind, making them brittle when faced with even slight variations of their environment (Yu et al., 2019; Meng & Khushi, 2019; Lu et al., 2020).
To address this limitation, recent research has increasingly focused on generalization capabilities of RL agents. Ideally, general agents should be capable of zero-shot transfer to previously unseen environments and robust to changes in the problem setting while interacting with an environment (Ponsen et al., 2009; Henderson et al., 2018; Cobbe et al., 2020; Zhang et al., 2021b; Fu et al., 2021b; Yarats et al., 2021; Abdolshah et al., 2021; Sodhani et al., 2021b; Adriaensen et al., 2022; Kirk et al., 2023). Steps in this direction have been laken by proposing new problem settings where agents can test their transfer performance, e.g. the Arcade Learning Environment's 's flavors (Machado et al. 2018) or benchmarks utilizing Procedural Content Generation (PCG) to increase task variation, c.g. ProcGen (Cobbe et al., 2020), NetHack (Küttler et al., 2020) or Alchemy (Wang et al., 2021). Furthermore, robustness to distribution shift as well as multi-lask learning have been long-standing topics meta-RL, both in terms of benchmarks (Yu al., 2019; Sodhani et al., 2021a) and solution methods (Pinto et al., 2017; Finn al., 2017; Zhu al., 2020; Zhang et al., 2021d).
While these extended problem settings in RL have expanded the possibilities for benchmarking agents in diverse environments, the degree of task variation is often either unknown cannot be controlled precisely. We believe that generalization in RL is held back by these factors, stemming in part from lack of problem formalization (Kirk et al., 2023). In order to facilitate generalization in RL, cRL proposes to explicitly take environment characteristics, the so called contert (Hallak et al., 2015), into account. This inclusion enables precise design of train and test distributions with respect to this context. Thus, cRL allows us to reason about which types of generalization abilities RL agents exhibit and quantify their performance on them. Overall, cRL provides framework for both theoretical analysis and practical improvements.
In order to empirically study cRL, we introduce a benchmark library for Context-Adaptive Reinforcement Learning: CARL. CARL collects well-established environments from the RL community and extends them with the notion of context. To ensure interpretability, CARL considers context which is mainly based on physical propertics and thus intuitive to humans. For example, CARL extends Brax (Freeman et al., 2021) environments with properties such as friction, gravity, or the mass of an object (see Figure 1). Through CARL's interface, it is possible to meticulously define the context distributions which RT. agents trained and evaluated. We
2

---

Published in Transactions on Machine Learning Research (06/2023)
use our benchmark library to empirically show how different context variations can significantly increase the difficulty of training RI. agents, even in simple environments. We further verify the intuition that allowing RL agents access to context information is beneficial for generalization tasks in theory and practice.
In short, our contributions are: (i) We provide a theoretical and empirical overview of why Contextual Reinforcement Learning (cRL. is useful for research into zero-shot generalization for RI; (ii) We introduce our benchmark library CARL which enables fine-grained context control in benchmarking cRL; and (iii) We demonstrate that even on simple environments, generalization is challenging for standard RL agents.
2 Contextual Markov Decision Processes
In order to facilitate generalization, we first have to rethink how model the RL problem. While we could follow the common notion of modeling environments as Markov Decision Processes (MDPs), this way of modeling typically assumes a single, clearly defined environment. We believe this problem formulation is overly restrictive. Agents trained under such an assumption fail when the underlying environment does not behave exactly as they have experienced during training. Modeling the problem as a contextual MDP (cMDP) following Hallak et al. (2015); Modi et al. (2018) instead, we assume that there are multiple related but distinet environments which an agent might interact with and which can be characterized through context. This notion of context provides us with the means necessary to study the generalization abilities of RL agents in principled manner.
What is Context? To help build intuition on what context is and how it might influence the learning problem, we first give an informal treatment of context. In essence, context characterizes how the environment behaves and what its goals look like. In contrast to the observations of an MDP, which describe the changes to the environment step by step, context allows us to reason about how the state will evolve without requiring access to the true transition and reward functions. Further, context features are typically static (i.e., do not. change during episode) or change at a much slower time scale than state observations.
In a robot, for example, joint friction could inform an RL controller how much torque to apply to execute some desired action. In the short horizon, the friction will not change. However, especially if the robot is not well maintained, the friction can increase due to mechanical degradation with the need to adapt accordingly. Context information can now help to appropriately compensate. Another example could be different payloads as context for a robot or different winds for a helicopter (Koppejan & Whiteson, 2009). Note that such a feature does not need to provide the exact change in transition dynamics, rather it needs to provide signal of how transition (and reward) functions relate to each other.
Context does not need to influence reward and transition functions at the same time, In goal-based reinforcement learning (c.g., Schaul et al., 2015; Eysenbach et al., 2019), the notion of goals inffuences the reward function, typically without changing the transition dynamics. For example, an agent needs to traverse an empty gridworld. A goal that is placed to the right of the agent yields high rewards by moving closer. Tf another goal is now on the left of the agent, the reward for actions switches without changing how the agent traverses the grid (i.e. the transition function).
By its nature, context enables agents to learn more discriminative and thus more general policies. This makes context very well suited to study generalization of RL agents (Kirk et al., 2023). However, context has yet to be widely explored or leveraged in reinforcement learning, and there are many open questions to be addressed. A prominent one among them is how to use context. during learning. We discuss contextual RL more formally in Section 4.
Contextual Markoy Decision Processes (cMDP) (Hallak et al., 2015; Modi al., 2018) allow to formalize generalization across tasks by extending the standard definition of an MDP in RL. An MDP M = (S, A. R., p) consists of state space S, an action space A, transition dynamics T, reward function R and a distribution over the initial states Through the addition of context, we can define, characlerize and parameterize the environment's rules of behavior and therefore induce task instances as variations on the problem. In the resulting cMDP, the action space A and state space stay the same; only the transition dynamics Te, the reward R. and the initial state distribution Pe change depending on the context c C. Through the context-dependent initial state distribution Pe, as well as the change in dynamics, the agent may
3

---

Published in Transactions on Machine Learning Research (06/2023)
furthermore be exposed to different parts of the state space for different contexts. The context space C can either be a discrete set of contexts or defined via context distribution pc- A cMDP M therefore defines a set of contextual MDPs M = =Mooto-pe.
cMDPs Subsuming Other Notions of Generalization Even though a lot of work in RL makes no explicit assumptions about task variations and thus generalization, there are extensions of the basic MDP models beyond cMDPs that focus on generalization. One of these is Hidden-PPraameter MDPs (Doshi-Veles & Konidaris, 2016), which allows for changes in dynamics just as in cRL, but keeps the reward function fixed. The reverse is true in goal-based RL (Florensa et al., 2018), where the reward function changes, but the environment dynamics stay the same. Block MDPs (Du et al., 2019) are concerned with a different form of generalization than cMDPs altogether; instead of zero-shot policy transfer, they aim at learning representations from large, unstructured observation spaces. An alternative approach is the epistemic POMDP (Ghosh et al., 2021) as a special case of cMDP. Here, transition and reward functions may vary, but the context assumed to be unobscrvable. The corresponding approaches then model the uncortainty about which instance of the cMDP the agent is deployed on during test. lime.
Settings that can be described as a collection of interacting systems, e.g. multi-agent problems, can be formalized as such through Factored MDPs (Boutilier et al., 1999; Guestrin et al., 2001). The generalization target here is again not necessarily zero-shot, policy transfer, but generalization with respect to one or more of the factored components, e.g. the behavior of a competing agent. Both Block MDPs and Factored MDPs are compatible with cMDPs, i.e, we can construct a Block cMDP or a Factored cMDP in order to focus on multiple dimensions of generalization (Sodhani et al., 2021a).
Apart from these MDP variations, there are also less formalized concepts in RL related to or subsumed by cMDPs. Skill-based learning, for example, relates to cMDPs (da Silva et al., 2012). Some varialions of the environment, and therefore areas of the context space, will require different action sequences than others. The past experience of an agent, i.c. its memory, can also be seen as a context, even though it is rarely treated as such. Frame stacking, as is common in, e.g., Atari (Bellemare et al., 2016) accomplishes the same thing, implicitly providing context by encoding the environment dynamics through the stacked frames.
Obtaining Context Features Not every task has casily defined or measurable context that doscribes the task in detail. Therefore, it is important to examine how context can be obtained in such cases. Often can only extract very simple context features for task. For example, even though procedural generation (PCG) based environments do not allow control over the training and test distributions, they can still be considered contextual environments since they are usually sceded. This would give us the random seed as context information (Kirk et al., 2023). However, the seed provides no semanntc information about the instance it induces. Obtaining more useful context features should therefore be a focus of cRL. Learned representations provide an opportunity (Jaderberg et al., 2017b; Gelada et al., 2019; Zhang et al., 2021a; Castro et al., 2021) to do this in a data-driven manner. As these representations encode the tasks the agent needs to solve, subspaces of the context space requiring different policies should naturally be represented differently. This idca has previously been applied to detecting context changes in contimuus environments (da Silva et al., 2006; Alegre et al., 2021) and finding similar contexts within a training distribution (da Silva el al., 2012). Thus, even without readily available context, representation learning can enable the reliable availability of information relevant for generalization to tasks both in- and out-of-distribation.
3 Related Work
Transferring and generalizing the performance of an RL agent from its training setting to some test variation has been at the center of several sub-communities within RL. Robustness, for example, can be seen subcategory of generalization where variations to the context are usually kept small, and the goal is to avoid failures due to exceptions on a single task (Morimoto & Doya, 2000; Pinto et al., 2017; Mchta et al., 2019; Zhang et al., 2021d). Policy transfer is also concerned with generalization in a sense, though here the goal has often been Ene-tuning a pre-trained policy, i..., few-shot generalization instead of zero-shot generalization (Duan et al., 2016; Finn et al., 2017; Nichol et al., 2018). The goal Multi-Task. Learning is to learn a fixed set of tasks efficiently (Yu et al., 2019), not necessarily being concerned with generalizing outside of this set. Meta-Learning in RL usually aims at zhro-shot policy generalization similar to Contextual
4

---

Published in Transactions on Machine Learning Research (06/2023)
Reinforcement Learning (cRL). This field is very broad with different approaches like learning to learn algorithms or their components (Schulman et al., 2016; Duan et al., 2016; Wang et al., 2017), generating task curricula (Matiisen et al., 2020; Nguyen et al., 2021) or meta-learning hyperparameters (Runge et al., 2019; Zhang et al., 2021c).
The previously mentioned methods were not conceived with cRT. in mind but use context implicitly. Many meta-RL methods, however, can or do make use of context information to guide their optimization, either directly (Klink et al., 2020; Eimer et al., 2021) or by utilizing learnt dynamics model (Kober et al., 2012). The idea of context-aware dynamics models has also been applied to model-based RL (Lee et al., 2020b). These approaches use context in different ways to accomplish some generalization goal, c.g. zero-shot generalization to a test distribution or solving a single hard instance. In contrast, we do not propose a specific Meta-Learuing method but examine the foundations of cRL and how context affects policy learning in general.
Zero-shot generalization across a distribution of contexts, specifically, has become a common goal standard RL environments, often in the form of generalization across more less randomly generated tasks (Juliani et al,, 2019; Cobbe et al., 2020; Samvelyan et al., 2021). The larger the degree of randomness in the generation procedure, however, the less context information and control are available for Meta-Learning methods (Kirk et al., 2023). Such underspecification of tasks can even make evaluations more challenging (Jayawardana et al., 2022). In contrast. to other works on generalization in RL, we therefore focus on sampling context without PCG but from explicitly defined distributions. This allows us to analyze the capabilities of our agents in a more fine-grained manner, e.g. how far away from their training distribution generalization performance starts to decrease, instead of relying only on the test reward across all task instances. Our benchmark is styled similarly to the early generalized helicopter environment, where Koppejan & Whiteson (2009) control and vary wind. Later, Whiteson et al. (2011) propose an evaluation protocol for general RL to avoid overfitting on particular training environments, where they argue for generalized methodologies assessing the performance of an agent on a set or distribution of environments. cMDPs easily fit into this line of evaluation protocols with the advantage of interpretable genoralization capabilitics because of the definition of interpretable context features. Kirk el. al. (2021) similarly propose evaluation protocols, but already with cMDPs in mind. For further ways cRL opens new directions in ongoing work, see Appendix G.
4 Reinforcement Learning with Context
In this scction, we provide an overview of how context can influence the training of RL agents. We discuss training objectives in the contextual setting and give a brief theoretical intuition of the implications of treating generalization problems that can be modeled cMDPs like standard MDPs This should serve as demonstration of why using the cMDP framework for generalization problems is beneficial and should be explored further. Note that we assume standard cMDPs in this section, mcaning the context, if provided to the agent, is fully observable and reflects the true environment behavior.
4.1 Solving cMPDs
Objectives Having defined context and cMDPs, we can now attempt to solve cMDPs. To this end, we must first formulate potential objectives being addressed by cRL. In contrast to standard RL, cRL offers several different objectives for the same training setting depending on what kind of generalization we are aiming for. We can provide this objective via a target context distribution that induces the target cMDP M. Depending on the relation between target and training distributions, we can measure interpolation performance, robustness to distribution shift, out-of-distribution generalization, and more, e.g., solving a single expensive hard task by only training on easy ones. Thus, the cRL objective is defined by the relationship between troin and test settings, similar to supervised learning, but on the level of tasks rather than the level of data points (sce Figure 2).
Optimality Regardless of the specific objective, we solve cMDPs in the same way we would solve standard MDPs, though we need to extend the definition of the return. Instead of maximizing the expected reward over time, we use the expected reward over both time and target context distribution. Therefore we define optimality in a cMDP in the following way:
5

---

Published in Transactions on Machine Learning Research (06/2023)
Genera Unsoen Floor Frictirs
Generalliing to Unseen Goals (Red)
Generaliting Across Seen Goals Unseen Foor Frictir
nextor
Frcken
RD ortneEn
Ticharclgg
Figure 2: Different train and test relationships result in different generalization tasks: interpolation between known friction levels (left), generalizing goals further away than seen in training (middle), generalizing to the goal distances in the training distribution with lower friction (right).
Definition 1 A policy x² is optimal for a given cMDP M with target context distribution pe iff n² optimally acts on every MDP in M (t.e. marimizes the return Ge for each context c):
Ve pe nt E arg max E, [G,,z]
sell
Note that such an optimal policy does not necessarily exist. Malik al. (2021) showed that the problem of learning a policy that satisfies Definition may be intractable for some context distributions, even if the contexts are similar.
In order to compare policies across a given target distribution, we propose to compare the gap between optimal and actual performance, what we call the Oplimalily Gop Og. Formally, we define Og as the gap between the optimal return Ge,z' over the target context distribution e and the return of the given policy m
Og = -BeelGerr - -G,
In settings where the optimal return is known, we can directly evaluate the optimality gap as the difference between the return of trained policy and the optimal return. However, in cases where the optimal return is either unknown or intractable, we can instead use an agent trained on each single context as an approximation of the optimal return. However, we have two sources of uncertainty: First, the specialized agent might not reach the best performance achievable in the MDP and second, the uncertainty of the optimal return also depends on the number of context samples for the specialized agent.
4.2 Optimal Policies Require Context
In this section, we give an intuition and a proof sketch of why conditioning the policy not only on the state space S but also on the context space can be beneficial. This is very much analogous to how larger degree of observability will enable better learning in POMDPs (Kurniawati, 2022). As a reminder, a standard RL policy is defined a mapping S > A from state observation s E S to action A." In contrast to standard RL, in cRI the state space S is shared among all MDPs within cMDP, meaning that states can occur in multiple MDPs even though the transition and reward functions might differ.
3-State cMDP A simple way to exemplify this is through the 3-state cMDP in Figure 3. For the first MDP on the left, the optimal action would be dg leading to state Si with a high reward of 10. The MDP in the middle variation of the same state and action space, where the transition function has changed: while the reward in stale S remains 10, action ag now leads to state S2 with lower reward of 1. Similarly, the MDP the right is another variation changing the reward function instead of the transition function: an still leads to S1, but the associated reward now is 1. An agent exposed to such changes would not be able to react appropriately unless the policy is conditioned on the context c;
In contrast, a conditiooen policy can distinguish between the contexts and thus receives more guiding foodback during training. Also, it can be more capable to perform optimally test time given an approximation of the context. We define context-conditioned policies follows:
Alternatively, one can also define policy probability distribution P(als) over the actions given state. The following line of arguments also holds for stochastic policies but we only outline for deterministic policies to not clutter notation.

---

Published in Transactions on Machine Learning Research (06/2023)
Definition 2 A content-conditionee policy is a mapping 8 C -+ A with state space S, contert space C and action space A.
In order to formalize this intuitive explanation of why context is helpful in generalization, let us first recall what optimal performance means in cMDPs. Definition 1 requires that there exists a policy x" that is optimal on every context. We will now show that optimal context-oblivions policies like this do not exist in general, but that to obtain optimality, the policy needs to have access to the context.
Proposition 1 For given cMDP = {Me,, Mog} defined over two possible conterts e₁ and C22 there is either a common s' at which the optimal policy differs between the conterts.
optimal policy H² for both conterts or there is at least one conftict state
Proof Sketch Let us look at a given state E S that is reachable in Ma. Further, let us assume TH is the optimal policy for Mc, that is defined only on states reachable in Ma. We have to consider the following three possible cases for cach state s E S: (i)
ao
S
a
a
reachable in Ma and कद, (s) 5o
So
So
is not optimal on Mezi (ii) is reachable in Mcy and nे (s) is optimal on Mezi (iii) is not reachable in Mo.. We assume that we act within one MDP.
ag
a1
52
S2
52
Co
C
C2
IC (i) is true for at least one state E S, the optimal policy obviously is different between the contexts in this state, and therefore we have found a conflict state s'. The other two cases do not produce such conflict. We can, however, construct a policy x² that is optimal on Me, and Mc, from Te if for all s E 8 either (ii) or (iii) is true. For any state 8 where (ii) holds, we simply set x" r*)) = = (s). For states that are not reachable in Mc, as stated in (ii), Th is not defined. We can therefore extend n' by these states without changing its optimality on Mar- Let. a" be the optimal action in such 8 on M Then, we define T*(s) = a". By
Figure 3: A sample cMDP with three contexts. The original one (left), one changing the transition function (middle) and another the reward function (right).
construction, x² is then optimal on all states s E S, and it exists iff there is no state reachable in both contexts where the optimal action for Ma, differs from the one for Mog.
Theorem An optimal policy n" for any given cMDP M is only guarnnleed to exist if il is conditioned on the contert: π S x C + A.
Proof Sketch Let us assume we know an optimal policy x. for any context c E C. As c induces an MDP and with it a corresponding optimal policy, we know that Th exists. Furthermore, let the sets of optimal policies between at least two MDPs Me, and Me, be disjoint. This means no policy exists that is optimal on both ej and 02.
Now let us examine the optimal policy H¹ and assume that exists for this cMDP M. By definition, x² is optimal on or and 62. If it is only conditioned on the state, (s) results in the same action independent of the context. Because the sets of optimal policies for and 02 are disjoint, there must be at least one state s', where T2 (s') # o (s') according to Proposition 1. As both are optimal in their respective contexts but not in the other and do not result in the same action for 8, x" cannot actually be optimal for both e and c2- Thus, the optimal policy πt" does not exist. On the other hand, if we can condition the policy on the context, such that क. x + A, we can circumvent this problem (for a discussion on how this relates to partial observability, see Appendix A). n (s, के is optimal for each context.
Discussion Theorem 1 raises the question of how performance changes if the policy is not conditioned the context. Apart from the fact that can construct cMDPs where a policy not conditioned the context may perform arbitrarily poorly, we intuitively assume the optimality gap should grow the broader pe becomes and the more impact slight changes in e have on the transitions and rewards. Formally assessing the optimality gap Og is another challenge in itself. Another question is how relevant the assumption of disjoint sets of optimal policies for different contexts in practice. For the case where the observations implicitly encode the context, a single policy can solve different contexts optimally by never entering the conflict above. Assuming this is how generalization is commonly handled in RI. However, we deem this
7

---

Published in Transactions on Machine Learning Research (06/2023)
not to be reliable mechanism to avoid conflicts between context-optimal policies the same observations. In conclusion, depending on the environment and on how context reflects on the observations, it might be beneficial to explicitly include context, especially on harder and more abstract generalization tasks.
5 The CARL Benchmark Library
To analyze how the context and its augmentation influence the agent's generalization capabilitios, learning, and behavior, we propose CARL: library for Context Adaptive Reinforcement Learning benchmarks following the Contextual Reinforcement Learning formalism. In our release of CARL benchmarks, we include and contextually extend classic control and box2d environments from OpenAl Gym (Brockman et al., 2016), Google Brax' walkers (Freeman et al., 2021), a selection from the DeepMind Control Suite (Tassa al., 2018), an RNA folding environment, (Runge al., 2019) well as Super Mario levels (Awiszus al., 2020; Schubert al., 2021), see Figure 4.
Benchmark Categories Often the physics simulations (brax, box2d, classic control and cm control) define a dynamic body in a static world with similar physical entities like gravity, geometry of the moving body and mass. In our example CARLFetch from Figure la the goal is to move the agent to the target area. The context features joint stiffness, gravity, friction, (joint) angular damping, actuator strength, torso mass as well target radius and distance define the context and influence the exact instantiation and dynamics of the environment. In principle, the designer is free select the context features from the set of paramcters defining the environment. For Fetch, this could also be limb length of the body, Out of practicality, to vary the most common physical attributes across the environments. When selecting an environment 's parameter to become context feature, it must be guaranteed that the physics and the environment's 's purpose are not violated, e.g. by setting a negative gravity such that the body flies up and is never able to reach the goal. Please see Appendix H for all registered context features per environment.
choose
Besides physical simulation environments, CARL pro- vides two
specific, challenging environments. The first is the CARLMarioEnv environment built on top of the TOAD-GAN level generator (Awiszus et al., 2020; Schubert et al., 2021). It provides a procedurally generated game-playing environment that allows customization of the generation process. This environment is therefore especially interesting for exploring representation learning for the pur- pose of learning to better generalize. Secondly, move closer to real-world application by including the CARLRNADesigubavironn
(Runge et al., 2019). The challenge here to design RNA sequences given structural constraints. As two different datasets of structures and their instances are used in this benchmark, is ideally suited for testing policy transfer between RNA structures.
Figure 4: The CARL benchmarks
5.1 Properties of Benchmarks
While the categorization of the CARL benchmarks above provides overview of the kinds of environments included, we also discuss them in terms of relevant environment attributes that describe the nature of their problem setting, see Figure 5.
State Space Most our benchmarks have vector-based state spaces, allowing to concatenate context information. Their sizes range from only two state variables in the CARIMountainCar environments 299 for the CARLHumanoid environment. The notable exceptions here are CARLVehicldRacing and CARIToodGAN, which exclusively pixel-based observations.

---

Published in Transactions on Machine Learning Research (06/2023)
classic control
box2d state space size
brax
RNA Mario
state space size
state space size
state space size
state space size
action
action
action
action space size
action space size
space
Per
space
Net
space
net
not
Net
nid, redard
dymamics
ncl reward
Def.oymamics
nef.romard
Netrwwaad
dyf.dmmamics
net, rewardd
dymamiic
Figure Characteristics of each environment of the environment families showing the action space size, state space size (log-scale), number of context features (Nef), the number of context features directly shaping the reward reeaar) and the ones changing the dynamics (fef dynamienm All axes are scaled to the global extrema and the state space size is additionally on a logarithmic scale.
Action Space We provide both discrete and continuous environments, with six requiring discrete actions and the other 14 continuous ones. The number of actions range from single action 19 different actions.
Quality of Reward We cover different types of reward signals with our benchmarks, ranging from relatively sparse step penalty style rewards where the agent only receives reward of -1 cach step to complex composite reward functions in e.g. the Brax-based environments. The latter lype is quite informative, providing updales on factors like movement economy and progress toward the goal whereas the former does not let the agents distinguish between transitions without looking the whole episode. Further examples for sparse rewards are the CARLCartPoleDnn and CARLVehicleRRccnngEns
Context Spaces While the full details of all possible context configurations can be seen in Appendix H, for brevity here we only discuss the differences between context spaces and the configuration possibilitics they provide. Depending on the environment the context features have different influences on the dynamics and the reward. Of all 145 registered context features, 99 % influence the dynamics. This means that a context. feature is changed then the transition from states to their successors affected and likely changed as well. Only % of the context features shape the reward. Most context features (91 %) are continuous; the rest are categorical discrete.
Summary Comparing benchmarks along these attributes,
wide spread in most of them (Figure 5). CARL focuses popular environments and will grow over time, increasing the diversity of benchmarks. Already now, CARL provides a benchmarking collection that tasks agents with generalizing in addition to solving the problem most common in modorn RL while providing a platform for reproducible research.
6 Experiments
Having discussed the framework of cRL and the implications of context in training, we now study several research questions regarding the empirical effects of context: (i) How much does varying context influence performance? Can agents compensate across context. variations the effects discussod in Section practice? Lc.,, is there an observable optimality gap on cMDPs, does the context visibility influence the performance and which role does the width of the context distribution play? assess generalization performance, and how does the test behavior of agents change when
zero-shot manner? (ii) Can we observe
(iii) How exposed to the context information?
To explore our research questions, we use our benchmark library CARL. Details about the hyperparameter settings and used hardware for all experiments are listed in Appendix In each experiment, if not specified otherwisc, we train and evaluate on 10 different random seeds and set of 128 uniformly sampled contexts. All experiments can be reproduced using the scripts we provide with the benchmark library https://github. com/ http/auutmm//A.....
9

---

Published in Transactions on Machine Learning Research (06/2023)
6.1 How Does Varying Context Influence Performance?
To get an initial understanding on the generalization capabilities, we train a well-known SAC agent (Haarnoja et al., 2018) on the default version of the Pendulum (Brockman et al., 2016) environment. Pendulum is very simple enviromment (see Appendix B for dynamic equations) compared to the majority of RL benchmarks and has been considered solved by deep RL for years. However, we show that we increase the difficulty of this environment substantially when considering even single context features, Note that this increase in difficulty also means the best return may decrease even for an optimal agent, in extreme cases making instances impossible to solve. The agent is not provided with any explicit information about the context, i.e, it is context-oblivious. Then, for evaluation, we vary each defined context feature by magnitudes A 0.1, .. 2.2 times the default value for 10 test episodes. In Figure 6, we plot the empirical cumulative distribution functions (eCDF) for the return showing the range of observed returns on the x-axis and the proportion on the y-axis. The further to the right the curve is, the better the observed returns. For the eCDF plots of other CARL environments, see Appendix D.1.
First and foremost we observe that some context features do not have an influence on the generalization performance when varied, see Figure 6. Even in this zero-shot sctting, the agent performs similarly well on all context variations of the initial_ aaallanglaaaaa max and initial_ settiiallllliityyy_cmm _acoccittyaaxtttaooo _max features. Yet, the agent's 's performance is very brittle to other context features, i.e. max max_speed, simulation timestep dt, gravity g and length The trained agent cannot compensate for the effect the context has on the policy and thus performs poorly on several context variations. It is worth noting that the performance curves across the variations depends on the context feature. max_ max_speed, for example, is hard for values below A = 0.7, but then the challenge level for the agent decreases abruptly. This transition is smoother for 1 where both very small and very large values are hard to solve. We conclude that it is not straightforward to estimate the impact of changing the environment on agent behavior, especially for physics simulations or similarly complex environments. We also clearly see that zero-shot generalization cannot compensate for variations environment dynamics. Context variations introduce a significant challenge, even on a simple environment like Pendulum. The next step, therefore, is to train the agent on varying contexts and re-evaluate its generalization performance.
max speed
m
1.0
Proportion
A
0.5
0.1
1.2
0.2
1.3
0.0
0.3
1.4
-10¹
-10² -10'
-100
0.4
1.5
Retur
0.5
1.6
initial angle max
initial velacity max
0.6
1.7
0.7
1.8
Proportion
0.8
1.9
0.5
0.9
2.0
1.0
2.1
1.1
2.2
0.0
-10
-10 -10' Return
0
-10² -10² -10' Return
0
-10²
-10² -10" Return
0
Figure 6: CARLPendulummEn eCDF Plot. A is the magnitude multiplied with the default value of each context feature. So, A = 1.0 refers to the standard environment.
6.2 Does the Optimality Gap Exist in Practice?
In the previous section, we saw that context variation heavily influences the test performance of a standard agent. Here, we take a closer look and connect this to the Optimality Gap Og (Equation (1)). In order demonstrate how significant this gap is in practice, we train C51 agent (Bellemare al., 2017) on our contextually extended Cort environment. Brockman et al. (2016) as well SAC agent (Haarnoja et al.,
10

---

Published in Transactions on Machine Learning Research (06/2023)
1.0
origin
anigin
Proporion
0.8
general
Madian
getieral
Median
0.6
speciaiized
spacialped genaral
0.5
specialized
geneew
0.4
0.4
specialoed
speciekzed
Mean
Mean
0.0
specialiedd genaral
specialized general
0
200
400
1500
1000
500
0
420
440
460
Return
-350
-325
-300
-275
(a) Return eCDF CartPole
(b) Return CartPole
(c) Return eCDF Pendulum
(d) Return Pendulum
Figure 7: Optimality Gap on CARLCartPole (left) and CARLPendulum (right).
2018) on Pendulum. Similarly as above, we use simple environments to demonstrate the induced difficulty by context variation.
To generate instances of both environments, we vary the length of the pole across a uniform distribution pc =00.5,, 0.75) around the standard pole length for CartPole and the pole length across pe = 4(1 2.2) for Pendulum. For training, we sample 64 contexts from this distribution and train a general agent which experiences all contexts during training in round robin fashion. However, we do not explicitly provide the context information to the agent. We approximate the optimal performance by training separate specialized agent on each context. Afterwards, each agent is evaluated on each context it was trained on for 10 episodes.
Comparing the general and specialized agents, we see a difference of at least 30 reward points in median, mean and estimated IQM performance (as proposed by Agarwal et al. (2021)) for CartPole and a smaller, but similar effect on Pendulum. While this is not a huge decrease performance, looking at how these rewards are distributed across the evaluation instances shows that the general agent solves significantly fewer instances than the specialized agent with a decrease of around 40% of finished episodes on CartPole. This shows that while most instances be solved by the agent at time, training an agent that solves all of them jointly is a significant challenge.
6.3 Does Access To Context Improve Training?
We have seen that agents without access to context information are not always able entirely solve even simple contextual environments. Can these agents improve with access to the context information? We choose very simple approach of adding either the whole context (concat all) only the actively changing context feature (concal, non-static) to the observation. Obviously this is a simplistic approach that in the case of concat all significantly alters the size of the observation. Though, even with this simple idea, training performance improves by a large margin in some cases, see Figure 8. Here, the agent, this time on F.RLDmeWalkor with changing viscosity (~ [1,,, 2.5] times the default value), learns faster, is more stable, and reaches a higher final performance with the additional information added the observation. In testing, we see significantly more failures in the hidden agent compared to the concat ones, with only the concat non-static agent learning to solve the contextual environment without large decrease in overall test performance. Effective generalization seems to be a matter of a reasonable feature set, similar to supervised learning.
On Pendulum, however, this is not the case; we see no meaningful difference in mean performance of concat (non-static) and hidden when varying length (~ [1,1 2.2] times the default value). Please note that the large confidence interval stems from runs where the algorithm hm did not find meaningful performance, see Appendix Figure 11. The concat agents perform better on some unscen contexts ovaluation, though the hidden agent is far superior on a slice of the instance set.
In both cases, the generalization behavior of contextual agents is different from the hidden one. Most CARL environments show that additional context information is indoed beneficial for train and test performance as in Walker (see Appendix E for full results). Additionally, the performances of concat all and concat non-static agents provide no clear pattern as to how many context features should be provided in training. We conclude that while context information is indeed useful for generalization performance, simply appending the context to the observation might not be the ideal way to communicate context features. Instead, context. embeddings could be a more potent way of capturing the way context feature changes an environment,
11

---

Published in Transactions on Machine Learning Research (06/2023)
-250
60
800
-500
Iiddon
visiillty
Relum
600
cEnEa yronslalec
(
Retum
Fresuanoy
consar (ron-saak)))
cuncer (al)
400
20
concal (41)
11000
soncel (al)
20
200
concal (non-staric)
concat (non-static)
hidsan
0.4
1,0
500
1002
20000 40000 60000 80000 100000
-1900
-1000
-500
Steps
Steps
Redumn
Figure 8: Train (lineplot) and test performance (histogram) of agents with visible and hidden context CARLDme Walker with different viscosity values and 5 seeds (left) and CARLPendulum with dilferent lengths and 20 seeds (right). Shown is the mean performance with 95% confidence interval and testing across 200 test contexts (metrics are computed using stratified resampled bootstrapping (Agarwal et al., 2021)).
88 we have seen in some prior work on incorporating goals (Sukhbaatar et al., 2018; Liu et al., 2022) into training. Since our goal was to show the potential of context information, we leave to future work to investigate better representations of context features.
6.4 How Far Can Agents Learn To Act Across Contexts?
As we saw different evaluation behaviors from hidden and visible agents in the last section, we want to further investigate their generalization capabilities in- and out-of-distribution. To this end, we follow a three mode evaluation protocol for Contextual Reinforcement Learning that tests the agent's interpolation capabilities and out-of-distribution generalization under different training distribution shapes (Kirk et al., 2023). This is in contrast to PCG environments where cannot define evaluation protocols and instead have to rely the given instance generalion procedure.
update interval concat + updale interval hidden
mode: A
mode:
mode:
10.001, 0,0C391]
(0,0068, 0.0077|
(0.0096.00 0.0115]
(0.0154. 0.0153
(0.0172. 00191)
(0,021 0.0229
a.287]
10.0286, 0.c30g
400
(0,0324, 0.043]
(0.0082, 0.0001]
300
Return
10.001, 0,003911
(0.0058,
(0,0006 0.0115
(0.0134, 0.0153
200
(0.0172. 0.101]
(0.021, 0.0229
[0.0248, 0.0257
(0.0286, 0.0005]
100
(0.0324. 0.0333)
(0.0362. 0.0381]
04, 0.11)
(0.23, 0.29)
10 88, a.ss)
(0.77, 0.83)
10.85, 1.1))
(1.13, (e
(0.448, 0.11)
10.20, 0.99
(0.4%, 0.47)
0.65)
10.77, 0.88)
(0.95, 1.01)
(t.18, 1.19)
00.46..0 0.111)
(0.20, 029)
(0.58, 0.65)
(0.77, 0.ss)
aass, 1.61)
11.3,1115 1.18)
10.4%,
pole pole_length
pole length pole_length
pole _llgthh
Figure 9: In- and Out-of-Distribution Goneralization on CartPole. We vary the pole length and up- date interval. The blue polygon marks the train context area. Black dots mark gaps the context. space due random sampling. First row: Context-oblivious agent, second row: Concat.
We define train and test distributions for each dimension of the context space individually, allowing us to study different relationships between train and test scttings. If at least parts of the context are within the train distribution, we speak of interpolation, if the whole context outside, extrapolation. By choosing two context. features and defining uniform train distributions both, construct convex training set in
12

---

Published in Transactions on Machine Learning Research (06/2023)
the context feature space (mode A: D). The context feature distributions can also be defined to allow only small variation (mode B: Lb] or a single value per feature (mode C: L), creating non-convex train sets. Thus, the convex hull of this non-convex set tests combinatorial interpolation.
To demonstrate this, we again choose contextual CartPole from CARL, and the C51 (Bellemare et al., 2017) agent known to perform well on it. We train the agent for 100000 timesteps and vary the update interval and the pole length in the environment, once without access to the context (hidden) and once concatenating pole length and gravity to the observation (concat). We repeat this with 10 random seeds and 5 test episodes per context. For the train and test context sets, sample 1000 contexts each for the train and test distributions defined in the evaluation protocol, see Figure 9. The test performances are discretized and aggregated across seeds by the bootstrapped mean using rliable (Agarwal et al., 2021).
In Figure 9, we show that both hidden (context-oblivious) and visible (concalenate) agents perform fairly well within their training distribution for evaluation mode A and even generalize to fairly large areas of the test distribution, more so for concat. Large update intervals combined with extreme pole lengths proves to be the most challenging area.
Interestingly, the concat agent is able to solve more of the large update intervals in contrast to the hidden agent which is most pronounced on train distribution C. This itsclf is counterintuitive, we would expect that the larger the train distribution, the larger the cnttof-diitributtin generalization. The hidden agent in general performs well for low update intervals. This resonates with the intuition that smaller update intervals are easier because there is more granularity (and time) to react to the current state. In addition, we provide results for varying the update interval and the gravity. Here, the results are similar but subdued, see Appendix Figure 12. Varying the pole length together with the gravity paints different image (Appendix Figure 13). In this case, the hidden agent performs much better. We suspect that the effects of gravity and pole length cancel out and thus context information is not needed to learn a meaningful policy. These three variations again show that providing context by concatenation can be helpful but not in every case, demanding further investigation on alternatives. Finally, neither agent shows reliable combinatorial interpolation performance, let alone out-of-distribution generalization. We see here a major open challenge for the RL community, for which CARL will support them in the development and precise studies of RL generalization capabilities.
7 Conclusion
Towards our goal of creating general and robust agents, we need to factor in possible changes in the environment. We propose modcling these changes with the framework of contextual Reinforcement Learning (cRL) in order to better reason about what demands Contextual Reinforcement Learning introduces to the agents and the learning process, specifically regarding the suboptimal nature of conventional RL policies in cRL. With CARL, we provide a benchmark library which contextualizes popular benchmarks and is designed to study generalization in Contextual Reinforcement Learning. It allows us to empirically demonstrate that contextual changes disturb learning even in simple settings and that the final performance and the difficulty correlate with the magnitude of the variation. We also verify that context-oblivious policies are not able to fully solve even simple contextual environments. Furthermore, our results suggest that exposing the context to agents even in naive manner impacts the generalization behavior, in some cases improving training and test performance compared to non-context-aware agents. We expect this to be first step towards better solution mechanisms for contextual RL problems and therefore one step closer to general and robust agents.
Broader Impact Statement
We foresee no new direct societal and ethical implications other than the known concerns regarding autonomous agents and RL (e.g., in a military context).
Acknowledgments
The work of Frederik Schubert, Sebastian Döhler and Bodo Rosenhahn was supported by the Federal Ministry of Education and Research (BMBF), Germany under the project LeibnizKlLabor (grant no. 01DD20003) and the AI service center KISSKI (grant no. 01IS22093C), the Center for Digital Innovations (ZDIN) and the Deutsche Forschungsgemeinschaatt (DFG) under Germany's Excellence Strategy within the Cluster of
13

---

Published in Transactions on Machine Learning Research (06/2023)
Excellence PhoenixD (EXC 2122). André Biedenkapp and Frank Hutter acknowledge funding through the research network "Responsive and Scalable Learning for Robots Assisting Humans" (ReScaLee) of the University of Freiburg. The ReScaLe project is funded by the Carl Zeiss Foundation.
References
M. Abdolshah, Н. Le, T. K. George, S. Gupta, S. Rana, and S. Venkatesh. A new representation of successor features for transfer across dissimilar environments. M. Mcila and T. Zhang (ods.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1-9. PMLR, 18-24 Jul 2021.
S. Adriaensen, A. Biedenkapp, G. Shala, N. Awad, T. Eimer, M. Lindauer, and F. Hutter. Automated dynamic algorithm configuration. J. Artif. Intell. Res., 75:1633 1699, 2022.
R. Agarwal, M. Schwarzer, P. Castro, A. Courville, and M. Bellemare. Deep reinforcement learning at the edge the statistical precipice. In Proceedings of the 35th International Conference on Advances in Neural Information Processing Systems (NeurIPS pp. 29304 29320, 2021.
L. Alegre, A. Bazzan, and B. da Silva. Minimum-delay adaptation in non-stationary reinforcement learning via online high-confidence change-point detection. In AAMAS '21: 20th International Conference on Autonomous Agents and Mulliagent Systems, pp. 97-105. ACM, 2021.
I. Arel, C. Liu, T. Urbanik, and A. Kohls. Reinforcement learning-based multi-agent system for network traffic signal control. IET Intelligent Transport Systems, 4(2):128-133) 2010.
M. Awiszus, F. Schubert, and B. Rosenhahn. TOAD-GAN: Coherent style level generation from a single exam- ple. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, October 2020.
A. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 18-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 507- 517. PMLR, 2020.
M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett (eds.), ) Proceedings of the 29th International Conference on Advances in Neural Information Processing Systems (NeurIPS'16) 16), pp. 1471-1479, 2016.
M. G. Bellemare, S. Candido, P. Samuel Castro, J. Gong, M. C. Machado, S. Moitra, S. S. Ponda, and Z. Wang. Autonomous navigation of stralospheric balloons using reinforcement learning. Nature, 588 (7836):77 02, 2020.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine mll. http://ss.m..pppeee/////e//lllee pptpss//70//elllareeeeeee
Learning Research, pp. 449 458. PMLR, 2017. URL html.
A. Biedenkapp, Marben, M. Lindauer, and F. Hutter. CAVE: Configuration assessment, visualization and evaluation. In Proceedings of the International Conference Learning and Intelligent Optimization (LION), Lecture Notes in Computer Science. Springer, 2018.
Craig Boutilier, Thomas L. Dean, and Steve Hanks. and computational leverage. J. Artif. Intell. Res., 11:1 94, 1999. doi: 10.1613/jair.575, URL https:
planning: Structural assumptions
www./////.....////... 577.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. CoRR, htts/1606.0154p) 2016.
14

---

Published in Transactions on Machine Learning Research (06/2023)
P. Castro, T. Kastner, P. Panangaden, and M. Rowland. Mico: Learning improved representations via sampling-basel state similarity for markov decision processes. CoRR, abs/2106.08229,, 2021.
K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference Machine Learning, ICML, volume 119 of Proceedings of Machine Learning Research, pp. 2048-2088 108-. PMLR, 2020.
B. da Silva, E. Basso, A. Bazzan, and P. Engel. Dealing with non-stationary environments using context detection. In Machine Learning, Proceedings of the Twenty- Third International Conference (ICML, volume 148 of ACM International Conference Proceeding Series, pp- 217 224. ACM, 2006.
B. da Silva, G. Konidaris, and A. Barto. Learning parameterized skills. In Proceedings of the 29th International Conference on Machine Learning, ICML, 2012.
J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. de Las Casas, C. Donner, L. Fritz, C. Galperti, A. Huber, J. Keeling, M. Tsimpoukelli, J. Kay, A. Merle, J. Moret, S. Noury, F. Pesamosca, D. Pfau, O. Sauter, C. Sommariva, S. Coda, B. Duval, A. Fasoli, P. Kohli, K. Kavukcuoglu, D. Hassabis, and M. Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning- Nature, 602(7897):414-47) 419,, 2022.
F. Doshi-Velez and G. Dimitri Konidaris. Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations. In Proceedings of the Twenty- Fifth International Joint Conference on Artificial Intelligence, IJCAI, pp. 1432-1440. IJCAI/AAAI Press, 2016.
S. Du, A. Krishnamurthy, N. Jiang, A. Agarwal, M. Dudík, and J. Langford. Provably efficient RL with rich observations via latent state decoding. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1665-1674. PMLR, 2019.
Y. Duan, J. Schulman, X. Chen, P. Bartlett, I. Sutskever, and P. Abbeel. RI$ Fast reinforcement learning via slow reinforcement learning. CoRR, abs/ 1611.02779 2016.
T. Eimer, A. Biodenkapp, F. Hutter, and M. Lindauer. Self-pacod context evaluation for contextual reinforcement learning. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning (ICMI, 21), volume 139 of Proceedings of Machine Learning Research, pp. 2948 2958.. PMLR, 2021.
B. Eysenbach, R. Salakhutdinov, and S. Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett. (eds.), Proceedings of the 32nd International Conference on Advonces in Neural Information Processing Systems (NeurIPS 19), pp. 15220-15231, 2019.
C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast. adaptation of deep networks. Tn
D. Procup and Y. Tch (eds.), Proceedings of the 94th International Conference on Machine Learning (ICML 17), volume 70, pp. 1126 1135. Proceedings of Machine Learning Research, 2017.
C. Florensa, D. Held, X. Geng, and P. Abbeol. Automatic goal generation for reinforcement learning agents. In Proceedings of the 35th International Conference on Machine Learning, 1CML 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1514-1523. PMLR, 2018
J. Franke, G. Köhler, A. Biedenkapp, and F. Hutter. Sample-effcient automated deep reinforcement learning. In 9th International Conference on Learning Representations, ICLR. OpenReview.ne .ec,, 2021.
C. Freeman, F. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax A differentiable physics engine for large scale rigid body simulation. CoRR, abs/2106. 132811 2021.
H. Fu, H. Tang, J. Hao, C. Chen, X. Feng, D. Li, and W. Liu. Towards effective context for meta- reinforcement learning: an approach based on contrastive learning. In Proceedings of the Conference on Artificial Intelligence (AAAT'21) pp. 7457-7465. AAAT Press, 2021a.
15

---

Published in Transactions on Machine Learning Research (06/2023)
X. Fu, G. Yang, P. Agrawal, and T. Jaakkola. Learning task informed abstractions. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3480 3491. PMLR, 18 24 Jul 2021b.
C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. Bellemare. Deepmdp: Learning contimuous latent space models for representation learning. In Proceedings of the 36th International Conference on Machine Learning, ICML, volume 97 of Proceedings of Machine Learning Research, pp. 2170 2179. PMLR, 2019.
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P. Adams, and Sergey Levine. Why generalization in RL is difficult: Epistemic pomdps and implicit partial observability. CoRR, abs/ hbs/2107.06277, 2021. URL attps://arxx wwwpwi.ww.aabaa......oooo wwtg/b///////eeee/.. http///b///2....0/0 .62777.
C. Guestrin, D. Koller, and R. Parr. Multiagent planning with factored mdps. In Advances in Neural Information Processing Systems 14 (Neural Information Processing Systems: Natural and Synthetie, NIPS, pp. 1523 1530. MIT Press, 2001.
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML, volume 80 of Proceedings of Machine Leurning Research, pp. 1856-1865. PMLR, 2018.
A. Hallak, D. Di Castro, and S. Mannor. Contextual markov decision processes. or 02259 [slat.MIJ,, 2015.
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In S. McIlraith and K. Weinberger (eds.), Proceedings of the Conference on Artificial Intelligence (AAAI'18). AAAI Press, 2018.
K. Holsheimer, F. Schubert, B. Beilharz, and L. Tsao. Coax: Plug-n-play reinforcement learning in python with gymnasium and jax. sttps://github com/ homm/coa--dee noax-dev/coax, 2023.
V. J. Parker-Holdee S. J. Nguyen, and Roberts. Provably efficient online hyperparameter optimization with population-basee bandits. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Proceedings of the 39rd International Conference on Advances in Neural Information Processing Systems (NeurIPS 20), volume 33, pp. 17200 17211, 2020.
M. Jaderberg, V. Dalibard, S. Osindero, W. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dumning, K. Simonyan, C. Fernando, and K. Kavukcuoglu. Population based training of neural networks ar arXiv:1711.09846 LC), 2017a.
M. Jaderberg, V. Mnih, W. Czarnecki, T. Schaul, J. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference on Learning Representations, ICLR. OpenReview.net, 2017b.
V. Jayawardana, C. Tang, S. Li, D. Suo, and C. Wu. The impact of task underspecification in evaluating deep reinforcement learning. CoRR, abs/2210.08607 2022.
A. Juliani, A. Khalifa, V. Berges, J. Harper, E. Teng, H. Henry, A. Crespi, J. Togelius, and D. Lange. Obstacle tower: A generalization challenge in vision, control, and planning. In Proceedings of the Twenty-Eights International Joint Conference on Artificial Intelligence, IJCAI, pp. 2684-2691. ijcai.org, 2019.
R. Kirk, A. Zhang, F. Grefenstette, and T. Rocktäschel. A survey of generalisation in deep reinforcement. learning. CoRR, abs/2111.09994, 2021.
R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. J. Artif. Intell. Res., 76:201 264, 2023.
P. Klink, C. D'Eramo, J. Peters, and Pajarinen. Self-pacod deep reinforcement learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS, 2020.
16

---

Published in Transactions on Machine Learning Research (06/2023)
J. Kober, A. Wilhelm, E. Öztop, and J. Peters. Reinforcement learning to adjust parametrized motor primitives to new situations. Aulonomous Robots, 33(4):361-38) 2012.
R. Koppejan and S. Whiteson. Neuroevolutionary reinforcement learning for generalized helicopter control. In Proceedings of the 11th Annual conference on Genetic and evolutionary compulation, GECCO '09, pp. 145-152. Association for Computing Machinery, 2009.
J. Kostas, Y. Chandak, Jordan, G. Theocharous, and P. Thomas. High confidence generalization for reinforcement learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Mochine Learning Research, pp. 5764 5773. PMLR, 18-24 Jul 2021.
H. Kurniawati. Partially observable markov decision processes and robotics. Annu. Rev. Control. Robotics Auton. Syst., 5:253-277 2022. doi: 10.1146/ www.aarree-cooooo000990000000000 /ttp:///00.///////oooooooooooo00 /amp://evcoooooooo00999000000000 URL
H. Küttler, N. Nardelli, A. Miller, R. Raileanu, M. Selvatici, E. Grefenstetle, and T. Rocktäschel. The nethack learning environment. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020. 2020.
M. Laskin, D. Yarats, H. Liu, Loe, A. Zhan, K. Lu, C. Cang, L. Pinto, and P. Abboel. URLB: Unsupervisod Reinforcement Learning Benchmark. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1, NewrIPS Datasets and Benchmarks 2021, December 2021, Virtual, 2021.
J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science in Robotics, 5, 2020a.
K. Lee, Y. Seo, S. Lee, H. Lee, and J. Shin. Context-aware dynamics model for generalization model-based reinforeement lcarning. In Proceedings of the 37th International Conference on Machine Learning, ICML, volume 119 of Proceedings of Muchine Learning Research, pp. 5757 5766. PMLR, 2020b.
X. Li, J. Zhang, Bian, Y. Tong, and T. Liu. cooperative multi-agent reinforcement learning framework for resource balancing in complex logistics network. In Proceedings of the 18th International Conference Autonomous Agents and MultiAgent Systems, AAMAS, pp. 980-988. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
M. Liu, M. Zhu, and W. Zhang. Goal-conditioned reinforcement learning: Problems and solutions. In Lue De Raedt (ed.), Proceedings of the Thirty-Firss International Joint Conference on Artificial Intelligence, IJCAI, pp. 3502-5511. ijcai.org, 2022.
M. Lu, Z. Shahn, D. Sow, F. Doshi Velez, and L. H. Lehman. Is deep reinforcement learning ready for practical applications in healthcare? A sensitivity analysis of duel-ddqn for hemodynamic management sepsis patients. In AMIA 2020, American Medical Informatics Association Annual Symposium, Virtual Event, USA, November 14-18, 2020. AMIA, 2020.
M. Machado, M. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. J. Artif. Intell. Res., 61: 523-562, 2018.
D. Malik, Y. Li, and P. Ravikumar. When is generalizable reinforcement learning tractable? In Marc' Aurelio Ranzato, Alina Beygekimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS, pp. 8032-8045, 2021.
T. Matiisen, A. Oliver, T. Cohen, and J. Schulman. Teacher-student curriculum learning. IEEE Trans. Neural Networks Learn. Syst., (9):3333 3740, 2020.
17

---

Published in Transactions on Machine Learning Research (06/2023)
B. Mchta, M. Diaz, F. Golemo, C. Pal, and L. Paull. Active domain randomization, In Lestic Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), 3rd Annual Conference on Robot Learning, CoRI, volume 100 of Proceedings of Machine Learning Research, pp. 1162 1176. PMLR, 2019.
T. Meng and M. Khushi. Reinforcement learning in financial markets. Data, 4(3):110, 2019.
A. Modi, N. Jiang, S. P. Singh, and A. Tewari. Markov decision processes with continuous side information.
In Algorithmic Learning Theory (8LT"18), volume 83, pp. 597 618, 2018.
J. Morimoto and K. Doya. Robust reinforcement learning. In Todd K. Leen, Thomas G. Dietterich, and Volker Tresp (eds.), Advances in Neural Information Processing Systems 13, Papers from Neural Information Processing Systems (NIPS) 2000, Denver, CO, USA, pp. 1061-1067. MIT Press, 2000.
S. Nguyen, N. Duminy, A. Manoury, D.e Duhaut, and C. Buche. Robots learn increasingly complex tasks with intrinsic motivation and automatic curriculum learning. Künstliche Intell., 35(1):81 10088,0000000 2021.
A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. CoRR, abs/ 0tss/1803.0990) 2018. URL atttp://arxin www.aawwwwwww..aaaa.....o.ooeeee
02999.
J. Holder, R. Rajan, X. Song, A. Biedenkapp, Y. Miao, T. Eimer, B. Zhang, V. Nguyen, R. Calandra, A. Faust, F. Hutter, and M. Lindauer. Automated reinforcement lcarning (autorl): A survey and open 2022.
problems. J. Arlij. Intell. Res.,
L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2817-2826 PMLR, 2017.
Kai Ploeger, Michael Lutter, and Jan Peters. High accelcration reinforcement learning for real-world juggling with binary rewards. In Jens Kober, Fabio Ramos, and Claire J. Tomlin (eds.), 4th Conference on Robot Learning, CoRL 2020, 16-18 Nouember 2020, Virtual Event / Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pp. 642 653. PMLR, 2020. URL attpps:/prooceedngs..tt pross/v155/ploeger222 html.
M. Ponsen, M. Taylor, and K. Tuyls. Abstraction and generalization in reinforcement learning: A summary and framework. In Adaptive and Learning Agents, Second Workshop, ALA 2009, Held as Part of the AAMAS 2009 Conference in Budapest, volume 5924 of Lecture Notes in Computer Science, pp. 1-32. Springer, 2009.
K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy mela-reiinforeement learning via probabilistic context variables. In K. Chaudhuri and R. Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning (ICML'19) volume 97, pp. 5331 5340. PMLR, 2019.
J. Rice. The algorithm selection problem. Advances in Computers, 15:65 118, 1976.
F. Runge, D. Stoll, S. Falkner, and F. Hutter. Learning to Design RNA. In Proceedings of the International Conference on Learning Representations (ICLR'19) 2019. Published online: iclr.cc
M. Samvelyan, R. Kirk, V. Kurin, J. Parker-Holder, M. Jiang, E. Hambro, F. Petroni, H. Küttler, E. Grefen- stette, and T. Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning rescarch. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NewrIPS Datasets and Benchmarks, 2021.
T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In F. R. Bach and D. M. Blci (eds.), Proceedings of the 32nd International Conference on Machine Learning (ICML '15), volume 37, pp. 1312-1320. Omnipress, 2015.
F. Schubert, M. Awiszus, and Rosenhahn. Toad gan: a flexible framework for few-shot level generation in tosen-based games. IEFE Transactions Games, pp. 1-1, 2021. doi: 10.1109/TG.200
18

---

Published in Transactions on Machine Learning Research (06/2023)
F. Schubert, C. Benjamins, S. Döhler, B. Rosenhahn, and M. Lindauer. Polter: Policy trajectory ensemble regularization for unsupervised reinforcement learning. Transactions on Machine Learning Research, April 2023. ISSN 2835-8856.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional contimuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Driessche, J. Schrittwieser, I. Antonoglou, V. Pan- neershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree scarch. Nature, 529(75877:4844488) 2016.
S. Sodhani, L. Denoyer, P. Kamienny, and O. Delalleau. Mtenv enviromment interface for mulit-task reinforcement learning. Github, 2021a. URL https://githhb thub. conn/faacebookreeeaccchhttttt
S. Sodhani, A. Zhang, and J. Pineau. Multi-task reinforcement learning with context- based representations. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9767-9779. PMLR, 18-24 Jul 2021b.
S. Sukhbaatar, E. Denton, A. Szlam, and R. Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning. CoRR, (bss/111.09000 2018.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. Riedmiller. Decpmind control suite. CoRR, abs/1801.00660 2018. URL //ap:vv/ org/ abs/1801.00690
J. van Rijn and F. Hutter. Hyperparameter importance across datasets. In Y. Guo and F.Farooq (eds.), Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 2367 2376. ACM Press, 2018.
J. Wang, Z. Kerth-Nelson, H. Soyer, J. Leibo, D. Tirumala, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning reinforcement learn. In G. Gunzelmann, A. Howes, T. Tenbrink, and Davelaar (eds.), Proceedings of the 39th Annual Meeting of the Cognétive Science Society. cognittiesciencesootttty.gg 2017.
J. Wang, M. King, N. Porcel, Z. Kurth-Nelson, T. Zhu, C. Deck, P. Choy, M. Cassin, M. Reynolds, H. Song, G. Buttimore, D. Reichert, N. Rabinowitz, L. Matthey, D. Hassabis, A. Lerchner, and M. Botvinick. Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Newrul Information Processing Systems Track on Dalasels and Benchmarks 1, NewrIPS Datasets and Benchmarks, 2021.
S. Whiteson, B. Tanner, M. Taylor, and P. Stone. Protecting against evaluation overfitting in empirical reinforcement learning. In 2011 IEEE Symposium on Adoplive Dynamic Programméng and Reinforcement Learning (ADPRL), pp. 120-127: 2011.
L. Xu, H. Hoos, and K. Browon Hydra: Automatically configuring algorithms for portfolio-based selection. In M. Fox and D. Poole (eds.), Proceedings of the founty-fourth National Conference on Artificial Intelligence (AAAI'10), pp. 210 216. AAAI Press, 2010.
D. Yarats, R. Fergus, A. Lazaric, and L.. Pinto. Reinforcement learning with prototypical representations. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11920-11931. PMLR, 18-24 Jul 2021.
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning (CoRL), 2019.
19

---

Published in Transactions on Machine Learning Research (06/2023)
A. Zhang, R. Thomas McAllister, R. Calandra, Y. Gal, and S. Levine. Learning invariant representations for reinforcement learning without reconstruction. In 9th International Conference Learning Represenlations, ICLR. OpenReview.net, 2021a.
A. Zhang, S. Sodhani, K. Khetarpal, and J. Pinean. Learning robust state abstractions for hidden-parameter block mdps. In Ith International Conference Learning Representations, ICLR 2021. OpenReview.ncs 2021b.
Zhang, R. Rajan, L. Pineda, N. Lambert, A. Biedenkapp, K. Chua, F. Hutter, and R. Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In The 24th International Conference Artificial Intelligence and Statistics, ATSTATS, volume 130 of Proceedings of Machine Learning Research, pp. 4015 4023. PMLR, 2021c
Zhang, H. Chen, D. Boning, and Hsich. Robust reinforcement learning on state obscrvations with learned optimal adversary. In 9th International Conference on Learning Represenlations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021d.
W. Zhou, L. Pinto, and A. Gupta. Environment probing interaction policies. Proceedings of the International Conference on Learning Representations (ICLR'19), 2019. Published online: iclr
Z. Zhou, X. Li, and R. Zare, Optimizing chemical reactions with deep reinforcement learning, ACS central science, 3(1)::1337 1344, 2017.
7. Zhu, K. Lin, and J. Zhou. Transfer learning in deep reinforcement learning: A survey. CoRR, /000/009007888 2020.
20

---

Published in Transactions on Machine Learning Research (06/2023)
Appendix
A Partial Observability in cMDPs
Discussing the visibility of context for an agent can be linked the partial observability we see in POMDPs. We believe it is useful to differentiate between the visibility of context and state features or observations both serve different function in cMDP. The state features describe the current state while the context. describes the current MDP. Therefore making dynamics in different ways. Therefore we define cMDP as special case of a POMDP, analogous to Kirk et al. (2023), where we have an emission function Φ S x Pc + O2 x O2 mapping the state space to some state obscrvation space O, and context observation space O & differentiates between state s and context c to allow different degrees of observability state and context, e-g. hiding the context completely but exposing the whole state, in order enable more flexible learning. It can also introduce the additional challenge of learning from imperfect noisy context information.
only partially observable should influence the learning
B Pendulum's Dynamic Equations
Because we use CARLPendulumEnv embedding gym's Pendulum (Brockman et al., 2016) for our task vari- ation experiment (see Section 6.1), we provide the dynamic equations to show the simplicity of the sys- tem. The state and observation consists of the angular position 0 and velocity 0 of the pendulum. The discrete equation defining the behavior of the environment is defined as follows:
tinestep
max. velocity B
contexta (default) gravity = -10, 'length' 1.,
gravity
M895
cantexti gravity -10.5, 'length' = 1.3,
length
B.+1 = 0 (- 39
sin sin + T) + m 12 uk At
context
Econtext2
ak+1 04 +0++1 At.
context 1
context 2
Here, h is the index of the iteration/step. The dynamic system is parametrized by the context, which consists of g the gravity, / and m the length and mass of the pendulum, w the control input and Al the timestep. Figure 10 shows how Pendulum is embedded in CARL.
CARLPendulumEnv
Figure 10: CARLPondulumEaw
C Hyperparameters and Hardware
Hyperparameters and Training Details We implemented our own agents using coax (Holsheimer el al., 2023) with hyperparaneters specified in Table All experiments can be reproduced using the scripts we provide with the benchmark library at https: //tp:////noomusu/ 4open. /ct.eencee/////AA/A//////////
Hardware All experiments on all benchmarks were conducted on slurm CPU and GPU cluster (see Table 2). On the CPU partition there are 1592 CPUs available across nodes.
D Additional Experimental Results
In this section, provide additional information and results for experiments section (section 6).
D.1 Task Variation Through Context
Following the experimental setup in Section 6.1 we conducted further experiments on representative CARL- environments.
21

---

Published in Transactions on Machine Learning Research (06/2023)
Table 1: Hyperparameters for algorithm and environment combinations
algorithm
cb1 CartPole
c51 Acrobos
RAC
c51
c51
EAC
AAC
KaC
env
Pendulum
MountainCar
Lenarlander
Dme Walleer
Quncuuudd
Halfchostah
n_step
ganoma
0.8
0.98
0-98
0.8
0.89
alpha
0.3
0-2
0.2
128
128
learning rate
0.001
0.001
001
warmap_
Frabica)
ta--aappp---mm Teatnee
7500
7500
updata freg
capacity
100000
100000
100000
100000
100000
100000
100000
1000000
network
(wndth': 256,
('width': 256,
('wideh'
Twidsh's 32,
{ ('width': 255,
('wndti'' 200)
('wndth's
F'rrdch''
"oum oiouss
"auun aloooo
200)
"rum atomss
stoums"
200]
1024)
51)
61)
51)
513
Pi tomperature
0.1
0.1
NaN
0.2
0.1
NaN NaN
NaN
NaN
q min value
0.0
-60.0
NaN
-100.0
-100.0
NaN
NaN
q m_____yal value
110.0
0.0
NaN
100.0
100.0
NoN
NaN
NaN
Table 2: GPU cluster used for training
Type
Model
Quantity
RAM CPU (G)
GPU
NVIDIA Quattro M5000
256
GPU
NVIDIA RTX 2080 Ti
384
GPU
NVIDIA RTX 2080 Ti
12
256
GPU
NVIDIA RTX 1080 Ti
512
GPU
NVIDIA GTX Titan X
128
GPU
NVIDIA GT 640
32
D.2 Adding Context to the State
When we concatenat all available context features to the observation for CARLPendulum, we often see that the algorithm fails to learn mcaningful policy on some seeds, see Figure 11.
-500
concat (all)
Return
concat (non-static) hidden
-1000
20000 40000 60000 80000 100000 Step
Figure 11: CARLPennulum with different lengths and seeds. Train performance.
22

---

Published in Transactions on Machine Learning Research (06/2023)
D.3 Generalization Results
Here provide two more combinations for CARLPendulun for the Kirk generalization protocol (Kirk el al., 2021) from Section 6.4 (same experimental setup).
update interval hidden
mode:
mode:
mode: C
(0.001, 0.00391]
(0.0058, 0.0077]
(0.0096, 0.0115]
(0.0134. 0.0153]
(0.0172. 0.01911
(0:021.
(0.0248 0.0267]
(0.0286 0.0305)
00.0324. 0.0343]
(0.0362, 0.0381]
update interval concat
(0.001, 0.003911
(0.0058. 0.0077]
00.00960.0111)
(0.0134, 0.0153]
(0.0172,00 0.0191]
(0.021. 0.0229]
(0.0248, 0.0267]
(0.0288. 0.0305]
(0.0324. 0.0343]
(0.0362, 0.0381]
2.8.
(4.6, 5.8]
(8.2. 9.4]
(11.8, 13.0)
(154, 16.6]
(19.0, 2]
(22 23.6)
(0.9, 2.2]
(4.6, 5.8
(8.2. 9.4]
(11.8 13.0)
6)
(19 20.2)
(22 23.8]
(0.8. 2.2]
(4.8. 5.8]
gravity y (15.4 18.6)
(8.2, 9.4]
(11.8, 13.0)
(18.0, 20.2)
(22 23.8)
gravity
gravity J
Figure 12: Varying gravity and update interval on CARLPendulmmEnn First row: Hidden agent, second
row: Concat agent.
23

---

Published in Transactions on Machine Learning Research (06/2023)
mode: A
mode:
mode: C
gravity hidden
(3.4, 4.6]
(5.8,
(8.2, 9.4]
(10.6, 11.8)
(13.0, 14.2]
(15.4, 16.6]
(17.8, 19.0)
21.4)
(22.6, 23.8)
(0.001,
gravity concat
4.6]
(5.8, 7.0]
9.4)
(10.6, 11.8)
(13.0, 14.2]
(15.4, 16.6]
(17.8, 19.0]
(20 21.4)
(22.6, 23.8)
0.10
0.29)
0471
0.65
(o81)
0:111
291
levo
toe se (eas
las 12r sz
Iss es
les 1z
Lo1 se
0AR
(2g)
(z
deg
pole_ length
pole_lengl length
pole pol__ength
Figure 13: Varying gravity and pole length on CARL.Pandulumnss First row: Hidden agent, second row: Concat agent.
24

---

Published in Transactions on Machine Learning Research (06/2023)
E
Baselines
Here we provide basclines for selected environments in CARL. For each environment we conduct the following experiments. First, we train a default agent on the default enviromment, i.e. with no context variation. This agent is then evaluated on context variations of single context features with magnitude E .0.....2...... This creates an initial performance profile and shows how sensitive the agent to which context feature. We plot this an cCDF, the further right the lines are, the more high returns the agent achieves. This plot features no aggregation. After that, we train agent. with context variation. For this we select context. feature with visible impact performance and determine the range by including magnitudes where the enviromment is still solvable which means reaching returns better than fail. We train the agent with varying context visibility: Once, the varying context feature is hidden, once the complete context set is concatenated to the state and once only the changing context feature. We plot the training curve. Finally, we test the agents trained on context variation on the same number contexts sampled from the train distribution and report the return histogram and return statistics. If not specified otherwise, we perform each experiment with 10 seeds and 128 training contexts.
We report on classic control (CARLPendulumnEw: Figure 14, CARLMounlainCarFEuw Figure 15, CARL- CartPoleEnv: Figure 16 and CARLAcrobotEnv: Figure 17) and box2d (CARLLamarLannderFnn Figure 18).
max speed
1.00
Proportion
0.75
0.50
1.2
0.25
02
1.3
0.3
1.4
0.00
10'
10'
10"
10
10
10' 10"
0.4
1,5
Return
1.6
initial initial_angle____ max
initial_velocity velocity_maa _mx
0.6
1.7
1.00
0.7
1.8
Proportion
0.75
1.9
0.50
0.9
2.0
1.0
2.1
0.25
2.2
0.00
-10' 110 -10
-16
10'
10'
-10"-10
11'
"0" Retum
10' 10'
-10" -10'
-10 "000 10' Return
10'
Return
(a) Default agent evaluated on context variation. eCDF Plot. A is the magnitude multiplied with the default value of each context feature.
visibility hidden
Median
hidden concat (all)
15
concat (all)
Frequency
concat (non-static)
concat (non-static)
-250
10
hidden
IQM
-800
concat (all)
Return
750
5
concat (non-static)
Mean
1000
(ail)
hidden concat (all)
concat (nonistatic)
-1250
hidden
0
1000
0
concat (non-static)
20000
40000
60000 80000 100000 Steps
Return
4000
-320
-240
(b) Training Curve
(c) Eval. train distr. (Histogram)
(d) Eval. on train distr. (Statistics)
Figure 14: CARLPendulum Env: Benchmark. Algorithm sac, 10 seeds. (a): Default agent (trained default context and context-oblivious) evaluated on context variations. (b-d): Training with varying context visibility. We vary the context feature(s) ['1'] (A ~ U(0.5,22 2.2)).
25

---

Published in Transactions on Machine Learning Research (06/2023)
min position
position
speed
goal position
1.00
Proportion
0.75
0.50
0.1
1.2
0.25
02
1.3
1.4
0.00
200
1000
0.4
1.5
Return
1.6
goal velocity
force
gravity
1.7
1.00
0.7
1.8
Proportion
0.75
1.9
0.50
0.9
1.0
2.1
0.25
1.1
2.2
0.00
-200
-100
-200
-100
-20
-100
Return
Return
Return
(a) Default agent evaluated on context variation. eCDF Plot. A is the magnitude multiplied with the default value of each context feature.
Median
60
visibility
hidden concat (all)
hidden
-120
50
concat (all)
concat (non-static)
Frequency
concat (non-static)
40
IQM
Retum
hidden
-60
context visibility hidden
30
concat (all)
concat (non-static)
20
-180
concat (all)
Mean
hidden concat (all)
concat (non-static)
-00
0
100000 200000 300000 Step
-200
-150
-100
-50
concat (non-static)
Return
136
128
-120
(b) Training Curve
(c) Eval. train distr. (Histogram)
(d) Eval. train distr. (Statistics)
Figure 15: CaRL Benchmark. Algorithm c51, 10 seeds. (a): Default agent (trained on default context and context-oblivious) evaluated on context variations. (b-d): Training with varying context. visibility. We vary the context feature(s) [' ['gravity'] (A ~ ((0.1,2)).
26

---

Published in Transactions on Machine Learning Research (06/2023)
gravity
masscart
masspole
pole length
1.00
Proportion
0.75
0.50
0.1
1.2
02
1.3
0.25
1.4
0.00
0.4
1.5
1.6
aace_mmanifier
update_ _interval
initial state_ lower
initial saall_sttt___aae upper
1.00
1.7
0.7
1.8
Proportion
0.75
1.9
0.50
1.0
2.1
0.25
1.1
2.2
0.00
0
200
400
0
200
400
200
400
0
200
400
Return
Return
Return
(a) Default agent evaluated on context variation. eDDF Plot. A is the magnitude multiplied with the default value of each context feature.
Median
200
visibility hidden
hidden concat (all)
450
Frequency
150
concat (non-static)
concat (all)
concat (non-static)
IQM
Return
hidden
100
400
context visibility hidden
concat (all)
concat (non-static)
350
concat (all)
Mean
concat (non-static)
hidden concat (all)
0
25000 50000 75000 100000 Step
200
400
concat (non-static)
Retum
420
450
480
(b) Training Curve
(c) Eval. train distr. (Histogram)
(d) Eval. train distr. (Statistics)
Figure 16: CARLCartPoleEnw: Benchmark. Algorithm c51, 10 seeds. (a): Default agent (trained on default context and context-obliviou) evaluated on context variations. (b-d): Training with varying context. visibility. We vary the context feature(s) ('pole_length') (A ~ ((0.2,2.2)).
27

---

Published in Transactions on Machine Learning Research (06/2023)
link length
link length_
link mass
link mass
1.00
Proportion
0.75
0.50
0.25
0.00
link_com_
link_com, com_2 _
link mooi
max max_velood velocity_ 1
1.00
Proportion
0.75
0.50
0.1
1.2
0.25
02
1.3
1.4
0.00
0.4
1.5
1.6
max velocity 2
torque noise max
initial angle lower
initial angle upper
1.7
1.00
0.7
1.8
Proportion
0.75
1.9
0.9
2.0
0.50
1.0
2.1
0.25
1.1
2.2
0.00
-400
-200
-400
-200
Retum
Return
initial welocity Jower
initial velocity upper
1.00
Proportion
0.75
0.50
0.25
0.00
-200
-400
-200
Return
Return
(a) Default agent evaluated context variation. eCDF Plot. A is the magnitude multiplied with the default value of each context feature.
40
Median
visibility
hidden concat (all)
hidden
30
concat (all)
concat (non-static)
Frequency
concat (non-static)
IQM
Return
20
hidden concat (all)
context visibility
-300
concat (non-static)
hidden
concat (all)
Mean
hidden concat (ail)
-400
concat (non-static)
50000 100000 150000
-400
-200
concat (non-static)
Step
Return
200
160
120
(b) Training Curve
(c) Eval. on train distr. (Histogram)
(d) Eval. on train distr. (Statistics)
Figure 17: CARLAcrobotEnw: Benchmark. Algorithm c51, 10 seeds. (a): Default agent (trained on default context and context-oblivious) evaluated on context variations. (b-d): Training with varying context. visibility. We vary the context feature(s) ['link_mase mass_2 2'] (A 2.2.),
28

---

Published in Transactions on Machine Learning Research (06/2023)
FPS
SCALE
MAIN_ENGIIN ENGINE POWERSIDE ENGINE POWER
1.00
Proportion
0.75
0.50
0.25
0.00
INITIAL RANDOM
GRAVITY X
GRAVITY Y
LEG AWAY
1.00
0.75
Proportion
0.50
0.1
1.2
0.25
02
1.3
1.4
0.00
0,4
1.5
0.5
1.6
LEG DOWN
LEG W
LEG H
LEG SPRING TORQUE
1.00
1.7
0.7
1.8
Proportion
0.75
1.9
0.50
1.0
2.1
0.25
1.1
2.2
0.00
SIDE ENGINE HEIGHT SIDE ENGINE AWAY
VIEWPORT W
VIEWPORT
1.00
Proportion
0.75
0.50
0.25
0.00
-10--1"'100--1" P10"
-1d--10'-nd-10" 10 1d Return
-10--t0--0--1d P10 -10-1o2-1d7-11 "10"p" 10' Return
Return
Retum
(a) Default agent evaluated context variation. eCDF Plot. A is the magnitude multiplied with the default value of each context feature.
Median
visibility hidden
hidden concat (all)
concat (all)
concat (non-static)
Frequency
concat (non-static)
IQM
Return
6
hidden
-100
context visibility hidden
concat (all)
4
concat (non (mon-static)
-200
concat (all)
2
hidden concat (all)
Mean
concat (non-static)
100000 Step
200000
2000
1000
concat (non-static)
Return
-25
25
50
(b) Training Curve
(c) Eval. on train distr. (Histogram)
(d) Eval. on train distr. (Statistics)
Figure 18: CARLLunarLandr-Ew: Benchmark. Algorithm c51, 10 seeds. (a): Default agent (trained on default context and context-oblivious) evaluated on context variations. (b-d): Training with varying context. visibility. We vary the context feature(s) [' (GGAVIIT_Y') (A ~ (((01,,.22)
29

---

Published in Transactions on Machine Learning Research (06/2023)
E.1
Context- Conditionnng CARLMarioEnv
Evaluation Episode Completion
0.50
Training Episode Completion
0.8
0.45
0.40
0.6
0.35
0.30
0.4
0.25
0.20
0.2
concat (non-static) hidden
0.15
0.2
0.4
0.6
1.0
0.0
0.4
0.8
Training Steps
1e7
Training Steps
1e7
Figure 19: CARLMarioEnv: Evaluation of the PPO agent on 16 TOAD-GAN training levels and 16 different test levels. The episode completion indicates the average distance from the level start that the agent is able to reach.
We train the Proximal Policy Optimization (PPO) agent on 16 distinct levels of the CARLMarioEnv environment and evaluate its performance on another 16 different levels for 10 seeds. In the context-aware variant of the agent, both the policy and value functions are conditioned on the provided context. Unlike the contexi-aware agent, the hidden agent only receives RGB frames from the environment as input. The context is encoded using a convolutional encoder and integrated into the hidden state representation, which is then fed into the policy and value heads. The context itself consists of noise tensor sampled from which TOAD-GAN (Awiszus et al., 2020; Schubert et al., 2021) generated the Super Mario Bros. levels. To ensure playability, the generated levels are filtered using static analysis.
As shown in Figure 19, the training performance of the context-aware agent and the agent without context is nearly identical. However, when evaluated, the context-awand agent outperforms the agent without context, demonstrating its ability effectively incorporate the noise map context into its policy. This observation underscores the value of the CARL benchmark in driving research on context representation.
E.2 Test Performance on Context Variations
In this section we show the test performance of an agent trained on the default context of additional selected environments which oblivious the context. For evaluation we rum 10 episodes on contexts with different magnitudes of variation. We vary each context feature by a magnitude A 0.1, 0.3, 2.2.
30

---

Published in Transactions on Machine Learning Research (06/2023)
LEG H
SPEED KNEE
TERRAIN STARTPAD D
TERRAIN STEP
1.00
Pro Proportion
0.75
0.50
0.25
0.00
MOTORS TORQUE
TERRAIN GRASS
TERRAIN LENGTH
GRAVITY
Proportion
0.75
0.50
0.25
0.00
A
LEG W
FPS
FRICTION
LIDAR RANGE
0.1
1.2
1.00
0.2
1.3
Proportion
0.75
1.4
0.4
1.5
0.50
0.5
1.6
0.6
1.7
0.25
0.7
1.8
0.00
0.8
1.9
0.9
2.0
SCALE
SPEED HIP
INITIAL RANDOM
LEG DOWN
LEG_DOWN
1.0
2.1
1.00
1.1
2.2
Proportion
0.50
0.25
0.00
500
500
Return
Return
TERRAIN HEIGHT
GRAVITY Y
1.00
Proportion
0.75
0.50
0.25
500
500
Return
Return
Figure 20: CARLBiiedaaWalkerrEn ECDF Plot. A is the magnitude multiplied with the default value of each context feature.
31

---

Published in Transactions on Machine Learning Research (06/2023)
actuator strength
density
joint damping
friction torsional
1.00
Proportion
0.75
A
0.50
1.2
0.25
02
1.3
0.3
1.4
0.00
500
1000
1.5
Return
0.5
1.6
viscosity
friction friclionrroliing
friction tangential
0.6
1,7
1.00
1.8
Proportion
0.75
1.9
0.50
0.9
2.0
2.1
0.25
1.1
2.2
0.00
500 Return
1000
500 Return
1000
500 Return
1000
Figure 21: CARLDmcFishEnw: ECDF Plot. A the magnitude multiplied with the default value of each context feature.
gravity
friction tangential
friction torsional
friction frictionrrolinng
joint damping
1.00
Proportion
A
0.1
0.50
0.2
0.25
0.3
0.4
0.5
0.8
1.00
soint_stiffnest
actuator_strre strength
density
viscosity
geom_density
0.7
Proportion
0.50
0.00
500 Return
1000
0
500
1000
Return
1.5
wind
wind
wind _z
1.6
Proportion
0.75
1.8
1.9
0.50
2.0
2.1
2.2
0.00
500 Return
1000
500 Return
1000
500 Return
1000
Figure 22: CARLDnnCQuadupedIInn ECDF Plot. A the magnitude multiplied with the default value of each context feature.
32

---

Published in Transactions on Machine Learning Research (06/2023)
gravily
friction tangential
friction torsional
friction rolling
timestep
Proportion
0.75
0.25
0.00
0.5
1.00
joint_dann damping
Joint_stiffness stiffness
actuator_strength sstrength
density
viscosity
0.7
0.8
Proportion
0.75
1.0
1.1
0.25
1.2
1.3
0.00
500 Return
1000
500
1000
1.4
Return
1.5
geom density
wind y
1.6
1.00
1.7
Proportin
0.75
1.8
1.9
0.25
500
1000
500 Return
1000
500 Return
1000
Retum
Figure 23: CARLDmeWalkerBan ECDF Plot. A is the magnitude multiplied with the default value of each context feature.
33

---

Published in Transactions on Machine Learning Research (06/2023)
F Hyperparameter Optimization in cRL
We have observed significant differences in learning performance for hidden and full visible context, but the same is also true for hyperparameter tuning in both of these settings. We use the same DQN and DDPG algorithms as in our other experiments with a narrow context distribution of 0.1 for the CARL Pendulum, Acrobot and LunarLander environments to show this point. To tune the hyperparameters, we use PB2 (J. Parker-Holder et al., 2020) for the learning rate, target update interval and discount factor.
As shown in Figure 24, the evaluation performances the found hyperparameter schodules differ significantly in terms of learning speed, stability and results per environment. Providing the context sometimes seems to increase the difficulty of the problem (see Section 6.4), i.e. finding a good hyperparameter configuration happens more often and more reliably when the policy is not given the context. We can only speculate on the reasons why this happens, but shows that context introduces complexities to the whole training process beyond simply the policy architecture.
G Open Challenges in cRL
We used the concept of Contextual Reinforcement Learning and its instantiation in CARL to demonstrate the usefulness of context information in theory and in practice. More specifically, showed that making such information about the environment explicitly available to the agent enables faster training and transfer of agents (see Section 6). While this already provides valuable insights to the community that increasingly cares about learming agents capable of generalization (see Sections 1 & 3) Contextual Reinforcement Learning and by extension CARL enables to study further open challenges for general RL.
G.1 Challenge I: Representation Learning
Our experiments demonstrated that an agent with access to context information can be capable of learning better than an agent that has to learn behaviors given an implicit context via state observations, but the naive method of including context. information in the observation is not reliable. We theorize that disentangling the representation learning aspect from the policy learning problem reduces complexity. As CARL provides ground truth for representations of environment propertics we envision future work on principled studics of novel RL algorithms that, by design, disentangle representation learning and policy learning (see, e.g., (Rakelly el al., 2019; Fu et al., 2021a; Zhang et al., 2021b) first works along this line of research). The ground truth given by the context would allow to measure the quality of learned representations and allows to relate this to the true physical properties of an environment.
Another direction of research under the umbrella of representation learning follows the work of environment probing policies (Zhou et al., 2019). There, exploratory policies are learned that allow one to identify which environment type an agent encounters. This complementary to the prior approaches as representations are not jointly learned with the behaviour policies in the previously discussed approaches but rather in separate offiine phasc. Based on CARL, huge amounts of meta-data could be collected that will enable the
visible
Noden
visible
-200
14600 PoAATE
-30
100
1000
-400
-200
1500
-300
0.5
16b
leb
240
(a) CaRLAcrobotEny
(b) CARLLamarLanderra
(c) Cen.ulnnuu
Figure 24: Hyperparameter Optimization with PB2 (J. Parker-Holder et al., 2020). Hidden means that the context is hidden, and visible means that the full context is appended to the observation.
34

---

Published in Transactions on Machine Learning Research (06/2023)
community to make use of classical meta-algorithmic approaches such algorithm selection (Rice, 1976) for selecting previously learned policies or learning approaches.
G.2 Challenge II: Uncertainty of RL Agents
With access to context information, we are able to study the influence of noise on RL agents in novel way. While prior environments enabled studies on the behavior of agents when they could not be certain about their true state in a particular environment, the framework of Contextual Reinforcement Learning further allows studying agents' behaviors scenarios with unccrtainty on their current contextual environment, o.g., because of noise on the context features. In the practical deployment of RL, this is a reasonable concern since context. features have to be measured somehow by potentially noisy sensors. As this setting affects the overall transition dynamics, Contextual Reinforcement Learning provides unique test-bed in which the infiuence of uncertainty can be studied and how RL agents can deal with such. This line of research can also be combined with the work on unsupervised RI. (Laskin et al., 2021; Schubert et al., 2023), where the agent Iearns a good policy initialization during a unsupervised pretraining phase, followed by a finetuning phase where the agent optimizing an external reward. CARL enables researchers to either guide the pretraining process via uncertainty measures that are based on the context information or evaluate the robustness of finetuned policies under context noise.
G.3 Challenge III: Interpretable and Explainable Deep RL
Trust in the policy is crucial factor, for which interpretability or explainability often is mandatory. With the provided ground truth through the explicit use of context features, Contextual Reinforcement Learning could be the base for studying the interpretability and explainability of (deep) RL. By enabling AutoRL studies and different representation learning approaches, Contextual Reinforcement Learning will contribute to better interpreting the training procedures.
Contextual Reinforcement Learning further allows studying explainability on the level of learned policies. We propose study the sensitivity of particular policies to different types of contexts. Thus, the value and variability of a context might serve as proxy to explain the resulting learned behavior. Such insights might then be used to predict how policies might look or act (e.g., in terms of frequency of action usage) in novel enviromments, solely based on the provided context features.
G.4 Challenge IV: AutoRL
AutoRL (Parker-Holder et al., 2022) addresses the optimization of the RL learning process. To this end, hyperparameters, architectures or both of agents are adapted either on the fly (Jaderberg et al., 2017a; Franke et al., 2021) or once at the begimning of a run (Runge al., 2019). However, as AutoRD typically requires large compute resources for this procedure, optimization most often done only on a per-environment basis. It is reasonable to assume that such hyperparameters might not transfer well to unseen environments, as the learning procedures were not optimized to be robust or to facilitate generalization, but only to improve the reward on particular instance.
As we have shown above in Appendix F, Contextual Reinforcement Learning provides an even greater challenge for AutoRL methods. On the other hand, as CARL provides easy-to-use contextual extensions of a diverse set of RL problems, could be used to drive rescarch this open challenge of AutoRI. First of all, it enables a large scale-study to understand how static and dynamic configuration approaches complement each other and when one approach is to be preferred over another. Such study will most likely also lead to novel default hyperparameter configurations that are more robust and tailored to fast learning and good generalization. In addition, it will open up the possibility to study whether it is reasonable to use a single hyperparameter configuration or whether a mix of configurations for different instances is required (Xu el al., 2010). Furthermore, with the flexibility of defining a broad variety of instance distributions for a large set of provided context features, experiments with CARL would allow rescarchers to study which hyperparamcters play a crucial role in learning general agents similar to studies done for supervised machine learning (van Rijn & Hutter, 2018) or AT algorithms (Biedenkapp et al., 2018).
35

---

Published in Transactions on Machine Learning Research (06/2023)
G.5 Challenge V: High Confidence Generalization
The availability of explicit context enables tackling another challenge in the field of safe RL. High Confidence Generalization Algorithms (HCGAs) (Kostas et al., 2021) provide safety guarantees for the generalization of agents in testing environments. Given a worst-casc performance bound, the agent can be tasked to decide whether a policy applicable in an aut-of-distributie for the deployment of RT. algorithms the real world where policy failures can be costly and the context of an environment is often prone change. Contextual Reinforcement Learning has the potential to facilitate the development of HCGAs that base their confidence estimates on the context of an environment.
context or not. This setting is especially important
H Context Features for Each Environment
We list all registered context leatures with their defaults, bounds and types for each environment family in Table 3 (classic control), Table 4 (box2d), Table (brax) and Table 7 (RNA and Mario).
Context Features: Defaults, Bounds and Types for OpenAl gym's Classic Control environ- ments (Brockman al., 2016)
Table
(a) CARLCartPoleEnv
(b) CARLPendulumEnr
Context Feature
Default
Bounds
Type
Context Feature
Default
Bounds
Type
force magnifier
10.00
(1, 100)
dt
0.05
(0, inf)
float
gravily
9.80
(0.1, inf)
float
10.00
(0, inf)
float
initial state lower
-0.10
(-inf, inf)
float
initial angle
3.14
(0, inf)
float
initial state upper
0.10
(-inf, inl)
float
initial _el___eellll_y_leeee max
1.00
(0, inf)
float
masscart
1.00
(0.1, 10)
float
1.00
(1e-06, inf)
float
masspole
0.10
(0.01,
float
pole Jength
0.50
(0.05, 5)
float
m max speed
1.00
(1e-06, inf)
float
8.00
(-inf, inf)
float
update interval
0.02
(0.002, 0.2)
float
(d) CARLActobotEny
(c) CARLMonntainCarFFn
Context Feature
Default
Bounds
Type
Context Feature
Default
Bounds
Type
initial aagll_ngeaaaa lower
-0.10
(-inf, inf)
float
initial _aglle upper
0.10
(-inf, inf)
float
force
0.00
(-inf, inf)
float
initial velloiit lower
-0.10
(-inf, inf)
float
goal position
0.50
(-inf, inf)
float
initial liciall_vllctt__uue upper
0.10
(-inf, inf)
float
goal velocity
0.00
(-inf, inf)
float
0.50
(0, 1)
float
gravity
0.00
(0, inf)
float
link com 2
0.50
(0,
float
max position
0.60
(-inf, inf)
float
link length 1
1.00
(0.1, 10)
float
max pos.tssiiion.... start
-0.40
(-inf, inf)
float
link length 2
1.00
(0.1, 10)
float
speed
0.07
(0, inf)
float
link_mass mass link mass 2 link_moi moi max uel_cielocity... max _el___eloiity___ torque noise max
1.00
(0.1,
float
max velooelloyyyyllll start
0.00
(-inf, inf)
float
1.00
10),
float
min position
-1.20
(-inf, inf)
float
1.00
(0,1
float
min position start
-0.60
(-inf, inf)
float
12.57
(1.257, 125.7)
float
min velocity start
0.00
(-inf, inf)
float
28.27
(2.827, 282.7)
float
0.00
(-1.0, 1.0)
float
36

---

Published in Transactions on Machine Learning Research (06/2023)
Table Context Features: Defaults, Bounds and Types for OpenAl gym's Box2d environments (Brockman
al., 2016)
(a) WaRLee
(b) CARLLunarZandeeEw
Context Feature
Defanlt
Bounds
Type
FPS
50.00
(1, 500)
float
Context Feature
Default
Bounds
Type
FRICTION
(0, 10)
GRAVITY X
0.00
(-20, 20)
Boat
FPS
50.00
(1, 500)
float
GRAVITY
10.000
(-20, -0.01)
Boat
GRAVITY
0.00
(-20, 20)
float
INITIAL, INITIAI_RANNIR RANDOM
5.00
(0, 50)
float
GRAVITY Y
-10.00
(-20, -0.01)
float
LEG DOWN
-0.27
(-2, -0.25)
float
INITIAL RANDOM
1000.00
[0, 2000)
float
LEG H
LEG AWAY
(0, 50)
float
LEG W
0.27
(0.25, 0.5)
Bloat
LEG DOWN
18.00
(0, 50)
flont
(0.5, 20)
(1, 20)
MOTORS TORQUE
80.00
(0, 200)
Bloat
LEG SPRING TORQUE
40.00
(0, 100)
float
SCALE
30.00
(1, 100)
Blat
LEG W
2.00
(1, 10)
flnat
SPEED HIP
4.00
(1e-06, 15)
float
MAIN ENGINE_
13.00
(0, 50)
float
SPEED KNEE
6.00
(1e-06, 15)
float
SCALE
30.00
(1, 100)
float
TERRAIN GRASS
(5, 15)
ENGINE AWAY
(1, 20)
float
TERRAIN HEIGHT
5.00
(3, 10)
float
SIDE ENGINE HEIGHT
14.00
(1. 20)
finat
TERRAIN LENGTH
200.00
(100, 500)
ENGINE, POWER
[0, 50)
TERRAIN STARTPAD
20.00
(10, 30)
int
VIEWPORT
400.00
(200, 800)
int
TERRAIN STEP
0.47
(0.25, I)
float
VIEWPORT W
600.00
(400, 1000)
VIEWPORT H
400.00
(200, 800)
VIEWPORT W
600.00
(100, 1000)
int
(c) CARLVehicleRacinngauy
Context Feature
Default
Bounds
Type
VEHICLE
categorical
37

---

Published in Transactions on Machine Learning Research (06/2023)
Table 5: Context Features: Defaults, Bounds and Types for Google Brax environments (Freeman et al., 2021)
(a) CARLAnt
(b) CARh.Halfcheetah
Context Feature
Default
Bounds
Type
Context Feature
Default
Bounds
Type
augular damping
actuator actustor_streogth
gravity
felction
300.00
-0.05
-9.80
0.60
(-inf, -0.1)
[-inf, inf)
(-lnf, inf)
Inf)
float
float
float
float
gravity
joint angular damping
angular damping
friction
-0.05
-9.80
0.60
(-inf, inf)
(-iuf, inf)
(-inf, -0.1)
inf)
float
float
float
float
joint sttifffssesss
joint angular damping
5000.00
35.00
(0, inf)
inf)
float
float
coint__stiffone
15000.00
(1, inf)
fooat
torso__ms mass
10.00
(0.1, inf)
float
torso_mass mass
9.46
(0.1, inf)
float
(c) CARLFetch
(d) CARLGrasp
Context Feature
Default
Bounds
Type
Context Feature
Default
Bounds
Type
actuator strength
300.00
[1, inf)
fooat
actuator strength
300.00
(1, inf)
float
nogular_dampal
-0.05
(-inf, inf)
float
pagular_ddmpiin
-0.05
(-inf, inf)
flaas
friction
0.60
(-inf, inf)
float
friction
0.60
(-inf, inf)
flot
gravity
-9.80
(-inf, 10.
float
gravity
-9.80
(-inf, -0.1)
floas
angular damping
35.00
inf)
float
joint angular damping
50.00
Bost
joint stiffaess
5000.00
(1, inf)
float
joint stiffness
5000.00
(1, inf}
float
target distance
(0.1, inf)
float
target distance
10.00
(0.1, inf)
target_radiie radius
2.00
(0.1, inf)
float
target_hhiige
8.00
(0.1, inf)
floas
torso mass
1.00
(0.1, inf)
float
target_raddus radlus
1.10
(0.1, inf)
float
(I) CARLUr5e
(e) CARL.Humanoid
Context Feature
Defsult
Bounda
Type
Context Feature
Default
Bounds
Type
actuator actuator_streng _sttength
100.00
(1, inf)
Hoat
ongular_damplnn
-0.05
(-inf, inf)
float
angular damping
-0.05
(-inf, inf)
float
frietion
0.60
(-inf, inf)
float
friction
0.60
(-inf, inf)
float
gravity
-9.81
(-inf, -0.1)
float
gravity
-9.80
(-inf, -0.1)
float
joint angular damping
50.00
(9, 360)
float
joint angular damping
20.00
(0, inf)
flout
joint stiffuess
40000.00
(1, inf)
foot
torso mass
8.91
(0.1, inf)
float
target distance
0.50
(0.01, inf)
foott
farget_radius radius
0.02
(0.01, inf)
float
1.00
inf)
float
38

---

Published in Transactions on Machine Learning Research (06/2023)
Table 6: Context Features: Defaults, Bounds and Types for Google Deepmind environments (Tassa et al.,
2018)
(a) CARLDuwWalkeraay
(b) CARLDucQuudruedeBrr
Context Feature
Default
Bounds
Type
Context Feature
Default
Bounds
Type
actuator_ actuator_strengge strength
1.00
(0, inf)
float
actuator _stenngt
1.00
(0, inf)
Hoas
density
0.00
(0, Inf)
float
density
0.00
(0. Inf)
float
friction rolling
1.00
(0, inf)
float
friction rolling
1.00
(0. inf)
float
1.00
(0, inf)
float
friction tangential
1.00
(0, inf)
flost
friction torsional
inf)
float
friction torsional
float
geom density
1.00
(0, inf)
float
geom density
1.00
(0, inf)
flout
gravity
(-inf, -0.13
gravity
-9.81
(-inf, -0.1)
flnas
joint_ _ain__dmpinggg
1.00
(0, inf)
float
1.00
(0, inf)
floas
joint stiffness
0.00
(0, inf)
float
joint stifluess
0.00
(0. inf)
float
timestep
0.00
(0.001, 0.1)
float
timestep
0.01
(0.001, 0.1)
flout
Viscosity
0.00
(0, inf)
float
viscosity
0.00
(0, inf)
float
wind
(-inf,
float
wind
(-inf, inf)
float
wind
0.00
(-inf, inf)
float
wind
0.00
(-inf, inf)
flout
wind_?
0.00
(-inf, inf)
wind_ x
(-inf, inf)
floas
(c) CnggDDmd'ingerEmn
(d) CARLDmoFishhaav
Context Feature
Default
Type
actuator strength
1.00
(0, inf)
float
Context Feature
Default
Bounds
Type
density
5000.00
(0, inf)
float
friction_rrollee rolling
1.00
(0, inf)
float
actuator actuator_strengtt sttength
1.00
(0, inf)
float
friction tangential
1.00
inf)
float
deusity
5000.00
inf)
float
friction torsional
1.00
(0, inf)
float
friction rolling
1.00
(0, inf)
float
geom deusity
1.00
(0, inf)
float
friction tangential
1.00
(0, inf)
float
gravity
-9.11
(-inf, -0.1)
float
friction torsional
1.00
(0, inf)
float
joint jonnt_dampigg
1.00
(0, inf)
float
goom density
1.00
(0, inf)
float
joint_stiffe Stint_stiffness
0.00
(0, inf)
float
gravity
-9.81
(-inf,
float
limb length_
0.17
(0.01, 0.2)
float
joint. damping
1.00
(0, inf)
float
limb length
0.16
(0.01,
joint stiffness
inf)
float
spinner_ ppinter_length
0.18
(0.01, 0.4)
float.
timestop
0.00
(0.001, 0.1)
float
spinuer radius
0.04
(0.01, 0.05)
float
viscosity
0.00
(U, inf)
float
timestep
0.00
(0.001, 0.1)
float
wind
0.00
(-inf, inf)
float
viscosity
0.00
(0, inf)
float
wind
0.00
(-inf, inf)
float
wind
0.00
(-inf,
float
wind_
0.00
(-inf, inf)
float
wind
0.00
(-inf, inf)
float
0.00
(-inf, inf)
Table 7: Context Features: Defaults, Bounds and Types for RNA Design (Runge et al., 2019) and Mario Environment (Awiszus et al., 2020; Schubert et al., 2021)
(a) Dasi.RnaDesignnFav
Context Feature
Default
Bounds
Type
mutation threshald
5
(0.1, inf)
float
reward exponent
1
(0.1, inf)
float
state radius
(1, inf)
float
dataset
eterna
categorical, 3 list. int
target_ structure
((dataset)
(0, inf)
(b) CARLMarioEnv
Context Feature
Default
Bounds
Type
level index index
calegorical, 15 float.
noise
f(level index, width, height)
(-1, 1)
mario state mario inertia
categorical, n = 3 float
0.89
(0.5, 1.5)
39


Reviews:
[{"review_id": "wgPOTMNof6", "reviewer": "Reviewer_Psm1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The problem of generalization is important in RL and we still don't fully grasp what it means, therefore the authors tackle an important problem.\n- The authors do a job at presenting the problematic in the introduction and motivate the rest of the paper\n- The authors study the effect of context in some of the environments in the benchmark that they propose\n\n**Weaknesses**\n- The important of the theoretical results are not clear and not clearly situated within the literature \n- The organization of the paper could be improved. A lot of space is spent on theory, while the presentation of the environments is totally deferred to the appendix.\n- The empirical results do not seem to corroborate the claims of the paper\n\nThe problem of generalization in RL is an important one, and also a fundamental one. In a sense, it is so fundamental that someone's definition of generalization may be quite different from another researcher. It all depends on the setting we care about. That being, the authors argue that cRL is a way to formalize this. I am not convinced that this is truly general. Indeed, cRL relies on a context. If this does not exist, where does the context come from? The authors make arguments in the end of section 2 that it could come from representation learning. However, then the problem is essentially one of learning representations and the benefit of the context is less clear. \n\nTo this end, the authors propose a benchmark for cRL. However, this takes about 20 lines in the main paper and all the details are relegated to the appendix. This is not great if the authors hope for the community to latch on such a benchmark. The presentation of all the environments should be clear and in the main paper in my opinion. The sections within the appendix refer to each other and we can get lost easily. In B.1, it is mentioned that \"joint stiﬀness, gravity, friction, (joint) angular damping, actuator strength, torso mass as well as target radius and distance deﬁne the context.\" What is left as features that are not part of the context? It would be great if this was clearly stated.\n\nAlthough space is an issue and it is hard to put everything in the main paper, I am not sure what the role of the theoretical section is in the paper. In particular, the main theoretical result seems not very surprising and I would be really interested in the authors clearly situating this result within the CMDP literature. \n\nThe empirical investigation is an important part of the paper as it illustrates the importance of the benchmark. However, looking closely at the results, it seems like there is not a very significant difference with the baselines. For example, in Figure 6bd, the x axis are \"zoomed in\" to make a point, but a difference of 20 points on Cartpole or even less on Pendulum is not that much. In the figure 7, it seems like there is little statistical difference between the methods, yet the text reads as \"agents without access to context information are not able to solve even simple contextual environments\" It seems like such claims could be rewritten to better reflect the results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors study the problem of generalization in RL through the lens of contextual MDPs, that is, through contextual RL (cRL). In particular, the authors argue that this framework allows for a clear formalization of the problem from the point of view of the theory as well as from the point of view of the experimental setups. To this end, the authors propose theoretical arguments that show only a policy conditioned on context can be optimal in contextual MDPs. Additionally, the authors provide a suite of environments that come with a context, which usually is not the default in RL. The authors conduct experiments on some of these environment to try to validate the cRL approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could the authors spend more time on clearly presenting the benchmark?\n\nCould the authors explain and situate the importance of the theoretical results?\n\nCould the authors address the experimental results concerns previously mentioned?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iBrJa9gjXx", "reviewer": "Reviewer_jRh1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Generalization to task variations is important for continual learning in RL\n- The proposed benchmark could be valuable to the community because it is easier and more interpretable than procedurally generated environments like ProcGen and NetHack.\n\nWeaknesses:\n- In motivating cMDPs, the comparison to HiP-MDPs, Goal MDPs and Block MDPs was useful. However, I am struggling to understand how the cMDP formulation is different from Parameterized Skills [1]---as far as I understand, Parameterized Skills also seek to solve a distribution of MDPs which vary in transition and reward functions; furthermore, they also try to maximize return in expectation over all task variation. Could you please clarify how the cMDP formulation is different?\n- The first contribution of the paper is supposed to be theoretical. My understanding is that the theoretical contribution is Theorem 1 which roughly states that the optimal policy in a contextual MDP needs to take context as input. This seems trivially true? If you need context to describe the transition and reward functions of the MDP, then you definitionally need it to describe the optimal policy.\n- The 3rd stated contribution: \"We demonstrate that context-oblivious agents are not suitable to solve contextual environments on their own.\" But by the authors' own admission, the empirical results do not support this claim --- the agent that has access to the context does *not* outperform the agent that is context-oblivious.\n\nQuestion:\nMeta Learning algorithms like MAML have also shown that policies can be adapted to new task instances. Can you also discuss how the cMDP framework differs? Would it make sense to compare cMDP approaches to Meta Learning?\n\n[1] Learning Parameterized Skills, Bruno Castro da Silva et al, ICML 2012", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of changing dynamics and rewards in RL. They cast the problem of generalization as a contextual MDP. Then they go on to create a new suite of benchmark tasks where the context can be varied via interpretable quantities. Finally, their experiments show that generalizing RL agents to new contexts is a challenging open problem.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please describe how the contextual MDP formulation is different from Parameterized Skills framework.\n2. Please comment on the significance of your theoretical contribution. \n3. If a context-aware agent does not outperform a context-oblivious one, consider dropping the claim that adding context is needed to adjust to task variations. \nI am happy to be told that I have misunderstood any of the 3 aforementioned points. However, based on my current understanding, all 3 changes are critical to securing my recommendation.", "broader_impact_concerns": "I do not have any ethical concerns about this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wU5XaSEi7f", "reviewer": "Reviewer_jbAE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The benchmark provided by this paper can provide a nice testbed to understand how to best utilize \"context\" information. The empirical results suggest that simply concatenating this context information to the feature vector is insufficient. The paper highlights that there is clear future work needed to improve the use of these special features and provides a good tool for approaching that future work.\n\nThere are several issues throughout the paper, however, that detract from its utility in its current state. A majority of the listed contributions are not necessarily unique to this work. The formalization of cRL seems to inhibit many of the claims made throughout the paper. And the empirical evidence does not provide sufficient support for claims.\n\n## Contributions\nIn the contributions list, the paper states:\n> In short, our contributions are: (i) We provide a theoretical and empirical characterization of Contextual Reinforcement Learning\n\nHowever, clearly Hallak et al., 2015 and Modi et al., 2018 both provide theoretical characterizations of cRL (the sole contribution of the Hallak paper is this theoretical characterization). The formalization in this paper does not appear to differ from Hallak's. I'm not sure this can be claimed as a contribution.\n\nAdditionally:\n> [...] (iii) We demonstrate that context-oblivious agents are not suitable to solve contextual environments on their own.\n\nLikewise, this was shown as the primary contribution of Malik et al. 2021, so again I doubt this should be claimed as a contribution of this work. Perhaps a better qualified version of this claim can be supported, since this paper does provide novel empirical evidence.\n\nThis work is also extremely similar to the work of Whiteson, Tanner, Taylor and Stone, 2011. In that work, they similarly discuss issues of generalization across environments where certain components of the environment are varied. Some examples include varying gravity in Mountain Car or joint length in Acrobot. In that work, the context information is discussed as forming a POMDP because the context was not provided to the agent. The work of Whiteson et al. should definitely be cited in this work.\n\nIn total, of the three stated contributions (i) and (iii) are entirely subsumed by prior works and (ii) is mostly subsumed by Whiteson et al. The primary novelty, I believe, is in applying the methodology of Whiteson et al. to a larger body of domains and comparing the performance of agents that include context information as part of state.\n\n## Formalization of cRL\n\nI understand the motivation to separate the context from the state, yielding a set of MDPs $(S, A, T_c, R_c, \\rho_c)$ indexed by context $c$. However, this is really no different than a single MDP $(S \\times C, A, T, R, \\rho)$ where the context is part of the state-space. In prior works (i.e. Hallak et al.) the context is known ahead of time and is static for all time. In this case, there can never be communication between states in context $c_1$ and states in context $c_2$. In that work, it makes sense to treat each as a (nearly) independent MDP.\n\nThe example of a cMDP given at the bottom of page 2 mentions that context can change over time. Later in the middle of page 3, it is mentioned that contexts can aid in life-long learning. These two claims are both strictly at odds with the given formalization of cMDPs. By treating the cMDP as a set of MDPs indexed by $c$, the context must be static and known at the beginning of time. In order to accommodate a context that is dynamic, the context must be included in the state-space yielding a single valid MDP.\n\nThe equivalent context-as-state formalization is fairly straightforward. The transition matrix $T$ is now $(|S| \\times |C|) \\times (|S| \\times |C|)$ as opposed to being a tensor of size $|S| \\times |S| \\times |C|$. In the case that the context is static, then we know that $T$ has at least $|C|$ unique communication classes which is what allows modeling this as $|C|$ different MDPs, with each $T_c$ being a valid stochastic matrix. However in the case that context is dynamic, then $T$ may have fewer than $|C|$ communication classes which means the system can transition from one context to the next. When $T$ is a tensor of valid stochastic matrices, there can be no communication between contexts (otherwise the matrices are not self-stochastic).\n\nFinally, by including context as state, it is clear that a cMDP with the context being hidden from the agent is really a specific form of POMDP (as noted in appendix A). In this case, the partially observable component is the context. From here, it is already well-established that a deterministic optimal policy may not exist in the presence of hidden information.\n\n## Proposition 1\nThe proof here is a bit unclear. It is stated that $\\pi^*_{c1}$ is defined only on MDP $M_{c1}$. Then the proof reasons about the optimality of $\\pi^*_{c1}$ on $M_{c2}$. However, this appears a contradiction: $\\pi^*_{c1}$ is not defined on $M_{c2}$. \n\nThis proof also reasons about states from one MDP that are reachable from another MDP. In the given formulation where the cMDP is a set of MDPs, this logic does not make sense. Instead, this proof appears to require the context-as-state formulation discussed above. As written, I argue this portion of the proof is vacuously true rendering the proposition not useful.\n\n## Section 5.1\nThis experiment is intended to study the generalization properties of a SAC agent when certain attributes of the environment change. However, changing attributes of the environment also changes the potential maximum achievable return in that environment. From figure 5, it is not possible to determine if the observed results are due to changing the maximum achievable return or due to the agent failing to generalize.\n\nWhiteson et al. have a nice discussion of a related issue. They use Mountain Car as an example. If the force of the car is changed to be too small, or the gravity too large, then it is possible to make Mountain Car no longer solveable --- the agent simply cannot achieve high enough velocity to overcome the hill. The benchmarks throughout this paper can likely suffer from this same limitation. Notably the first subfigure of Figure 5 (max_speed) suggests that the blue lines are unable to solve the domain. Is this because of the agent failing to generalize, or is this because the max_speed is now too low that no agent can solve the environment?\n\n## Section 5.3\nThe `Concat-all` agent is adding several bias units to the learning process, as several of the context variables never change. It isn't too surprising, then, that this harms out-of-sample generalization as there are several collinear features. \n\nI'm curious how confidence intervals were computed in Figure 7. Clearly the distributions are highly skewed and/or multimodal, which makes computing confidence intervals challenging. It seems worth pointing out that the performance between no-context and context is identical on Pendulum in Figure 7. This suggests the variable being modified in the benchmark has little effect.\n\n## Section 5.4\nA prominent conclusion of these experiments is that neither agent generalizes particularly well. It appears the claim:\n> [...] the visible-context agent can sometimes generalize better [than the hidden-context agent] to out-of-distribution contexts.\n\nis not supported by the evidence in Figure 8. In all three columns, it seems rather clear that the hidden-context agent outperforms the visible-context agent for a substantially larger surface area.\n\n## Questions\n\n1. What is the x-axis in Figure 7 (labeled $k$)? Likewise the y-axis (labeled $R$)? \n1. In Figure 7, how are confidence intervals calculated and about what statistic (e.g. percentile bootstrap about the mean)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper formalizes Contextual RL (cRL) in order to study agents' abilities to generalize across MDPs where certain properties of the MDP vary. An example from the empirical investigation is an agent's ability to generalize across instances of the Cartpole domain where the length of the pole is randomly assigned at the beginning of each episode. Agents are then provided \"context\" features which define the components of the MDP that are parameterized---in the above example, the agent is provided a \"pole-length\" feature. The paper provides theoretical and empirical support for the need for contextual information on a certain class of MDPs (called cMDPs).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Changes I view as necessary:\n\n1. The stated contributions need to clarify what this paper uniquely contributes beyond the current state of the field.\n1. A more in-depth discussion about the relationship between cMDPs and POMDPs is needed, especially towards the goal of dynamic contexts and life-long learning.\n1. Reconsider the formulation of cMDPs, perhaps instead writing the context as belonging to the state. \n1. Rework the proof of proposition 1, clarifying what it means for states across different MDPs to communicate. Or, preferably, using the context-as-state formulation instead.\n1. Change Figure 5 to show regret (or similar). Find a near-optimal policy for each configuration, then show the difference in performance between the SAC agent and the near-optimal performance. \n1. Early in the paper, the concept of optimality gap is defined. Utilizing this measure in the experiments section would resolve several issues of confounding variables. Perhaps optimality gap should be reported for most (if not all) figures. Figures 5 and 8 appear to have the greatest need (i.e. the claims cannot currently be supported).\n1. If only using 10 seeds, use a better aggregate statistic than IQM. It isn't clear that throwing away the top 2-3 and bottom 2-3 performing agents is justified in this setting---there is no reason to believe these are noisy samples or corrupt. I believe this ultimately hides information. Personally, I prefer the mean since it matches the evaluation proposed on page 5 eqn 1. At the very least, Agarwal et al. 2021 should be cited as I suspect that's where the paper got this choice. Also, use more than 10 seeds; the distributions in Figure 7 are _wild_.\n\nChanges I view as improving the work:\n\n1. Clean up some of the excess ink in all figures. For example, remove the grey background lines from Figure 5. Label all axes instead of a subset, or alternatively stack the plots in a neater way to share axes.\n1. The robot example on page 2 does provide some clarity on terms. However, the example relies on context variables that are likely impossible to measure (such as friction). As such, this rather seems an example of a general POMDP and not an example of a cMDP.\n1. On page 3 it is claimed \"Context enables agents to learn more general policies\". The opposite appears to be true as stated in the conclusion. Besides, providing additional context information can allow learning more discriminatory policies (using the context information to select which policy is used) suggesting exactly the opposite as well. Better ability to discriminate, not better ability to generalize.\n1. Page 4: the frame-stacking example does not seem to fit the context definition. It is not clear how providing a history relates to providing a context. Instead frame-stacking appears to provide access to second order information, such as velocity.\n1. Page 5: to estimate the optimality gap, the paper suggests learning an agent per context in order to approximate the best achievable performance. This either limits the formalization of cRL to discrete contexts, or adds a source of variance to the optimality gap metric. This limitation should likely be discussed.", "broader_impact_concerns": "Not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Both the authors and reviewers should be commended for the effort put in during the discussion period. All three reviewers agreed on acceptance, and the paper clearly meets TMLRs criteria for publication.

The authors should consider reviewer jbAE's comment here (https://openreview.net/forum?id=Y42xVBQusn&noteId=wb6g5CSlg8)---in the end its a request for an additional line of text in the paper---and make any final edits.
Decision: Accept (poster)