Grounding Aleatoric Uncertainty for Unsupervised Environment Design
Authors: Minqi Jiang
Michael D Dennis
Jack Parker-Holder
Andrei Lupu
Heinrich Kuttler
Edward Grefenstette
Tim Rocktäschel
Jakob Nicolaus Foerster
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
curriculum learning
generalization
environment design
procedural content generation

Abstract: Adaptive curricula in reinforcement learning (RL) have proven effective for producing policies robust to discrepancies between the train and test environment. Recently, the Unsupervised Environment Design (UED) framework generalized RL curricula to generating sequences of entire environments, leading to new methods with robust minimax regret properties. Problematically, in partially-observable or stochastic settings, optimal policies may depend on the ground-truth distribution over aleatoric parameters of the environment in the intended deployment setting, while curriculum learning necessarily shifts the training distribution. We formalize this phenomenon as curriculum-induced covariate shift (CICS), and describe how its occurrence in aleatoric parameters can lead to suboptimal policies. Directly sampling these parameters from the ground-truth distribution avoids the issue, but thwarts curriculum learning. We propose SAMPLR, a minimax regret UED method that optimizes the ground-truth utility function, even when the underlying training data is biased due to CICS. We prove, and validate on challenging domains, that our approach preserves optimality under the ground-truth distribution, while promoting robustness across the full range of environment settings.

Grounding Aleatoric Uncertainty for
Unsupervised Environment Design
Minqi Jiang" UCL & Meta Al
Michael Dennis UC Berkeley
Jack Parker-Holder University of Oxford
Andrei Lupu MILA & Meta Al
Heinrich Küttler* Inflection AI
Edward Grefenstette UCL & Cohere
Tim Rocktäschel UCL
Jakoh Foerster FLAIR, of Oxford
Abstract
Adaptive curricula in reinforcement learning (RL) have proven effective for producing policies robust to discrepancies between the train and test environment. Recently, the Unsupervised Environment Design (UED) framework generalized RI. curricula generating sequences of entire environments, leading to new methods with robust minimax regret properties. Problematically, in partially-observable stochastic settings, optimal policies may depend on the ground-trui -trut distribution over aleatoric parameters of the environment in the intended deployment setting, while curriculum learning necessarily shifts the training distribution. We formalize this phenomenon curriculum- aurriculun-induced induced covariate shift (CICS), and describe how its occurrence in alcatoric parameters can lead to suboptimal policies. Directly sampling these parameters from the ground -tuth distribution avoids the issue, but thwarts curriculum learning. We propose SAMPLR, a minimax regret UED method that optimizes the ground-truth utility function, even when the underlying training data is biased due to CICS. We prove, and validate on challenging domains, that our approach preserves optimality under the ground-truth distribution, while promoting robustness across the full range of environment settings.
Introduction
Adaptive curricula, which dynamically adjust the distribution of training environments to facilitate learning [26, 35], have played key role in many recent achievements in deep reinforcement learning (RL). Applications span both single-agent RL [34, 53, 58, 22, 27J, where adaptation occurs over environment variations, and multi-agent RL, where adaptation can additionally occur over co-players [43, 52, 44]. These methods demonstrably improve the sample efficiency and robustness of the final policy [25, 8, 21, 20], c.g. by presenting the agent with challenges the threshold of its abilities.
In this paper we introduce and address fundamental problem relevant adaptive curriculum learning methods for RI, which we call caniicullu--inuucer covariate shift (CICS). Analogous to the covariate shift that occurs in supervised learning (SL. [18], CICS refers to a mismatch between the inpur distribution at training and test time. In the case of RL, we will show this becomes problematic when the shift occurs over the aleatoric parameters of the environment-thos aspects the environment holding irreducible uncertainty even in the limit of infinite experiential data [9]. While in some cases, CICS may impact model performance SL, adaptive curricula for SL have generally not been found to be impactful in RL [55]. Therefore, focus addressing CICS specifically as it arises in the RL setting, leaving investigation of its potential impact in SL to future work.
Correspondence to maj @meta. com. Com. Work done while at AI.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

carrindan Arpreemeed
U(ry) U
Campleed Levela
U 0(%)]
AMEPLE
Vpln Uln.)
P10 (F)
Figure I: Adaptive curricula can result in covariate shifts in environment parameters with respect the ground-truth distribution P(θ) (top path), e.g. whether road is icy or not, which can cause the policy to be optimized for a utility function U' differing from the ground ground-truth truth utility function based on P (See Equation 1). Here, the policies zg. and ग * drive assuming ice and no ice respectively. SAMPLR (bottom path) matches the distribution of training transitions to that under P(0|r) (pink triangles), thereby ensuring the optimal policy trained under a biased curriculum retains optimality for the ground-trum distribution P.
To establish precise language around adaptive curricula, cast discussion under the lens of Unsupervised Enviromment Design [UED, 8). UED provides a formal problem description for which an optimal curriculum the solution, by defining the Underspecified POMDP (UPOMDP: see Section 2). which expands the classic POMDP with a set of free parameters O, representing the aspects of the environment that may vary. UED then seeks adapt distributions over θ to maximize some objective, potentially tied to the agent's performance. UED allows us to view adaptive curricula as emerging via a multi-player game between a teacher that proposes environments with parameters 0 ~ P()) and a student that learns to solve them. In addition to notational clarity, this formalism enables using game theoretic constructs, such as Nash equilibria [NE, 28), to analyze curricula.
This game-theoretic view has led to the development of curriculum methods with principled robustness
guarantees, such PAIRED [8] and Robust Prioritized Level Replay [PLR- 20], which aim to maximize a student's regret and lead to minimax regret [40] policies at NE. Thus, at NE, the student can solve all solvable environments within the training domain. However, in their current form the UED robustness guarantees are misleading: the UED curriculum deviates from a ground-truth distribution P()) interest, i.c. the distribution at deployment, with respect to alcatoric parameters O/ C θ, the resulting policies may be suboptimal under the ground-truth distribution P.
For concrete example of how CICS can be problematic, consider the case of training a self-driving car to navigate potentially icy roads, when icy conditions rarely occur under P. When present, the ice is typically hard spot in advance; thus, the aleatoric parameters O' correspond whether each section of the road is icy. A priori, curriculum should selectively sample more challenging icy settings to facilitate the agent's mastery over such conditions. However, this approach risks producing an pessiiipssiiiiitic agent (i.e. one that assumes that ice is common), driving slowly even in fair weather. Such a policy leads to inadequate performance on P, which features ice only rarely.
We can preserve optimality on P by grounding the policy-thal that is, ensuring that the agent acts optimally with respect to the ground-truth utility funcction for any action-observation history r and the implied ground-truth posterior over O:
U U Eg-P(1)))) [U(T]T, 0)].
(1)
where the ground-truth utility conditioned on X, (T is defined to be E, F,,0) Ei= rrre]. for rewards r; and discount
We can ground the policy by grounding the training distribution, which means constraining the training distribution of aleatoric parameters P(O') match P(O'). This is trivially accomplished by directly sampling 8' P(e'), which we call naive grounding. Unfortunately, this approach makes many curricula infeasible by removing the ability to selectively sample environment settings over aleatoric parameters. Applying this strategy to the self- agent may result in policy that is optimal in expectation under P where there is rarely ice, but nevertheless fails to drive safely on ice.

---

We wish to maintain the ability to bias training distribution, since it is required for curriculum learning, while ensuring the resulting decisions remain optimal in expectation under P. This goal is captured by the following objective:
= ((]]7)),
where D is the training distribution of т. Under naive grounding, D is equal to P(r) and Equation 2 reduces to (T). To overcome the limitations of naive grounding, we develop an approach that allows D to deviate from P(T). e.g. by prioritizing levels most useful for learning, but still grounds the policy by evaluating decisions following potentially biased training trajectories according to (TTl)). Figure summarizes this approach, and contrasts with an ungrounded adaptive curriculum.
In summary this work presents the following contributions: i) We first formalize the problem of CICS in RL in Section 3. ii) Then, we present SAMPLR, which extends PIR- state-of-the-art UED method, to preserve optimality on P while training under usefully biased training distribution in Section 4. iii) We prove in Section 5 that SAMPLR promotes Bayes-optimal policies that are robust over all environment settings 0 P(O). iv) Our experiments validate these conclusions in two challenging domains, where SAMPLR learns highly robust policies, while PLR* 1 fails due to CICS.
2 Background
2.1 Unsupervised Environment Design
Unsupervised Environment Design [UED, is is the problem of automatically generating an adaptive distribution of environments which will lead to policies that successfully transfer within a target domain. The domain of possible environment settings is represented by an Underspecified POMDP (UPOMDP), which models each environment instantiation, or level, as specific setting of the free parameters that control how the environment varies across instances. Examples of free parameters are the position of walls in a maze or friction coefficients a physics- based task. Formally UPOMDP is defined as tuple M = (A, O, T, I, R. 7), where A is the action space, O is the observation space, θ is the set of free parameters, S is the state space, T
A x θ - ASS) is the transition function, I S + O is the observation function, R. S + E is the reward function, and is the discount factor. UED typically approaches the curriculum design problem as training a teacher agent that co-evolves an adversarial curriculum for student agent, e.g. by maximizing the student's regret.
2.2 Prioritized Level Replay
We focus on a recent UED algorithm called Robust Prioritized Level Replay [PLR- 21], which performs environment design via random search. PLR maintains a buffer of the most useful levels for training, according to an estimate of learning potential-typically based on regret, approximated by a function of the temporal-difference (TD) errors incurred on each level. For each episode, with probability p. PLR actively samples the next training level from this buffer, and otherwise evaluates regret a level 0 P(O) without training. This sampling mechanism provably leads to minimax regret policy for the student at NE, and has been shown to improve sample-efficiency and generalization. The resultant have no regret. We provide implementation details for PLR- - in Appendix A.
curricula naturally avoid unsolvable levels, which
3 Curriculum-Inducel Covariate Shift
Since UED algorithms formulate curriculum learning multi-agent game between teacher and student agent, we can formalize when CICS becomes problematic by considering the equilibrium point of this game: Let be the environment parameters controlled by UED, P(O), their ground- truth distribution, and P(@), their curriculum distribution at equilibrium. We use to refer to the joint action-observation history (AOH) of the student until time / (and simply T when clear from context). Letting V V(T denote the value function under the curriculum distribution P(O), we characterize an instance of CICS over θ as problematic if the optimal policy under P(0) differs from that under the gruthd-truth P()) for some Tr, so that
arg max (TT(TT) o arg max V (TTTT))
x
3

---

The value function (T) with respect to P(O) can be expressed as marginalization over B:
V(TTT)) = Σ P(OT)) 0) & Σ P(0
Here, the notation P(0) means P(θ 0), and the tilde on the and V terms indicates independence from any distribution over θ, as they both condition on 0. Importantly, the value function under the curriculum distribution (T) corresponds to Equation with P replaced by P. We see that ((x(T)) is unchanged for a given Tt when P(0) is replaced with P(0) if 1) P(0' (m.) 1 for some 0" and 2) P shares support with P. Then ((re(0) = 1 iff 0* and zero elsewhere. this case, the sums reduce to V= regardless of changing the ground-trutt distribution P to P. In other words, when ④ is fully determined given the current history т, covariate shifts over θ with respect to P(0) have no impact on policy evaluation and thus the value function for the optimal policy, If the first condition does not hold, the uncertainty over the value of some subset O' + is irreducible given T, making O' aleatoric parameters for the history т. Thus, assuming the curriculum shares support with the ground-truue -trth distribution, covariate shifts only alter the optimal policy at when they occur over aleatoric parameters given Such parameters can arise when the environment is inherently stochastic or when the cost of reducing uncertainty high.
Crucially, our analysis assumes P and P share support over O When this assumption is broken, the policy trained under the curriculum can be suboptimal for environment settings 6, for which P(0) = 0 and P(0)= > O. In this paper, we specifically assume that P and share support and focus on addressing suboptimality under the ground-truth due to CICS over the aleatoric parameters O'
This discussion thus makes clear that problematic CICS can be resolved by grounding the training distribution, i.e. enforcing the constraint P(0'(T)= = P()'(r) for the aleatoric parameters of the environment. This constraint results in grounding the policy, i.c. ensuring it is optimal with respect to the ground-truth utility function based on P (Equation 1). As discussed, naive grounding satisfies this constraint by directly sampling 0' ~ P(O'), at the cost of curricula over O', This work develops an alternative for satisfying this constraint while admitting curricula over O¹
4 Saaple-Matched PLR (SAMPLR)
We now describe general strategy for addressing CICS, and apply to PLR- resulting in Sample-Machhed PLR (SAMPLR). This new UED method features the robustness properties of PLR- while mitigating the potentially harmful effects of CICS over the aleatoric parameters O'.
Algorithm 1: Sample-Matched PLR (SAMPLR)
Randomly initialize policy T(Ø), an empty level buffer A of size K. and belief model (s)(TT).
while not converged do
Sample replay-decision Bernoulli, d ~ PD(d) if d = 0 or |A| = 0 then
Sample level 0 from level generator Collect x 's trajectory T on 8, with stop-gradicnt Φ1
As discussed in Section 3, CICS become problematic when the covariate shift occurs over some aleatoric subset O' of the environment parameters O, such that the expectation over O' influences the optimal policy. Adaptive curriculum methods like PLR- prioritize sampling of environment settings where the agent experiences the most learning. While such a curriculum lets the agent focus correcting its largest errors, the curriculum typically changes the distribution over aleatoric parameters O', inducing bias in the resulting decisions. Ideally, we can eliminate this bias, ensuring the resulting policy makes optimal decisions with respect to the ground ground-truth utility function, conditioned on the current trajectory:
else
Use PLR to sample a replay level from the level store, ~ A
Collect fictitious trajectory r' on f, based on s/ ~ B
Update æ with rewards R(7')
end
Compute PLR score, S score(r', T)
Update A with using score S
end
U(ITT) Bg-7((1) U U
naive solution for grounding is to simply exclude O' from the set of environment parameters under curriculum control. That is, for each environment setting proposed by the curriculum,

---

resample 8' ~ P. We refer to this approach as naive grounding. Naive grounding forces the expected reward and next state under each transition at the current AOH r to match that under P. Thus, optimal policies under naive grounding must be optimal with respect to the ground-trutt distribution over 0'.
While technically simple, naive grounding suffers from lack of control over O'. This limitation is of no concern when the value of G' does not alter the distribution of r until the terminal transition, e.g. when O' is the correct choice in a binary choice task, thereby only influencing the final, sparse reward initial experiment in Section 6 shows naive grounding
when the right choice is made. In fact, performs well in such cases. However, when the value of @' changes the distribution of r before the terminal transition, the agent may benefit from curriculum that actively samples levels which promote learning robust behaviors under unlikely events. Enabling the full benefits of the curriculum in such cases requires the eurriculum to selectively sample values of O'. Instead of naive grounding, we aim to ground only the policy updates, allowing the curriculum to bias the training distribution. This can be accomplished by optimizing the following objective:
(5) To achieve this, we replace the reward r, and next state 81+1 with counterfactual values that would be experienced if I' were consistent with T and P. so that 8' (() This substitution occurs by simulating fictitious transition, where the fictitious state is sampled as s/ ~ Bs((TT), the action as a₂ ~ x(-[7) (as per usual), the fictitious next state as 6/41 = T at), and the fictitious reward as r/ R(( The belief model ((s((T) (T) is the ground-truth posterior of the current state given T:
(T) Er~D (T]Tr]]
((&lT) = ((
(6)
Fictitious transitions, summarized in Figure 2. ground the observed rewards and state transitions to P. Should training on these transitions lead to an optimal policy over O this policy will also be optimal with respect to P. We prove this property Section 5. Fictitious transitions thus provide the benefit of naive grounding without giving up curriculum control over O'
In general, we implement B as follows: Given P(e') as a prior, we model the posterior P(0'(7) (r) with Bayesian inference. The posterior could be learned via supervised learning with trajectories collected from the environment for a representative selection of f'. Further, we may only have limited access to P(O) throughout training, for example, if sampling P()) is costly. In this case, we can learn an estimate P(O/) from samples we do collect from P(e), which can occur online. We can then use P(0') to inform the belief model.
?
Tr
Ais0 R.)
Te
T
a
SAMPLR, summarized in Algorithm 1, incorporates this fictitious transition into PLR by replacing the transitions experienced in replay levels sampled by PLR with their fictitious counterparts, as PIR only trains on these trajectories. PLR- uses PPO with the Generalized Advantage Estimator [GAF, 41] as the base RL algorithm, where both advantage estimates and value losses can be written in terms of one-step TD errors 6, at time t. Training on fictitious transitions then amounts to computing these TD errors with fictitious states and rewards: & r/ + (s{) V (5(44)). Importantly, because PLR L provably leads to policies that minimize worst-case regret over all at NE, SAMPLR enjoys the same property for 0 P(0). A proof of this fact is provided in Section 5.
a
Figure 2: transition (top) and a fictitious transition used by SAMPLR (bottom). A is the advantage function.
A standard RL
Applying SAMPLR requires two key assumptions: First, the simulator can be reset to a specific state, which is often true, RL largely occurs in resettable simulators those that can be made to do so. When a resettable simulator is not available, a possible solution is to learn a model of the environment which we leave for future work. Second, we have knowledge of P(O'). Often, we know P a priori, e.g. via empirical data or as part of the domain specification, as in games of chance.
5 The Grounded Optimality of SAMPLR
Training fictitious transitions is a method for learning an optimal policy with respect to the ground- truth utility function U (T) over the distribution D of training trajectories 7, defined in Equation 5.
5

---

When D corresponds to the distribution of trajectories on levels 0 ~~ P(O), (D(T) reduces to the ground-truth utility function, ( For any UED method, our approach ensures that, in equilibrium, the resulting policy is Bayes-optimal with respect to P(0) for all trajectories in the support of D. Remark If is optimal with respect to the ground-truth utility function U ( then it is optimal with respect to the ground-trum truth distribution P(O) of environment parameters on the support of D.
Proof. By definition we have n² E Earg max max (T)( = arg max
m (Er~D [T[T] Since л can
xeII
condition on the initial trajectory т, the action selected after each trajectory can be independently
optimized. Therefore, for all 7 E D, x² E arg max ( (T(T)) implying that x is the optimal policy
sell
maximizing (TTT))
Thus, assuming the base RL algorithm finds Bayes-optimal policies, UED method that optimizes the ground-truth utility function, as done by SAMPLR, results in Bayes-optimal performance over the ground-truth distribution. If the UED method maximizes worst-case regret, we can prove even stronger property we call robust 6-Bayes optimality.
Let (T) be the ground-true truth utillty function for π on the distribution Dj of initial trajectories sampled from level 0, so that U (T) = Dz (rz). Given policy maximizing Uz(z), we say that m is robustly e-Bayes optimal iff for all 0 in the domain of P(e) and all n², we have
(π 2 Uo(x') -
Note how this property differs from being simply e-Bayes optimal, which would only imply that ( 2 U(T) c.
Robust c- c-Bayes optimality requires रे to be c-optimal on all levels @ in the support of the ground- truth distribution, even those rarely sampled under P(0). We will show that at e-Nash equilibrium, SAMPLR results in a robustly c-Bayes optimal policy for the ground-truft utility function (T) In contrast, training directly on levels @ P()) results a policy that is only c-Bayes optimal.
Theorem 1. Tf is e-Bayes optimal with respect to U (T) for the distribution D of trajectories sampled under over levels maximizing the worst-case regret of n. occurs under SAMPLR, then n' is robustly e-Bayes optimal with respect to the ground-tran truth utility function, U(T).
Proof. Let nt be e-optimal with respect to U (T) where D is the trajectory distribution under on levels maximizing the worst-case regret of क. Let .* be an optimal grounded policy. Then for any 0,
Uo(T") - Uo(m') < U (TT)) Us(T") Sc
The first inequality follows from D being trajectories from levels that maximize worst-case regret with respect to n², and the second follows from n² being e-optimal on Up (T). Rearranging terms gives the desired condition.
Experiments
Our experiments first focus on discrete, stochastic binary choice task, with which validate theoretical conclusions by demonstrating that CICS can indeed lead to suboptimal policies. Moreover, show that naive grounding suffices for learning robustly optimal policies in this setting. However, as we have argued, naive grounding gives up control of the aleatoric parameters O' and thus lacks the ability actively sample scenarios helpful for learning robust behaviors especially important when such scenarios are infrequent under the ground-truth distribution P(O). SAMPLR induces potentially biased curricula, but retains optimality under P(0) by matching transitions under P(O') with those under P(e'). We assess the effectiveness of this approach in our second experimental domain, based on the introductory example of driving icy roads. In this continuous-contre driving domain, we seek to validate whether SAMPLR does in fact learn more robust policies that transfer to tail cases under P(O"), while retaining high expected performance on the whole distribution P(O').
All agents are trained using PPO [42] with the best hyperparameters found via grid search using set of validation levels. We provide extended descriptions of both environments alongside the full details of our architecture and hyperparameter choices in Appendix C.

---

DR
PLRU
Naiva grounding! SAMPLR
25
0.00
5774
2.0
Limoura0 Limerppcoade woooodoooooooooooooooooo docc-pooooocoo Leop
Schadd room count
reom court
8.25
4.75
4.50
0.5
4.25
30M
10064
150M
2024
5034
100M/
100M
2005
505
100M
15001
20074
Steps
Figure 3: Left: Example Stochastic Fruit Choice levels. The plots show mean and standard error (over 10 runs) of episodic returns (left); room count of solved levels (middle), during training (dotted lines) and test on the ground-truth distribution (solid lines), for q 0.7; and the room count of levels presented at training (right).
6.1 Stochastic Fruit Choice
We aim to demonstrate the phenomenon of CICS in Stochastic Fruit Choice, binary choice task, where the aleatoric parameter determines the correct choice. This task requires the agent to traverse up to cight rooms, and in the final room, decide to cat either the apple or banana. The correct choice I' is fixed for each level, but hidden from the agent. Optimal prcision-making depends on the ground tround-truth distribution over the correct fruit, P(e'). This task benefits from a curriculum over the number of rooms, but curriculum that selectively samples over both room layout and correct fruit choice can lead to suboptimal policies. Figure 3 shows example levels from this environment.
This domain presents a hard exploration challenge for RL agents, requiring robust navigation across multiple rooms. Further, this environment is built on top of MiniHack [39]. enabling integration of select game dynamics from the NetHack Learning Environment [23], which the agent must master to succeed: To go from one room to the next, the agent needs to learn to kick the locked door until opens. Upon reaching the final room, the agent must then apply the eat action on the correct fruit.
Let T be the policy that always chooses the apple, and TB. the banana. If the probability that the goal is the apple is P(A) = q, then the expected return is Rag under and R The optimal policy is TA when q > Rg/(RA + RB). and Ty otherwise. Domain randomization (DR), which directly samples each level P(0). optimizes for the correct ground truth P(e'), but will predictably struggle to solve the exploration challenge. PLR may induce curricula easing the exploration problem, but can be expected make the correct fruit choice oscillate throughout training to maximize regret, leading to problematic CICS.
under nn
We set RA = 3. Rs = 10, and q = 0.7, making n optimal with an expected return of 3.0. We compare the train and test performance of agents trained with DR, PLR- and PLR- with naive grounding over 200M training steps in Figure 3. In this domain, SAMPLR reduces to naive grounding, as 8' only effects the reward of a terminal transition, making fictitious transitions equivalent real transitions for all intermediate time steps. We see that DR struggles to learn an effective policy, plateauing at a mean return around 1.0, while PLR- performs the worst. Figure in Appendix B shows that the PLR curriculum exhibits much higher variance in q, rapidly switching the optimal incentive, making learning more difficult. In contrast,
choice of fruit to satisfy its PLR with naive grounding constrains g = 0.7, while still exploiting curriculum over an increasing number of rooms, as visible Figure This grounded curriculum results in a policy that solves more complex room layouts at test time. Figures 5 and 6 in Appendix B additionally show how the SAMPLR agent's 's choices converge to TB and how the size of SAMPLR' improvement varies under alternative choices of in {0.5, 0.33.
6.2 Zero-Shot Driving Formula 1 Tracks with Black Ice
We now turn to domain where the aleatoric parameters influence the distribution of at each t, thereby creating opportunities for a curriculum to actively sample specific 0' to promote learning on biased distributions of 7,. We base this domain on the black ice driving scenario from the introduction of this paper, by modifying the CarracingBerzier environment in [20]. In our version, each track tile has black ice with probability g, in which case its friction coefficient is 0, making acceleration and
7

---

DR
grarddrg
SANPLR
000
500
0.05
Meain www.nndeeeeeeeeee retiam
Proporton
0.04
300
200
0.00
0.02
0.01
Stops
Proportion ice
Figure 4: Charts show mean and standard error (over 10 runs) of fraction of visited tiles with ice during training (left) and zero-shot performance on the full Formula benchmark as a function of ice rate (right). Top row screenshots show the agent approaching black ice (q- 0.4) and example training track (g 0.6). Bottom row shows a Formula track (q = 0.2) at two zoom scales.
braking impossible. This task is especially difficult, since the agent cannot see black ice in its pixel observations. Figure 4 shows example tracks with ice rendered for illustration purposes. The episodic returns scale linearly with how much of the track is driven and how quickly this is accomplished. As success requires learning to navigate the challenging dynamics over ice patches, curriculum targeting more difficult ice configurations should lead to policies more robust to black ice. Here, the ground-truth distribution P(0') models the realistic assumption that most days see little to no ice.
We therefore model the probability of ice per tile g Beta {), where = 1, 3 15.
We test the hypothesis that SAMPLR's segret-maximizing curriculum results in policies that preserve optimal performance on the ground-truth distribution P(O'), while being more robust to tail cases compared to DR and PLR 1. with naive grounding. We expect standard PLR I to underperform all methods due CICS, leading to policies that are either too pessimistic or too optimistic with respect
Table 1: Icy F1 returns, mean = standard error over 10 runs.
Condition
DR
PIR
Naive
SAMPLR
Grond trath -- Beta(1,
581 +: 23
543 + 21
618 + 6
616 + 6
Zem-shot
g 0.2
332 + 63
328 60
363 15
393 13
g G 0.4
94.7 41
43 38
75
+ 39
195 11
-76.3
-115 12
-79
0.8
-11.11 1:1.1:10 11
-151 6.0
-139
9
-111 7
to the amount of ice. These baselines provide the controls needed to distinguish performance changes due to the two grounding approaches and those due to the underlying curriculum learning method.
We train agents with each method for 5M and test zero-shot generalization performance on the Formula (F1) tracks from the CarRacingF1 benchmark, extended to allow each track segment to have black ice with probability q in {0.0. 0.2, 0.4, 0.6, 0.8). These test tracks are significantly longer and more complex than those seen at training, as well as having a higher rate of black ice.
To implement SAMPLR's s belief model, we use a second simulator as perfect model of the environment. At each time step, this second simulator, which refer to the fictitious simulator, resets to the exact physics state of the primary simulator, and its icy tiles are resampled according to the exact posterior over the aleatoric parameter g = 0', such that 0' ~ P(0'(7), ensuring the future uncertainty is consistent with the past. The agent decides on action a based on the current real observation Ot, and observes the fictitious return rf and next state s determined by the fictitious simulator after applying ag in state s/ P(s [r, 0'). This dual simulator arrangement, fully detailed in Appendix A.2, allows us measure the impact of training on fictitious transitions independently of the efficacy of model-based RL approach. Further, as the training environment in RL is most often simulation (e.g. in sim2real), this approach is widely applicable.
SAMPLR outperforms all baselines in zero-shot transfer to higher ice rates on the full F1 benchmark and attains statistically significant improvement at p < 0.001 when transferring to q 0.4 and = 0.6, and < 0.05 when g 0.8. Importantly, SAMPLR outperforms PI. with naive grounding, indicating that SAMPLR exploits specific settings of O' to better robustify the agent against rare icy conditions in the tail of P(O'). Indeed, Figure 4 shows that on average, SAMPLR exposes the agent to more ice per track tile driven, while PLR- underexposes the agent to ice compared to DR and naive grounding, suggesting that under PLR agents attain higher regret on ice-free tracks- a likely outcome as ice-free tracks are easier to drive and lead to returns, with
8

---

which regret scales. Unfortunately, this results in PLR being the worst out of all methods on the ground-truth distribution. SAMPLR and naive grounding avoid this issue by explicitly matching transitions to those under P at т. As reported in Table 1, SAMPLR matches the baselines in mean performance across all F1 tracks under P(O'), indicating that despite actively sampling challenging 8', it preserves performance under P(H"), i.e. the agent does not become overly cautious.
7 Related Work
The mismatch between training and testing distributions of input features is referred to as covariate shift,, and has long served as a fundamental problem for the machine learning community. Covariate shifts have been extensively studied in supervised learning [51. 18, 5, 21. In RL, prior works have largely focused on covariate shifts due to training on off-policy data [47, 37, 11, 14, 13, 49] including the important case of learning from demonstrations [6]. Recent work also aimed learn invariant representations robust to covariate shifts [56, 57]. More generally, CICS is form of sample-selection bias [15]. Previous methods like OFFER [7] considered correcting biased transitions via importance sampling [46] when optimizing for expected return on a single environment setting, rather than robust policies over all environments settings. We believe our work provides the first general formalization and solution strategy addressing curriculum-induced covariate shifts (CICS) for RI.
The importance of addressing CICS is highlighted by recent results showing curricula be essential for training RL agents across many of the most challenging domains, including combinatorial gridworlds [58], Go [43], StarCraft II [52], and achieving comprehensive task mastery in open-ended environments [44]. While this work focuses on PLR other methods include minimax adversarial curricula [32. 53, 54] and curricula based on changes in return [25, 34]. Curriculum methods have also been studied in goal-conditioned RL [12, 6, 45, 291, though CICS does not occur here as goals are observed by the agent. Lastly, domain randomization [DR, 38. 31] can be seen as a degenerate form of UED, and curriculum-bass based extensions of DR have also been studied [0),
Prior work has also investigated methods for learning Bayes optimal policies under uncertainty about the task [59, 30]. based on the framework of Bayes-adaptive MDPs (BAMDPs) [3, 10]. In this setting, the agent can adapt to an unknown MDP over several episodes by acting reduce its uncertainty about the identity of the MDP. In contrast, SAMPLR learns a robustly Bayes-optimal policy for zero-shot transfer. Further unlike these works, our setting assumes the distribution of some aleatoric parameters is biased during training, which would bias the a posteriori uncertainty estimates with respect the ground-truth distribution when optimizing for the BAMDP objective. Instead, SAMPLR proposes a means to correct for this bias assuming knowledge of the true environment parameters, to which we can often safely assume access in curriculum learning.
Deeply related, Off-Belief Learning [OBL., 16) trains cooperative agents in self-play using fictitious transitions assuming all past actions of co-players follow base policy, e.g. a uniformly random one. Enforcing this assumption prevents agents from developing conventions that communicate private information co-players via arbitrary action sequences. Such conventions hinder coordination with independently trained agents or, importantly, humans. SAMPLR can be viewed as adapting OBL to anggle curriculum learning, where a co-player sets the environment parameters the start of each episode (see Appendix D). This connection highlights how single-agen agent curriculum learning is inherently a multi-agent problem, and thus problems afflicting multi- ageet learning also surface in this setting; moreover, methods addressing such issues in one setting can then be adapted to the other.
8 Conclusion
This work characterized how curriculum-induced covariate shifts (CICS) over aleatoric environment parameters O' can lead to suboptimal policies under the ground-truth distribution over these parameters, P(O'). We introduced a general strategy for correcting CICS, by training the agent on fictitious rewards and next states whose distribution is guaranteed to match what would be experienced under P(O)). Our method SAMPLR augments PLR with this correction. By training on fictitious transitions, SAMPLR actively samples specifie values of 8' that induce trajectories with greater learning potential, while still grounding the training data to P(O'). Crucially, our experiments in challenging environments with aleatoric uncertainty showed that SAMPLR produces robust policies outperforming those trained with competing basclines that do not correct for CICS.
9

---

A core assumption made by SAMPLR and all other UED methods is the ability to reset the environment arbitrary configurations of some set of free parameters. While such resets can be difficult or impossible to perform in real world environments, in practice, this assumption is nearly always satisfied, as RL training largely occurs under a sim2real paradigm due to the additional costs of training in the wild. Most RL simulators can either be directly reset to specific environment configurations or be straightforwardly made to do so. SAMPLR thus provides a means to more fully exploit the affordances of simulator to produce more robust policies: Policies trained with SAMPLR retain optimality when transferred to the ground-truth distribution aleatoric parameters in the real environment- crucial property not satisfied by prior UED methods. Importantly, the approach based on fictitious transitions used by SAMPLR can, in principle, be generally applied to prior UED methods to provide them with this desirable property.
Acknowledgments and Disclosure of Funding
We thank Robert Kirk and Akbir Khan for providing useful feedback on earlier drafts of this work. Further, we are grateful to our anonymous reviewers for their valuable feedback. MJ is supported by the FAIR PhD program at Meta AI. This work was funded by Meta.
References
[1] M. Andrychowicz, A. Raichuk, P. Stanczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters in on-policy reinforcement learning? large-scale empirical study. CoRR, abs/2006.05990,, 2020.
[2] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. CoRR, abs/1907.002 002893,, 2019.
[3] R. Bellman. A problem in the sequential design of experiments. Sankhyā: The Indian Journal of Statistics (1933-1960), 16(3/4):221-22,, 1956.
[4] D. S. Bernstein, R. Givan, N. Immerman, and S. Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 2794):819-840, 2002.
[5] S. Bickel, M. Brückner, and T. Scheffer. Discriminative learning under covariate shift. Journal of Machine Learning Research, 10:2137-2155, 2009.
[6] A. Campero, R. Raileanu, H. Kuttler, B. Tenenbaum, T. Rocktäschel, and E. Grefenstette. Learning with AMIGo: Adversarially motivated intrinsic goals. In International Conference on Learning Representations, 2021.
[7] K. A. Ciosek and S. Whiteson. OFFER: off off-ennviroment reinforcement learning. In S. P. Singh and S. Markovitch, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 1819-1825. AAAI Press, 2017.
[8] M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, Russell, A. Critch, and Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances in Neural Information Processing Systems, volume 33, 2020.
[9] A. Der Kiureghian and O. Ditlevsen. Aleatory or epistemic? does it matter? Structural safety, 31(2):005-112, 2009.
[10] M. O. Duff. Optimal Learning: Computational procedures for Bayes-adaphe Markov decision processes. University of Massachusetts Amherst, 2002.
[11] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1407-1416. PMLR, 10-15 Jul 2018.
[12) C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1515-1528. PMLR, 10-15 Jul 2018.
10

---

[13] C. Gelada and M. G. Bellemare. Off Off-policy deep reinforeement learning by bootstrapping the covariate shift. The Thirty- Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2019, Honolulu, Hawaii, USA, January 27- February 1, 2019, pages 3647-3655. AAAI Press, 2019.
[14] A. Hallak and S. Mannor. Consistent on-line off- off-policy evaluation. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1372-1383. PMLR, 06-11 Aug 2017.
[15] J. J. Heckman. Sample selection bias as a specification error. Econometrica: Journal of the econometric society, pages 153-161, 1979.
[16] H. Hu, A. Lerer, B. Cui, L.. Pineda, N. Brown, and J. N. Foerster. Off-belief learning. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4369-4379. PMLR, 2021.
[17] H. Hu, A. Lerer, A. Peysakhovich, and J. Foerster. "Other-play" for zero-shot coordination, In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 13999-4110 PMLR, 13-18 Jul 2020.
[18] J. Huang, A. Gretton, K. Borgwardt, B. Schölkopf, and A. Smola. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19:601-608, 2006.
[19] N. Jakobi. Evolutionary robotics and the radical envelope-of-noise hypothesis. Adaptive Behavior, 6(2):325 -38, 1997.
[20] M. Jiang, M. Dennis, J. Parker-Holder J. Foerster, E. Grefenstette, and T. Rocktäschel. Replay- guided adversarial environment design. Advances in Neural Information Processing Systems, 34, 2021.
[21] M. Jiang, E. Grefenstette, and T. Rocktäschel. Prioritized level replay. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4940-4950. PMLR, 2021.
[22] N. Justesen, R. R. Torrado, P. Bontrager, A. Khalifa, J. Togelius, and S. Risi. Procedural level generation improves generality of deep reinforcement learning. CoRR, abs/1806. 106.1 2018.
[23] H. Küttler, N. Nardelli, A. H. Miller, R. Railcanu, M. Selvatici, E. Grefenstette, and T. Rocktäschel. The NetHack Learning Environment. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2020.
[24] X. Ma. Car racing with pytorch. https: //github/ com/ http:yyttooccrrrcepeee
2019.
[25] Matiisen, A. Oliver, T. Cohen, and Schulman. Teacher-student curriculum learning. IEEE
Transactions on Neural Networks and Learning Systems, PP, 07 2017.
[26] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 2 21:18 1::1::1-111::00 2020.
[27] S. Narvekar, J. Sinapov, M. Leonetti, and P. Stone. Source task creation for curriculum learning. In Proceedings of the 2016 international conference on autonomous agents & multiagent systems, pages 566-574, 2016
[28] J. F. Nash et al. Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36(1):48-43 49, 1950.
[29] O. OpenAI, M. Plappert, R. Sampedro, T. Xu, I. Akkaya, V. Kosaraju, P. Welinder, R. D'Sa, A. Petron, H. P. de Oliveira Pinto, A. Paino, H. Noh, L. Weng, Q. Yuan, C. Chu, and W. Zaremba. Asymmetric self-play for automatic goal discovery in robotic manipulation, 2021.
[30] I. Osband, D. Russo, and B. V. Roy. (more) efficient reinforcement learning via posterior sampling. In C. J. C. Burges, L. Bottou, Z. Ghahramani, and K. Q. Weinberger, editors, Advunces in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013, Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 3003-3011, 2013.

---

[31] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. CoRR, abs/1710.06537, 2017.
[32] I.. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning.
In International Conference on Machine Learning, pages 2817-2826. PMIR, 2017.
[33] D. Pomerleau. ALVINN: an autonomous land vehicle in neural network. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems 1. INIPS Conference, Denver, Colorado, USA, 1988], pages 305-313, Morgan Kaufmann, 1988.
[34] R. Portelas, C. Colas, K. Hofmann, and P.-Y. Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 835-853. PMLR, 30 Oct-01 Nov 2020.
[35] R. Portelas, C. Colas, L. Weng, K. Hofmann, and P.-Y. Oudeyer. Automatic curriculum learning for deep rl: A short survey. In C. Bessiere, editor, Proceedings of the Twenty-Ninh International Joint Conference on Artificial Intelligence, IJCAI-20, pages 4819-4825, International Joint Conferences on Artificial Intelligence Organization. 7 2020. Survey track.
[36] S. Ross and D. Bagnell. Efficient reductions for imitation learning. In Y. W. Teh and M. Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 661-668, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.
[37] M. Rowland, W. Dabney, and R. Munos. Adaptive trade-off's in off-policy learning. In Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 34-44. PMLR, 26-28 Aug 2020.
[38] F. Sadeghi and S. Levine. CAD2RL: real single-imen single--iggeeeee flight without a single real image. In N. M. Amato, S. S. Srinivasa, N. Ayanian, and S. Kuindersma, editors, Robotics: Science and Systems XIII. Massachusetts Institute of Technology, Cambridge, Massachusetts, USA, July
12-16, 2017, 2017.
[39] M. Samvelyan, R. Kirk, V. Kurin, J. Parker-Holder, M. Jiang, F. Hambro, F. Petroni, H. Kuttler, E. Grefenstette, and T. Rocktäschel. Minihack the planet: A sandhox for open-ended reinforcement learning research. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[40] J. Savage. The theory of statistical decision. Journal of the American Statistical association, 46(253):55-66 67, 1951.
[41] Schulman, P. Moritz, S. Levine, M. I. Jordan, and Abbeel. nigh-dimensiooal continuous control using generalized advantage estimation. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Truck Proceedings, 2016.
policy optimization [42] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy algorithms, 2017.
[43] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 2016.
[44] A. Stooke, A. Mahajan, C. Barros, C. Deck, J. Bauer, J. Sygnowski, M. Trebacz, M. Jaderberg, M. Mathieu, N. McAleese, N. Bradley-Schmieg, N. Wong, N. Porcel, R. Raileanu, S. Hughes- Fitt, V. Dalibard, and W. M. Czarnecki. Open-ended learning leads to generally capable agents. CoRR. abs/2107. 12808, 2021.
[45] S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and R. Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations, 2018
[46] R. S. Sutton and A. G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998.
12

---

[47] R. S. Sutton, A. R. Mahmood, and M. White. An emphatic approach to the problem of off- off-policy curporalddifference learning. Journal of Machine Learning Research, 17(73):1-29. 2016.
[48] Y. Tang, D. Nguyen, and D. Ha. Neuroevolution of self-interpretable agents. In Proceedings of the Genetic and Evolutionary Computation Conference, 2020.
[49] P. Thomas and E. Brunskill. Data-efficient off off-po policy evaluation for reinforcement learning. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2139-2111100 2148, New York, New York, USA. 20-22 Jun 2016. PMLR.
[50] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSI International Conference on Intelligent Robots and Systems, IROS 2017, Vancouver, BC, Canada, September 24-28, 2017, pages 23-30. IEEE, 2017.
[51] V. N. Vapnik and A. Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, pages 264-280, 1971.
[52] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang. L.. Sifre,
T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. Paine, Ç. Gülçehre, Z. Wang. T. Pfaff, Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. P. Lillicrap, K. Kavukcuoglu,
D. Hassabis, C. Apps, and D. Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature, 575(7782):350-35)5 354, 2019.
[53] R. Wang. J. Lehman, J. Clune, and K. O. Stanley. Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. CoRR, abs/1901.00 .1.0111 153,,, 2019.
[54] R. Wang. J. Lehman, A. Rawal, Zhi, Y. Li, J. Clune, and K. Stanley. Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In Proceedings of the 37th International Conference on Machine Learning, pages 1940-9951, 2020.
[55] X. Wu, E. Dyer, and B. Neyshabur. When do curricula work? In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nee, 2021.
[56] A. Zhang, 7. C. Lipton, L. Pineda, K. Arizzadenesheli, A. Anandkumar, L. Itti, J. Pineau, and T. Furlanello. Learning causal state representations of partially observable environments. CORR, abs/ Abs/1996.10 10430,77222 2019.
[57] A. Zhang, R. T. McAllister, R. Calandra, Y. Gal, and S. Levine. Learning invariant representations for reinforcement learning without reconstruction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[58) V. Zhong, T. Rocktäschel, and E. Grefenstette. RTFM: generalising to new environment dynamics via reading. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net 2020.
[59] L. M. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson. Varibad: very good method for bayes-adaptive deep RL via meta-learning In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 30, 2020. OpenReview.net, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix E.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section In particular, we highlight that our proof of SAMPLR's s robustly e-Bayes optimal property assumes the underlying optimization procedure reaches an approximate Nash cquilibrium.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 5.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] The code reproducing the experimental results is included in the supplemental material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 6 and Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] See figure captions in Section 6.
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See compute estimates in Appendix C.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [Yes] We include the license for the code
reproducing our experiments in the supplemental material.
(c) Did you include any assets either the supplemental material
URL? [Yes]
We include the code reproducing our experimental results in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "sh2RSvU0XI", "reviewer": "Reviewer_dr8D", "summary": "This paper aims to generate curricula to 1) preserve the performance under the target distribution, and 2) improve the generalization to multiple levels by avoiding over-conservativeness due to the worst-case scenario. An algorithm is proposed based on Prioritized Level Replay and Bayesian inference. Some theoretical results are presented to give insights into the practical implementation. The experiment is conducted on a gridworld hard-exploration task and a continuous control task.", "questions": "See Weaknesses 2, 3 and 5.\n\nAdditional question:\nHow can SAMPLR be combined with UED methods other than PLR, such as PAIRED, RARL [1], etc.?\n\n[1] Pinto, Lerrel, et al. \"Robust adversarial reinforcement learning.\" International Conference on Machine Learning. PMLR, 2017", "limitations": "I see some of the limitations are discussed in the paper. I do not see any potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\nThis work looks at an interesting extension of Unsupervised Environment Design, which is associated with a target distribution of environments. The perspective is novel and intriguing. The experimental results seem to be significant.\n\n**Weaknesses:**\n1. The assumption to have an access to full access to the exact perfect simulator is very restrictive. It requires not only the parameter of the environment but the entirety of transition dynamics and initialization, random seed, and the ability to reset to any state. Most of the curriculum RL assumes access to the parameter of the environment, but not the latter parts.\n2. Could the author provide more intuition about how the fictitious trajectories help mitigate the curriculum-induced covariate shift? I do not think the explanation is sufficient in the current draft.\n3. Could learning a posterior $\\bar{P}\\left(\\theta^{\\prime} \\mid \\tau\\right)$ using supervised learning require tons of data? The curriculum itself could produce limited pairs of trajectories and $\\theta$. Are the data/environment interactions required to obtain this estimator included in the plots?\n4. Although the objective is to generate a curriculum, there is no visualization of what kind of curriculum is generated and how they improve the performance. If $\\theta$ controls the goal identity, showing the number of training rooms (Figure 3) is more of the result due to the curriculum but not how SAMPLR helps. Similar arguments go to Figure 4. \n5. How is SAMPLR scaled to environments with a higher dimension of environment parameters? Both of the experiments have only one dimension and it is hard to see the potential.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BsA6L9odeb8", "reviewer": "Reviewer_tyft", "summary": "This paper proposes a method that addresses the covariant shift problem that can happen between the training parameter distribution induced by curriculum and test-time parameters in adaptive curriculum learning methods. Naive choice would be to sample parameters from ground-truth distribution but this would lose the benefit of curriculum learning, so the paper proposes a method that adjusts the policy learning to be optimal with ground-truth simulator while still sampling environment parameters with some curriculums.", "questions": "- Improving clarity of the paper would be nice (see above for details)\n- Experiments in more complex setups instead of simple setup of fruit choice or single-parameter setup in racing env would be helpful\n- Clarification on the agent design and discussion on how it would affect the analysis and the messages in the paper would be helpful\n", "limitations": "Limitation or potential negative impact are not discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- This paper investigates an interesting problem of environment design with curriculum\n- Experimental results show that the proposed method can outperform baselines potentially suffering from covariate shift\n\n**Weaknesses**\n- One of weaknesses in the paper is in the clarity especially Introduction, where it is difficult to understand what 'grounding' means in the title and also throughout the paper. It was also a bit difficult to parse the last sections in introduction without having a prior knowledge about the area; I could understand it after looking at the definitions from method section. Making it much more abstract in intro or more formal would be helpful for readability.\n- In Figure 1, it would be nice to explain what U means and matching this during training would help, in caption.\n- In Equation 3, It's not clear to me what does $\\bar{P}(\\tau_{t}^{O}|\\theta,\\tau_{t}^{A})$ mean and what would this being equal to 1 mean?\n- It's curious to see how the method would work when considering more complex parameter configurations. Currently it's just a single parameter setup and it would be nice to show that the method could cover more complex setups.\n- Is PPO agent based on recurrent architecture? The reason why I ask this question is because it is a bit questionable whether the ice parameter is indeed aleatoric -- if the agent has an information about the previous observations and actions where actions were not applied then agent might have more information about the road even without the visible pixel observation in roads, which makes it difficult to call icy parameter as aleatoric? It would be nice to clarify this point.\n\n---\n[2022-08-05 Update] Increased the score (5->6) per the discussion during the rebuttal phase", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1zZPeGXDxmz", "reviewer": "Reviewer_EQPK", "summary": "This paper focuses on the unsupervised environment design (UED) framework. They first formalized the covariate shift issue that the training distribution differs from the test distribution in curriculum learning. To address this issue, they proposed a new algorithm SAMPLR, which can preserve optimality on the ground-truth distribution while training under a usefully biased training distribution. Finally, they evaluated SAMPLR on two tasks.", "questions": "1. In the related work part, the connection between SAMPLR and Off-Belief Learning in multi-agent RL is established. It is not intuitive to understand this connection. Can the authors elaborate more on this point?", "limitations": "The limitations of their algorithm are discussed clearly in sections 4 and 5.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper studies the covariate shift issue in UED, which is an important problem.\n2. The proposed algorithm SAMPLR outperforms baselines on two tasks. \n\nWeakness:\n1. I am most concerned about the assumptions required by SAMPLR. First, SAMPLR requires access to a simulator that can be reset to any state. In many RL tasks, the simulator cannot be reset to an arbitrary state. Second, SAMPLR requires knowledge of the ground truth distribution.  \n2. The experiments are insufficient to support their algorithm. First, they tested the proposed algorithm only on two tasks. I am not sure whether the covariate shift issue is common in curriculum learning and whether SAMPLR can consistently outperform baselines. Second, it seems that the stochastic fruit choice task is a bit too simple. In this task, the proposed algorithm reduces to naïve grounding. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept