Causally Aligned Curriculum Learning
Authors: Mingxuan Li
Junzhe Zhang
Elias Bareinboim
Venue: ICLR 2024 poster (2024)
Keywords: Causal Inference
Curriculum Learning
Reinforcement Learning
Primary area: causal reasoning

Abstract: A pervasive challenge in Reinforcement Learning (RL) is the ``curse of dimensionality'' which is the exponential growth in the state-action space when optimizing a high-dimensional target task. The framework of curriculum learning trains the agent in a curriculum composed of a sequence of related and more manageable source tasks. The expectation is that when some optimal decision rules are shared across source tasks and the target task, the agent could more quickly pick up the necessary skills to behave optimally in the environment, thus accelerating the learning process. 
However, this critical assumption of invariant optimal decision rules does not necessarily hold in many practical applications, specifically when the underlying environment contains unobserved confounders. This paper studies the problem of curriculum RL through causal lenses. We derive a sufficient graphical condition characterizing causally aligned source tasks, i.e., the invariance of optimal decision rules holds. We further develop an efficient algorithm to generate a causally aligned curriculum, provided with qualitative causal knowledge of the target environment. Finally, we validate our proposed methodology through experiments in confounded environments.

Published as a conference paper at ICLR 2024
CAUSALLY ALIGNED CURRICULUM LEARNING
Mingxuan Li and Junzhe Zhang and Elias Bareinboim Causal Artificial Intelligence Lab, Columbia University, USA (ml, junzhez, eb) @cs. columbia.ee.
ABSTRACT
A pervasive challenge in Reinforcement Learning (RI.) is the "curse of dimen- sionality" which is the exponential growth in the state-action space when optimiz- ing a high-dimensional target task. The framework of curriculum learning trains the agent in a curriculum composed of sequence of related and more manage- able source tasks. The expectation that when some optimal decision rules are shared across source tasks and the target task, the agent could more quickly pick up the necessary skills to behave optimally in the environment, thus accelerating the learning process. However, this critical assumption of invariant optimal de- cision rules does not necessarily hold in many practical applications, specifically when the underlying environment contains unobserved confounders. This paper studies the problem of curriculum RL through causal lenses. We derive a suffi- cient graphical condition characterizing causally aligned source tasks, i.e., the in- variance of optimal decision rules holds. We further develop an efficient algorithm generate causally aligned curriculum, provided with qualitative causal knowl- edge of the target task. Finally, we validate our proposed methodology through experiments in discrete and continuous confounded tasks with pixel observations.
INTRODUCTION
As Roma not built in one day, learning achieve complex task (e.g., cooking, driving) directly can be challenging. Instead, the human learning process is scaffolded with incremental difficulty to support acquiring progressively advanced knowledge and skills. The idea of training with increas- ingly complex tasks, known as curriculum learning, has been applied in reinforcement learning when Selfridge et al. (9855) used a carefully curated sequence of tasks to train agents to solve modified Cart Pole system. In recent years, there has been a growing interest in automatically generating curricula tailored the agent's current capabilities, which opens up a new venue called "Automatic Curriculum Learning" (Portelas et al.) 2020). An automatic curriculum generator requires two com- ponents: an encoded task space and a task characterization function (Narvekar et al., 2020 Wang et al. 2020). Task space encoding is often bijective function that maps task to low dimensional
vector Porker-Holder et al.)
2022 Klink et al. 2022 Florensa et al.] 2018: Jiang et al. 2021 Porte-
2020 Cho al. 2023 Huang al. 2022a). A proper task space encoding lays the foundation of a reasonable task characterization function measuring the fitness
las et al. 2019:; Wang et al.
2019
2018 Dennis et al. 2020 Andreas al.) 2017 Sukhbaatar et al. 2018 Jiang et al.) 2021). New training tasks, called source tasks, are generated by changing the target task's state space parameters of transition functions in the encoded task space. A system designer then determines in which order the agent should be trained in these source tasks, following the task characterization function. The set of generated source tasks and the training order defined upon this set defines curriculum for the learning agent. Please see App. G for more related work.
of tasks (Florensa et al.
While impressive, most curriculum RL methods described so far rely on the assumption that gener- ated source tasks are aligned with the target. Consequently, the agent could pick up some valuable skills by training in such source tasks, allowing it to behave optimally in certain situations in the target environment. However, this critical assumption does not necessarily hold in many real-world decision-making settings. For concreteness, consider a modified Sokoban game shown in Fig. [ in- spired by Schrader (2018) where unobserved confounder U, randomly determines the box color C, (0 for yellow, for blue) at every time step t. The agent receives a positive reward Y, only when it pushes the box to the goal state when the box color appears yellow (U, 0); otherwise,

---

Published as a conference paper at ICLR 2024
Misaligned Source Task
(b) Aligned Source Task
"Push the box"
"Push the box, little further"
"Push the box, even further"
"Target Task"
(c) Aligned Curriculum
Figure I: Examples of dap full episode misaligned source task that intervenes in the box color, (b) full episode of aligned source task that only changes the initial box location, and (ch an aligned curriculum where none the source tasks intervenes in the box's 's color.
gets penalized (U, = 1). We apply several state-of-the-ard curriculum generators that construct source tasks by fixing the box color to yellow or blue, including ALP-GMM (Portelas et al. 2019),
PLR (Jiang et al. 202I). Goal-GAN (Florensa et al., 2018), and Currot (Klink et al.J 2022). ). Fig. la shows example of the generated source tasks. We evaluate agents' performance trained by those generated curricula and compare it with the one directly trained in the target task. Surprisingly, sim- ulation results shown in Fig. 2 reveal that agents trained by the curricula failed to learn push the yellow box to the destination. This suggests source tasks generated by intervening in the box color misaligned; that is, training in these source tasks harms the agents' target task performance.
Several observations follow from the Sokoban example. (1) A curriculum designer generates source tasks by modifying the data-generating mechanisms in the target tasks. (2) Such mod- ifications could lead to a shift in system dynamics between the target task and source tasks. When this distribution shift is sig- nificant, training in source tasks may harm the agent's learning. (3) The agent must avoid misaligned source tasks to achieve optimal learning performance. There exist methods attempting to address the challenges of misaligned source tasks leverag- ing a heuristic similarity measure between the target and source tasks (Svetlik et al. 2017: Silva & Costa 2018). Yet, systematic and theoretically justified ap- proach for exploiting other types of knowledge, e.g. qualitative, about the target task is missing.
Goil-GAN
Reward
PLR
ALP CMM
100K
150
Step
Figure 2: The average perfor- mance of curriculum generators.
This paper aims to address the challenges of misaligned source tasks in curriculum generation by exploring causal relationships among variables present in the underlying environment. To real- ize this, formalize curriculum learning in the theoretical framework of structural causal models (SCMs) (Pearl 20099.. This formulation allows us to characterize misaligned source tasks by exam- ining the structural invariance across the optimal policies obtained from the target and source tasks. More specifically, our contributions are summarized as follows. (1) We derive a sufficient graphical condition determining potentially misaligned source tasks. (2) We develop efficient algorithms for detecting misaligned source tasks and constructing source tasks that are guaranteed to align with the target task. (3) We introduce a novel augmentation procedure that enables state-ofthhe-ar currieu- lum learning algorithms generate aligned curricula to accelerate the agent's 's learning. Finally, validate the proposed framework through extensive experiments in various decision-making tasks.
1.1 PRELIMINARIES
This section introduces necessary notations and definitions that will be used throughout the discus- sion. We use capital letters (X) to denote random variable, lowercase letters (x: to represent specific value of the random variable, and ?(-) to denote the domain random variable. We use bold capital letters (V) to denote set of random variables and V to denote its cardinality.
2

---

Published as a conference paper at ICLR 2024
The basic semantical framework of our analysis rests on structural causal models (SCMs) (Pearl 2009 Bareinboim & Pearl 2016). An SCM M is a tuple (U, VU...9. S&P,P, P), where U is a set of exogenous variables and V is set of endogenous variables. F is set of functions s.t. each fv E & decides values of an endogenous variable V E V taking as argument a combination of other variables in the system. That is, V < fv (Av, Uv), PAy C V. Uv U. Values of exogenous variables U are drawn from the exogenous distribution P(U). A policy π over a subset of variables X C V is a sequence of decision rules {x(X]Sx)) Sx]]xex where every (X)Sx) is probability distribution mapping from domains of set of covariates Sx S V to the domain of action X. An intervention following a policy π over variables X, denoted by do(rr), an operation which sets values of every X E X to be decided by policy X ~ xX|[xx) (Correa & Bareinboim, 2020], replacing the functions fx = (/x : V E X) that would normally determine their values. For an SCM M, let M,, be submodel of M induced by intervention do(x). For a set Y C V. the interventional distribution P (T: is defined as the distribution over Y in the submodel Mg, i.c.,
PM (Y; T) A PM, (Y); restriction MM is left implicit when it is obvious.
Each SCM M is also associated with causal diagram G (e.g., Fig, 3aj which is directed acyelic graph (DAG) where nodes represent endogenous variables V and arrows represent the arguments PAv, Uy of each structural function fv Φ. Exogenous variables U are often not explicitly shown by convention. However, a bi-directed arrow V. <> V, indicates the presence of an unob- served confounder (UC), UL, E U affecting Vi, Vj. simultaneously (Bareinboim al.) 2022). We will use standard graph-theoretic family abbreviations to represent graphical relationships, such as parents (pa), children (ch), descendants (de), and ancestors (an). For example, the set of parent nodes of X in G is denoted by pa(X) gg = Uxex (X)g. Capitalized versions Pa, Ch, De, An include the argument as well, e.g., g
pa g U X. A path from a node X to node Y in G is sequence of edges that does not include a particular node more than once. Two sets of nodes X, Y are said to be d-separated by a third set in a DAG G, denoted by (X Y 2)g, every edge path from nodes in X to nodes in Y is "blocked" by nodes in Z. The criterion of block- age follows Pearl 2009 Def. 1.2.3). For more details on SCMs, we refer readers to Pearl (2009); Bareinboim al. 2022 For the relationship between (PO)MDPs and SCMs, please see App. H
CHALLENGES OF MISALIGNED SOURCE TASKS
This section will formalize the concept of aligned source tasks and provide an efficient algorithmic procedure to find such tasks based on causal knowledge about the nata-genneaating process. Formally, planning/policy learning task (for short, task) is a decision-making problem composed of an environment and an agent. We focus on the sequential setting where the agent determines values of sequence of actions X = [X1, ( XH} } based on the input of observed states {S1..... SH). The mapping between states and actions defines the space of candidate policies, namely,
Definition 1 (Policy Space). For an SCM M
(U, 1, ,,P). policy space 11 is set of Each policy π is a sequence of decision rules } where for every i 1, H,
policies π over actions X = (XI,...,,XH) TH (X (X ...
(i) Action X, is a non-descendent of future actions X Xu. i.. X, E V \ (e(X++)H); (ii) States S; are non-descendants of future actions X....., XH. i.e. S. C V De u.n) Henceforth, we will consistently denote such a policy space by II = ((X1,
(XH, Sn)}.
The agent interacts with the environment by performing intervention do(T), VT E Il to optimize reward function R(Y) taking a set of reward signals Y C V as input | A policy space, a reward function, and an SCM environment formalize a target decision-making task. We will graphically describe a target task using an augmented causal diagram & constructed from the SCM M; actions X are highlighted in blue; reward signals Y are highlighted in red; input states S, for every action X; E X are shaded in light blue. For instance, Fig. 3a shows a causal diagram representing the decision-making task in the Sokoban game (Fig. [T. For every time step i 1,..., H. L stands for the agent's location, B; for the box location, and C2 for the box color.
For instance, cumulative discounted reward is defined as R(Y)- EiL, 1 where V, V, i 1...oy H, endogenous variables, and ry E (0, 1] a discount factor.
3

---

Published as a conference paper at ICLR 2024
11.
B
B2
(R)
(a) g
(b) g() and g())
Figure 3: Causal diagram for (a) the target task T; and (b) comparing domain discrepancies between the target task T and source tasks (()) and ((2), (b) is (a) augmented by edit indicators.
Definition (Target Task). A target task is a tuple T= (M,TI,), where M F. an SCM, II policy space over actions X S V, and R is reward function over signals Y C.V
The goal is to find an optimal policy क* E II that maximizes the expected reward function E (YY)); ; m] evaluated in the underlying environment M, i...,
π arg max EM [R(Y):
FEll
When the detailed parametrization of the SCM M is provided, the optimal policy x' is obtainable by applying planning algorithms, e.g., dynamic programming (Bellman 1966) or influence diagrams (Koller & Milch 2003). However, when underlying system dynamics complex or the state-action domains are high-dimensional, it might be challenging to solve an optimal policy even with state-of-- the-art planning algorithms. We will then consider the curriculum learning approach (Selfridge al., 1985), where the agent is not immediately trained in the target task but provided with sequence of related yet simplified source tasks.
Definition 3 (Source Task). For a target task T (M, II, R), a source task TD) a tuple (M6) II, R, AG) where M6) is an SCM compatible with the same causal diagram M, i.c., GM GMini of variables AG) C V is called edits where there might exist a discrepancy that fv * 1 or P(Uv)) + (Uv) for every V E AC)
In practice, source tasks are constructed from the target task by modifying parameters of the under- lying structural functions I or exogenous distributions P(U). Consider again the Sokoban game described in Fig. 1 The system designer could generate source task (1) by changing the agent and box's 's initial location L1, B,. Fig. shows a causal diagram ((1) representing the source task (11;; (1)) is an edit indicator representing the domain discrepancies source tasks T1), Here, arrows (1)) + L1, 7(1) + B₁ suggest that structural functions fr, f', or exogenous distributions PUUr, Un, have been changed the source task (1) while other parts of the system remain the same as the target task T.
between the target and
By simplifying the system dynamics, learning an optimal policy in the source task T()) could be easier than in the target task T. The expectation here is that the optimal decision rules n00 over some actions ()) C X remain invariant across the source and target tasks. If so, we will call such source tasks aligned. Training in aligned source task thus guides the agent to move toward an optimal policy π. For example, Fig. Tb] shows an aligned source task for the Sokoban game where the agent and box's locations set close to the goal state. By training in the simplified task, the agent learns the optimal decision rule to push the yellow box to the goal state in this game.
However, modifying the target task could lead misaligned source task whose system dynamics differ significantly from the target. Interestingly and more seriously, training in these source tasks may "harm" the agent's 's performance, resulting in suboptimal decision rules, as illustrated next.
We consistently use superscript (7) to indicate a diagram g() A GMan associated with source task 76), Similarly, we write p(i) (Y; PMiis (Y; and g(5) arg max Byts [R())=))
4

---

Published as a conference paper at ICLR 2024
Example (Misaligned Source Task). Consider the Sokoban game T (M, R) described in Fig. T Fig. Ba shows its causal diagram G. Specifically, the box color C₁ (0 for yellow, 1 for blue) is determined by unobserved confounder U, E {0, 1) randomly drawn from distribution P(U, 1) = 3/4. Box location B; and agent location L, are determined following system dynamics in deterministic grid worlds (Chevalier-Boiss) Boisvert et al. 2018). The reward signal Y; is given by,
10
if B; = "next to goal" A X; "push" A (U: = 0)
Y
-
10
if B; = "next to goal" A X; = "push" A (U; = =)-
(2)
0.1 otherwise
If the agent pushes the box into the goal location (top right corner in Fig. [ it receives positive reward when the box appears yellow; gets penalized when the box appears blue. Since C₁ t- U. evaluating the conditional reward E Y b;, Gz do(x;)] in the Sokoban environment M gives.
10
if C, =
E
Yi B; = "next to goal", C;;d (X; = "push")] =
(3)
10 if C; =1
Thus, the agent should aim to push yellow boxes to the goal location in the target. The curriculum designer now attempts to generate source task ((2) by fixing the box color to yellow, i.c., C₁ + 0. Fig. 3b) shows the causal diagram g(2) associated with the source environment M2 where edit indicators 7(2) denote the change in the structural function Sc, determining the box color C.- Evaluating the conditional reward E [y; do(c, xx:]] in this manipulated environment M gives
E (2) [Y, B, = "next to goal"; ; do (C: 0, X; "push")] -5.
Detailed computations are provided in App. B Perhaps counter- counter-intuitivee intuitively, pushing the yellow box to the goal location in the source task (2) results in negative expected reward. This is because box color C₁ is only a proxy the unobserved Ug that controls the reward. Fixing C; won't affect Y but only breaks this synergy, hiding the critical information of U; from the agent. Consequently, when training in the source task (2)), the agent will learn to never push the box even when it is next the goal location, which is suboptimal in the target Sokoban game T.
2.1 CAUSALLY ALIGNED SOURCE TASK
Example [ suggests that naively training in a misaligned source task may lead to suboptimal per- formance in the target task. The remainder of this section will introduce an efficient strategy to avoid misaligned source tasks, provided with the causal knowledge of the underlying data- gettraennra mechanisms in the environment. For a target task T = (M, II, R), let 9 be the causal diagram as- sociated with M. Let G.r be intervened diagram obtained from G by replacing incoming arrows action X; E X with arrows from input states S; for every action X; E X. We first characterize a set of variables (j) C V amenable to editing (for short, editable states) using independence relationships between edit indicators T() and reward signals Y. Formally,
Definition 4 (Editable States). For target task T (M, II, R), let & be a causal diagram of M and X ()) C X be subset of actions. A set of variables A C VV) x is editable w.r. X if and only if VX, E x()), the following independence holds in the intervened diagram G..
(~(0 IL Y n De(X,) S.).
(5)
where -0) is the set of added edit indicators pointing into nodes in (1)
For example, consider again the Sokoban game described Example The initial agent and box's position (1) {Bi, L is editable with regard all actions X following Def. Precisely, in the augmented diagram ((2) of Fig. 3b for every action X; E X, the edit indicators ( are d-separated the reward signals Y n (e(X;)
YY...., Yn given input states {L,, B,, C.}. On the other hand, {CI...., CH } are not editable w.r.t actions X since in the augmented diagram ((2) of Fig. 3b for every action X, E X, there exists an active path between edit indicators (2) and reward signals {Y..... Yn given action X, and input states {L,, B,, C.),
the set of box color variables
violating the criterion given by Def.
5

---

Published as a conference paper at ICLR 2024
For fixed policy E II, for any subset S C V, denote by (S: (As E ((S) PM (s; > 0} the set of reachable values of S, which is the set states that are possible to reach in a source task TU) under intervention do( The following proposition establishes that modifying functions and distributions over a set of editable states A leads to an aligned source task.
Theorem (Causally Aligned Source Task). For a target task T [M,1I,R let TÚ) (M II R. A ()) be source task T by modifying states A C V. If editable w.r.1 some actions X()
(d) C X, then for every action X; E X
9i (Xx 8:) 7.2) (X: st), vs; E ( (Q((STT)
(6)
where at" m(1) E II optimal policies in the target T and source T(d) tasks, respectively.
Thm. Iimplis that whenever states ()) is editable w.r.t. some actions X(i,, one could always construct an aligned source task TG) such that the optimal decision rules π over (J) is invariant across the target T and source T tasks. Consequently, one could transport these optimal decision rules trained in the source task T() without harming the agent's performance in the target domain
T. F For example, in the Sokoban game of Example 1 since initial states A (B1, L1) is editable w.r.t. actions X, moving the agent and box's location leads to an aligned source task, which allows the agent to learn how to behave optimally when getting closer to the goal state. However, the performance guarantee in Thm. [ does not necessarily hold when states A (j) are not editable. For instance, recall that A (2) (C1. CH) } are not editable in the Sokoban game. Modifying the box's color could lead to misaligned source task (2). An agent trained in this source task could pick up undesirable behaviors, as demonstrated Example
Algo. describes an algorithmic pro- cedure, FINDMAXEDIT, to find a maximal editable set A in causal diagram G w.r.t. a set of actions X C X. set of editable states A is maximal w.r.t. X if there is no other editable states A strictly containing AU) We always prefer maximal editable set since it offers the maximum freedom to simplify the system dynamics in the target task. Particularly, FINDMAXEDIT it- eratively adds endogenous variables V (X(UY) to the editable states AU) and test the inde- pendence criterion in Def. 4 This procedure continues until it cannot add any more endogenous variables. Evidently, FINDMAXEDIT returns maximal editable set (i) w.r.t. X(), A natural question arising at this point is whether the ordering of endogenous variables V changes the output. Fortunately, the next proposition shows that this is not the case.
Algorithm 1: FINDMAXEDIT
Input: causal diagram 9T and set of actions, X Output: The maximal editable states A (1) w.r.t X
Let (J) < 0:
for V E V XUYA do A + u
for every X, E X do if if(6)) H. Y De(,)) X S:) in In then Remove V from A and break:
return AU;;
Theorem 2. For a target task = (M, II, R), let Gn be an intervened causal diagram of M and let ( C X be subset of actions. FINDMAXEDIT (9r,X returns maximal editable set A () w.r actions X(i); moreover, such a maximal set A (i) is unique.
Let n and m denote the number of nodes and edges in the intervened diagram GN and let d be the number of actions X. Since testing d-separation has time complexity of O(n + m)) FIND- MAXEDIT has a time complexity of O (d(n + m)). We also provide other algorithmic procedures for directly deciding set's 's editability and constructing editable sets for a target task T in App. C
3 CURRICULUM LEARNING VIA CAUSAL LENS
Once a collection of source tasks is constructed, the system designer could organize them into an ordered list, called curriculum, defined next:
Causal aligned source tasks (Thm. 1 and editable states (Def. 4 are related to the concept of transportabil- iry in causal inference literature (Bareinboim & Pearl 2016). which generalizes estimation of unknown causal effects from different domains. Here study the generalizability of optimal decision policy.

---

Published as a conference paper at ICLR 2024
Definition 5 (Currieulum). For target task T (M, 11, R.,, curriculum C. for T is sequence of source tasks (TO)) N f=1 where T (MG), 11, R, A
For instance, Fig. Tc describes a curriculum in the Sokoban game where the agent and the box are placed increasingly further away from the goal location. Given a curriculum C, a typical curriculum learning algorithm trains the agent sequentially in each souree task, following the curriculum's ordering. Algo. 2 shows the pseudo-code describing this train- ing process. It first initializes arbitrary baseline policy (()) For every source task T()) E C. the algorithm updates the current policy -(-1) such that the new policy π(() is optimal in the source task TÚ), This step could be performed using standard gradient-based algorithm, e-g., the policy gradient (Sutton & Barto, 2018). The expectation is that, as the agent picks up more skills in the source tasks, it could consis- tently improve its performance in the target task or at least not regress.
Algorithm 2: CURRICULUM LEARNING
Input: A curriculum C.
Output: A policy ((N) E II.
Initialize a baseline policy x0(); for 1,... NN do Update policy π() from T(1-1) such that
T() arg max MuI [R(Y); (7)
Tlll
return (M);
Definition 6 (Causally Aligned Curriculum). For a target task T (M, let C - be a curriculum for T. Curriculum C' is said to be causally aligned with ' if for every j = 1, 1, the set of invariant optimal decision rules across the source task and the target task expands, i.e.,
(x())=+)) S ( (+(+)))zz))
where n= E II is an optimal policy in the target task T.
A naive approach to construct a causally aligned curriculum is to (1) construct a set of aligned source tasks by modifying editable states (Thm. [T] and (2) organize these tasks in an arbitrary ordering. However, the following example shows this is not a viable option.
Example 2 (Overwriting in Curriculum Learning.. Consider a sta-s target task where action X1 takes input H and X₂ takes input Z. The task SCM is. H = Un Z = X1 + Uz, Yi = 0.5 * (H + X1). Y2 = -H @> X₂ A where P(Ux = 1) = 1/2, P(U" = 1) = 1 1/100. Other than the reward Y1, Y2. all other variables are binary. The optimal policy for the target task is n (X1 --H HH) = 1,
rX22 = 0|2)= = 1. We create two source tasks. For T1)), let P(UH
1) 9/10 while other parts stay the same as target task T. For 7(2), let Z -X1 while other parts stay the same as target task T. From the causal diagram, we see that A (1) = {H} is editable w.r.t X (1) = {X,} and A = {Z} is editable w.r.t X {X2}.
Now if the agent is trained in a curriculum C = (T(), ( 7(2),
its target task performance will 1|
deteriorate instead of improving. To witness, the optimal policy for X₂ in ((1) is ())
1 and the optimal policy for X1 in (2) is (X)) 0) H) = 1. After training in T π has an expected target task reward of 0.55 since z( (X2 = 2) is not optimal in the target yet. So, the agent proceeds to train in (22), It will learn the optimal target policy for X2, T" (X2 0 1. But in the mean time, optimal policy of X, learned from (1), T (X1 -H (H)
is also overwritten by (2),, The agent will only receive 0.5 in the target task, which is even worse than before training in (2), This suggests that curriculum C is not causally aligned.
In the above example, the agent fails to learn an optimal policy due to "policy overwriting". Fig. provides graphical representa-
Farget x]
tion of this phenomenon. Particularly, each source task (1), (2) covers one of the optimal decision rules over action X1, X2. re- spectively. An agent trained in one of the source tasks, say T(),
T(2) N1,73
Farget K
learns the optimal decision rule गं for action X1, but forgets the decision rule πद for the other action X₂ learned previously in (2).
Figure 4: Policy overwriting
described in Example 2
The same overwriting also occurs when the agent moves from task T(2) to T()) This means that regardless of how the system designer orders the curriculum, e.g., C = (T(), T(2), T(1,, T(2),.. .....,... }. the agent will always forget useful skills it picked up from
7

---

Published as a conference paper at ICLR 2024
previous source tasks, thus making it unable to achieve satisfactory performance in the target task. This example implies that there are more conditions for curriculum to be "causally aligned"..
3.1 DESIGNING CAUSALLY ALIGNED CURRICULUM
We will next introduce novel algorithmic procedure to construct causally aligned curriculum while avoiding the issue of overwriting. We will focus on a general class of soluble target tasks, which generalizes the perfect recall criterion (Koller & Friedman, 2009) in the planning/decision- making literature (Lauritzen & Nilsson 2001).
Definition 7 (Soluble Target Task). A target task T = (M, R) is soluble if whenever < i, (Y n (e,,, TL Ty|S; X;) Gs. where Ty is a newly added regime node pointing to Xj.
In words, Def. 7 says that for a soluble target task T, for every action X; E X, the input states S summarizes all the states and actions' history S, X1,
..... X;- If this is the case, an optimal policy x² for task T is obtainable by solving series of dynamic programs (Lauritzen & Nilsson 2001 Koller Milch 2003). For instance, the Sokoban game T graphically described in Fig. Balis is soluble. For every time step = ...... H, given input states S; = { {Le B;, Ci} and action ......,, X,, regime variables Wi...., ., 1 are d d-spaaraeed from subsequent reward signals Y,...., .u.
1,
Theorem 3 (Causally Aligned Curriculum). For a soluble target task T = (M, II, R), curriculum (T fj=1 is causally aligned if the following conditions hold,
(i) Every source task T()) E C is causally aligned w.r.t. actions X (Def. 4:
(ii) For every ,,.... N 1, actions X (5) x(++1).
Consider again the Sokoban game described in Fig. 3a Let [T6)} be curriculum such that for every source task TU) is obtained by editing the agent and box's location A A = {Le, B.) time step i H - j + We now examine conditions in Thm. 3 and see if C causally aligned. First, Condition (i) holds since every source task TG) is causally aligned w.r.. actions X {X X following discussion in the previous section. Also, Condition (ii) holds since for every j 1,.... H 1, actions ( C ( This implies that could construct a causally aligned curriculum in the Sokoban game by repeatedly editing the agent and box' location following reversed topological ordering; Fig. [c describes such an example.
The idea in Thm. 3 Suggests natural procedure for constructing causally aligned curriculum, FINDCAUSAL.
Algorithm 3: FINDCAUSALCURRICEE
Input: A target task T. causal diagram G.r Output: A causally aligned curriculum C Let C + 6: for j = H,..., ... do
which is implemented RICULUM (Algo. 3) Particularly, it assumes access curriculum generator GEN(T, A which generates source task T by editing set of states A C V in the target task T. follows a reverse topological ordering .........k, For every step j = H.... 1. the algorithm call the sub- routine FINDMAXEDIT (Algo. D to find of editable states A w.r.t. actions ())
Let x +
XH3:
Let A <- FINDMAXEDIT X());
Let T()) + GEN(T, AA));
actions X
Let = -CU(TU); return
{X,....,XHI)
J. }. then calls the generator GEN to generate a source task TG) by editing states ( The conditions in Thm. Jem.ure that Algo. returns causally aligned curriculum.
Corollary 1. For a soluble target task T (M, II, R), let GR be an intervened causal diagram of M. FINDCAUS AT. WTT.SSAAACCCRCUUTIITI (T, 9-) returns a causally aligned curriculum.
A more detailed discussion the additional conditions under which combination of Algs and
is guaranteed to find an optimal target task policy is provided in App. D
EXPERIMENTS
In this section, we build on Algo. B and different curriculum generators to evaluate causally aligned curricula for solving challenging tasks in which confounding bias is present and previous, non- causal generators cannot solve. In particular, four best-performing curriculum generators:
8

---

Published as a conference paper at ICLR 2024
Colored Sokoban
1.0
1.0
1.0
1.0
0.8
0.8
onginnl
orghtl
ofgiral
04
orgital
cuupal
02
canool
02
000551
62
25CA
E307200002 Stape
InArEReR: Saga
DIvEnocees Stape
1.0
3.0
Button Maze
0.8
0.5
0.8
0/8
0.6
origtnal
02
crajital ceisoo
ungand
cousal
0.0
0.0
0.0
5.0
200, 25ck
JK
15A8 2004
1056 1933, 2009 5006
3tk
1006 2008 25CK
(a) ALP-GMM
(b) PLR
(c) Goal-GAN
(d) CurroL
Target task performance of the agents at different training stages in Colored Sokoban (Row 1) and Button Maze (Row using different curriculum generators (Columns). The horizontal green line shows the performance of the agent trained directly in the target. "original" refers unaugmented curriculum generator and "causal" refers to its causally augmented version.
Figure 5:
refers to the
ALP-GMM (Portelas et al. 2019). PLR (Jiang al. 202T), Goal-GAN (Florensa et al.) 2018), and Currot (Klink al. 2022) two confounded environments with pixel observations: (a) Colored Sokoban, (b) Button Maze, (c) Continuous Button Maze (App. Fb. All experiments are conducted with five random seeds and reported in Interquartile Mean (IQM) normalized w.r.t the minimum and maximum rewards with 95% confidence intervals shown in shades. See App. F for more details.
Colored Sokoban. Consider the same Sokoban game as shown in Example I The curriculum generators are allowed to vary the initial location of the agent, to vary the initial box location, and to intervene the box's 's color. Without editing, the box color synes with the true underlying rewards, i.e., pushing yellow box always yields a positive reward. However, after intervening the box color, this sync is broken and the agent has no information on the right time to push the box. As shown in Fig. [] agents trained by original curriculum generators failed to converge due to this. After causal augmentation. those misaligned source tasks with intervened box color are all eliminated from the search space during curriculum generation. The causal versions of those generators successfully train the agent to converge efficiently and surpass those trained directly in the target task.
Button Maze. In this grid world environment (Chevalier- Boisvert et al. 2018), the agent must navigate to the goal location and step onto it the right time. Specifically, after pushing the button, the goal region will turn green and yield positive reward if the agent steps onto it. However, before pushing the button, there is only 20% chance the agent gets positive reward for reaching the goal, and the goal randomly blinks between red and green, independent of the underlying rewards. Curriculum generators can intervene the goal color and vary the agent's initial location but intervening goal colors creates misaligned curricula (Thm. (3). As shown in Fig. 5 agents trained by vanilla curriculum generators failed learn
Figure 6: But- ton Maze.
at all, while the agents trained by their even surpassing the one trained directly in the target task.
versions all converged to the optimal,
CONCLUSION
We develop formal treatment for automatic curriculum design in confounded environments through causal lenses. We propose sufficient graphical criterion that edits must conform with to generate causally aligned source tasks in which the agent is guaranteed to learn optimal decision rules for the target task. We also develop practical implementation of our graphical criteria, i.e., FIND- MAXEDIT, that augments the existing curriculum generators into ones that generate aligned source tasks regardless of the existence of unobserved confounders. Finally, we analyze causally aligned curricula' design principles with theoretical performance guarantees. The effectiveness of our ap- proach is empirically verified in two high-dimensional pixel-based tasks.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENTS
This research was supported in part by the NSF, ONR, AFOSR, DoE, Amazon, JP Morgan, and The Alfred P. Sloan Foundation.
REPRODUCIBILITY STATEMENT
For all the theorems, corollaries, and algorithms, we provide proofs and correctness analysis in App. E] To implement the algorithms, we provide pseudo-code in the main text and App. C We also provide experiment specifications, environment setup, and neural network hyperparameters in App. F Colored Sokoban and Button Maze are implemented based on Sokoban (Schrader) 2018 and GridWorld Sntevalier-Boisvee et 2018), respectively.
REFERENCES
David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. Advances in Neural Information Process- ing Systems, 34:7799-781 7812, 2021.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Belle- mare. Deep reinforcement learning the edge of the statistical precipice. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Process- ing Systems, 2021. URL
U41KBe
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 166-175. PMLR, 2017. URL seaerediingsmm...... mmr. www.preesssvv. hat/////nnaaaaaaaa 17a.mmmll
Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Is- abelle Guyon, Ulrike von Luxburg, Samy Bengio. Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5048-5058, 2017. URL http:::tpp::toooooo paper 201 hwwash/aahaae httphhh/33faabbbbbce ofw.aaacoeeaaaaaaaaaaaaoaafoSfS.
neur meurips..... cc/ Abst html
Minoru Asada, Shoichi Noda, Sukoya Tawaratsumida, and Koh Hosoda. Purposive behavior ac- quisition for real robot by vision-based reinforcement learning. Machine Learning, 23(2-3); 279-303, 1996. doi: 10.1023/A:101823700833,, URL https:: wrw. 100 1023/A: 1018237005823
Adrien Baranes and Pierre-Yvel Oudeyer. Active learning of inverse models with intrinsically tivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):99-73, 2013. doi:
10.10166f.oboo.2012....0o URL htttps://oii.on http:////ooi.o0010.1111111///1// robott
robot 2012.00.... 05.
008
Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 113(27):7345-7332) 2016. doi: 10.10773/mas..51000711 1510507113.
URL https: / www. pnas http:///ea///bb//111000/aaaaaa http://oa...1.111111111.11..
Elias Bareinboim, Juan D. Correa, Duligur Ibeling, and Thomas Ieard. On Pearl's Hierarchy and
the Foundations of Causal Inference, pp. 507-556. Association for Computing Machinery, New
York, NY, USA, 1 edition, 2022. ISBN 9781450395861. URL Ihtpps:/ /oi. org/10.
1145/35011 710171110 114 3501743)
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying coset-based exploration and intrinsic motivation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.),
10

---

Published as a conference paper at ICLR 2024
Advances in Neural Information Processing Systems 29: Annual Conference Neu- ral Information Processing Systems 2016. December 5-10, 2016. Barcelona, Spain, pp.
1471-1479, 2016. URL htppp:///proeeeeeoossossooo htt.a//awwaaaaaaaaaa/e/////////a 92
Richard Bellman. Dynamic programming. Science. (099773))334-7,, 1966. doi: 10.
1126/science. wat.pmec...13333111. 153.3731.34.
URL
1126/
sC ccceece... 1 .34
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Or- leans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
net./
Forum? d== JJJnR5Ym
Maxime Chevalier- Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium, 2018. URLhttpss: /ttp:////ttttbbt/ com/ Faa Famama Foundat innidaatio//MM n nigrid
Daesol Cho, Seungjae Lee, and H. Jin Kim. Outcome-directed reinforcement learning by uncertainty \& temporal distance-aware currieulum goal generation. In The Eleventh International Confer- ence on Learning Representations, 2023. URL
net/ tene/fforumi forum?id
V69itrHLEs
Juan Correa and Elias Bareinboim. General transportabilitty of soft interventions: Complete- ness results. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10912. Curran Associates. Inc., 2020. URL https://proceeenggg..so neurips cc/paper files/
Steve Dahlskog and Julian Togelius. A multi-level level generator. In 2014 IEEE Conference on Computationnll Intelligence and Games. CIG 2014. Dortmund, Germany, August 26-29, 2014. pp. 1-8. IEFE, 2014. doi: 1010///I..2114.622220
URI. httpss://o: https////ooo.oo1111000000
CTG. 2014 4. 6932909
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised envi- ronment design. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Bal- can, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, Virtual, 2020. |RL http:tppppspppppppeeessssssssss neurs cc/ppaperr/222//aaas 0B663386660033351111112996533--3
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arxiv, 2019. URL httt:://axxiv.oxtt abs//
1901.109951
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need:
Learning skills without reward function. In 7th International Conference on Learning Repre- sentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreviie net www.rooroouoooooooooo fRqFm
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In 1st Anmual Conference on Robot Learn- ing, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings, vol- ume 78 of Proceedings of Machine Learning Research, pp. 482-495. PMLR, 2017. URL hett::/pproeediing
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmssmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1514 1523. PMLR, 2018. URL http:// [ttp::/prrceedings mlr. wwwps/eeessep....... /vcsss/vv/// /teess//80/lozensaaa /wt.///oooeeeaa...mmm html

---

Published as a conference paper at ICLR 2024
Xing Hu, Rui Zhang, Ke Tang, Jiaming Guo, Qi Yi, Ruizhi Chen, Zidong Du, Ling Li, Qi Guo, Yunji Chen, et al. Causality-drivee hierarchical structure discovery for reinforcement learning. 2022.
Advances in Neural Information Processing Systems,
Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum rein- forcement learning using optimal transport via gradual domain adaptation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Pro- cessing Systems, 2022a. URL
Shengyi Huang. Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022b. URL
track wit.tbithbbu..... 1000022/03302222 htw.
25/ppo- www.e//peeeeettttooooooeooo implemmemattio-ddetllll..
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep rein- forcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022c. URL
Minqi Jiang. Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning ICML 1021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 4950. PMLR, 2021. URWhttp:: htttt:::ppppoceeiiiieee mlr. pp395//1397 jiang21b. html
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018. December 3-8, 2018, Montréal, //ttpp::/dooii orgg 0 48554455//aa /0.50//rxiv.. 1 10729
Canada, 2018. https::::::
Leslie Pack Kaelbling, Michael Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.
Faisal Khan, Xiaojin (Jerry) Zhu, and Bilge Mutlu. How do humans teach: On curricu- lum learning and teaching dimension. In John Shawe-Taylor, Richard S. Zemel, Peter L.. Bartlett, Fernando C. N. Pereira, and Kilian Q. Weinberger (eds.). Advances in Neural In- formation Processing Systems 24: 25th Annual Conference on Neural Information Process- ing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pp. 1449 1457, 2011. URL https: http::::ppppooceedccco neur ips.. cc/papprr/211/haah E9 E9028faese (0025aaar77eeeeee55500003333 -ww.4444222223-9---aa3co.. html
Pascal Klink, Haoyi Yang, Carlo D) ErEmo,, Jan Peters, and Joni Pajarinen. Curriculum reinforce- ment learning via constrained optimal transport. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 11341-11358, PMLR, 2022. URL https: //proceddings mlr wtw.::aww.......seeeeeeeeee wtwps///w///kkiileeeeeee
html
Daphne Koller and Nir Friedman. Probabilistic Graphical Models Principles and Techniques. MIT wwt. wwt.mem/caaaall...g..o/oo//
Press, 2009. ISBN 978-0-262-01319-2. URL aatp::/mmi htt::://miiitrt http:/twrrrrs..mteeeeeeeeoeeeeee
item/ item/defaule httpd/dfflllaaacccccm htt.aaut....atttytyyeeeeee...--
Daphne Koller and Brian Milch. Multi-agent influence diagrams for representing and solving games.
Games and Economic Behavior, 45(1):181-221, 2003. doi: 10.
httptt:::://// h0tp::/://06.l..rg000000000////0 8216 (02) 00544-4)
Steffen Lauritzen and Dennis Nilsson. Representing and solving decision problems with limited information. Management Science, 4709):125--12511 2001.
12

---

Published as a conference paper at ICLR 2024
Adam Li, Amin Jaber, and Elias Bareinboim. Causal discovery from observational and interven- tional data across multiple environments. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Diego Perez Liebana, Spyridon Samothrakis, Julian Togelius, Tom Schaul, and Simon M. Lucas. General video game AI: competition, challenges and opportunities. In Dale Schuurmans and Michael Wellman (eds.), Proceedings of the Thirtieth AAAI Conference on Artificial Intelli- gence, February 12-17, 2016, Phoemix, Arizona, USA, pp. 4335 4337. AAAI Press, 2016. URI. AAAT wttp://appppee//e///e/pppp 11853
http:/ / www. aaai org/ /c////c////////// wwaessiindeee nnde.. php/AAAT wtt../ppp/....AT....
Patrick MacAlpine and Peter Stone. Overlapping layered learning. Artificial Intelligence, 254: 21 21-43, 2018. doi: 10.101//j.rrtint2217...
URL https: http:///ooooro//../// 0001///00.111/1111 6/.6/
artint 201 2017.09.00 00.
Sanmit Narvekar, Jivko Sinapov, Matteo Leonetti, and Peter Stone. Source task creation for cur- riculum learning. In Catholijn M. Jonker, Stacy Marsella, John Thangarajah, and Karl Tuyls (eds.), Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, Singapore, May 9-13, 2016, pp. pp.566-555 566-574, ACM, 2016. BRLhttp::// acm. org/
citation. cfm?id 2937007
Sanmit Narvekar, Jivko Sinapov, and Peter Stone. Autonomous task sequencing for customized curriculum design in reinforcement learning. In Carles Sierra (ed.). Proceedings of the Twenty- Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melhourne, Australia, August 19-25, 2017 pp.
ijcai.org, 2017. doi: 0..24963/jcaai.2117/33 2017/333. httpss:::
//doi. org/ 229.0.24933322
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21:181:1-181:50, 2020. URL http:// http:://jmlr.orc/pppprrs. org/papers/
v21/20-212. html
OpenAI, Iige Akkaya, Marein Andrychowiez, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang, Solving rubik's cube with robot hand. arxiv, 2019. URL http://arxiv. bxq//bs//911.00
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob N. Foerster, Ed- ward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Re- search, pp. 17473-17498 PMLR, 2022. URL https: v162/par www.eh22a................aa.aaaa html
/ http://pooceeeeineeee/e www.nnm.mmm..........mm..... mmlr, prreserrsssnsssm..rrssss///
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2 edition, 2009. doi: 10.1017/CB0978051 111999878811111111
1803161.
Judea Pearl and Elias Bareinboim. Transportability of Causal and Statistical Relations: A Formal Approach. Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, vol- ume 25, pp. 247-254, 2011. doi: 10. 10.11609/aai..2251..
URL https: pettps//oojas //ojs///sss../... www..aaaa..oo..oo...o......
index. php
Bei Peng, James MacGlashan, Robert Loftin, Michael L Littman, David L Roberts, and Matthew E Taylor. Curriculum design for machine learners in sequential decision tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 2(4):268-21) 2018. doi: 10. 1099/TCTT
2018.2829980. URL
1ww11111991T1... 2018 002.1182822500 2829980
Ronan Perry, Julius Von Kügelgen, and Bernhard Schölkopf. Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Sys- fems, 2022. URL https::/ http::///pperevvieeee net. forum? www.mmidddkccccco d=
13

---

Published as a conference paper at ICLR 2024
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Oudeyer. Teacher algorithms for
curriculum learning of Deep RL in continuously parameterized environments. Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), 3rd Annual Conference Robot Learn- ing, CoRI. 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pp. 835-853. PMLR, 2019. URL http:
p/pocceddiggs.. M.l.... prwss/
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Yves Oudeyer. Automatic curriculum learning for deep RL: A short survey. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninte International Joint Conference on Artificial Intelligence, I/CAI 2020, pp. 4819 4825. ijcai.org, 2020. doi: 00.24963/iccaii2220/7777 URL https:://oi.ooo http:///oii..r11...49.00
ijcai. 2020/671
Sébastien Racanière, Andrew K. Lampinen, Adam Santoro, David P. Reichert, Vlad Firoiu, and Timothy P. Lillicrap. Automated curricula through setter-solver interactions. arxiv, 2019. URL
http://arx: www.or//aaaa. http:/bb/11900 12892
Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence, 2(8):428-44 -4288443,,65 2020.
Tim Salimans and Richard Chen. Learning montezzma's revenge from single demonstration. arxiv, 2018.
C3381
Terence D Sanger. Neural network learning control of robot manipulators using gradually increasing task difficulty. IEEE transactions on Robotics and Automation, 1033):323-333, 1994. doi: 10.
1109/70.294209 UtL.htttss://dol. http:///ddi..0011..111111 1109/70 294207
Tom Schaul. A video game description language for model-based or interactive learning. In 2013 IEEE Conference on Computational Inteligence in Games (CIG), Niagara Falls, ON, Canada, August 11-13, 2013, pp. 1-8. IEEE, 2013. doi: 10.1109/CIG.2013..63336
URL https:
//doi. 11t9//16...... 2013. 6633610
Max-Philipp B. Schrader. Gym-sokoban, 2018. URL.tttp..//ghhhbb..mmmmphhhaaae 1ymm-sokoban
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arxiv, 2017.
Oliver G. Selfridge, Richard S. Sutton, and Andrew G. Barto. Training and tracking in robotics. In Proceedings of the 9th International Joint Conference on Artificial Intelligence Volume 1, IJCAI'85, 85, pp. 670-672. Morgan Kaufmann Publishers Inc., 1985. ISBN 0-934613-02-8.
Felipe Leno Da Silva and Anna Helena Reali Costa. Object-oriented curriculum generation for reinforcement learning. In Proceedings of the 17th International Conference on Aulonomous Agents and MultiAgent Systems, AAMAS '18, pp. 1026-1034, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Satinder Singh, Andrew G. Barto, and Nuttapong Chentanez. Intrinsically motivated reinforcement learning. In Advances Neural Information Processing Systems 17 [Newral Information Pro- cessing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada), volume 17, pp. 1281-1288, 2004. URL 2200///ahh/beetlccccaaaaa2222333
wtw.:oceeeeiinns..ee....ppppppps Abstract...ttttttt.m..
cc/paper/
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In 6th Inter- national Conference on Learning Representutions, ICLR 2018, Vancouver, BC, Canada, April May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018. URL https:
7 /openreveew. http://www.....ttoooooadd----aee
Richard Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. Bradford Book, second edition, 2018. ISBN 0262039249.
14

---

Published as a conference paper at ICLR 2024
Maxwell Svetlik, Matteo Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, and Peter Stone. Auto- matic curriculum graph generation for reinforcement learning agents. In Proceedings the the AAAI doi: 10.1609/aaii.v.. v3lil. 10933. URL httthhtts:://ot. aaa.. index http:///w.....aa..////a/aa/aaaaa
Conference Artificial Ineellgencee volume 31, 2023/06/09 2017.
Benito van der Zander, Maciej Liundefinedkiewicz. and Johannes Textor. Constructing separators and adjustment sets in ancestral graphs. In Proceedings of the UAI 2014 Conference on Causal Inference: Learning and Prediction Volume 1274, CT 14, 11-24. CEUR-WS.oo 2014.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. arxiv, 2019.
abs 00t//9991.111 01753,
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth O. Stanley- Enhanced POET: open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In Proceedings the 37th International Conference on Ma- chine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1951. PMLR, 2020. URL http://proceiingg. m press/vl / 9/wang2011 html
Junzhe Zhang and Elias Bareinboim. Designing optimal dynamic treatment regimes: A causal reinforcement learning approach. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11012-11022. PMLR, 13-18 Jul 2020. URL ptttps://proceeenngs.... press/v119/ 19/ 2hwai//gg///aggggaaaaa html
15

---

Published as a conference paper at ICLR 2024
Task Contest Space
Task Contect Spitoo
Confounded Target Tasks
Curriculum Generator
Misaligned Source Tasks
Causal Diagram
Algned Source
Tasks Contexts
Causal Diagrams
Augmentation (Our Work)
Curricllum Generator
Allgned Source Tasks
An overview of our proposed approach. In the upper half, the non-causal curriculum generator will produce misaligned source tasks failing to train the agent convergence. While in our approach (the bottom half), we utilize the qualitative causal knowledge (causal diagrams) to filter out those misaligned source task contexts such that applying same curriculum generator can now produce aligned source tasks.
Figure 7:
Table 1: Time complexity of FINDMAXEDIT and its variations.
Algorithm
Description
Runtime
ISEDIT
For given set of -(), X0) and graph Gr decide editability
Odd(n +))
FINDEDIT
Find an admissible set of r(i) w.r.t X (i) in G
(dn
FINDMAXEDIT
Find the maximal admissible set of -() w.r. X (f) in 9
(dn")
LISTEDIT
List all admissible sets of (1) w.r.t X() in 9
(n) delay
A METHOD OVERVIEW
In this section, we will summarize our proposed approach from a high-level view and provide figure for better illustration of the pipeline.
B DETAILED COMPUTATIONS OF EXAMPLE I
For the reward of pushing the box, we have,
E Y B: "next to goal", C2 do (X; "push")]
(9)
E [Y, B, "next to goal", Ug do (X, "push")]
(10)
10 1(U, 0) 10 I(U 1)
(11)
[10 if = 0
(12)
10 if C; =
After fixing the box color, we have,
E Yı B, = "next to goal"; do (Ci, X; = "push")]
(13)
= 10 P(U, = 0) - 10 x P(U: =1) 1)
(14)
--
(15)
16

---

Published as a conference paper at ICLR 2024
Algorithm 4: ISEDIT
Input: A set of edit indicators r(), set of actions X and graph 9.- Output: Whether the input edits are admissible. for X in X do
(T IL Y De(X) X, Sx) in 9v then
return False;
return True;
Algorithm 5: LISTEDIT
Input: A causal diagram Gr, set of actions X
Output: All possible editable sets.
A FINDMAXEDITT X());;
mask 21 1;
while mask do
A 0:
tmpmask mask;
i i- |
while tmpmask do if tmpmask &1 then
A Au [A
i
tmpmask
m mask -1 1; yield A;
C VARIATIONS OF ALGORITHMS
The first task of deciding editability is direct application of the criterion in Def. 4 and can be implemented by using TESTSEP (Algo. 4p. Finding an editable set and finding the maximal editable set are both solved by proposed Algo [ After find the maximal editable set. by Thm. 2 the power set of it is indeed all the possible editable sets w.r.t X in Gn (Algo. 5). Note that there are exponentially many subsets so we focus on the time complexity of yielding each output. We implement this by bit manipulation. Specifically, we denote the existence of variable in the output set as 0 or 1. Then listing all the elements in power set of A is equivalent to decrease a bit mask from 2
O. Each mask in this process represents unique subset of A))
D ALGORITHM FOR CAUSALLY ALIGNED CURRICULUM LEARNING
Algo. construct the curriculum before any training is done. However, more common practice to construct the curriculum while adapting to the edge of the agent's capability. this section, we will discuss more involved version of Algo. 3 that implements this idea. As have seen in the main text, two source tasks T6, T() in soluble curriculum satisfy X C X if < <.j- This incremental nature of the action sets actually brings us opportunities to reduce further the computation of finding editable states. Thm. 4 reveals nice property of (6) such that if set of state variables. ( editable w.r.t an action X, it is also editable w.r.t any actions X' precedent to X in the soluble ordering, X x X,e. (X' n De(Y)) IL X', Sx.) in Go where π is new regime node pointing to X.
Theorem 4 (Expanded Action Sets). Let T (M, be soluble target task and T() (M, II, R, A0) aligned source task of T. Then AG) is also editable w.r X X X E
17

---

Published as a conference paper at ICLR 2024
Algorithm 6: CAUSAL CURRICULUM LEARNING
Input:
target task T. curriculum generator GEN.
Output: A policy trained in the curriculum C, π (C).
Generate causal diagram 9- of 'T w.r.t policy space TI;
Sort the set of actions w.r.t the soluble ordering (ascending) and save the result into X';
Randomly initialize the agent's ' policy π² (C);;
j 1; for X; in X' do
A A-FINMMAXEDITCC. {X,)):
c[X.]
+ 0:
while *T) & Urwegix ((S;:=(CC) do
Generate source task TŪ) with GEN(T, A);
Train x x'(C)) (C) in TÚ) to converge;
c[x;] < C(X]U((T)):
j j+1;
return xt (C(;;;
Thm. Dimplies that when checking editability w.r.t a of actions, we only need check w.r. the single action that ranked highest in the soluble ordering (e.g., for actions X3 x X2 x X1, finding the editable states for {X1} is equivalent to finding editable states for {x,)==)
Based on the above discussion and Algo. [ we propose causal curriculum learning algorithm (Algo. (6) to find the optimal target task policy given curriculum generator GEN that generates source tasks by editing given set of editable states AG) This algorithm resonates with the heuris- ties that the agent should be trained in a sequence of more challenging source tasks (Portelas et al.) 2020 Narvekar et al., 2020). More specifically, after sorting the actions in the soluble ordering. the first generated source task only requires the agent to learn the optimal target policy of action XN (Say the target task T is an N-step one). The next stage of learning will be grasp the optimal target policy of both action XN and X but since the agent has already learned the optimal policy of XN previously. This won't be too much harder for it. We will continue this procedure until the agent learns the optimal policy for all the actions in the target task. Algo. 6 basically a combination of CURRICULUM LEARNING(Algo.) 2 and FINDCAUSALCURRICU LUM(Algo.) 3) except that for each A, there will be multiple source tasks T(i) being generated until ((S Urinectx,) SSSS; n (C)) where ((S,; m²) is the space of possible S, values in the target task under the optimal target task policy N², ( (C)) is the space of possible S values in source tasks under policy x² (C) and C[X,) is the set of source tasks T() such that X; X This condition ensures that the input space of an action X; is thoroughly traversed during training in the curriculum. If this is satisfied, means that the agent has learned the optimal decision rule for X, in every possible situation in the target task, as we show formally in the following theorem.
Theorem 5. If the set of actions in target task satisfies X C (N) and VX, E X, M(S T-) S UTonectx S2S,; x (C)), the following solution (C) is an optimal policy x in the targer task
π (C) = arg max MON; [R(Y);
(16)
HEIl
where MNN) is the SCM of source task (M), and (M) is aligned to T w.r. X
Nonetheless, there are other practical roadblocks before realizing optimal curriculum learning. For example, we don't have exact measurement of when the input space of an action is traversed thoroughly by the curriculum, and the agent may not converge in every source task TU) to learn all the optimal decision rules of x(), But still, as we have shown in the experiments, simply augmenting the existing curriculum generators already gives satisfying performance.
E PROOF FOR THEOREMS AND ALGORITHM CORRECTNESS
In this section, we provide proof sketches for all the theorems proposed in the previous sections.
18

---

Similar definitions hold for the optimal policy m of source task TG), Our goal is to show that once the graphical criterion holds for ((ر and action X,, the optimal decision rule of X; on those shared input states of the source task and the target task is the same in these two tasks. We first Published as a conference paper at ICLR 2024
Lemma 1. If the set of edit indicators satisfy (r(1) IL Yx Ex|X,Sx) Sx) in soluble source task, for any X' X the soluble ordering, it satisfies (rli) I1 Yx "x' Sx.).
Proof Lem. 7 We will prove this by contradiction. Suppose (r0) M.. (xx)). There are three possibilities based on the type of node V to which ا(0) corresponds.
i) V E Sx, we can show that the source task cannot be soluble by finding an open path from pseudo parent of X to Yx'. Firstly, X' x X in soluble source task means X' E De(X). When adding pseudo parent V' to X, the path V - X f- V + ا is open under X', X',Sx By the assumption, there also exists an open path from T(5) to Yx' under X', S'', which means that path V + +x + V +- -(6) is also open under X', Sx Thus, this contradicts the definition of soluble task and X' X.
ii) If V & Sx and V An(X), we can find a similar open path as in the previous case. By the assumption, there exists an open path p from r Yxx under X', Sxx. But we have X' E De( and thus Yx S Yx. This open path must be blocked under X. Sx, which means that X must observe a non-collider variable Z on p. Without observing Z, p will be open under X, Sx. Note that any colliders p will have X' as their descendant since p is open under X', X', SS', which creates new open paths. So, p won't 't't be blocked because of colliders. Now, a pscudo parent of X, say V will again have open path towards Yx under X' Sx' V" + X t Z... - Yx', where Z... + Yx is part of the p path. Thus, this contradicts the definition of a soluble task and X" < X.
iii) If é Ax(X), we prove contradictions by checking all the possible path types between V and Yx.. If the open path from V to Yx' under X', S is causal, p must be blocked under X, Sx by having X observe variables on p. which contradicts with the condition that V & An(X). If is not causal and colliders exist, say Z, X' must be descendant of such colliders. Let Z be the leftmost collider p. This leads to another causal path p', x() + V + + Z - X' - - Yx.. Clearly, X cannot block y' by observing any variables on Thus, p' is open under X, Sx which contradicts with the condition that (7 11 Yx | Sx).
Thus, we have proved that if (7()) IL Yx|x,, Sx) in a soluble source task, for any X' < X, also satisfies (r() 11 Yx Xx,
Proof of Thm.
By definition, the optimal policy of the target task satisfies,
π arg max E. (R(Y)::].
(17)
#ENl
simplify the calculation of the optimal decision rule at action X; using the concept of "relevance graph" (Koller& & Milch 2003).
Definition 8 (Relevance Graph of Tasks). The relevance graph, Gr, of a target task T= (M,
directed graph whose nodes representing action variables X C V are connected by directed edges X' + X if and only if (I' 1L LYDDeXX)SSx where x' is an added regime node pointing X'.
This relevance graph specifies the order in which those actions should be optimized. Intuitively, X' is optimized before X because it can affect X's reward signal while X's inputs, Sx. and X itself cannot block such causal effects. We denote the topological order over actions in G, by < where X' < X if and only if X' should be optimized before X. It is also possible that the relevance graph contains Strongly Connected Components (SCCs) where each pair of actions in the same SCC is connected by directed path. Semantically, actions in the same SCC affect the rewards of each other. Thus, one must consider all actions in the same SCC to find the optimal policy for these actions (Koller & Milch, 2003). Formally speaking, let L be equivalence relationship over X such that action X, X' E X belong to the same partition, X ~e X', (equivalently, X e [X'Te or X' (X]c) if and only if they are in the same maximal SCC of target task T relevance
19

---

Published as a conference paper at ICLR 2024
graph, G- W
When there is SCC in the relevance graph, say this target task is soluble Def. 7
which can be solved by a series of dynamic programs (Koller & Milch 2003 Lauritzen & Nilsson 2001). When there is an SCC, we can treat each SCC as a single high-level action taking value as the combination of all the actions in that SCC, i.e., 2((x,) = xx (2()), After eliminating all such SCCs in the relevance graph, we transform the task back to soluble one. So, the same
solver can be used. Now we show that given an optimal target task policy, the decision rule at [X,]
optimal if and only if for every input 8[x.] E (SSxx.]
(ssxx)))
arg max EM (R(Yx.)) ((X))T[( U U 1 (ixa(-lexxa)
(18)
where Yix, = De((x:)) n Y is the set of rewards that are descendants of actions in the SCC of X; and x²xx, is the set of optimal decision rules of actions that precede [X;] w.r. the relevance graph. By definition, the decision rule at [X,] is optimal if it maximizes the expected reward function when other parts of the optimal policy are given,
arg max EM : ( ( \
(19)
X.J
We can expand the expected reward as follows,
EM [ Tx.))
(T"
mxaa)]
Σ Spxc
P(s (R(Y) |(xx]i U (T* \ (ixaa]
(20)
Clearly, the input distribution of S|x, is fixed given other parts of the optimal policy so Eq. equivalent to for every input s[x,] E [((((x,]==*)
19 is
(1xx(( = arg max EM R(Y) PX U (x* m xpx,0(-1exx1)
(21)
arg max *xxa(-lexxa) [x(x
R(y))(y)ssx.) [ (ix.) ((xx)))((= (1x1])((x))
(22)
Since non-descendants of [X, are independent of [X.] given Six,1 and actions are not confounded, we can further reduce the reward function to focus on only Y(..). rewards that are descendants of [X:), assuming that R(-) is cumulative,
(+x1((1))))))++ = arg max
Σ R(xxx)) [ms]i"" ((xi)))(( (23)
From the relevance graph definition, we know that only actions that precede [X,] in the relevance graph will affect M(x.) given Spx.1s [Xi]. Thus, we can simplify the conditioning further,
arg max
Σ ((xx)))
[mn] -xx ) ((ae]](s)))))))))))))
(24)
max EM ((((x,)) 8|x.] [xa] =xx]((xx11))
(25)
which is exactly Eq. (18). We can do a similar derivation for the optimal decision rule at [X.] in source task TU), for every input six,1 120 (SxxF==7)))
( [ix] (1spxa)
arg max
Σ
[2,), (1); (3) (T
(26)
arg max EMO x(xx))))x
R(( Spx,)TT(x)) U π x[x
(27)
where 70) the edit indicator ( is the optimal policy of source task TG) In practice we can let actions in each SCC X, have an edge into every reward node associated with the SCC and still
for simplicity, will use [X] to denote [X] in the following discussion.
20

---

Published as a conference paper at ICLR 2024
the graph is compatible with the original task. Then, only need to show that once the graphical
criterion is satisfied in such a graph, for any input 8; E (S; n ((S;; the following holds,
P(yx, ((x.) [ma]]; " (x.)) P(ux, [xx.) [m:]], =0; G) <(x.))-
(28)
If this is true, the optimal decision rule at X; will be invariant across both the target task and the source task. We first consider the simpler case when there is no SCC in the relevance graph. So, [X:] {X;}. In this case, we can apply the result of Lem. I directly and know that the graphical criterion is also satisfied by any action X < X,, Then we prove Eq. (28) holds by induction on action X; The base case is there is no action preceding X; in the relevance graph. So, there will be no policy dependencies in Eq. (28). and will be trivially true given the graphical criterion. Now assume when there are k actions preceding X, in the relevance graph, and the graphical criterion holds, Eq. (28) will hold. When there are k + actions preceding X; in the relevance graph, by the inductive hypothesis and the fact that graphical criterion also holds for X < X; when it holds for X,, we know the optimal decision rules of X < X; stay the same across the source task and the target task. Thus, Eq. (28) is reduced to P(yx, 8X,, zi) = P(yx, sx,, Lz, r(0),, which is true when the graphical criterion holds, (7(2 TL Yx, X,, S.) in Gn.
When there are SCCs in the relevance graph, the graphical criterion only shows us that (r() IL Yx, S, X:) in 9r. But we can show that (7(5 IL Yx, | [X.], the difference between these two criteria is that the latter includes more variables in the conditioning set. If the latter one doesn't hold, that means adding these variables opens up at least a collider path p that is blocked under the original criterion. Say this collider on p is M, and it's ancestral to an action X; E [X:]. Clearly, there is an active path from -() to M and there is also active causal path from M to Yx, under S,, X,. But we also know that by the condition that X, E [X,].
in Gr also holds. Notice that
Yx, nY, This means that there is also an active path from -()) to Yx, under S, X; which clearly contradicts with the fact that (v(2) 1L Yx, X,, S,) in Gn. Thus, (7() 11 Yx, (X in 9- holds. Now, we can view the whole [X,] one high-level action and follow a similar induction
procedure as in the simpler case, which completes the proof.
Proof of Thm.
We first show that the maximal editable set w.r.t. a set of actions is unique. Let K- = max ( V If there are two maximal admissible sets V1, V/2 w.r. X(),, they satisfy [V] = [/ = K but V1 * Vz- Interchangeably, we can assume a state variable V E V, but V & V/2 exists. Since V satisfies our criterion, so does every variable in V2. Then by Def. 4 we know the set V V2 U [V} is admissible w.r.t Xú), which contradicts with the fact that V22 is the maximal set since V = K + 1. Thus, this is impossible to happen.
2
By the uniqueness, we only need search for one maximal editable set w.r.t given set of actions X6), For each variable, by Def. 4 it has to be admissible w.r.t ( before it can be added to the admissible set AG), Thus, we loop through all state variables and check their editability. If a single state variable is not admissible w.r.t an action, X E X(), we don't need to check its editability w.r.t other actions further. So, we can break the loop there. The editability check is done on the augmented graph G- where a pseudo edit indicator is added, pointing to V. the state variable being checked. The correctness of this step is guaranteed by the correctness of TESTSEP (van der Zander et al., 2014).
Proof of Thm. Since every source task we use is causally aligned, the optimal decision rules of
actions in X G) will be invariant across the target task and the source task T()) The same is true for the set of actions XÚ+1), By our construction, we have X6) C x(+1), and each action corresponds to exactly one element in m0)) Oir". mb+1) n=" respectively. Thus, the set of invariant optimal decision rules is also expanding.
Proof of Corol. By the correctness of FINDMAXEDIT, we know that every source task generated by GEN(T, A)) will be causally aligned w.r.t X)), Then by the way we construct ( it is
21

---

Published as a conference paper at ICLR 2024
guaranteed that x C x Thus, the returned curriculum of FINDCAUSALCURRICULI will be causally aligned.
Proof of Thm. This is direct result of applying Lem. | to soluble target tasks.
Proof of Thm. 5
The algorithm works by creating source tasks with an expanding set of ac- tions. The set of actions expands in the direction that follows the soluble ordering, X' = {XN; (XN:NN-L NN-1,-.,,1}> Then, by Thm. 4 the editable set can be calculated w.r.t only the newly added action in this round (X²). The while loop ensures we generate enough source tasks to cover all the possible state inputs to X,- Note that we don't require the final output =² to be optimal in all source tasks. It is still the optimal target task policy, Because given the expanding action sets when constructing source tasks, optimal decision rules of X learned T(- are still optimal in TD) and the final output policy πt" (C) will contain optimal decision rules for all actions X in the target task.
F EXPERIMENT AND IMPLEMENTATION DETAILS
This section introduces the details of our experiments, including environment specifications, agent hyper-parameters. training/testing protocols, and more experimental results. For the Colored Sokoban and the Button Maze, we implemented Proximal Policy Optimization (PPO) agent with independent actor and critic networks in Py (Schulman et al. 2017). Both networks have three convolutional layers and two linear layers with rectified linear unit (ReLU) activation functions. The number of output channels of the convolutional layers is [32, 64, 64 with each layer. For convolu- tional kernels in those three convolutional layers, we use 8 x 8 with a stride of 4, 4 x 4 with a stride of 2, and 3 x 3 with a stride of 1, respectively. We flatten the output of the convolutional layers and feed it into the two linear layers with the intermediate feature dimension set to 512. The input for the network is an image observation of size 84 x 84 for both environments. For hyper-parameters, we follow the default hyper-parameters from Huang et al.] (022b)) on which the implementation also heavily based. For the Continuous Button Maze environment, we use an Soft Actor Critic (SAC) agent with wwwdimmnsional state vector inputs following the implementations by Huang et al. and adopting the hyper parameters setting from Klink al.) For the four curriculum generators used, we adopted the implementation from Klink et (2022) 's official implementations (https:/gittuu..comm) cok/pssllkkk///crroo))) Note that even though three environments are confounded, we still use MDP-based policy learning algorithms. Because for those environments, in both target task and aligned source tasks, the confounder revealed by other variables without intervention. Thus, the agent has perfect information to decide which state it is in exactly. So, we can still use PPO and SAC to find the optimal policy.
F.1 ENVIRONMENT SPECIFICATIONS
For Colored Sokoban, the target task definition is already specified in Example
For Button Maze, at each time step, let C2 be the target location's color, B, be the button status of whether it has been pushed or not, Lg be the agent's 's current location, Y, be the reward for this step and X, be the agent's 's action in this step. B, -B,-1 when the agent pushes the button. The goal location's 's color C = U, before the button is pushed but C; = Uc after the button is pushed, where P(U, 1) 1/2, P PUC 1) 1/5. The reward function is specified as follows,
if B; = "next to goal" A X, = "move forward" A (Uc = =1)
-1
if B, "next to goal" A X, "move forward" A (Uc = =0)
Y
(29)
-0.1
if L I
0.01
otherwise
In the Continuous Button Maze environment Fig. similar to the grid-world button maze, the agent also must navigate the target region at the right time. The only difference is that this time the environment is an open area and all states and actions are in the continuous domain, which is exponentially large. The optimal strategy for the agent is still push the button first then step onto
22

---

Published as a conference paper at ICLR 2024
Figure 8: Continuous Button Maze. The agent (the grey dot) needs to navigate the room and step into the goal region (the region in the bottom left) at the right time. The goal region flashes between red and green. And after pushing the button, it will be always green.
the green goal region. Note that the goal region flashes between red and green color before pushing the button. Even if the agent step onto it when shows green, is still possible that the agent gets negative reward. The environment is defined the same as the Button Maze with the only difference that we remove the not-moving penalty from the reward function,
if B, "next to goal" A X, "move to the goal" A (Vc 1)
Y
if B; "next goal" X, "move the goal" (Uc 0) 00001 otherwise
(30)
The curriculum generators are allowed pick the initial location of the agents, whether to toggle the button at the beginning, and whether to interven the goal region color.
F.2 ADDITIONAL EXPERIMENT RESULTS
When reporting results, instead of using the cumulative rewards directly, as promoted in Agarwal et al. (2021), we report the Interquartile Mean (IQM) normalized by the maximum and minimum rewards in the corresponding environment. Specifically, for each experiment, we run five random seeds. Then, the normalized IQM is calculated as,
2 4
(x:
NORMALIZEDIQM
(31)
71
(u - 1)
7
where 24 are the original data points we collected and sorted, e.g., cumulative rewards, and Vr,, E [b, u], u R the bounds for the data points. In our experiments, the bounds for the rewards are easily obtainable as they are both artificially designed game environments. In the implementation, edit the environment by editing set of predefined parameters. Each vector of parameters corresponds to a unique instance of the environment.
In Fig. [ we see that agent trained by causal agnostic curriculum generators fail avoid misaligned source tasks and unable to converge to the optimal policy. Those agents trained by the curricu- lum generators perform even worse than those directly trained in the target task which verifies the necessity of avoiding misaligned source tasks empirically. After augmenting the same curriculum generators with our algorithm, agents trained by them all successfully converge to the optimal sur- passing those trained directly in the target task. The result demonstrates that our method is indeed widely applicable to both high-ding digh-iimensiona and continuous domains and that utilizing qualitative causal knowledge properly is crucial to the successful application of curriculum learning in the founded environments.
During the training process, also counted the portions of misaligned source tasks generated by those con-causal curriculum generators. From T we see clearly that in all three environments, those curriculum generators fail to avoid misaligned source tasks and most of the source tasks pro- posed by them are actually misaligned. We report those portions with 95% confidence intervals.
We also show examples of the curricula generated by those augmented generators in Figs. 10 12
23

---

Published as a conference paper at ICLR 2024
20.0
54
0.5
0.5
504
original
orgiea
seggtaa
caroa
counall
1K. 506, J066, 192), 2008, 2006 3331
JK 30K 100L I5A6, 2006, 2506 100K
JK
50k
100L 1906, 2006 2506 100%
18
50k
100L 1291, 2005 2005 2005
(a) ALP-GMM
(b) PLR
(c) Goal-GAN
(d) Currot
Figure 9: Target task performance of the agents at different training stages in the Continuous Button Maze using different curriculum generators (Columns). The horizontal green line shows the perfor- mance of the agent trained directly in the target. "original" refers to the unaugmented curriculum generator and "causal" refers to its causally augmented version.
Table 2: Misaligned Source Task Portion.
Env.Alg.
ALP-GMM
PLR
Goal-GAN
Currot
Colored Sokoban
68.10 + 0.71%
69.10 + 2.77%
66.41 + 1.84%
68.36 + 1.06%
Button Maze
88.41 + 0.43%
3.77% 3.77% 87.33 =
90.30 = 0.27%
88.62 = 1.00%
Con. Button Maze
85.77 = 0.83%
83.57 = 14.5%
80.55 = 19.6%
47.70 = 32%
G DETAILED RELATED WORK
Research in generating suitable curricula for reinforcement learning agents can date back to the 90s when people manually designed subtasks for robot controlling problems SSnner] (1994). In recent works, a general curriculum generation framework requires two components, an encoded task space and a task characterization function (Narvekar et al. 2020 Wang et al. 2020). Each component may be either hand-coded as inputs (Khan et al. | 2011
Peng et al. 201 8; MacAlpine & Stone 2018 Portelas et al. 2019) or automatically learned from data along with training the agents (Parker-
Holder et al.] 2022 Klink et al. [002]] Florensa et al.
2018 Jiang et al. 202T Florensa et al.) 2017
Risi & Togelius 2020 Cho al. 2023 Huang al. 2022a). straightforward task space encoding
is to split the observation with common patterns (e.g. pixel tiles) and re-combine them to create new tasks (Dahlskog & Togelius 2014), which can be intractable in the face of a rich observation
space and adds extra representational burdens to the curriculum generator. Recent work usually uses vectored parameters as task space encoding, each of which can be grounded into a unique task instance -Parker-Holder et al.) 22022] Klink et al.] [202]] Florensa et al., [2118] Diang et al., 2021 Portelas et al. 20199 Dennis et al.) 2020] Wang et al.) 2019 2020; Cho et al.) 2023 Huang et al,, 2022a). While parameter-based encoding is widely applicable in various decision-making tasks, using a dedicated domain description language such as Video Game Description Language (VGDL) for task space encoding provides finer granularity and readability in task generation (Schaul 2013 Liebana et al. 2016 Justesen et al., 2018). A suitable task space encoding then lays the foundation of a reasonable task characterization function, which is either task difficulty measure (Florensal
let al.
2018: Paaker-Holder et al.] 2022, Dennis et al., 2020 Andreas et al., [2017] [Sukhbaatar et al. 2018 Jiang et al. 2021] or task similarity function (Svetlik al.) 2017: Silva Costa 2018 Jiang et al. 2021 Eysenbach et al. 2019) in general. For example, task similarity can be measured
domain knowledge based heuristics SSetllk et al. 2017 Andrychowicz et al. 2017 Silva &
Costa 2018 and agent's performance can be used as a direct indicator of the task difficulty (Florensa
al. (2022). Curriculum generator relies heavily on these task characteristic functions to measure the quality of the task, schedule the training
al.
2017 2018 Narvekar al. 2017| Parker-Holdee
process, and evaluate the agent's performance (Narvekar et al.) 2020 Portelas et al. 2020).
Given task space encoding and task characteristic functions, the remaining central problem of cur- riculum learning is how exactly one could generate new tasks efficiently. The most intuitive idea of training agents on increasingly harder tasks has been verified in various works, which can be imple- mented as setting different goals (Florensa et al. 2018 Racanière et al. 2019 Baranes & Oudeyer, 2013) or changing starting state distributions (Florensa et al.) 2017 Salimans & Chen 2018 Asada et al. 1996; Narvekar et al.] 2016). Another major branch task generation is to change task's state space task parameters. This approach usually works together with parametrized task space
24

---

Published as a conference paper at ICLR 2024
(a) ALP-GMM@II IK
(b) PLR @ IK
(c) Goal-GANe 1K
(d) Currot IK
(c) ALP-GMM@ 100K
r PLR @ 100K
Goal-GAN @ 100K
(h) Currot 100K
ALP-GMM CCMM@2002 200K
PLR 200K
(k) Goal-GAN 200K
Currot 200K
(m) ALP-GMM @00K
PLR@ 300K
(0) Goal-GAN 300K
(p) Currot@300K
Figure 10: Curricula generated by the (causal) augmented curriculum generators for Colored
Sokoban. Each column shows curriculum from one generator at different training steps.
where all state factors and transition function parameters are fully encoded as a latent vector Parker- Holder al. 2022
Klink al. 2022 Florensa al., 2018 Tiang et al. 202T. Portelas al. 2019
Dennis et al. 2020: Wang et al. 2019 2020: Cho et al.) 2023 Huang et al. 2022a). When more flexibility is desired, source tasks can be generated by changing the transition reward func- tion forms. In curiosity-driven agents, exploration is encouraged by an intrinsic reward added upon the original reward from the task (Bellemare et al. 2016 Ecoffet al. 2019), and this intrinsic re- ward signal can be learnable which evolves agents progress the task (Burda et al. 2019 [Singh ctt al. 2004). Another telling illustration function forms change is the Sim2Real problem. When generating tasks for robot in simulation, other than simulated perfect dynamics model, one needs to take into account extra fractions, inaccurate sensors, motor latency, and poorly executed actions for successful deployment into the real-world target task (OpenAl al. 2019).
The crux of curriculum reinforcement learning is transfer useful knowledge from subtasks to target tasks (Narvekar al. 2020), which can be modeled transportability problem in the causal
25

---

Published as a conference paper at ICLR 2024
(a) ALP-GMM IK
(b) PLR@ 1K
(c) Goal-GANe 1K
(d) Currot IK
(e) ALP-GMM@ 100K
(f) PLR @ 100K
Goal-GAN 100K
(h) Currot @ 100K
(i) ALP-GMM DIP-GMM@20NM 200K
200K
(k) Goal-GAN 200K
Currot @ 200K
(m) ALP-GMM @300K
(n) PLR@@ 300K
(o) Goal-GAN 300K
(p) Currot 300K
Figure 11: Curricula generated by the (causal) augmented curriculum generators for Button Maze.
Each column shows curriculum from one generator at different training steps.
literature (Correa & Bareinboim 2020 Pearl & Bareinboim 2011 Bareinboim & Pearl 2016)
The literature transportability studies broadly how to answer queries with data from disparate domains. By examining shared causal mechanisms across seemingly dissimilar domains, formal graphical conditions are established identify what queries can be answered and how those should be answered. In our curriculum reinforcement learning setting, good curriculum elicits policy that performs well not only subtasks but, more importantly, target tasks. Policy pieces from various tasks in the curricula constitute data from which quest is to construct an optimal target task policy. Since subtasks are generated from target tasks, they share certain aspects in principle. Analyzing how as transportabilityy problem.
transfer those policy pieces the target tasks can thus be viewed
Throughout the paper, an orthogonal line of research dedicated in learning the causal structure directly from the task from which causal diagram can be derived naturally (Hu et al. 2022 Li et al. 2023 Perry al. 2022
to the causal diagram of the task. However, there is also
26

---

Published as a conference paper at ICLR 2024
(a) ALP-GMM@ IK
(b) PLR@ 1K
(c) Goal-GA 1K
(d) Currot IK
(e) ALP-GMM@ 100K
(l) PLR 100K
(g) Goal-GAN 100K
(h) Currot 100K
(i) ALP-GMM@2000 PLR@200K
(k) Goal-GAN 200K
Currot 200K
(m) ALP-GMM @300K
PLR@ 300K
(o) Goal-GAN 300K
(p) Currot @00K
Figure 12:
Curricula generated by the (causal) augmented curriculum generators for the Continuous Button Maze. Each column shows a curriculum from one generator at different training steps.
H MARKOV DECISION PROCESSES (MDPS), PARTIALLY OBSERVABLE MDPs (POMDPs), AND STRUCTURAL CAUSAL MODELS (SCMs)
An MDP is defined to be four-tuple AS R, T") where S is finite set of states, A finite set of actions, T S A - IT(S") the transition function mapping from state action pair to the distribution over the set of states and R S x A - IR the reward function (Kaelbling et al. 1996). And POMDP defined to six tuple with two additional elements than the MDP, (S, A, R, T), O,p where O is finite set of observations and p S II(O) is the observation function mapping from the true underlying state to the distribution of observations (Kaelbling et al. 1998). Recall the definition of SCMs in the preliminary section, the transition function and inherent structural assumptions in MDPs and POMDPs. We can encode all the state/actiin/ddservattooooeee
that the definition of SCMs subsumes
variables endogenous variables and the randomness of the
27

---

Published as a conference paper at ICLR 2024
S
S
X
X
Y
Y
Figure 13: rewards.
The causal diagram of a standard MDP. We use X to denote actions and Y to denote
O
O
S
S
X
Y
y
Figure 14: The causal diagram of a standard POMDP. We use X to denote actions and Y to denote rewards.
reward and transition functions as exogenous variables. Each variable's value is either determined by structural equation specified by the environment or policy specified by the agent.
Graphically speaking, the causal diagram of a typical MDP is shown in Fig. 13 The next state S only takes the action and state the current time step input, which means that in the form of structural equations, St+1 fs... (S,,X,, Us.,1) where US is an exogenous variable representing the inherent randomness in the transition function. In Fig. 13 we can see clearly the Markov assumption embedded inside the graphical structure, i.e., (S.+1 II S St, X2). Similarly, we can also ground the Markovian reward assumption with precise graphical criteria. As stated by Abel et al.) Markovian reward assumption assumes that the state factors that are affecting the reward are fully observable to the agent. This can be interpreted as, pa(Y;) S S, where all parent nodes of the reward node is given input to the agent (S,). Note that here denote the state as set of variables instead of one single variable for clarity and we implicitly assume that the agent can observe all S, when making decisions.
For POMDP, the causal diagram is shown in Fig. The causal diagram faithfully reflects the fact that the agent cannot observe the state variables directly but only the observations. And importantly,
28

---

Published as a conference paper at ICLR 2024
the underlying transition dynamics between the true states and actions still following the standard MDP. The representation of SCMs is more versatile in modeling decision making scenarios in that it is amenable to represent any data generating processes without casting structural assumptions like MDP nor POMDP to the problem. By introducing the notion of confounders, we can better utilize the graphical structure construct optimal agents efficiently (Zhang & Bareinhoim 2020).
In this work, we utilize the qualitative causal knowledge to ensure that the causal effect changing certain aspects of the target task won't affect the optimal policy the agent will learn from the gen- erated source task. Even without confounders, if the state space is partially observable, the same situation as in the Colored Sokoban could happen since not all factors affecting the reward can be observed by the agent. But when the Markovian reward assumption holds, where the agent can observe all the parents the reward variable, the reward cannot be confounded with any other vari- ables. Under this stronger assumption, as our Theorem 1 indicates, all state variables are editable. On the other hand, we are dedicated solving the curriculum generation problem in the presence of unobserved confounders. Thus, the setting of Markovian rewards actually falls into the traditional curriculum reinforcement learning problem studied in the literature, which our work is trying to relax.
I LIMITATIONS
As our theorems and algorithms require causal diagrams inputs. it might not be easily applicable when the domain specific knowledge is hard to retrieve or the input diagram is inaccurate. Thus, we will combine method with causal discovery methods that automatically learn the underlying causal structure from the data to loosen this constraint in the future (Hu et al. 2022 Li et al.) 2023 Perry al. 2022).
29


Reviews:
[{"review_id": "bfEzo3VzGP", "reviewer": "Reviewer_C4uc", "summary": "The paper delves into the challenges of designing a curriculum of source tasks to tackle a complex target task in the presence of unobservable confounding variables within the environment. The authors leverage the structural causal model framework (Pearl, 2009) to define causally aligned source tasks for a given target task. They propose a causally aligned curriculum that incorporates qualitative causal knowledge of the target environment, and they validate their approach through experiments in two confounded environments.", "questions": "Re. the Colored Sokoban example:\n\nIn the baseline, the context/task space is restricted to tasks where the box color remains constant throughout the game. However, considering a more extensive task space that encompasses all possible combinations of box colors might lead the state-of-the-art curriculum strategies to automatically select relevant/aligned tasks during training. In contrast, the proposed causal curriculum limits the task space to initial agent and box positions, with the environment determining box colors based on intrinsic randomness. \n\nFormally, let us denote the initial positions of the agent and the box as $(a_0^x, a_0^y)$ and $(b_0^x, b_0^y)$, respectively. Furthermore, let $c_t$ represent the color of the box at time step $t$, and designate $H$ as the maximum number of steps allowable in the game. In the results presented, the baseline methodology confines the context/task space to instances of the form $[a_0^x, a_0^y, b_0^x, b_0^y, c_0 = c_1 = \\cdots = c_H]$. The extensive task space contains all possible task configurations denoted as $[a_0^x, a_0^y, b_0^x, b_0^y, c_0, c_1, \\dots, c_H]$. The proposed causal curriculum imposes a task space restriction, characterized by tasks of the form $[a_0^x, a_0^y, b_0^x, b_0^y]$, delegating the selection of $[c_1, c_2, \\dots, c_H]$ to the environment, predicated upon intrinsic randomness. \n\nPlease provide your insights on this aspect.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important problem in curriculum reinforcement learning, highlighting the potential negative impact of inadequate task space design on target task performance. \n\nThe writing quality is commendable, even though the paper features heavy notation due to its nature. The repeated use of the Sokoban game aids comprehension and clarity regarding the paper's contributions and claims.", "weaknesses": "The strategy for avoiding misaligned source tasks, as proposed in this work, relies on causal knowledge about the underlying data-generating mechanisms within the environment. This requirement limits the broader applicability of the strategy, making it dependent on domain-specific knowledge. The practicality of obtaining a causal diagram G for a general target task remains uncertain. \n\nIt remains unclear whether the proposed causal curriculum strategy can be extended to domains with continuous state and action spaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4cUSTdDVCO", "reviewer": "Reviewer_StU4", "summary": "Curriculum reinforcement learning is to train the agent on a sequence of simpler, related tasks in order to gradually progress the learning towards a difficult, often sparse-reward target task. The hope is that if there are common optimal decisions among these simpler intermediate tasks and the target task, the agent will learn transferrable skills during the process and accelerate the learning process. However, this assumption may not hold if there are unobserved confounders in the environment. This paper delves into this problem from a causal perspective, defining conditions under which the optimal decisions remain consistent. It also introduces a method to create a curriculum that aligns causally. The method is tested in two grid-world environments with unseen confounders to validate the effectiveness of the proposed algorithms.", "questions": "* In Algorithm 3, how is the actions sequence $\\{X_1, \\ldots, X_H\\}$ obtained in the first place? If we have this optimal action sequence before the learning starts, why do we need RL?\n* In Theorem 3, one of the conditions is that: For every $j=1, \\ldots, N-1$, actions $\\boldsymbol{X}^{(j)} \\subseteq \\boldsymbol{X}^{(j+1)}$. Does this mean that the transition must be deterministic and there only exists one unique optimal solution?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The structure and flow of the paper are easy to follow. Several simplified examples are provided to support the argument and illustration. To my knowledge, the perspective of confounders in curriculum reinforcement learning is novel.", "weaknesses": "* Toyish experiments. The experiments conducted are limited to grid-world environments, which significantly narrows the scope of application. For the conclusions to be generalized, the experiments should ideally be extended to a more diverse set of environments.\n* Scalability. The paper does not adequately address how the proposed method scales to continuous environment variables. This is a significant concern as many practical applications in continuous or high-dimensional environment variable space. By not tackling this challenge, the authors leave a gap in understanding the full potential and limitations of their method.\n* Missing related work:\n    * Hu, Xing, et al. \"Causality-driven Hierarchical Structure Discovery for Reinforcement Learning.\" Advances in Neural Information Processing Systems 35 (2022): 20064-20076.\n    * Cho, Daesol, Seungjae Lee, and H. Jin Kim. \"Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation.\" arXiv preprint arXiv:2301.11741 (2023).\n    * Huang, Peide, et al. \"Curriculum reinforcement learning using optimal transport via gradual domain adaptation.\" Advances in Neural Information Processing Systems 35 (2022): 10656-10670.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LQS6NlrHVK", "reviewer": "Reviewer_eG2G", "summary": "The paper tackled the problem of curriculum design in multi-task Reinforcement Learning. A challenge in the previous curriculum learning literature is that they all assume that the optimal decision rules are shared across different tasks. From a causal point of view, this expectation may not hold when the underlying environment contains unobserved confounders. To tackle this issue, the paper proposed a causal framework based on structural causal models and rigorously defined the notion of aligned tasks. The paper then proposed an algorithm that only generates aligned tasks. Simulation studies on Maze and Sokoban environments showed the advantage of the proposed algorithms.", "questions": "1. I was a bit confused by the wordings in abstract. The authors first state that invariant\noptimal decision rules does not necessarily hold, then they propose condition characterizing causally aligned source tasks, i.e., the invariance of optimal decision rules holds. The flow does not seem right here.\n\n2. Need extra clarifications on Figure 1 and 2. What does it mean by fixing the color? When the different algorithms are trained, are they only allowed to choose tasks from misaligned source tasks? Should this example be too artificial? It would be interesting to see the results when ALP-GMM can generate tasks with unknown color, while the algorithm may reuse these tasks. In this way, it is possible for ALP-GMM to discover that certain tasks can be more helpful for learning the target task. This is in fact more aligned with the situations in the real-world.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper finds a significant issue in the current literature of automatic curriculum learning and provides a clear framework to discuss the issue of non-alignment. Thus, the paper has great significance.\n\nThe paper is well-written and all the important messages are clearly delivered.", "weaknesses": "1. The authors mentioned that curriculum learning could overcome the curse of dimensionality. However, all the examples discussed in this paper are tabular. There seems to be a mismatch. It would be interesting to discuss and experiment on an example with exponentially large state and action space.\n\n2. What is the computation-complexity of FindCausalCurriculum? Does it scale with the size of state and action space? If that is the case, it seems to be against the motivation of applying curriculum learning.\n\n3. Comments on experiments: In general, I believe the environment tested in this paper are rather naive.\n\nThe authors implement ALP-GMM by fixing the color. I don't think this is a good way to construct baseline. One can run ALP-GMM as a task sampler on a fixed set of tasks. ALP-GMM essentially assigns a probability for different tasks at the each round. If one task has the underlying U that is not aligned with the target task, ALP-GMM will adaptively reduce the weight for that task. \n\nI understand that fixing color matches the general story of the whole paper. However, I don't think this is a natural choice in examples where we have access to a simulator. It would be interesting to demonstrate a non-trivial example where C is a proxy of the underlying U and choosing C by running ALP-GMM could be harmful. \n\n4. An important way to improve the paper is to propose an algorithm for large (or continuous) state and action space. The current version seems to only work with tabular MDPs, which limits the complexity of environments the algorithm can be applied to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h8ybid6q86", "reviewer": "Reviewer_1poU", "summary": "This paper presents a curriculum design method that produces causally aligned curriculums. Given a causal graph, a curriculum generator, and a target task, the proposed method creates a causally aligned curriculum using two main ideas. First, the FindMaxEdit algorithm, which constructs a set of causally aligned source tasks based on the modifying the editable variables of the target task. Importantly, optimal decision rules can be transported across causally aligned source tasks. Second, the authors propose ordering causally aligned source tasks with an expansion criterion on the optimal decision rules (i.e. tasks should be ordered such that the set of optimal decision rules expands with each task).", "questions": "See the weaknesses section for the most substantial questions and comments. \n\nMinor clarity comments are below: \n- Exogenous vs endogenous should be defined at some point in the preliminaries section on SCMs. \n- \"che\" \"de\" \"an\" meanings were not immediately clear to me. \n- In Def 1, the SCM M* is labelled with a *, yet this notation does not appear elsewhere in the paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "i. The method is well-founded and rigorously specified. Further, the main idea of this paper - using knowledge of causal relationships to improve curriculum generation methods - is clearly useful to the community. \nii. The paper is clear, with all details necessary to reproduce the work present in the paper. \niii. The experimental results on Colored Sokoban and Button Maze are strong.", "weaknesses": "Most of my concerns + comments are related to the motivation + presentation of the paper. \n\n- Motivation: Example 1's motivation for the paper is very contrived. The only reason a curriculum designer would design such a source task (where the box color is fixed to yellow) is if the curriculum designer was completely unaware of the causal dependencies of the reward. While this is possible in the case of causally-unaware automated curriculum generation methods, I find it a bit unlikely unless the crucial confounder variable U_i was either excluded from the state space (which would violate the usual Markovian reward assumption) or it is included in teh state space, but the agent cannot observe it (a partial observability assumption).  This leads to my questions: \n\t1)  Can the authors experimentally verify how often existing non-causal curriculum generation methods generate causally unaligned tasks? \n\t2) Can the authors comment on the expected benefit of their method in a domain with Markovian reward, or a fully observable domain w/no unobserved confounders? \n\n-  Acknowledging limitations and situating their work in presence of related work \n\t1) I saw that the related works section was relegated to the Appendix. There should at least be a pointer in the main paper to the related works. I found it helpful to read it, to understanding the positioning of this work, and think that it should be included in the main paper. The paper provides almost too many details about the method in the main paper, and at places is rigorous to the point of being pedantic. I think that the space would be better spent on situating the work properly. \n\t2) The inputs/outputs of the authors' method is not presented in plain language. As I understand it, the authors' method assumes access to the SCM of the task, and a curriculum generator such as GoalGAN. Such details should be made much clearer, perhaps through a summary methods figure or by adding a discussion of limitations. \n\n- Figures are not standalone: it is common practice to read a paper by skimming the figures + captions first, but I found that it was not possible to get the main idea of the figure, method and experimental results by doing so. Even after reading most of the accompanying text, I still found the figures + captions alone ambiguous. I had to find the specific part of the text referencing the figures -- sometimes needing to jump to other parts of the paper-- and read very carefully to understand what was happening. The authors should rewrite the figure captions and perhaps add a new \"methods\" figure summarizing the flow of their method. My specific comments on two figures are below, but all figures can be improved: \n\t1) Figure 5: Specifically, I was confused about why the curves labelled \"causal\" performed differently across the four columns? Also, the names of \"causal\" vs \"original\" were confusing. I think this is because the fact that the proposed method augments existing curriculum generation methods was presented only in a single sentence towards the end of the intro, and the knowledge was assumed in the rest of the paper. Seeing as this information is crucial to understand the paper, perhaps the authors can add this crucial piece \n\t2) Figure 3: It's confusing that figure (a) and (b) are identical except for the edit indicators. I needed to visually trace all arrows of both figures to verify this. The presence/absence of edit indicators should be explained in the caption, and the figure can be modified so that it is much quicker for the reader to notice that the only addition is the edit indicators (perhaps through the use of opacity / shading).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)