SCALE: Causal Learning and Discovery of Robot Manipulation Skills using Simulation
Authors: Tabitha Edith Lee
Shivam Vats
Siddharth Girdhar
Oliver Kroemer
Venue: CoRL 2023 Poster (2023)
Keywords: skill discovery
causal learning
manipulation

Abstract: We propose SCALE, an approach for discovering and learning a diverse set of interpretable robot skills from a limited dataset. Rather than learning a single skill which may fail to capture all the modes in the data, we first identify the different modes via causal reasoning and learn a separate skill for each of them. Our main insight is to associate each mode with a unique set of causally relevant context variables that are discovered by performing causal interventions in simulation. This enables data partitioning based on the causal processes that generated the data, and then compressed skills that ignore the irrelevant variables can be trained. We model each robot skill as a Regional Compressed Option, which extends the options framework by associating a causal process and its relevant variables with the option. Modeled as the skill Data Generating Region, each causal process is local in nature and hence valid over only a subset of the context space. We demonstrate our approach for two representative manipulation tasks: block stacking and peg-in-hole insertion under uncertainty. Our experiments show that our approach yields diverse skills that are compact, robust to domain shifts, and suitable for sim-to-real transfer.

SCALE: Causal Learning and Discovery of Robot Manipulation Skills using Simulation
Tabitha Edith Lee" Shivam Vats Siddharth Girdhar
Oliver Kroemer
The Robotics Institute
Carnegie Mellon University
(tabithalee, svats, sgirdhar, okroemer } @cmu. edu
Abstract: We propose SCALE, an approach for discovering and learning di- verse set of interpretable robot skills from a limited dataset. Rather than learning a single skill which may fail to capture all the modes in the data, we first iden- tify the different modes via causal reasoning and learn separate skill for each them. Our main insight to associate each mode with unique set of causally relevant context variables that discovered by performing causal interventions in simulation. This enables data partitioning based the causal processes that generated the data, and then compressed skills that ignore the irrelevant variables can be trained. We model each robot skill as a Regional Compressed Option, which extends the options framework by associating a causal process and its rele- vant variables with the option. Modeled as the skill Data Generating Region, each causal process local in nature and hence valid over only subset the context space. We demonstrate our approach for two representative manipulation tasks: block stacking and peg-in- peg-in-hole insertion under uncertainty. Our experiments show that our approach yields diverse skills that are compact, robust to domain shifts, and suitable for sim-to-real transfer.
Keywords: skill discovery, causal learning, manipulation
Introduction
We want robots help and work alongside humans in their homes, kitchens, and restaurants. How- ever, outside of structured environments, robots currently struggle at reliably performing even some of the basic manipulation tasks that humans can do with ease. Why humans much better despite the vast diversity of objects and their complex interactions that they potentially need to rea- son about? First, humans usually know multiple ways to solve a task to be robust to failures and variations in the environment. For example, if a tight jar doesn't open with our bare hands, we may use a piece of cloth to improve our grip. Second, humans excel at selectively attending [1] to only a small part of the environment that relevant to the task. Selective attention significantly reduces the computational complexity of reasoning and allows us to handle complicated situations.
Prior works in manipulation skill learning have leveraged these two observations separately. Most methods [2. 3, 4, 5, 6] learn skills by associating each skill with a sub-goal, where, the sub-goals are hand-de or learned from demonstrations. Once the sub-goals have been assigned, feature se- lection [7, 8] and abstraction selection [9, 10] can be used to reduce the complexity of skill learning. However, such approaches are quite sensitive to the sub-goals and struggle to distinguish between different strategies achieve the same goal. Our main insight is to associate a skill with not just sub-goat but also with the variables that causally relevant to it. For example, opening a jar with our bare hands is skill distinct from opening it with the help of a piece of cloth. Only hand and
"Equal contribution. Conresponding author.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

jar are relevant to the former, while the latter also relies on the properties of cloth. Hence, these two strategies should be represented as two distinct skills even though they achieve the same goal.
Based on this principle, a manipulation task involving n variables can have up to 2 skills, based on variable subsets being causally relevant. This is very large space to search for skills and not all sub- sets may correspond to useful skill. Hence, we propose SCALE (Skills from CAusal LEarning), an efficient approach for robot skill learning through causal feature selection in simulation. Instead of naīvely generating data, in our approach, the robot interacts with the simulator by conducting causal interventions. This elicits the causal features for completing a task under different settings, yielding a diverse and compact library of skills. Our approach learns skills that are described by physically meaningful properties without spurious variables that would be related irrelevant objects.
Our contributions of this work are two-fold. First, we introduce SCALE, algorithm for learning robot skill library from causal interventions in simulation. Second, we conduct variety of experi- ments that demonstrate that SCALE outperforms baseline approaches for two manipulation domains of block stacking and sensorless peg insertion. As part of these experiments, we also demonstrate sim-to-real transfer of the skills learned by SCALE for block stacking.
2
Related Work
Robot skill learning. Building robots that can solve wide variety of complex tasks is one of the fundamental problems in robotics. A popular approach is to learn skills parameterized by the task parameters as these can generalize over related tasks. Prior works [11, 12] show such parameter- ized skills lie on a low dimensional piecewise-smoodt manifold in the context space and identify this structure using ISOMAP [13]. For nigher-dimensional problems, becomes infeasible to learn directly in the full context space. One approach is to leam a library of simple parameterized skills which can be composed to solve more complex tasks [14, 15, 16, 17]. Recent works [18, 19] pro- pose a differentiable attention mechanism to learn context-specific attention, but these have been evaluated only in relatively small domains. Popular methods for unsupervised skill discovery in- clude graph-based methods [20, 21] that seek to build a graph of skills to cover the task space and ceermattoo-theoreete methods [22, 23, 6] that seek to maximize the diversity of skills.
Causality in robotics and reinforcement learning. Causality is the science of cause and ef- feet [24. 25, 26). Although the advantages of causal inference and discovery within the biomedical sciences, economics, and genomics have been well-established [27, 28), the integration of causality within machine learning is nascent [29, 30]. In robotics, causality-based approaches are partic- ularly under-explored despite the potential advantages of greater reasoning and learning capabil- ities [31], particularly through structure and transfer learning [32]. Most similar our work is that of CREST [33], an algorithm for identifying features for a robot policy through causal inter- ventions. Our algorithm SCALE leverages the work of Lee al. on CREST for determining the causally relevant variables for each robot skill. Causality has also empowered learning the structure of physical systems from videas [34] and explanations for robot failures [35]. Within reinforcement learning more broadly, causality plays a central role for improving performance through greater structure [36], learning latent factors in dynamics via causal curiosity [37], learning invariant poli-
cies [38], and learning a dynamics model that can yield state abstractions [39].
Intuitive physics. Please see App. B for a discussion of how SCALE relates to intuitive physics.
3 Preliminaries
The robot learns a set of skills K= [K,,...., where each skill solves a distribution of manip- ulation tasks. Each task modeled as a manipulation MDP [40], M = (S, R. T, T,, where s S is the state space, a A is the action space, R. is the reward function, T is the transition
/ebbsite https://sites: htt.a//ee.ooo...oooo///////eeeae

---

function, y is the discount factor, and T is additional task information. Tasks are solved if the final reward Ry > Rg, where Rg is a solved threshold.
Options. Each skill IC is parameterized option [41, 10]. An option O = (I., 1. is defined using three components: (1) the option control policy (aass); (2) initiation set I = {R₁ Rs|s) that specifies where the final reward R, solves the task when taking option O in state using option policy т: and (3) termination condition ((s) that specifies when the option concludes. For the termination condition of this work, our skills execute open-loop with fixed duration.
Context. We define the context e E as a set of variables C that fully specify the manipulation task. The context space time-invariant task properties defined by Each skill requires only subset of the full context, determined via causal feature selection (c.f., Sec. 5.1).
S generalizes the state space to include geometric and other
Contextual policies. We use a hierarchical approach [42] to decompose the option control policy into an upper-level policy (0lc) and a lower-level policy Ti(als, 0). Given a context E C, Ta : + / specifies the parameters / for the lower-level policy कt- For example, л could be a Cartesian-space impedance controller, where specifies the sequence of waypoints to be followed by the robot end-effector. For our work, we assume the lower-level controller is given, and learn the upper-level policy Tuk- The lower-level controller is shared across the different skills.
Compressed Context and Feature Selection. SCALE learns compressed skills that only use causally relevant context variables, as many will be unimportant. For our work, we disregard di- mensions of the context space that are not chosen by causal feature selection (c.f., Sec. 5.1), leading compressed context space è that is obtained by selecting dimensions of the full context space that correspond to the relevant variables of interest.
Causal Reasoning in Simulation. SCALE leverages a simulator with the key capability of inter- acting with scenes through context interventions, which enables the causal learning in SCALE. For this reason, we formalize the simulator as a causal reasoning engine W = (Es, T), where Es is the scene structural causal model (SCM) and T is the transition model. App. C provides greater discussion of this formalization. This formalism addresses SCALE's assumption that the simulator is capable of answering questions scene interventions, i.e., constructing new scenes with change to one variable to assess if there is a change (c.f., Sec. 5.1). These variables are required to be in- tervenable within the simulator, but not all variables need to be intervenable. For instance, gravity a simulation variable, but for this work, it is not considered as a candidate for causal reasoning; therefore, does not need to intervenable.
4 Skill Formulation
4.1 Regional Compressed Option
In our work, we formalize each robot skill IC as a Regional Compressed Option (RCO), where K (Mx, Pre, B, ②) and nu is the option control policy, Pre is the precondition, B is the termination condition, and 9 is the data generating region (DGR). In this model, the policy (a(a)ēsa) ) uses compressed context ČA2, which is obtained by selecting dimensions of the context space according to the relevant variable set A& C C (c.f., Sec. 5.1). The learned, upper-level policy is (0)(O(A).) The precondition Pre(c) = P(R₂ > Rs(c) is a probabilistic initiation set [43].
4.2 Data Generating Region
Our goal is to learn an upper-level policy म, c - 8, i.e., a mapping that generates the correct parameters for solving the task from any initial context c. We refer to this unknown mapping as a data generating process or causal process. Instead of trying to learn this data generating process directly, which may be difficult when many variables are involved, our main insight is model it as mixture of multiple causal processes. Each such process is likely to have smaller set of relevant variables and thus would be casier learn. For example, consider the task of opening jars, where
3

---

Task Description
Causal Feature Selection using CREST
Skill Learning
Content
Polly-relevent Parsmettars
Generating
Ubbrary
Simulation
Salver
Interventions
C
x F
A
D
Polley
A
Proje))
milo in 0)
Geunal Reasoning Engine
(1/(5.0)
A²D
Polley Learning
Precondition Learning
Taak Reward/GGal (x.4))
Train - x2 A
D
Train - 2,
Staoking
Figure The figure shows an overview of proposed framework applied to a block stacking task. The robot is given a context space, control policy, task simulator, and task reward. The robot samples a set of contexts to create task instances, which it subsequently solves for that instance. The robot then applies interventions the contexts to identify skill-relevant parameters. Contexts with the same set of policy-relevan come from the same causal model and are hence combined to form data generation regions. Here, we have two causal models: e with relevant variables from the yellow, blue, and red blocks and DGR 21; and C2 with relevant variables from the yellow and blue blocks and DGR Ø2. Each region then used to learn separate skill policy with the corresponding set of policy-relevant parameters. For each skill, we finally learn set of preconditions within the context space determine where the skill ultimately be applied. The pairs of policies and preconditions are then combined to create a skill library for completing the given task.
parameters
the jar could be tight not tight. We can model the data generating process for this task combination of two simpler causal processes: E₁ which uses only your hand, and e₂ which uses piece of cloth along with your hand. However, these causal processes don't 't hold for all jar opening tasks. e, holds when the jar is not tight, while C₂ holds when the jar is tight. Thus, every causal process is valid only in a subset of the context space. We refer to this subspace 2 C as the data generating region (DGR) of the causal process. Here, 21 == {not tight) and 9 := (tight). The robot learns separate skill for every such causal process. Furthermore, each skill should be trained by only using data from inside its DGR; data lying outside the DGR are generated by different causal process and is hence out-of-distribution. The DGR uses compressed context éD, obtained from relevant variable set D C C (c.f., Sec. 5.1).
5
Skill Discovery through Causal Reasoning in Simulation
The SCALE algorithm (Fig. 1) comprises two steps: 1) skill dataset generation and 2) skill training. These steps are described in Sec. 5.1 and 5.2, respectively. Algorithm descriptions are in App. E.
5.1 Batch Data Generation
First, the robot interacts with the simulator W to collect skill training data. This is done by collecting a batch dataset Dg. The robot samples random scenes represented by c, and attempts to determine the lower-level controller parameters 0, that solve the specific task. In practice, we use Relative Entropy Policy Search (REPS) [44], but any suitable planner, trajectory optimizer, or reinforcement learning algorithm would suffice. Unsolved tasks disregarded and not collected in Dnz
Causal feature selection. For successfully solved scenes, the relevant variables for the policy A; and DGR D; are selected using the CREST algorithm [33]. CREST conducts feature selection through causal interventions. Intuitively, a variable is causal if, for all other variables held equal, interventions upon this variable induce change the final obtained reward R,, spurious variable has no effect on the reward and thus can safely be ignored. To summarize CREST, the process begins by solving scene, which refer to the non-intervened scene. For each context variable, value is randomly sampled from a distribution (in the CREST work, this distribution is the context

---

variable's 's possible values). A scene is constructed with that intervened value, with all other context variables holding the same, non-intervened value. Then, the robot executes the solution to the non- intervened scene in this intervened scene to obtain an intervened reward. This process repeats a given number of times, and a statistical test assessed to determine how often the intervened rewards differ from the non-intervened reward. If the intervened rewards are frequently no different than the non-intervened reward, the context variable is considered spurious (and causally relevant otherwise).
In this work, CREST performs interventions I over local (e.g.. 10%) fraction of context space C to yield A, Similarly, D; is obtained through interventions over the entire space C. Finally, the batch dataset is appended by the dataset point (c, A,, D.). Note that CREST is not a strict requirement of SCALE. In principle, SCALE requires only a determination of which variables are causally relevant, which CREST provides. Other approaches, such as using causal discovery, are also possible. An important consideration of choice of approach is whether the context space is disentangled. In our work, we assume a disentangled context space, and so the variable-by- -avarbbl intervention process of CREST (which also assumes disentangled variables) will suffice. If the context space is entangled, then causal disentanglement approaches could first be used.
Splitting batch data into skill data. After dataset collection, batch dataset DB split into different skill datasets according the relevent variable sets. In this work, we assign highly occurring batch data that contain the same relevant variables A into the same skill dataset Dx. while also taking the union over all associated D. This assumption may not always hold, but is sufficient for the tasks we examine in this work. More sophisticated ways of splitting the batch dataset is left for future work.
5.2 Skill Training
The second phase of SCALE trains each skill Kg using Dr. Each skill has relevant variable sets A& and D3 with task solution datapoints (c, 0).
DGR. The DGR S is first trained on ÊD using Dg. For this work, we use a one-class SVM to model the DGR, but in principle, any one-class classification algorithm would suffice.
Policy. The policy is trained next. The skill dataset contexts are filtered through the DGR 9 to obtain inliers ct for policy training data. This ensures policy training data are consistent with the underlying causal process. Then, the policy Tak (0) (A.) is trained using é (using A₂) and the corresponding parameters #+. For our work, policies are learned using regression, but reinforcement learning could also be used [33]. With Tue learned, the final skill policy F.. (x)))) is determined.
Preconditions. The preconditions Pre are learned last through policy evaluation. Using the simula- tor W, contexts are re-sampled and evaluated with policy T& to obtain rewards Ry. This evaluation data (c, Ry) is used to train a precondition classifier to obtain Pre. For our work, we use a nonlinear SVM classifier that has probability estimates.
Experimental Results
We conduct skill learning experiments with SCALE for block stacking and peg-in peg-in-hole insertion tasks with the Franka Emika Panda robot (Fig. 2). Both tasks are emblematic of high-precision con- trol that is desirable in many industrial applications [45]. We conduct our experiments in NVIDIA IsaacGym [46, 47], high-fidelity physics simulator that also serves our causal reasoning engine W. We use custom library that implements the scene SCM Es to facilitate scene creation and interventions. The forward simulation of physics provides the transition model T.
Baselines. We compare SCALE to baseline approaches with monolithic policies (without any skills) for either the full-dimensional context space ("monopolicy") or a reduced context space obtained by using the most commonly occurring CREST result ("crest-monopolicy") The CREST monopolicy represents naïvely using CREST, ignoring that CREST provides locally different results within the underlying data (a property that SCALE leverages).
5

---

Thit
(b)
(c)
(d)
(c)
Figure SCALE discovers skills for the Franka Emika Panda robot using causal learning simulation for two manipulation tasks: (a) block stacking and (b) peg-in-hole insertion. In addition to skill learning experiments, we also show how SCALE can yield skills (c) for sim-to-real transfer (App. I); (d) for generalization in down- stream tasks, such stacking block tower (App. J); and (e) for robustness to task domain shifts (App. L..
Table 1: Skills Katookk that discovered for the block stacking task. A and D the variables used for the skill's policy and DGR, respectively. Data is the quantity of data used for each skill (from a batch dataset of 585 samples, 340 samples were used to train skills). Tsk. Sv. %, shown for both scale-lin and scale-nonlin, is the rate of task solves over the entire context space using only that skill. Skill
Data
Tsk. Sv. %. Lin
Tsk. Sv. %, Nonlin.
{a Vi 32 92
Kaa
53 (9.06%)
65.36% (200)
18.36% (56)
D: (1t yi. J2 h2]
A: [T V.. 222
K₂
272 (46.50%)
78.76% (241)
55.88% (171)
D: (xy Vi h1, U2 h2
A: [Ø] Vi ; th,, 22 1/2 lv
Kis
15 (2.56%)
34.31% (105)
1.31% % (4)
D: {xf v1. 21, 12, h2
6.1 Block Stacking
Task representation. In the block stacking task, the robot starts with a source block (B1) grasped, and it learns place it on top of target block (B2). To do this, the robot uses a controller x that defines the trajectory for the robot end-effector to traverse via impedance control. This trajectory is parameterized by 0, [. Oay. 0 O IT Et, which specify waypoints the robot follows sequentially. Specifically, these parameters characterize trajectory where the robot lifts the source block vertically, moves horizontally, descends vertically, and releases the block.
For this task, the context variables CB are {CB,,..., CB hx), which the union of context variables for each of NB blocks plus the table height h upon which the blocks are placed. The context variables for each block b are {=r] V8 4/s, ha, R2, G₁, Bb), yielding 36-dimensional context space for this problem. Here, 2 and y are the world and y-positions of the block, and the block's 's orientation is represented by a rotation angle ths, around the block's 's vertical axis (2). The z-dimension (height) of the block is hss Additional experimental details are available in App. H.
Skill learning results: variable selection. From a batch dataset 585 samples, SCALE found the skill library Kitocke = [K1, K2,KKa) that is shown Tab. I that were learned using 340 samples of the dataset. These 340 samples were selected for being the most commonly occurring within the dataset, based on a heuristic threshold. Even though there are five blocks and 36 possible variables, the skills generally consisted of a much smaller subset of variables, relating to the geometry of the source and target blocks. Note that K2's relevant variables for the policy, Ax,, are consistent with earlier work by Lee et al. [33] for this domain. This is generally considered to be the "ground truth" variable result for unobstructed block motion this case. Skill Kr, could be seen as version of K₂ when h2 is not needed. Rarely, the source block's rotation i become important (e.g., the source block's 's final pose not fully stable when stacked on the target block), and thus skill emerges with this variable (KCz). Variables for neither block color nor table height are observed expected.

---

Table 2: Task evaluation results for using the skill library Kotocks for the block stacking task. Ctrl. is the approach control (skills or one monolithic policy). Fn. CL. is the approach's function class, Linear approaches use Bayesian ridge regression, whereas nonlinear methods consist of a multilayer perceptron with 16x16xII 16 architecture using ReLU activations. Task Solve % is the rate of task solves over the entire context space using the approach. Methods within +2% (the stochasticity of the simulator) of the best approach bold. |A] is the quantity of input variables used for the approach's policy. Data is the amount of training data used for the approach. ground truth policy also shown, using all context variables and additional domain knowledge.
Approach
Ctrl.
Fn. CI.
Task Solve %
Data
scale-lin (ours)
3 skills
Linear
90.49% % (276)
4/5/6
340
monopolicy-lin-all
policy
Lincar
85.95% (263)
36
585
crest-monopolicy-line
policy
Linear
89.87 % (275)
5
585
scale-nonlin (ours)
3 skills
Nonlinear
63.40% % (194)
4/5/6
340
canopollccy-onlin--al
I policy
Nonlinear
10.13% (31)
36
585
trest-monooolicynoooe nonlin-all
policy
Nonlincar
60.78% (186)
585
ground-truth-policy
policy
Nonlinear
95.75% (293)
Skill learning results: task evaluation. We evaluate the skill library Kotocks over the entire task distribution and show the results in Tab. 2. That is, for each context sample, the robot evaluates each skill's precondition and selects the skill with the highest probability of success. The suffix "-all" denotes that the entire batch dataset is used for the approach. For both function classes, SCALE yields approach that outperforms full-dimensional policies and is generally comparable to CREST-reduced policies. However, the CREST-reduced policies only learn one approach to solving the task, whereas SCALE learns three. The overall best performing approach was scale- lin (90,49%) with similar performance to the CREST baseline. Performance across all nonlinear approaches was generally lower. App. H details the SCALE skill selection and further ablations.
Sim-to-real experiment. We transfer the skills learned by SCALE and our baselines to real Franka Panda robot without any fine-tuning, As diseussed in App. I, SCALE outperforms the baselines.
6.2 Sensorless Peg-in-Hole Insertion
Our second domain peg-in-hole insertion under sensing uncertainty. requires the robot to insert cuboidal peg of cross-section cm
into cuboidal hole of cross-section 1.3 em x 1.3 cm. The robot gets noisy initial position of the hole with the noise sampled from Gaussian distribution (0, 0.3² cm?). No further sensory observations are available. Due to this uncertainty, a naive strategy of directly trying to push the peg down at the observed location of the hole achieves a success rate of only 34%. To address this, the robot should take uncertainty reducing [48] actions by initiating contact with the environment (e.g., a fixture next to the hole). Our goal in this experiment to learn such skills autonomously.
Task representation. Each assembly task has 4 axis-aligned cuboidal fixtures (i.c., walls) of fixed dimensions around the hole. The 8-dimensional context variables Cp are (#1, J1,.,....... I4, 34), con- taining the (Ir y) coordinates of these fixtures with respect to the hole. The positions are different in every task, but it is always possible for the robot to localize against any of the walls to complete the task. We use a 6-parameter policy space: three (Ar, Ay, Az) actions executed in sequence in the robot's end-effeetor frame. In every policy, 22's are designed to move the peg down while Az and Ay are parameters that are learnt using RL. Additional experimental details are available in App. K.
Skill learning results: variable selection. Table 3 enumerates the skills Kyes = (KI,..., Kr}} discovered by SCALE. Skills K2-5 localize against one of the 4 walls. For each such skill, only the wall being used for localization is relevant to the skill and the other walls can be ignored. Con- sequently, the set of relevant variables for these skills contains only the distance to the wall that localizes against. For the linear case, all of these skills except for K3s have high success rate, whereas success rates are slightly lower for the nonlinear approach. Interestingly, SCALE also discovers skill KC that has an empty set of relevant variables. For more discussion of this skill, see App. K.
7

---

Table 3: Skills Keeg that discovered for the peg-in-hole insertion task. Columns the in Tab.
except represents which samples were used to train skills (from batch dataset of 210 samples).
Skill
A
Data
Task Solve %, Lin
Task Solve %6, Nonlin.
K1
21,
U2. U3, m4
56 (26.67%)
64.84% (166)
61.72% (158)
K2
r4
25 (11.90%)
97.66% (250)
84.38% (216)
K3
{x1
[21]]
27 (12.86%)
44.53% (114)
84.77% (217)
K4.
{ya
{ya
28 (13.33%)
94.14% (241)
82.81% (212)
Ks
[y2}
{22)
32 (15.24%)
98.44% (252)
79.69% (204)
Table 4: Task evaluation results for using the skill library Kpey for peg insertion (columns in Tab. 2).
Approach
Ctrl.
Fn. Cl.
Task Solve
|A|
Data
scale-lin (ours)
skills
Linear
96.48% (247)
0/1/1/1/1
168
monopolicy- lin-all
policy
Linear
62.50% (160)
210
crest-monopolicy-iie lin-all
I policy
Linear
62.89% (161)
210
scale-nonlin (ours)
5 skills
Nonlinear
88.67% (227)
0/1/1/1/1
168
monopolicy-none
policy
Nonlinear
12.89% (33)
210
crest-monopolicy nonlin-all
policy
Nonlinear
55.47% (142)
210
Skill learning results: task evaluation. Table 4 presents the task evaluation of the skill library Kegg for 256 randomly sampled tasks. For both linear and nonlinear cases, SCALE outperforms is likely due insufficient data owing to a larger network. The most common CREST result was variable x, (21.90%), so this was used for the CREST baselines. However, it only localizes against one wall. The improvement of SCALE over the CREST baselines implies SCALE skills benefit from the DGRs through greater quality training data, whereas the CREST approaches use the entire dataset despite most samples having differing CREST result than T44. For details of SCALE skill selection and further ablations, see App. K.
both baselines. The low success of monopolicy-nonline
Domain shift experiment. To evaluate the out-of-distributtinn generalization capabilities SCALE, evaluate the skills All approaches degradation in performance, but ours is more robust. See App. L for details.
distribution that is significantly harder than the training distribution.
7 Conclusion
We present SCALE, an approach for discovery of compact, diverse robot manipulation skills from causal interventions in simulation. These skills arise from the skill DGR: region that captures the underlying data generating process. We demonstrate the advantages of skill libraries discovered with SCALE for two simulation domains as well as on a real robot system.
Limitations and future work. SCALE assumes the robot has access causal reasoning engine. We provide this via simulation and scene structural causal models, but these models could be learned via causal discovery. SCALE primarily learns from batch dataset collection; active learning of skills would reveal useful behaviors that are statistically uncommon in the batch setting. Lastly, SCALE assumes that the context variables are defined, intervenable, and disentangled. For tasks and domains where these assumptions do not currently hold, future work in adjacent fields may ultimately provide a path forward. Specifically, causal representation learning [29] - learning high- intervenable variables from low-level observations could construct state representation that SCALE use, and, if representation is available but entangled, causal disentanglement could be used.
Acknowledgments
We gratefully acknowledge support from the National Science Foundation (Grant No. CMMI- 1925130), U.S. Office of Naval Research (Grant No. N00014-18-1-2775) U.S. Army Research Lab- oratory (Grant No. 18F-188--20221 part of the A212 Program), and the NVIDIA NVAIL Pro- gram. We also gratefully thank reviewers, whose helpful comments strengthened this work.

---

References
[1] R. Desimone and J. Duncan. Neural Mechanisms of Selective Visual Attention. Annual Review of Neuroscience, 18(1):193-22 1995.
[2] G. Konidaris, S. Kuindersma, R. Grupen, and A. Barto. Robot Learning from Demonstration by Constructing Skill Trees. The International Journal of Robotics Research, 3163:360-375, 2012.
[3] Z. Su, O. Kroemer, G. E. Loeb, G. S. Sukhatme, and S. Schaal. Learning Manipulation Graphs from Demonstrations Using Multimodal Sensory Signals. IEEE International Conference on Robotics and Automation (ICRA), 2018.
[4] O. Kroemer, C. Daniel, Neumann, H. Van Hoof, and Peters. Towards Learning Hierar- chical Skills for Multi-Phase Manipulation Tasks. IEEE International Conference on Robotics and Automation (ICRA), 2015.
[5] A. Chenu, N. Perrin--iiie Gilber,, and O. Sigaud. Divide & Conquer Imitation Learning. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[6] Achterhold, M. Krimmel, and Stueckler. Learning Temporally Extended Skills Contin- uous Domains as Symbolic Actions for Planning, arXiv preprint arXiv:2207.05018 2022.
[7] C. Devin, P. Abbeel, T. Darrell, and S. Levine. Deep Object-Centric Representations for Gener- alizable Robot Learning. IEEE International Conference on Robotics and Automation (ICRA), 2018.
[8] 7. Kolter and Y. Ng. Regularization and Feature Selection Least-Squares Temporal Difference Learning. International Conference on Machine Learning (ICML), 2009.
[9] N. Jiang, A. Kulesza, and S. Singh. Abstraction Selection in Model-Based Reinforcement Learning. International Conference on Machine Learning (ICML), 2015.
[10] G. Konidaris and A. Barto. Efficient Skill Learning using Abstraction Selection. International Joint Conference on (ICCAI), 2009.
[11] B. C. Da Silva, G. Konidaris, and A. G. Barto. Learning Parameterized Skills. International Conference Machine Learning (ICML), 2012.
[12] B. C. Da Silva, G. Baldassarre, G. Konidaris, and A. Barto. Learning Parameterized Motor Skills on a Humamoid Robot. IEEE International Conference on Robotics and Automation (ICRA), 2014.
[13] J. B. Tenenbaum, V. De Silva, and J. C. Langford. A Global Geometric Framework for Non- linear Dimensionality Reduction. Science, 290055000:2319-233,, 2000.
[14] L. P. Kaelbling and T. Lozano-Pérez. Learning Composable Models of Parameterized Skills. IEEE International Conference on Robotics and Automation (ICRA), 2017.
[15] Z. Wang, C. R. Garrett, L. Kaelbling, and T. Lozano-Pérez. Learning Compositional Models of Robot Skills for Task and Motion Planning. The International Journal of Robotics Research, 40(6-7):866-98 8644 2021.
[16] J. Peters, J. Kober, K. Mülling, O. Krämer, and G. Neumann. Towards Robot Skill Learning: From Simple Skills to Table Tennis. Joint European Conference on Machine Learning and Knowledge Discovery Databases, 2013.
[17] R. Pahič, Z. Lončarević, A. Gams, and A. Ude. Robot Skill Learning in Latent Space of Deep Autoencoder Neural Network. Robotics and Autonomous Systems, 135:103690, 2021.

---

Foundations and [18] K. Khetarpal, M. Klissarov, M. Chevalier-Boisver,, P.-L. Bacon, and D. Precup. Options of Interest: Temporal Abstraction with Interest Functions. Proceedings of the AAAI Conference on Artificial Intelligence, 34(4):4444 4451, 2020.
[19] M. Abdulhai, D.-K. Kim, M. Riemer, M. Liu, G. Tesauro, and How. Context- Rep- resentation Abstraction for Deep Option Learning, arXiv preprint 0rXiv:2109.09876 2021.
[20] A. Bagaria and G. Konidaris. Option Discovery using Deep Skill Chaining. International Conference Learning Representations (ICLR), 2020.
[21] A. Bagaria, J. K. Senthil, and G. Konidaris. Skill Discovery for Exploration and Planning using Deep Skill Graphs. International Conference on Machine Learning (ICML), 2021.
[22] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is All You Need: Learning Skills without a Reward Function. International Conference on Learning Representations (ICLR), 2018.
[23] A. Sharma, S. Gu, S. Levine, V. Kumar, and Hausman. Dynamics-Aware Unsupervised Discovery of Skills. International Conference on Learning Representations (ICLR), 2020.
[24] Spirtes, C. N. Glymour, R. Scheines, and D. Heckerman. Causation, Prediction, and Search. MIT Press, 2000.
[25] J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2nd edition, 2009.
[26] J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference: Learning Algorithms. MIT Press, 2017.
[27] G. W. Imbens and D. B. Rubin. Causal Inference Statistics, Social, and Biomedical Sciences. Cambridge University Press, 2015.
[28] C. Glymour, K. Zhang, and P. Spirtes. Review of Causal Discovery Methods based on Graph- ical Models. Frontiers in Genetics, 10:524, 2019.
[29] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Towards Causal Representation Learning. Proceedings of the IEEE, 109(5):612 634, 2021.
[30] J. Kaddour, A. Lynch, Q. Liu, M. Kusner, and R. Silva. Causal Machine Learning: A Survey and Open Problems. arXiv preprint 15X4:22206.14755 2022.
[31] K. C. Stocking, A. Gopnik, and C. Tomlin. From Robot Learning to Robot Understanding: Leveraging Causal Graphical Models for Roboties. Conference on Robot Learning (CoRL), 2022.
[32] O. Ahmed, F. Träuble, A. Goyal, A. Neitz, Y. Bengio, B. Schölkopf, M. Wüthrich, and S. Bauer. CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning. arXiv preprint arXiv:2010.04296 2020.
[33] T. E. Lee, J. Zhao, A. S. Sawhney, S. Girdhar, and O. Kroemer. Causal Reasoning in Simula- tion for Structure and Transfer Learning of Robot Manipulation Policies. IEEE International Conference on Robotics and Automation (ICRA), 2021.
[34] Y. Li, A. Torralba, A. Anandkumar, D. Fox, and Garg. Causal Discovery in Physical Sys- tems from Videos. Advances in Neural Information Processing Systems, 33:9180 18099192, 2020.
[35] M. Diehl and K. Ramirez-Amaro, Why Did I Fail? A Causal-Based Method to Find Explana- tions for Robot Failures. IEEE Robotics and Automation Letters, 7(4):8925-8922, 2022.
10

---

[36] N. R. Ke, A. Didolkar, S. Mittal, A. Goyal, G. Lajoie, S. Bauer, D. Rezende, Y. Bengio, M. Mozer, and C. Pal. Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning. Proceedings of the NeurlPS 2021 Datasets and Benchmarks Track, 2021.
[37] S. Sontakke, A. Mehrjou, L. Itti, and Schölkopf. Causal Curiosity: RL Agents Discov- ering Self-supervised Experiments for Causal Representation Learning. International Confer- ence Machine Learning (ICML), 2021.
[38] A. Sonar, V. Pacelli, and A. Majumdar. Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning. Learning for Dynamics and Control, 2021.
[39] Z. Wang, X. Xiao, Z. Xu, Y. Zhu, and P. Stone. Causal Dynamics Learning for Task- Independent State Abstraction. International Conference on Machine Learning (ICML), 2022.
[40] O. Kroemer, S. Nickum, and G. Konidaris. A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms. Journal of Machine Learning Research, 22: 30-1, 2021.
[41] R. S. Sutton, D. Precup. and S. Singh. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence, 1120--1)181-211, 1999.
[42] M. Deisenroth, G. Neumann, J. Peters, et al. Survey on Policy Search for Robotics. Foundations and Trends Robotics, 2(1-2):1-142, 2013.
[43] G. Konidaris, L.. Kaelbling, and T. Lozano-Perez. Symbol Acquisition for Probabilistic High- Level Planning, International Joint Conference on Al (IJCAI), 2015.
[44] J. Peters, K. Mülling, and Y. Altun. Relative Entropy Policy Search. AAAI Conference on Artificial Intelligence, 2010.
[45] J. Luo, E. Solowjow, C. Wen, J. A. Ojea, A. M. Agogino, A. Tamar, and Abbeel. Reinforce- ment Learning on Variable Impedance Controller for High-Precision Robotic Assembly. IEEE International Conference on Robotics and Automation (ICRA), 2019.
[46] J. Liang, V. Makoviychuk, A. Handa, N. Chentanez, M. Macklin, and D. Fox. GPU- Accelerated Robotic Simulation for Distributed Reinforcement Learning. Conference Robot Learning (CoRL), 2018
[47] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. arXiv preprint arXiv:2 108. 10470, 2021.
[48] R. Brost. Automatic Grasp Planning in the Presence of Uncertainty. IEEE International Conference Robotics and Automation (ICRA), 1986.
[49] J. R. Kubricht, K. J. Holyoak, and H. Lu. Intuitive Physics: Current Research and Controver- sies. Trends in Cognitive Sciences, 21:749-759, 2017.
[50] P. W. Battaglia, J. B. Hamrick, and J. B. Tenenbaum. Simulation as an Engine of Physical Scene Understanding. Proceedings of the National Academy of Sciences, 110(45):18327- 18332, 2013.
[51] D. Ha and J. Schmidhuber. World Models. arXiv preprint arXiv: 2018.
[52] D. Hafner, P. P. Lillicrap, M. Norouzi, and J. Ba. Mastering Atari with Discrete World Models. International Conference Learning Representations (ICLR), 2021.

---

[53] J. Wang, C. Hu, Y. Wang, and Y. Zhu. Dynamics Learning With Object-Centric Interaction Networks for Robot Manipulation. IEEE Access, 9:68277-68288, 2021.
[54] O. Kroemer and G. Sukhatme. Meta-level Priors for Learning Manipulation Skills with Sparse Features. 2016 International Symposium Experimental Robotics (ISER), 2017.
[55] Katz, A. Tal, and R. Basri. Direct Visibility of Point Sets. ACM SIGGRAPH, 2007.
[56] M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al. A Density-based Algorithm for Discover- ing Clusters in Large Spatial Databases with Noise. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD), 96(3))222- 231, 1996.
[57] K. Zhang, M. Sharma, J. Liang, and O. Kroemer. A Modular Robotic Arm Control Stack for Research: Franka-Interface and FrankaPy. arXiv preprini urXiv:2011.003330 2020.
12

---

A SCALE and Appendices Overview
Fundamentally, SCALE is a causal learning algorithm for discovering compact, diverse skills through interventions in simulation. Figure provides an overview of the approach.
Structure of appendices. These appendices structured follows. Appendix B describes how SCALE connects to related work in intuitive physics Appendix C provides greater details into the formalization of the simulator and its role as causal reasoning engine. Appendix E formalizes the SCALE algorithm using nomenclature introduced in App. D. A discussion of nigher-dimensional context spaces and SCALE is then provided in App. F. Next, App. provides a toy experiment that designed to convey greater intuition and visualization of the mechanisms that underlie SCALE. Appendix H presents additional experimental details of the block stacking experiment presented in Sec. 6. Following this, Apps. and provides two additional experiments in the block stacking do- main: sim-to-real transfer experiment and a downstream task evaluation experiment, respectively. The next two appendices concern the peg-in inr-iihhhe insertion domain. Appendix K details additional experimental details first presented in Sec. 6.2, and App. I. presents an additional experiment that shows the robustness of SCALE under task domain shift. Lastly, Appendix M contains a primer on causality for readers who are new to this area of research.
(c)
(6)
KG
K K
K
Context
π
Action
Figure 3: In SCALE, the robot discovers skills in simulation using causal learning. (a) The simula- tion is used to solve task instances and conduct interventions to determine causally relevant context variables. (b) Simulation data are used to train library of skills, (c) which are suitable for sim-to- real transfer learning. (d) Each skill that learned is parameterized by the relevant variables selected in simulation. Here, red context variables are unnecessary for the skill policy and can be safely ig- nored. The boundary encireling the policy represents the skill DGR and precondition, which are also learned.
B Related Work for Intuitive Physics
This appendix describes the connections between SCALE and the intuitive physics literature. Intu- itive physics is the ability to approximately predict and model the physical world without explicit understanding of the underlying dynamics [49]. Literature in cognitive psychology has suggested that humans develop mental intuitive physies models to support fast prediction and understanding of complex physical scenes which enables physical reasoning [50]. Computational learning of intuitive physics have been successful, enabling reinforcement learning and planning applications owing to the models ability for forward prediction [51, 52, 53]. In work, causal reasoning engine can
13

---

be viewed as an internal model that uses interventions to elicit the physical mechanisms by which the data arise.
C Simulation as Causal Reasoning Engine
Scene SCM
0.1
Es
0.1
C
0.1
C S O O-PO
C
0.6
context
Fb
b=
context
0.4
0.4
0.4
interventions
scene
I I= {do(hs = 0.6)) C
(a)
(b)
Figure 4: Illustrations of the scene structural causal model used in the simulator W. (a) From context space C and robot interventions I, the scene SCM Cş generates a context vector that represents a particular scene that defines objects and their properties. (b) this block example, E3 is defined using scene variables L 2U2₁ and context variables C == faa, hs, hr J. where 26 is block x x-position, ht is block height, hg is table height upon which the block rests, and 24 = h, + h, is block z-position. Normally, values of C are sampled from context space C, but the robot performs an intervention = {do(hs = 0.6) to force the value of h& to be 0.6. As a result, the dependent variable 2; is determined as 0.7 using this intervened value. Lastly, the scene is constructed and represented as context vector c = [0.1, 0.6, 0.4]".
This appendix provides greater discussion of the simulator formalization used by SCALE. The sim- ulator model, W = (Cs: T), is formalized as follows:
1. scene structural causal model Es (Fig. 4) that, given context space C and interventions I, instantiates a scene that can be represented as a context vector, c C C;
2. the transition model T that captures the domain forward dynamics as the robot interacts with the world through 0 starting from the scene initialized from Cg.
A structural causal model (SCM) [25, 26] can be represented directed acyelic graph that is driven by exogenous variables (functional inputs of the graph) that produces the solution for all variables within the graph. These two components of the simulator capture the spatial structure inherent to the scene itself (Cs), and the spatiotemporal structure of the robot interacting with the world (T"). The simulator model W, including the scene SCM and transition function, is provided for the robot to use. principle, the scene SCM could be learned via causal representation learning [29], e.g., world models approach that admits causal interventions.
The scene SCM Cş is defined by structural equations with scene variables v, where C C V. In the graph induced by Cs. the scene variables are the nodes, and context variables C are the root nodes and exogenous variables (functional inputs) of the SCM. The value of the context variables is given by interventions I {do(C: C = c if specified, or otherwise sampled from the context space C. The robot only conducts interventions with respect to C that would yield a steady-state solution and are physically realizable, excluding physically invalid scenes (c.g.. object penetration).
The transition model T is the same typical simulators. The forward dynamics are simulated through the initial state su, obtained from the scene created by Es. and 6, the inputs to the low-level controller Ty. With these inputs, the system temporally evolves as usual until the end of the episode, where reward R is obtained and compared to threshold Rs to determine if the task was solved.
D Nomenclature
Table 5 summarizes the nomenclature used in this paper and, in particular, the SCALE algorithm (c.f., App. E). Note the use of italics and bold type to disambiguate certain symbols. For example,
14

---

Table 5: Table of nomenclature.
Symbol
Meaning
X
set of d random variables, i.e., X = (Xi,.... Xa) space X,
X
i.e., A' = 1 ...... Xg] TT
vector instantiation of X
i.e., I == [x] E X1 C , aa Xa S Xx]"
K
set of k robot skills.
i.e., K. = (KI,.... .
D
dataset containing A Rmxx samples from set A with size and By E gmxp labels from set B with size
X is set of random variables, but X refers to dataset matrix. The notation for variable and its instantiation scalar may also be overloaded depending on the context.
E SCALE Algorithm
As explained See. 5, the SCALE algorithm (Alg. 1) describes how the skills are learned through batch dataset collection and skill training. The procedure for batch dataset collection used by SCALE (SKILLTRAINDATA) is described Alg. 2.
Note that the number of skills is not a hyperparameter of the SCALE algorithm. Rather, the skill quantity emerges from AALTIINNTSKIIL.AAAAAASAA from groups of highly-occurring CREST re- sults, where each group becomes the dataset for particular skill.
F SCALE and Higher-Dimensional Context Spaces
The SCALE algorithm scales linearly with the dimensionality of the context space, i.e., O([C),, due to the necessity of performing interventions each context variable. In the experiments examined in this work, the dimensionality of the context space was 36 and 8 for the block stacking and peg insertion domains, respectively. For other applications where the context space is very large, heuris- ties can be incorporated to first downselect the context space into smaller candidate space that can be provided to SCALE. Example heuristics could include a distance metric (objects closer to the goal may be more likely be relevant than those further away) or using other approaches such as meta-level priors [54].
G Block Stacking Intuitive Example
To provide greater intuition for SCALE and the causal skill learning problem, present the Height- Height experiment (Fig. 5): a simple example in the block stacking domain that can be easily visu- alized.
Task and policy description. The Height-Heeigh experiment contains 3 blocks: source block; 2) target block; and obstructing block between the source and target block. As in Sec. 6. the task is to place the source block on top of the target block. The same controller is used in Sec. 6.1, which is parameterized by e I Specifically, each parameter of the controller is defined as follows:
1. Aa:: the distance the source block is moved along the world coordinate frame's is +r-axis once it picked up.
2. dAy: the distance the source block is moved along the world coordinate frame's +y-axis once it is picked up.
15

---

Algorithm 1: SCALE: SKILLS FROM CAUSAL LEARNING
Input: causal reasoning engine W, context space C, controller TT, reward solved threshold Rg. number of samples n, skill policy function f., numher of evaluations m, skill timestep
Tj
Initialize: skills IC +- 0
11 Collect training data
(D....., Dž) f- SKILLTRAINDATAAWWCCC mi
1/ Train skills
for j 1 to & do
(xx, YY AY, D) +- D
// Train DGR
P REDUCEDIMS( (
9 TRAINDGR(PX)
// Train Policy
AX REDUCEDIMS(
(Axt, 'Y+) < DGRINLIERS(? 9 Ax, Dx Py)
Tu (f,, Axt yY+)
π & nıTT
// Train Preconditions
("x,, "YY) + EVALUATEPOLICY(WWCC. C, m)
Pre t TRAINPRECONDIION("X "Y, Rs))
// Set Termination Conditions
3 4- Т₁
1/ Construct Skill
K = (T, Pre, 3, Φ)
end
Result: learned skills KC
3. f...: : the distance the source block lifted (moved along the world coordinate frame's
+ z-axis) during pick-up motion.
4. 0azyi the distance the source block descends (moved along the world coordinate frame's
-z-axis) during the set-down motion.
The controller behaves as follows:
Move robot end-effector to source block and grasp it.
2. Lift up the source block according to 0az..
3. Move the source block in the x-y plane according policy parameters Os. and Oay
4. Set down the source block according Oazz
5. Ungrasp the source block.
The context space of this experiment just variables, h and h facilitating 2-dimensional visual- izations. For greater clarity, we refer to block properties by whether they belong to the target block ()) the obstructing block (o), instead of their index (as in Sec. 6.1). For this experiment, only linear approaches are considered.
Skill learning results. The SCALE results for the Height- experiment are shown in Tab. and Fig. 6. The dataset size for skill learning was 569 samples, from an original size of 581. The remaining 12 samples consisted of CREST results that occurred rarely (2.07%), and thus they were used for skill learning. Additionally, Fig. 7 visualizes the policy parameters of the dataset. Two primary behaviors were leamned: free motion (Kfree), and obstructed motion (Kobset)). These behaviors emerge because of the causal relationships between context variables.
16

---

Algorithm 2: SKILLTRAINDATA
Input: causal reasoning engine W, context space C, controller иy, reward solved threshold Rg. number of samples n, local region fraction f. minimum dataset size d
Initialize: batch dataset DB 0
// Collect training data
for i to do
c SAMPLEVALIDSCENNWW.CO
(0, R2) TRYTOSOLVETASK(MM m))
Task:Solved 6- R2 > Rg
if TaskSoveed then
- CREST(W, c, T. 0, R fC)
D - CREST(W,CC e, Ti, 0, C)
DB F (c,0,,,))
end end
// Separate into k skill datasets (D....., D&) S d)
Result: skill training data (D1....,Dy,
obstr.
target
block
block
height
height
Figure 5: The Height-Hleight experiment is intuitive example for SCALE in the block stacking domain. In this experiment, only two context variables can vary: the height (z-dimension) of the obstructing block (he) and the height of the target block (h₂). All others variables (e.g.. features of the source block) do not change throughout this experiment.
When the obstructing block shorter than the target block (i.e., by > ha), then the obstructing block height safely be ignored in the robot action (thus, ho & A for Kfree). This is reflected by the values of fa=,. and f in Fig. 7. In the region corresponding to Kffree, 0.... varies linearly with respect to the target block height, but not with the obstructing block height. Thus, Os is generally The result that the robot tends to lift the block to value that depends on the target block height, and no set-down motion (0sz,) is needed.
However, when the obstructing block is taller than the target block (i.e., h ho). the obstructing block's 's geometry interferes with the robot's 's motion, and the robot must take this into account when
17

---

taking action. Specifically, the robot must first lift the source block over the obstructing block. After moves laterally, the robot must descend to set the source block down; dropping the block would typically lead to inadequate reward to solve the task. Because both the heights of these blocks are needed to perform this action, (hr C. A for Kohatt In Fig. 7, the effect of ht, appears in the aas.. parameter values, where the variation in the Kobstr region arises because of needing lift above the obstructing block height, ha (and thus, this parameter no longer depends on h₂). However, for Daa both h4 and ht are needed, as the distances the robot descends through 0 OZ:,, arises from the difference between h2 and ho Thus, the gradient here shows components for both h4 and ho-
These two skills encode the two distinct data generating processes within this context space. These processes - the reason why the data are generated certain way - fundamentally depend on whether the obstructing block is shorter or taller than the target block. Whether condition holds for given context requires the value of both of the blocks heights, so both block heights needed to define each skill's data generating region (i.e., {he, ho} C D).
Note that neither skill can robustly solve the entire task space (55.63% for Kfrre and 57.50% for Kobatr). However, when using the entire library Kии = {K, free, Kobatrr (Tab. 7), the success rate becomes 100.00%, with each skill being selected at approximately 50% chance (49.38% for Kfree= and 50.62% for Kohhstr). This is expected because the relationship hi ho holds for half of the context space and Krree should be used, whereas h2 < ho (Kobatr) holds for the other half.
Table 6: Skills 1 that were discovered for the Height experiment. A and D are the variables used for the skill's policy and DGR, respectively. Data is the quantity of data used for each skill (from a batch dataset of 581 samples, 569 samples were used to train skills). These samples used train a linear policy (Bayesian ridge regression) using the features from variables in A. Task Solve % is the rate of task solves over the entire context space using only that skill.
Skill
A
D
Data
Task Solve
Krree Kohstr
Thet (he,ho)
The, ho (he, ho)
253 (43.55%)
55.63% (178)
316 (54.39%)
57.50% (184)
Baseline comparisons. In addition to scale-lin, Tab. 7 shows comparisons against several baselines. The "monopolicy" baselines monolithic policies (without skills). The "-sk" and "-all" suffixes denote whether the monolithic policy uses the same data as the SCALE library ("-sk", 569 samples) or the entire batch dataset ("-all", 581 samples). Given the similar amount of data, it is unsurprising that monopolicy-lin- and monopolicy-lin-al are essentially the same up to the stochasticity of the simulator (+2%).. Note that, unlike in Sec. 6.1 and Sec. 6.2, CREST monopolicy baselines are not examined in this experiment; they are functionally equivalent to the monopolicy approaches because the most common CREST result {he, ho), which is the same as the entire context space used for the monopolicy baselines.
As shown in Tab. 7, the skill library obtained by SCALE vastly outperforms the baselines, provid- ing task evaluation performance similar to that of a ground truth policy. This outcome is possible because SCALE learns underlying regions of similar causal structure within the data, whereas mono- lithic policies ignore such structure. As shown in Fig. 7c-7d, this domain is nonlinear, but can be represented by two smaller linear regions (hę > ho and h < h2). Learning to regress to both re- gions with monolithic linear policy is not possible, but SCALE can solve this domain with separate linear skills, one per region.
Summary. Our approach for SCALE - learning skills that encode distinct causal processes empowers the robot with a diversity of specialized behaviors to use, depending on the context. Generalization of the context space can be achieved then through the composition of these behaviors, rather than attempting learn a monolithic skill policy that can capture the entire variation. In this example, two skills each with a linear policy is sufficient for generalization with SCALE, whereas monolithic approach would require nonlinear policy.
18

---

1.0
0.000
01000
0.9
nbatr: block height (m)
0.05
obstr: black height (m)
0.00
0.8
00
0.7
0.050
0.030
0.6
0.5
0.045
0.045
0.1
0.040
0.040
60
0.3
0.2
0.035
0,055
6
0.1
0 80
0.0
0.035
0.040
0.045
0.050
0.035
0.059
0:035
0.040
0,045
0.050
0.055
0.060
target block height (mn)
target block hoight (m)
(a)
(b)
0.060
0.060
88
obett: block height (m)
obstr. bloek height (m)
0.056
00
0.7
0.050-4
0.6
0..5
0.045
0.04b
0.000
80
0.3
0040
0.2
0,035
0.1
0p
0.040
6.045
0.050
0,055
0.060
0.0
0.035
0.040
0.045
0.050
0.55
0.060
target block height (m)
target block height (m)
(c)
(d)
Figure 6: SCALE results for the Feight-Heiigt experiment. Two skills were found: Kfrre (free block motion), stylized blue with rectangular markers, and Kobstr (obstructed block motion), stylized in orange with diamond markers. (a) Learned data generating regions. Each datapoint is result from CREST. Datapoints that are crossed out are considered outliers and not used for training the policy for that skill. (b-c) Preconditions for Krree and Kobstr. respectively. The black line is the decision boundary for the prediction of whether the task would or would not be solved with that skill. Note that each skill's DGR generally falls within the positive precondition boundary. Train- ing and test data for learning the preconditions are indicated by circle and thin diamond markers, respectively. Datapoints that result in a different prediction than observed are crossed out. (d) Task evaluation when using the skill library of each datapoint indicate which skill was selected for completing the task based on the skill pre- conditions (i.e., the skill with the highest probability of success). Note that the separation between selecting K free and Kobstr is consistent with each skills' underlying precondition and DGR. Data- points that were not solved by the chosen skill are crossed out.
} to solve the task. The marker and color
H Additional Details for Block Stacking Experiment
This appendix provides greater information for the block stacking experiment first presented in Sec. 6.1.
Context. Note that the block vertical position = v is not part of the context, cases where the scene can be initialized into steady state condition. Thus, 2 = -h, + hgg
only consider
- Reward function. The reward function for the task is R = - a, L-yL-a,, a.e a-.--aa- ayd where R = 10
is a bonus term obtained when the block is successfully stacked, L is the total end-effector path of the robot (a,z 1), e the 1.2 norm error between the source block at the time of release and the goal (a. 1), and d is the distance the source block travels between the point it was ungrasped to its final position (a = 1). The task is considered solved if the final reward Ry exceeds solved threshold Rş- = 5.
19

---

0.000
0.000
-0.008
0.301
abatr: blookk height (m]
0,055
0.003
obstr: block height (ml
0.55
0.290
0.003
0.050
0.296
0.002
0.045
0.000
0.055
0.292
0.002
0.040
0.000
+0.003
0.287
0.005
0.285
40.036
0.283
0.030
0.035
Largel block height (in)
Largel block height (mi
(a)
(b)
0.020
0.062
obsttr black height (m)
0.056
0.06
block gerttt
0,022
0.030
0.035
0.050
-0014
0.045
0.045
-0.000
0040
0.007
0.05
0.000
0.03
0.000
0.035
0.000
0.045
0.050
0.055
0.060
0.035
0,40
0,015
0,050
0.055
0.066
target hnight [m)
target block height (m)
(c)
(d)
Figure 7: Policy parameters for the Height-Height experiment (shown as interpolated across the 569 dataset samples to better visualize the gradients). The units of the parameters are in meters. The parameters Arr (a) and Aay (b) are generally constant they are unaffected by the variation con- text variables. The notable variations occur in fa.,, (c) and a (d). Specifically, the relationship changes whether the obstructing block is taller or shorter than the target block (above below the he Nt = 0 line, respectively).
Table 7: Task evaluation results for using the skill library KH# for the block stacking task. Ctrl. is the approach control (skills one monolithic policy). Fn. Cl. is the approach's 's function class. Linear approaches use Bayesian ridge regression. Task Solve % is the rate of task solves over the entire context space using the approach. Methods within +2% (the stochasticity of the simulator) of the best approach are bold. A is the quantity input variables used for the approach's 's policy. Data is the amount of training data used for the approach. A ground truth policy is also shown, using all context variables and additional domain knowledge.
Approach scale-lin (ours)
Ctrl.
Fn. Cl.
Task Solve %
A
Data
2 skills
Linear
100.00 % (320)
1/1
569
monopolicy-line lin-sk
policy
Linear
64.06% (205)
2
569
monopolicy-ling -in--al
policy Linear
62, 19% (199)
2
581
ground-truthe truth-policy
policy
Nonlin.
100.00% (320)
SCALE skill selection. In all SCALE approaches, the skills were complementary; using the entire skill library afforded greater coverage (greater task solve rate) than any single skill alone. For scale- lin, the skill selection distribution was almost even between K₁ (43.28%) and K2 (56.72%), with K3 never being chosen. The skill K3 is dominated by the other two skills for this task, but K₂ could nonetheless be useful for a different downstream task. Empirically, was observed that Ki chosen for shorter target block heights, whereas K2, used elsewhere (see Fig. 8). In the nonlinear case, only 10p was selected.
20

---

0.09
0.08
target block height (m)
0.07
0.06
0.05
0.04
0.04
0.05
0.06
0.07
0.09
source block height (m)
Figure Skill selection for the scale- lin approach for the block stacking task. Skill K₁ generally selected when h2 is short, whereas taller h2 values perform better with K2 because hg S A. Skill K3 dominated by the other two skills and is not selected. Datapoints that were not solved are crossed out.
Policy and training data ablations. We provide additional experiments to investigate the effect of different policy functions and training data usage. The results are shown in Tab. 8, which ex- pands Tab. 2. For the linear function class, we conduct experiments with Bayesian ridge regression (B. ridge reg.) and ordinary least squares linear regression (OLS lin. reg.). Both linear policy fune- tions used an intercept term and were trained using unnormalized data. For the nonlinear function class, we conduct experiments with a multilayer perceptron (MLP, 16x 16x16 architecture using ReLU activations) and support vector regression with a radial basis function (RBF) kernel (SVR (RBF)). The nonlinear policy functions were trained with normalized data. Additionally, we present ablations in terms of training data usage. Methods ending in "-all" use the entire batch dataset. For the full-dimensional monopolicy approaches, the "-sk" ablation uses same training data as used by the SCALE skills (340 samples). For the CREST baselines, the "-subs" ablation randomly downse- lects the batch dataset to the same number of samples used by SCALE (340 samples).
In general, we see that SCALE generally outperforms the full- dill-mmmniii monopolicy methods and matches the performance of the CREST baselines in most (but not all) cases. We see that increasing the amount of training data available for the baselines usually improves performance. For the linear function class, both Bayesian ridge regression and ordinary least squares linear regression produced capable approaches. For ordinary least squares linear regression, SCALE (scale-lin-ols) outper- forms the full-dimensional monopolicy on a sample-adjusted basis. For the nonlinear function class, the performance of approaches was lower overall. The similarity in performance of scalc-nonlin to the full-data CREST baseline is strictly due to sample size; on a sample-adjusted basis, scale- nonlin is slightly more performant. However, for support vector regression with a RBF kernel, although SCALE maale-nonlin-svr-bbe exceeds the performance of the full-dimensional monopol- icy approaches, the CREST approaches perform more strongly (although modestly overall). Thus, sensitivity for the nonlinear function class the selection of policy function used for this task.
21

---

Task evaluation results for using the skill library Kitocks for the block stacking task for variety of policy functions and training data ablations. This table expands upon Tab. 2. Ctrl. is the approach control (skills or one monolithic policy). Fn. CL. is the approach's s function class. P. Fn. the policy function. Task Solve % is the rate of task solves over the entire context space using the approach. Methods within +2% (the stochasticity of the simulator) of the best approach are bold. |A| is the quantity of input variables used for the approach's policy. Data is the amount of training data used for the approach. ground truth policy is also shown, using all context variables and additional domain knowledge. The abbreviation "mp" stands for monopolicy. Fn.
Table 8:
Approach
Ctrl.
Fn. CI.
Task Solve %
A
Data
scale-lin (ours)
3 skills
Linear
B. ridge reg.
90.49% (276)
4/5/6
340
monopolicy-lin-sk
1 policy
Linear
B. ridge reg.
80.72% (247)
36
340
monopolicy-lin-al
policy
Linear
B. ridge reg.
85.95% (263)
36
585
crooooioyyyylicy -iniccy----uuss
policy
Linear
ridge reg.
89.87 (275)
340
crest mecsst-oooollicy-in--..
policy
Linear
B. ridge reg.
89.87 % (275)
585
linellin-ols (ours)
3 skills
Linear
OLS lin. reg.
90.85% (278)
4/5/6
340
monnopolicylinnoos--
policy
Linear
OLS lin. reg.
83.33% (255)
36
340
monopolicy-Iin -in-oolii--i-----allllll lin-ols-all
policy
Linear
OLS lin. reg.
90.16% (275)
585
crest- crest-monopolicy -at-ooooiicc-i--oo--sbbbe -in--os--uus
policy
Linear
OLS lin. reg.
90.52% (277)
340
policy
Linear
OLS lin. reg.
90.20% (276)
585
scale-nonlin (ours)
skills
Nonlin.
MLP
63.40% (194)
4/5/6
340
monopolicy-nonlin--
policy
Nonlin.
MLP
1.31% (4)
340
moonopllic-noonli-alll
policy
Nonlin.
MLP
10.13% (31)
36
585
crest-monooolicy-noniii conpoiicc--oonnnuusee subs
policy
Nonlin.
MLP
58.17% (178)
5
340
policy
Nonlin.
MLP
60.78% (186)
585
scale-nonlin-svee (ours)
skills
Nonlin.
SVR (RBF)
19.61% (60)
4/S/6
340
policy
Nonlin.
SVR (RBF)
1.63% (5)
36
340
monopolicy-nonlin-vvee
nonlin-svr-rbi svr-rbf-all all
policy
Nonlin.
SVR (RBF)
7.19% (22)
36
585
crest-mp- nonlin svr-rbf-subs
policy
Nonlin.
SVR (RBF)
41.64% (127)
340
crest-mp-none nonlin-soenneeeee rvr-rbf-all
policy
Noulin.
SVR (RBF)
56.86 (174)
585
ground truth truth-policy
policy
Nonlin.
95.75% (293)
I Sim-to-Real Block Stacking Experiment
In this appendix, demonstrate that the skills learned by SCALE are suitable for sim-to-real trans- fer. As skills are constructed using only the relevant causal variables, this is a form of structural sim-to-real transfer. For this experiment, we evaluate the skill library Kotookks for real block stack- ing domain with a Franka Emika Panda robot manipulator (Fig. 2c). This experiment is generally similar to task evaluation simulation, except with a smaller subset of the context space. We assess the SCALE approaches, scale-lin and scale-nonlin, against their monopolicy counterparts. We only consider the "-all" monopolicy approaches. as they were generally better performing.
I.1 Experimental Setup
For this experiment, smaller subset of the context space is varied, as compared to the variation across the entire context space tested in Tab. 2. From pool of 20 blocks, 5 were randomly chosen to be used for each experimental trial. The 20 blocks consisted of variations of 10 different colors and 2 different heights (5.7 em or 7.6 cm). The length and width of the blocks were 4.2 cm. The randomly chosen blocks were placed into the Panda robot workspace and randomly shuffled, producing variation in block z-position, y-position, and orientation. The table height h was determined from manual measurement and was not varied for this experiment.
Perception. An Intel RealSense camera mounted to the robot wrist provided RGB-D perception of the x-position, y- positioon, and orientation of the blocks in the workspace. A depth observation was collected by commanding the robot above the workspace. This point cloud was then processed to yield five clusters via hidden point removal [55], RANSAC-based table plane fitting, and density- based clustering using DBSCAN [56]. Averaging the colors within each cluster yielded the block color. A least-squares optimization procedure fit cuboid of known length and width to each cluster, yielding the position and orientation of the blocks. Block height provided by manual input
22

---

because of inaccuracies with estimation from depth alone. The camera extrinsics were obtained via computer-aided design models of the Panda robot and wrist mount, which were confirmed via manual measurement. The camera intrinsics were used as directly reported by the camera.
Control. The FrankaPy library [57] is used to provide impedance-based control of the Panda robot.
1.2 Experimental Results
Table 9 presents the results. For each function class, the skill library learned by SCALE outperforms the full-dimensional monopolicy baseline and is generally comparable to or slightly outcompetes the CREST monopolicy baseline. The ground truth policy matched the linear SCALE approach and is only slightly better than the nonlinear SCALE approach. Compared to the task solve rate in simula- tion (Tab. 2), scale-lin performed consistently, and scale-nonlin had slightly better performance. All baseline approaches generally matched their evaluation in simulation, except for monopolicy-lin which had marked degradation. This may arise from domain differences between simulation and reality. Full-dimensional approaches are more susceptible to domain shifts due to their reliance the entire context space (all 36 variables), whereas SCALE approaches are compressed, using only minimal subset. Error was only loosely correlated with task solve rate, and likely explains the poor performance of monopolicy-nonlin--e
Even though their errors were similar, it was observed that monopolicy-lin--a tended to underpredict the height needed to clear the target block as compared to scale-lin. This caused the target block to pushed away from where it should have been for the goal position, leading to block stacking failures.
For both scale-lin and scale-nonlin, skill K2 was always chosen, its precondition was on average greater than that of the other skills. Specifically, for scale-lin, the average preconditions were 58.88% for J,, 75.77% for 12, and 36.99% for X3-- As the block heights used were only 5.7 em and 7.6 cm, it is reasonable to expect that skill K₂ would have been chosen more for shorter target block heights (per Fig. 8). For scale-nonlin, the average preconditions were Ky; 20.17%, K2: 51.84%, K33: 1.21%.
Table 9: Sim-to-real evaluation results for using the skill library Kitocks for real block stacking domain. Table columns are as described in Tab. 2. Task Solve % is the rate successful block stacks. Error is the mean error (+1 standard deviation) in meters between the block position when the block is ungrasped and the goal position determined at the beginning of the trial. Ctrl.
Approach
Fn. Cl.
Task Solve %
Error
A
scale-lin (ours)
3 skills
Linear
90.00% (9)
0.010 = 0.003
4/5/6
monopolicy-lin-
policy Linear
50.00% (5)
0.008 0.003
36
crest-monopolicy-linn..
policy
Linear
90.00 % (9)
0.004 + 0.001
5
scale-nonlin (ours)
skills
Nonlinear
80.00 % (8)
0.007 0.002
4/5/6
conopolicy-nonlinge
policy
Nonlinear
10.00% (1)
0.093 + 0.040
36
messt-oonoooicy-ooe
nonlinll nonlin-all
policy
Nonlinear
70.00% (7)
0.013 0.012
5
ground-truth-
policy
Nonlinear
90.00% (9)
0.002 = 0.003
J Skill Library Use in a Downstream Task: Stacking a Block Tower
To demonstrate the utility of re-using skills learned by SCALE, follow-up experiment is conducted wherein the skill library KMMokk is used for a task in which it was not specifically trained: stacking block tower (Fig. 9). This long-horizon task can be decomposed into a number of sequential actions that must be performed correctly, so an approach that can capture the essence of large problem and re-use smaller, modular components should perform best. Moreover, we do not perform any additional training or fine-tuning; we intentionally use the skills off-training data to test their gener- alization capability. This is a challenging task: in addition to the fong-horizon precision involved, the skills being evaluated increasingly out-of-distribution each step, the effective block heights increase beyond what is seen in training.
23

---

(e) Figure 9: The block tower task. As previously, five blocks are initially available to the robot. However, after each stack attempt, the task does not reset. Instead, the block enumeration changes, so that the previous source block becomes the new target block. This happens four times, after which the task resets. The robot must complete each of the four individual steps successfully, as failure in any step renders the entire block tower task failure. (a) Initial task scene. (b- d) Successful block stacks for intermediate attempts. (e) A successfully stacked block tower.
(a)
(b)
(C)
(d)
Experimental setup. For this experiment, we assume that the robot has access to a planner and ad- ditional domain knowledge as part of this downstream task. We assume that the robot understands that at any step, the target block should be adjusted in the following manner. First, the target block's s r- and y-position should be substituted with the bottom-most block's z- and y-position. Then, the target block's 's height should be substituted with the sum of all heights of the previous blocks, plus small offset (1.5 cm). Effectively, this can be seen as treating each new step as stacking upon one, increasingly taller block. We leave the development of such planner that can provide this additional information for future work, but suffices for this experiment that this information is available.
Block tower results. Table 10 shows the results for stacking the block tower. For this experiment, we use the same linear and nonlinear approaches and baselines from Sec. 6.1, including the training data ablations. Included is ground truth policy with access to oracle information.
Overall, we see that the scale-lin approach does best for stacking tower with five blocks, although a notable gap exists between the ground truth policy. However, a block tower success rate of 48.29% unreasonable, given that even the ground truth policy fails almost 30% of the time. The linear approaches are all comparable for the first stacking step, and for the second step with a NB = 3 tall tower, three baseline methods slightly outperform scale-lin. However, for the last two steps, baseline approaches become markedly less performant, leading to scale-lin emerging as the best overall approach despite modest performance in an absolute sense. Fach step requires successively greater extrapolation out of the training data, so an approach that can capture the smaller process well should perform best, assuming that this process also holds outside the training data. For the case of the block tower, this is generally true, so the skills learned by scale-lin are best suited for this downstream task despite the challenge of generalization yet-unseen data.
For the nonlinear function class, performance across all approaches suffers beyond the first stacking step, where the CREST baselines outperform scale-nonlin. The challenge of extrapolation for non- linear functions is evident here; the best linear approach for each step was better performing than any nonlinear approach (and markedly so for taller towers). Thus, duito--distribution generalization not observed for any nonlinear approach, whereas scale-lin exhibits modest performance in this area.
For SCALE approaches, the skill selection rate is intriguing. The skill K,, does not contain the target block height, which is likely why it was only selected during the first block stack attempts. However, K2 continues to demonstrate its robustness, as it was used for all remaining block stack attempts in the linear case and for all attempts in the nonlinear case. Its inclusion of target block height in Ax, is in fact the reason this skill can extrapolate to taller towers. Like K2, Kg also contains the block height, but this skill was generally dominated, and thus it is not surprising it not selected.
24

---

In summary, in addition to the benefits of SCALE described previously for task learning, the capa- bility for SCALE to learn smaller, modular skills is evident in this experiment. Although out-of- distribution generalization was not observed in the nonlinear function class, we see that in principle SCALE does offer these benefits under certain conditions, such as in the linear case. We suggest that this aspect of causal learning often overlooked for experiments that only concern single- task learning. However, the benefits of modularity become advantageous for re-using behaviors for downstream tasks at a later time in the robot's operational lifetime.
Table 10: Results for re-using learned behaviors in a representative downstream task: stacking block tower. The task solve percentage is shown for stacking a tower of at least Ng blocks tall. The sequence is executed in one attempt, so a fully stacked tower (Ng = 5) requires successful block stacking attempts. Methods within +2% (the stochasticity of the simulator) of the best approach at each step are bold. For SCALE approaches, the skill selection rate at each step (not cumulative) is also shown. The abbreviation "mp" stands for monopolicy.
Approach
NB 2
NB 33
NB 1
NB
scale-lin (ours)
92.20% (272)
80.73% (222)
65.23% (167)
48.29% (113)
K1
15.59% (46)
0.00% (0)
0.00% (0)
0.00% (0)
K2
84.07% (248)
100.00% (275)
100.00% (256)
100.00% (234)
Kis
0.34% (1)
0.00% (0)
0.00% (0)
0.00% (0)
monopolicy-line
93.22% (275)
87.23% (239)
55.08% (141)
1 1.27% (3)
monopolicy-liae lina-lll
93.56% (276)
76.36% (210)
2.33% (6)
0.00% (0)
geet-mpplin--supe
93.20% (274)
85.40 % (234)
5.84% (15)
0.00% (0)
crest-mp-lin-all
93.92% (278)
85.51 (236)
5.84% (15)
0.00% (0)
scale-nonlin (ours)
67.46% (199)
2.55% (7)
0.00% (0)
0.00% (0)
K
0.00% (0)
0.00% (0)
0.00% (0)
0.00% (0)
K2
100.00% (295)
100.00% (275)
100.00% (256)
100.00% (235)
K
0.00% (0)
0.00% (0)
0.00% (0)
0.00% (0)
monopolicy- sonnoyy--oonn----n
2.72%
0.00% (0)
0.00% (0)
0.00% (0)
monopolicy conlecy--oonnnnnan all
11.86% (35)
0.00% (0)
0.00% (0)
0.00% (0)
crest-mp-nonlin-suus
84.75% % (2.50)
27.37% (75)
0.78% (2)
0.00% (0)
crest-mp-nomiinall
75.59% (223)
11.31% (31)
0.00% (0)
0.00% (0)
ground-truth--ee policy
96.25% (282)
90.48% (247)
83.14% (212)
69.96% (163)
K
Additional Details for Sensorless Peg-in-Hole Insertion Experiment
This appendix serves provide greater detail for the peg insertion experiment that described in Sec. 6.2.
Reward function. Our reward function consists of two terms: 1) a penalty hased on the Euclidean distance of the peg from the hole, and 2) a bonus of 10 for successful insertion. We also add a regularization term based on the norm of the policy parameters. The task is considered solved if the final reward Ry exceeds solved threshold Rs- = 8.
SCALE skill K1. Unlike the other skills in Kpey that were discovered by SCALE, skill Kr, has an empty set of relevant variables. This is surprising as it is difficult to solve this task reliably without taking the help of one of the walls, in which case the wall should show up as a relevant variable. However, we observed that Ic₃ actually localizes against walls instead of just 1. Hence, when SCALE intervenes on any one of the two walls, the skill is still able to complete the assembly by taking advantage of the other wall. In other words, our assumption that the context space is disentangled does not hold in this case which leads to this erroneous relevant variable set. However, the precondition would limit where this skill would be applied, as skills Ka-s are generally more performant.
SCALE skill selection. For scale-lin, skills K₂ (48.44%) and Kss (51.56%) were chosen nearly equally. Conversely, the skill selection was more distributed for the nonlinear case: Kg: 46.48%, Kg: 35.16% K₁: 3.91%, KG: 14.45%. For both approaches, 1r, not chosen it was dominated by the other skills.
25

---

Task evaluation results for using the skill library Kpeg for peg insertion for a variety of policy functions and training data ablations. This table expands upon Tab. 4. Ctrl. is the approach control (skills or one monolithic policy). Fn. CL. is the approach's function class. P. Fn. is the policy function. Task Solve % is the rate of task solves over the entire context space using the approach. Methods within +2% (the stochasticity of the simulator) of the best approach are bold. |A] the
Table 11:
quantity of input variables used for the approach's ' policy. Data is the amount of training data used for the approach. The abbreviation "mp" stands for monopolicy.
Approach
Ctrl.
Fn. Cl.
P. Fn.
Task Solve %
A
Data
scale-lin (ours)
5 skills
Linear
ridge reg-
96.48% % (247)
0/1/1/1/1
168
monopolicy-Iinn
1 policy
Linear
ridge reg.
67.19% (172)
168
monopolicy- lin-alll
policy
Linear
B. ridge reg.
62.50% (160)
210
crest-monopolicy -iny--ii--uuuu lin-subs
1 policy
Linear
B. ridge
66.80% (171)
168
crest-monopolicy lin-all
policy
Linear
B. ridge reg.
62.89% (161)
210
linoll (ours)
5 skills
Linear
OLS lin. reg.
96.88% (248)
0/1/1/1/1
168
monopolicy- -in-oss-kk
1 policy
Linear
OLS lin. reg.
50.78% (130)
168
monopolicy- sinoy---i-------aa
policy
Linear
OLS lin. reg.
67.19% (172)
210
crest-monppolicy wiw.oooollic---ooo---saee----e-- lin-ols-subs
policy
Linear
OLS lin. reg.
63.67% (163)
168
mossso-onopllcy---ooo all
policy
Linear
OLS lin. reg.
60.55% (155)
210
scale-nonlin (ours)
5 skills
Nonlin.
MLP
88.67 % (227)
0/1/1/1/1
168
caaopolicy-nonlinn-oe
1 policy
Nonlin.
MLP
18.36% (47)
8
168
monopolicy-nonie
policy
Nonlin.
MLP
12.89% (33)
210
crest-mowt cest-moonpollcy -nnnin--uubee
policy
Nonlin.
MLP
56.64% (145)
168
crest- cossst-ooopolicy-non nonlin--ninn
policy
Nonlin.
MLP
55.47% (142)
210
scale-nonlin-swr-i (ours)
5 skills
Nonlin.
SVR (RBF)
94.53% (242)
0/1/1/1/1
168
monopolicy -onnin--ve wr-rrf-sk
I policy
Nonlin.
SVR (RBF)
53.52% (137)
8
168
monopolicy-nonn sirii---------al--- all
policy
Nonlin.
SVR (RBF)
58.20% (149)
210
crest-mp cettmppnooniin-ee nonlin-svr-rbf -vr-ri------b-ss reest-mp-nonlin-sv---f-ll
1 policy
Nonlin.
SVR (RBF)
57.81% (148)
168
policy
Nonlin.
SVR (RBF)
60.94% (156)
210
Policy and training data ablations. As with the block stacking domain, we conducted experi- ments with several policy functions and training data ablations. Table 11 details the experimental results, which expand upon Tab. 4. In the linear function class, two policy functions were investi- gated: Bayesian ridge regression (B. ridge reg.) and ordinary least squares linear regression (OLS lin. reg.). An intercept term was used for both approaches, and the training data were unmormal- ized. In the nonlinear function class, experiments were conducted with a multilayer perceptron (MLP, 16x16xX architecture using ReLU activations) and support vector regression with radial basis function (RBF) kernel (SVR (RBF)). For the nonlinear policy functions, the training data were normalized. Methods with the "-all" suffix use the entire batch dataset. For the full-dimensional mo- nopolicy approaches, the "-sk" suffix indicates that the same training data as SCALE was used (168 samples). The "-subs" suffix for the CREST baselines denotes that the batch dataset was randomly downselected to the same number of samples used by SCALE (168 samples).
Overall, we observe that the SCALE skills are highly performant across function class and policy function type. Moreover, SCALE significantly outperforms both the full-dimensional monopolicy approaches and the CREST baselines. Indeed, SCALE exceeds the performance of the baselines by around 30% for each policy function type. The success of SCALE is attributed to capturing the four modes in the data - localizing against each of the four walls - found by exploiting the underlying causal structure. The baselines, which are agnostic to such structure, do not leverage this property and are therefore limited. Unlike in the block stacking domain, we see that the effect of training data size does not necessarily yield an increase in performance for the baseline approaches.
L Sensorless Peg-in-Hole Insertion: Domain Shift Experiment
We evaluate the generalization capability of SCALE by evaluating it under domain shift. All tasks generated by uniformly sampling the relative position of the center of each wall with respect to the hole from given range. The ranges used to generate the training and test tasks are specified in
26

---

Tab. 12. We transfer all the policies zero-shot to the test distribution. However, we do re-learn the preconditions of the scale-lin policies for the test distribution.
The evaluation results are summarized in Tab. 13. All approaches witness sharp drop in perfor- guaranteed to be feasible and (b) the ranges
mance. This is expected as (a) the test tasks used generate the test task more than double those used in training. However, our multi-skill approach scale-lin performs much better than the baselines. This highlights a key benefit of learning multiple skills. A skill may perform well on the training distribution, but it can be rendered invalid due to an unforeseen domain shift. Having a repertoire of different skills allows the robot to still complete the task by switching to a different skill. This makes our multi-skill approach more robust than single single-skill approaches.
Table 12: Training and test distributions of the domain shift experiment in the sensorless peg-in- hole domain. The relative position of the center each of the 4 walls is uniformly sampled from the given (min, max) range. The ranges used to generate test tasks are more than double the ranges used generate training tasks in the domain shift experiment. All values are in meters.
Train
Test
x-min
2-max
y-min
y-max
r-min
r-max
y-min
y-max
Wall 1
0.01
0.05
-0.02
0.02
-0.04
0.10
-0.07
0,07
Wall 2
-0.02
0.02
-0.05
-0.01
-0.07
0.07
-0.10
0.07
Wall 3
-0.02
0.02
0.01
0.05
-0.07
0.07
-0.04
0.10
Wall 4
-0.05
-0.01
-0.02
0.02
-0.10
-0.04
-0.07
0.07
Table 13:
Task evaluation results under domain shift for sensorless peg-in-hole insertion. We eval- uate only linear policies as nonlinear policies perform worse in this domain. Table columns are as described in Tab. 4.
Approach scale-lin (ours) monopolicy-line ain-
Ctrl.
Fn. CL.
Task Solve %
A
5 skills
Linear
64.84%
0/1/1/1/1
policy
Linear
44.92%
8
crest-monopolicy-ling lin-all
policy
Linear
39.83%
I
M A Primer on Causality
For readers who are unfamiliar with causality, this appendix serves as a gentle "on-ramp" for under- standing SCALE.
What's 's data generating process? A data generating process (DGP) is dynamical process that generates data in physical system. The process is usually described by variables that characterize the system. Consider the following examples: turning a light switch on a lamp to illuminate the lightbulb; inserting a car key into an ignition and turning the starter to start a vehicle; rain showers causing rainfall. These examples can be considered data generating processes if system variables were instrumented, such as instrumenting rain gauge measure rainfall.
What's 's Structural Causal Model? A Structural Causal Model (SCM) [25, 26] is representation of a data generating process. Usually, the SCM consists of variables of a system, graph (which is usually directed with no eycles) that describes how the variables depend on each other, and functions that describe how each variable characterized based on that variable's s causes. These functions are also called structural equations or functional equations, and each function will have its own noise variable. Noise variables (also referred to exogenous variables) are generally jointly independent.
What's 's an example of an SCM, and how can it be used? Consider the following example of SCM e;:
X = Nx
. Y = 2X + Ny
27

---

Here, X and Y are variables of our SCM, and Nx and Ny are the noise terms. This SCM can also be characterized by its underlying graph, where X - Y because X a cause of Y. For this example, consider that Nx and Ny are (independently) sampled from the uniform distribution from -10 to +10. Then, if Nx = and Ny -3, then by the mechanies of the SCM, X 2 and therefore Y =
We now introduce the concept of an intervention, where we set the value of variable to be particular value (usually regardless of its causes or noise variables), holding all other variables equal. We can formalize this using the do operator [25]. Thus, an intervention do = 5) means that no matter what value Nx. X, or Ny take, Y = 5. In the previous example, under this intervention, (and not 1). This type of intervention called "hard" since
Nx = 2, then X 2, but Y induces structural change; other intervention types are possible, such "soft" interventions where the functional equation of a variable changes (but not its parents).
What's 's the difference between a DGP and an SCM? In the case where the SCM captures the DGP exactly, there is no difference. However, often times we wish to learn the data generating process, and the SCM encodes the knowledge of the DGP that is currently known. these cases, the SCM an approximation of the underlying DGP in the physical world.
In SCALE, what's 's the Data Generating Region and how does it differ from DGP? The Data Gen- erating Region (DGR) introduced by SCALE provides locality to the data generating process. Con- sider a physical system where SCM e co-exists with the following new SCM, E2:
X = 3Nx
2 = -
However, is also noticed that according to a fourth variable A, when A < O, E₁ applies, whereas when > 0, C2 applics. The condition where these causal models apply is equivalent to how the DGR specifies where particular skills are defined in the context space. Note that X, Y. and Z not needed define where the models apply (only A). A learning algorithm could use all four variables specify where the models apply, but minimal, compressed representation only requires one (A).
Moreover, Z and A are not needed to specify the mechanics of C₁ (similarly, Y and A for C2). This is similar to how SCALE learns which variables of the context space to use for modeling the skill policy. Even though a learner could potentially use all four variables it knows about, irrelevant variables are not needed in minimal representation.
I' ' interested in learning more about causality. Where should I start? There are many important and useful textbooks in this area. We use Pearlian causality and SCMs as the basis of our formalism, so we recommend the reader reviewing Causality (Pearl 2009, 2nd edition) [25], in particular, chapters 1-3. Then, we recommend the reader reviewing Elements of Causal Inference (Peters, Janzing, and Schölkopf, 2017) [26], in particular, chapters and 6.
28


Reviews:
[{"review_id": "a37b6NT0ZOb", "reviewer": "Reviewer_zy8g", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method for causal learning in simulation by performing sample-based causal interventions (randomizing the environment variables) measuring task success. The learned causal relationship is captured by a dimensionality reduction matrix that filters out unimportant environment variables. Experiments validate that the method is able to extract a very compact skill library from large a number of environments.", "summary_of_recommendation": "Valid approach for learning causal features and improves policy interpretability. However, readability is a weakness and the paper could benefit from a rewrite. I will re-assess if the authors could address the issues in the review.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Can you provide more clarity on D_{A_k} and D_{D_k} in section 4?\n2. Can you provide more clarity on figure 1? Currently there is no explanation in the main text on the block illustrations, making it very hard to interpret.\n3. Can you comment on the sample complexity of random scene generation, and in general, how many scenes need to be generated for causal structures to be found?\n4. Can you incorporate the toy example in the main text as a running example? It can serve as a running example and help ground all the notations in the method section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nPro:\n1. Causal learning in the face of distractor features is an important and often understudied task in robotics. While most existing methods rely on hand-engineering of causal features, this paper provides a feasible way of learning such relationships automatically in simulation. \n2. The method offers more interpretability to robot learning, as it is able to categorize learned policies into different skills, and pinpoint the relevant features.\n\n\nCons:\n1. Clarity on main skill formulation. The main method sections (section 4 and 5) are very heavy on notations. Yet, many of the notations are only loosely defined, without a concrete example to ground them. More specifically, how are D_{A_k} and D_{D_k} computed and what are their differences? While I was able to infer them from the tables in the experiment section, I was left confused after reading section 4.1 and 4.2. \n2. Clarity on figures. Would be great if the authors could elaborate on the causal feature selection section in Figure 1. More specifically, why in certain cases, the red block is kept in the final policy-relevant parameters and in other cases removed? Why is the final example failed? What is the implication of the final example? While some of these are illustrated in appendix E, it would be good to provide explanations in the main text.\n3. Sample Complexity. The causal structure is learned through preforming random scene generation (section 5.1) and policy learning in all generated scenes. However, for more complicated causal structures — for instance, multi-step block stacking with complex rules — this would likely result in very  large number of scenes in order for the causal structure to be extracted. I wonder if the authors could shed any light on the sample complexity of batch data generation.\n4. Visualization of the learned skills. While the experiments on learning causal policies in block stacking and peg insertion are very essential parts of explaining the main approach, the paper glosses over the resultant skills (k1-kn) in the main text, leaving it very difficult to interpret table 1 and 2 without going back and forth between the main text and the appendix.  The authors could consider moving the explanatory toy example in the main paper, and additionally, offer more insight into the individual skills in each experiment.\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Ea63Uf1T7B", "reviewer": "Reviewer_yTw8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "SCALE is a representation learning algorithm for skills which are (1) generalizable, (2) account for multi-modality of robot skills. To achieve (1), they use causal feature selection to extract out a relevant subset of state variables, used for that skill policy's input. To achieve (2), they define a mode to be a set of action parameters which share the same relevant state variables from causal discovery. SCALE proposed to learn skill policies by partitioning a collected dataset by action relevant variable set and data generating region relevant variable set. The final system is a tri-level policy:\n - In the first level, a learned pre-condition classifier is used to pick one of regional compressed options.\n - In the second level, the chosen regional compressed option's upper level policy infers the low-level (third-level) policy's parameters given the compressed context.", "summary_of_recommendation": "The paper, in its current state, is inaccessible to people outside of causality and robotics. I think with some rewriting, many more could learn from this paper's findings.\n\n**Update**\n\nMy main concern was the original writing and presentation was poor and inaccessible to the wider CoRL community beyond causality robotics researchers. The authors have significantly improved the writing, sufficiently addressed all my questions, and explained clearly some of the limitations/scope of their approach. I think their attempt at defining skills is interesting and insightful, although clearly limited by its dependency on an interven-able state representation. My final recommendation is a `weak accept`.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- The data generation process in 5.1 remains unclear to me. It sounds either extremely expensive (e.g., requiring an inner reinforcement learning loop) or it just another way of saying its a high-level action space that can take care of the low-level heavy lifting.\n - There is nothing in the framework which constrains or assumes much about $\\pi_{l}$ or $\\theta$. How well would data generation perform when $\\pi_{l}$ is a joint position controller (e.g. pure pursuit) and $\\theta$ is the robot's joint space? Is the careful design of this action space crucial for good performance?\n - The DGR is trying to estimate the support for the policy's context distribution (L144), which can be used to filter out data relevant for policy training of that specific skill. Since the union of the DGRs are used to filter out the dataset, my interpretation is that this accounts for cases where the skill policy's context distribution support is more complicated (e.g. as in Figure 8 in the appendix)?\n - Once you train a pre-condition, it seems like DGRs are not necessary? It seems like the motivation is to ensure the data is in distribution (L175-L176), but if its not and $\\pi_{uk}$ outputs bad $\\theta$ then the pre-condition classifier will just give low probabilities for those contexts?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\n - Video is informative, and was crucial for me to fully understand the approach. However it could still be improved by defining terms from CREST before using them.\n\n\n## Weakness\n \n### Accessibility to the Robot Learning Community\n\nIn its current state, the paper is not educational to those who are not fully familiar with the causal learning literature (the majority of robot learning researchers).\nThis is because many causal learning technical terms are used without informative examples, and assumptions deviating from the typical robot learning paper are not correctly spelled out.\nBecause this makes it challenging for many to understand the approach, appreciate its contributions, and grasp its limitations, people will be less likely to enter the space, extend the current frameworks, and relax existing assumptions in causal robotics.\nThis will create a greater divide between causality robotic researchers and the rest of the robotics learning community.\nThis reason is the key reason for my strong rejection.\n\nTo address this, I recommend re-writing with the goal of educating the average robot learning researcher.\nI think if the paper do this in a self-contained manner and the figures are clear, many more can learn from and be excited by its contributions.\nFor instance, I find Figure 1 and its caption confusing because it contains symbols and uncommon words like data-generating regions and instance solvers without definitions.\nFurther, without watching the supplementary video, I would not have understood that the colored rectangles were blocks, not bar charts.\n\nI understand that causality & robotics is still nascent, therefore many assumptions needs to be relaxed before it can be widely applied to robotic systems.\nThe limitations has listed all of these major challenges of integrating causality into robotics.\nHowever, before causality & robotics become as mainstream as control & planning or neural networks, I think interleving these limitations in the paper writing will help the average reader more.\n\n\n### Skill Definitions Depends on States\n\nSCALE's formulation of clustering action parameters into different skills by the set of causally-relevant state variables is a interesting attempt at defining what a skill is. It is a useful and interesting application of CREST. \nHowever, this approach relies on an interprettable, controllable state representation (an requirement inheritted from CREST's context interventions).\n\nThe current approach can't be applied to general robotic systems (e.g, deformable/fluid manipulation), and causal representation learning's goal of learning disentangled features do not imply they will be interven-able.\nThis means SCALE's skill learning formulation will either remain unapplicable to domains for which state-estimation is difficult or require an even stronger notion than causal disentangle feature to be achieved.\n\n### Minor Suggestion\n\nMany notation is introduced in 3.1 that are never used again. I recommend removing them for clarity sake. In other cases, I would recommend making sure all definitions are in 3.1. For instance, pre-conditions is used throughout the paper, yet only defined in L135.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "AKwFQi5rJEv", "reviewer": "Reviewer_7iCw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the authors introduce a novel method for skill discovery, which explicitly manipulates task parameters, referred to as the 'context.' This method employs simulators to observe the resultant effects of such manipulation and identify skills by the relevant subspace of context.\nThe authors evaluate the method's efficacy using various manipulation tasks, where it demonstrates an improved success rate in comparison to other ablation methods.\nFurthermore, the authors showcase the applicability of the framework with a real-world robot.", "summary_of_recommendation": "The manuscript at hand builds upon previous context compression methods to extract skills in an option-oriented manner. The proposed framework, validated through multiple robotics tasks, certainly sparks interest. However, the paper falls short in providing a comprehensive discussion and comparison with other skill discovery methods.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. Can you elaborate on the differences between DGR training and precondition training in skill training? Is it possible to use the trained \\mathcal{D} to determine preconditions?\n2. Could you provide insights on how your method compares with other self-supervised skill discovery methods such as those referenced in [22, 23]?\n3. Why do you categorize this algorithm as a self-supervised learning method?\n4. What is the rationale behind using the entire context space to identify D_k?\n5. Is there potential to extend this work to deal with pixel observations or long-horizon tasks?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n\n1. **Interpretable Skills Discovery**: A major strength of the paper is its capability to discover interpretable skills.\n2. **Context Space Subspace Discovery**: Another commendable feature is the ability to discover a subspace in the context space, making the framework more transferable to real-world applications.\n\n**Weaknesses**\n\n1. **Dependency on Prior Work**: The manuscript relies heavily on the understanding of CREST [33], making it difficult for readers not familiar with this work to grasp the concepts and significance fully. Reducing this dependency would make the paper more accessible.\n2. **Lack of Comparative Discussion**: The paper falls short in discussing how the proposed method stacks up against other skill discovery methods.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "8mCdjYqXpd", "reviewer": "Reviewer_H8Po", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work takes the options framework and provides an option discovery and skill-learning framework. It uses factor-specific interventions to determine which causal factors are relevant for reward in a given state. It then fits a one-class algorithm to the dimension-reduced factor-involving data to get the initiation set, regresses a policy and uses evaluation data to determine the preconditions.", "summary_of_recommendation": "The work introduces some interesting ideas, but the presentation is somewhat terse and the experiments do not provide sufficient evidence that this method will work generally, though the algorithm is described as being general.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Is there a clear overview of the design decisions and ways that the algorithm is implemented in practice, along with justification for these components?\n\nCan ablatives be added to assess these design decisions?\n\nIs it possible to propagate reward through factors to get relevance or are skills only possible when directly connected to reward in the causal graph?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "While the overall intuition is relatively understandable, the details of the algorithm itself are sparse enough to the overall algorithm is difficult to follow. For example, It is not clear how the initiation set is differentiated from the initiation set. It is not clear how the policies are cleared using regression, nor what reward function is being used to train them. It is also not clear why a matrix is used to represent feature selection, instead of x_i notation, because it is unclear if a true dimensionality reduction method would be applicable to this case.\nFigure 1 should help provide some of the details, but it does not really provide much beyond the same high-level overview.\n\nThe experiments provide some ambiguous results. In particular, in the block stacking task it is not clear why none of the obstacle blocks are used in any of the skills. If they are completely irrelevant to the task, then it seems to limit ---wouldn't the context specificity be useful for detecting the particular times when the distractor blocks actually inhibit motion? If not, then it is not clear why a policy composed of skills that are agnostic to these objects performs better than baselines. \n\nBy using final reward for the interventions, it seems like this could be a significant limiting factor. If a causal chain is required to achieve high reward (ex: An agent has to grab a stick to push a block to get reward), then interventions on one factor alone would give no information about the final reward. \n\nThis work could benefit from being more upfront about assumptions. Not only is the decoupled simulator assumption quite strong, but also the one-class initiation set, the regression-trained policies and the ability to find clear separations in the dataset, and the assumption f a Factorized dataset in the first place. \n\nSome discussion of the causal relationships and the nature of the interventions would provide valuable context that this work currently lacks.\n\nThe work could benefit from additional ablatives to determine the effectiveness of some of the components: The policy regression instead of RL, the SVM one-class model, and the counterfactual testing region. \n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The general consensus is that this is a good step towards an important topic (causality in robotics) with a reasonable set of experiments. However, the reviewers are generally confused about when the proposed methodology can be applied. The updated paper improved the understanding and all reviewers granted weak acceptance. 

I believe the proposed method makes an explicit assumption that the world can be modeled as a *factored* MDP where a discrete set of variables (ex. gravity) is used to define the model and the intervention is done using these variables. Perhaps stating this clearly and giving examples of interventions would mitigate the issue regarding the applicability of the method.
Decision: Accept (Poster)