Granger Causal Interaction Skill Chains
Authors: Caleb Chuck
Kevin Black
Aditya Arjun
Yuke Zhu
Scott Niekum
Venue: Accepted by TMLR ()

Abstract: Reinforcement Learning (RL) has demonstrated promising results in learning policies for complex tasks, but it often suffers from low sample efficiency and limited transferability. Hierarchical RL (HRL) methods aim to address the difficulty of learning long-horizon tasks by decomposing policies into skills, abstracting states, and reusing skills in new tasks. However, many HRL methods require some initial task success to discover useful skills, which paradoxically may be very unlikely without access to useful skills. On the other hand, reward-free HRL methods often need to learn far too many skills to achieve proper coverage in high-dimensional domains. In contrast, we introduce the Chain of Interaction Skills (COInS) algorithm, which focuses on \textit{controllability} in factored domains to identify a small number of task-agnostic skills that allow for a high degree of control of the factored state. COInS uses learned detectors to identify interactions between state factors and then trains a chain of skills to control each of these factors successively. We evaluate COInS on a robotic pushing task with obstacles—a challenging domain where other RL and HRL methods fall short. We also demonstrate the transferability of skills learned by COInS, using variants of Breakout, a common RL benchmark, and show 2-3x improvement in both sample efficiency and final performance compared to standard RL baselines.

Kevin Black Published in Transactions on Machine Learning Research (03/2024)
Granger Causal Interaction Skill Chains
Caleb Chuck
calebo@es.ulee edu
University of Teras at Austin
University of California Berkeley
Aditya Arjun University of Texos Austin
Yuke Zhu
University of Teras Austin
Scott Nickum
University of Massachusetts Amherst
Reviewed on OpenReview: https: /ttps://opense optpr/view.ee.e... net/ forum? 4A2KQyoun]
Abstract
Reinforcement Learning (RL) has demonstrated promising results in learning policies for complex tasks, but it often suffers from low sample efficiency and limited transferability. Hicrarchical RL (HRL) methods aim to address the difficulty learning long-horizon tasks by decomposing policies into skills, abstracting states, and reusing skills in new tasks. How- ever, many HRT methods require some initial task success to discover useful skills, which paradoxically may be very unlikely without access useful skills. On the other hand, reward-free HRL methods often need to learn far too many skills to achieve proper cover- age in high-dimensional domains. In contrast, (COInS) algorithm, which focuses on controllability in factored domains to identify small number of (ask-agnostic skills that still permit a high degree of control. COInS uses learned detectors to identify interactions between state factors and then trains a chain of skills to control cach of these factors successively. We evaluate COInS on a robotic pushing task with obstacles-a a challenging domain where other RL and HRL methods fall short. We also demonstrate the transferability of skills learned by COTnS, using variants of Breakout, common RL benchmark, and show 2-3x improvement in both sample efficiency and final performance compared to standard RL basclines.
introduce the Chain of Interaction Skills
1 Introduction
Reinforcement learning (RL) methods have shown promise learning complex policies from experiences on various tasks, from weather balloon navigation (Bellemare et al., 2020) to Starcraft (Vinyals et al., 2019). Nevertheless, they often struggle with high data requirements and brittle generalization in their learned controllers (Nguyen & La 2019). One promising avenue to improve sample efficiency and generalization is by incorporating hierarchical skills.
To address the limitations of vanilla RL, hicrarchical RL (HRL) methods exploit temporal and state abstrac- tions. Standard "flat" RL methods learn monolithic policy, while HRL constructs a temporal hierarchy in which higher-level policies invoke lower-level policies (also called skills or options) and the lowest-level skills invoke primitive actions. This structure offers three major benefits: First, skills can represent temporally extended behaviors, decomposing long-norizon tasks into a sequence shorter temporal segments. Second, appropriate selection of skills fosters useful state abstractions, reducing the state-space complexity by

---

Published in Transactions on Machine Learning Research (03/2024)
Reward
Initial
Intermediate
Reward
Optimization
Optimization
Govls
Actions
Ball
Block
Actions
Bouncing
Control
Gouls
Final
Actions
Paddle
Gripper Control
control
Actions
Primitive Actions
Primitive Actions
Figure 1: Left: The chain of COInS goal-based skills for Breakout, from primitive actions to final reward optimization. The goal space of the skills for one factor is the action space of the next factor controlling skill in the chain. COInS uses Granger-causal tests to detect interactions and construct edges between pairs of factors and their corresponding skills. Right: The Robot Pushing domain with negative reward regions. The objective to push the block (red) from a random start position to random goal (green) while avoiding the negative regions (shown in yellow, -2 reward), which are generated randomly 15 grid spaces 5x5 grid over the workspace.
merging states through skill-based characteristics (e.g., skill preconditions and/or effects). Third, the skills can be reused across different tasks to enable multi-task learning and fast adaptation (Kroemer et al. 2021
Skills can be acquired through either reward- based methods reward-free methods, each with challenges. Using reward for skill learning (Konidaris & Barto 2009 Levy et al. 2019b) limits these HRL algorithms to update only when the agent achieves meaningful progress. For sparse-reward, long-horizon tasks, this often means that the learner flounders indefinitely until the first success is observed. On the other hand, many reward- free methods utilize state-covering statistics (Eysenbach et al., 2019). They struggle to learn high- dimensional environments because learning skills that cover the entire state space is inefficient. Furthermore, even after learning those skills, the number of skills can end up excessively large for high-level controller. While handful of methods suggest ways distinguish Machado et al. 2017), this prioritization is open research area.
state another (Simşek & Barto 2008
While HRL methods have shown promising results, the existing limitations lead to low sample efficiency and hinder transfcrability. One reason for this is that noither novelty nor reward-based methods explicitly identify key states that mediate controllability. In many euvirouments, these states, characterized by object interactions, bottleneck the high reward states: without reaching and performing precise control of these interaction states, the policy cannot perform well. For example, consider the game of Breakout (Figure [ left) with the game state decomposed into paddle, ball, and blocks. A strong policy in this domain control each of these objects. However, learning this requires identification and credit assignment of rare and intermittent ball bounces. Similarly in the robot pushing scenario (Figure Tright), the agent must learn complex manipulation of the block. Intuitively, manipulating interactions gives a policy a high degree of control, while failure to produce interactions often results in poor performance.
This intuition drawn from the observation that human behavior dirceted causing factor intcractions. "factor" to describe either collection of features like the state of an object non-state young infants, often
In this work, factors such actions and rewards. Unlike reward-bas based skills, humans, exhibit exploratory behavior without obvious top-down rewards. Unlike novelty-based skills, this behavior is not state-covering but directed towards causing particular effects between factors (Rochat 1989). The framework of directed behavior causing one factor to influence another decouples a small subset of interacting factors from the combinatorial state space of all factors.
The proposed Chain of Interaction Skills (COInS) algorithm identifies pairwise interactions factored state spaces to learn hierarchy of factor-controlling skills. COTnS extends the Granger-causal test. (Granger)
2

---

Published in Transactions on Machine Learning Research (03/2024)
Tank et al. 2021 with learned forward dynamics models and state-specific interactions. It uses Granger-causal relationships between factors to construct chain of skills. Intuitively, Granger-causality determines if adding information about a new factor, which we refer to as the parent factor, is useful for prodicting another factor, the "target" factor (see Figure 2 Then, goal-conditioned reinforcement learning learns interaactio--controllnng skills.
1969
Using interactions, COInS automatically discovers hierarchy of skills starting with primitive actions. These skills control progressively more difficult-o--control factors using already acquired skills, ending with reward optimization. In Breakout, COInS discovers the intuitive sequence of skills described in Figure [ We demonstrate that the COInS algorithm not only Iearns the skills efficiently in the original version of the task but that the learned skills can transfer to different task variants even those where the reward structure makes learning from scratch difficult. We show sample efficient learning and transfer in variations of the game Breakout and a Robot Pushing environment with high variation of obstacles, which are challenging domains for conventional RL agents.
Our work has three main contributions:
1) An unsupervised method of detecting interactions via an adapted Granger Causality criterion using learned forward dynamics models;
2) A skill-chain learning algorithm (COInS) driven by these discovered interactions; and
3) Empirical results demonstrating how COInS, a proof-of-concept instantiation of interaction-guudeed goal- based HRL, can sample efficiently learn transferable, high-performance policies in domains where skills controlling pairwise factors can achieve high performance.
2 Related Work
The literature on skill learning is often divided into two categories: reward-free (task-agnostic) skills and reward-based (lask-specifie) skills. Reward-free skills often utilize values derived from state visitation, such as information-theoredd
skills (Eysenbach et al. 2019). to ensure that the skills cover as many states as possible while still being able to distinguish skills apart. Other reward-frec skill learning uses tools such as the information boutleneck (Kim al., 2021), transition Lagrangian (Machado al. 2017) and dynamics (Sharma et al. 2020). While COTnS also learns skills based on state information, prioritizes controlling interaction-related states instead of arbitrary coverage. Interactions bear some resemblance to curiosity [Burda et al. 2018 Savinov et al. 2019) and surprise-based reward-free exploration methods (Berseth et al. 2021 in how it uses model diserepancy to set goals. COInS introduces Granger-causal interaction detection to learn reward-ffiie free skills that are not state-covering but capture the space of controllable factor relationships.
Another line of HRL methods backpropagate information from extrinsic rewards or goals (Barto & Mahade- van 2003). Hierarchical Actor Critic (Levy et al., 2019b a) uses goal-based rewards in locomotion tasks and bears similarities to COInS. Unlike COInS, it uses complete states as goals and does not scale to complex object manipulation where there is a combinatorial explosion of configurations. Other hindsight goal-based hierarchical methods propagate information from the true reward signal through distance metrics (Ren et al. 2019) or imitation learning (Gupta et al.) 2019 neural architectures (Bacon et al.] 2017). option switch- ing (termination) cost (Harb et al. 2018), or variational inference (Haarnoja et al. 2018a). However these struggle with the same sparse, long-horizon reward issues. COTnS shares similarities to causal dynamics learning (Wang et al.) 2022 Seitzer et al. 2021), , though these methods are non- hieraieealliaaal and focus on general causal relationships instcad of interaction events. COInS builds skill chain similar to HyPE (Chuck et al. 2020), which uses strong inductive biases with changepoint- interactions. COInS employs fewer biases using a learned interaction detector and goal-based RL. In summary, COInS presents novel approach to defining skills for hierarchical reinforcement learning leveraging interactions as a key mechanism.
3
Overview and Background
3.1 Overview
This work is motivated by an observation derived from the game Breakout: the agent Breakout typically executes a lengthy series of 20 100 actions between the bounces of the ball. Identifying the precise sequence of actions that correlates with the reward from hitting a block often necessitates large amount
3

---

Published in Transactions on Machine Learning Research (03/2024)
of data, and the resulting policy may become overly tailored to the specific environment settings. However, by breaking down Breakout. into sequence of intermediate interactions firsst between the actions and the paddle's 's position, then between the paddle and the ball, and finally between the ball and the block (similar to Figure [ - the task becomes significantly simpler. This work introduces a general strategy for docomposing control into intermediate, factor-based interactions that can be applied to a wide range of tasks, instantiated as Chain of Interaction Skills (COInS).
At each level of the hierarchy, COInS constructs a temporally abstracted MDP where the agent selects different interactions. For example, consider a version of Breakout where the action space consists of ball angles, and the state transitions occur at ball bounces. This abstraction trivializes the basic challenge of hitting blocks and transfers to more challenging versions of Breakout where the agent must strike specific blocks for positive reward.
This work aims to get this abstracted MDP by (1) identifying interactions, (2) training skills to produce those interactions, (3) building interaction skill chains from the bottom up, beginning with primitive actions and progressing to more complex factor relationships. In this section we formalize the chain of skills used in this abstracted MDP, starting from the Factored Markov Decision Process (FMDP). In Section 4.2 we introduce the Granger- cansal method for detecting interactions, in Section 4.4 we formalize skill chain learning, and in Section 4.5 we describe how the skill chain is iteratively constructed.
3.2
Factored Markov Decision Processes
A Markov Decision Process (MDP), is described by tuple E (S,p.r, Aprim. 7). S is the state space, S. is the initial state space, S is particular state, and S denotes the random voriable for state. We use script notation (8) lo represent sets, uppercase (S) for random variables, and boldface (s) for vectors. The primitive action space Aprim. which can be either discrete or continuous, comprises the available actions
prrim E Aprim In this work, we extend the factored MDP (FMDP) formulation (Boutilier et al. 1999 Sigand
& Buffet 2013 where the state space is factorized into n factors: S = S1 x... x Sn. Each factor represents a distinct component of the overall state, such as the state of an individual object. Fach state factor s; E S
is represented by fixed length vector of real-valued features, denoted with sy[k] for the feature indexed by l of factor s,, For this work, we assume that the factorization is predefined, as this problem is being actively investigated in vision (Voigtlaender et al. 2019 Kirillov et al., 2023) and robotics (Lin et al. 2020 but is not the focus of this work. Our goal is to identify controllable state factors and the Granger-causal relationships between controllable factors.
The transition function is a probability distribution over S' the next state, given s the current state, and aprim the current primitive action. is defined as p x Aprim X P(-|S s, Aprim = aprim). In the FMDP formulation, the transition function is represented with a Bayesian network connecting state factors S; and aprim at the current time step with the next state factors s!. Previous works utilize the sparsity of this network for efficient RL. In this work we extend this to state-conditional sparsity: in certain states, the connectivity is especially sparse, ex. when a ball is moving through free space in Breakout
The reward function is a mapping from states, primitive actions, and next states to real-valued rewards: r : S x Aprim x S + R. A policy is a function mapping states to the probability distribution over actions, such that π S + P(|SS = s). A trajectory is a length T sequence of state action pairs:
T (s) a prim (1) a prim..... s( a The probability of trajectory under a particular policy is (T) P(S. s (0)) II T(a paap (s(*) ((s(+++))))))))))) (()) apdim) where S. is the initial state distribution. We represent the trajectory distribution given a particular policy as p(x). The objective of RL is to learn a policy that maximizes the expected return, the 7-discounted sum of rewards, for y E [0, 1). defined as:
(1)
rel[m] Eypp(r)
Σ rr(ss a0))
3.3 Skills
This work builds on the skills/options framework described by the semi-MDP (sMDP) (Sutton al. 1999) formulation. A skill or option in sMDP is defined by the tuple w (I, mw, Φ). A_ is the action space
4

---

Published in Transactions on Machine Learning Research (03/2024)
of option w, and E A_ could be primitive action or different skill. I C S is the initiation set where skill can be started. COInS uses the common assumption that the initiation set for a skill covers the entire state space, though future work could revisit this assumption. The skill policy π : :: + P(-SS = s) defines the behavior of the skill as a conditional distribution over A.. Φ: S - [0, 1] is the termination function, which indicates the probability of skill ending in a particular state. In this work we use deterministic terminations @ @:S + (0, 1), which terminate if a condition (eg. interaction) is met.
This work also builds from goal-based skills, that parameterize the skill policy and termination function by goals cu C.. The termination function is thus binary check of proximity to the goal @(s) ||c, s|| and the skill policy is augmented to be π: S + P(-|S = s. C. = cw), that is, parameterized by the goal cw. Pactored goal-based skills utilize factorization in the goals, so the goal space is a subset of a particular factor: C. S,, and the termination function is factor proximity: b(s,c,) = ||cw - s;ll <c
A chain of factored goal-based skills combines levels of skills (Konidaris Barto 2009) a sequence {wo.... ..o...,,,,]] where the goals of wi is the action space of wi-1. Cu,, C S is the space of goals used for (s C, Tw, (s. a, Cu,), and is the same as the action space of the skill policy WH+1 Awits C.,, This structure exploits compositionality to reduce the effective horizon of upper-level skills. This work efficiently learns, unsupervised, a chain of factorized goal-based skills that can achieve high performance with good sample efficiency and transfer to similar tasks.
4 Chain of Interaction Skills
The Chain of Interaction Skills (COInS) algorithm is an unsupervised HRL algorithm that constructs a chain of factored goal-based skills using interactions identified using adapted Granger Causality. In this section, we first identify how interactions are identified using Granger Causality, then how the signal can be used to define a skill, and finally how goal-based RL methods can be used to learn that skill. COInS iteratively learns pairwise skills where one factor (the "source factor" with state Sa S,) is controlled to produce interaction in the "target factor" with state E Sg. si denotes the next state of the target factor.
4.1 Granger Causality
The Granger cansal test (Granger 1969 is a hypothesis test to determine if the state of the source factor s , - is useful in predicting the target factor , Without the Markov assumption, it utilizes history window w, the number of past steps needed identify an interaction. The test compares the null hypothesis, which is the passive autoregressive (self- seelf-predicting affine model of b, where 0 are parameters Icarned by affine regression to best fit s
...
(1) with noise cb:
Das,G (s) (s
(t-w)
. (-"') 000_
Σ
0
cb
Then, the hypothesized relationship with signal is modeled by the active distribution with learned than v/a
ar
s 1)- (t-w) m ( ......! s
s (- (.0 vo) = -/000 +
J Σ
Ea
(2)
Signal is said to Granger-cause (G-cause) signal b if the regression model in Equation 2 met yields statistically significant improvement in prediction over the autoregressive distribution in Equation [
4.2 Granger Causal Factor Tests
Granger Causality with dynamics models in FMDPss introduces two differences: First, transition dynamics in FMDPs are not always affine, so we replace the affine models with a function approximator, following recent work by Tank al. (2021). We model pass. m with neural conditional Gaussian models. These models use factored states as input and output the conditional mean μ and diagonal variance Σ of normal distribution N N(o, E) over Sy -e. predicting the next target state. Second, the Markov property allows
5

---

Published in Transactions on Machine Learning Research (03/2024)
to collapse the history window to just the last state: 1. Combined, we describe the autoregressive and pair-regressive distributions with passive model maas and active model m
m (st: 0) S + (N, 2) m 86: v) S x S. N(,, E)
(3)
(4)
We call matt the "active model" since m can capture when the source factor affects the dynamics of the larget PS S.). We call mpas the "passive model" because it uses only S, to predict Sy- Figure 2 illustrates the passive and active Granger models in the paddle-ball case. Using dataset D of state, action, next state tuples (s, Aprim: s') E DD collected during skill learning as described in Section 4.5] we train mPss and mact as variational models to maximize the log-likelihood of the observed data, where m=))ss denotes the probability of next factored state s/ under the modeled distribution:
pas (8a: s/; 0) == log m PSssss 0) [+(]
(5)
lact (Sa: Sb,, s/1; vb) = log m act (Sa, So; [s({
(6)
The Markov-Grange (MG) score identifies if a source factor affects target factor according to dataset D:
SDMa(D)= = max V
Σ
(Sc Sb, Sb, (6: vb) (b)1),
max
Σ
epas s6:
(7)
D
D
(B..8())D
A high score indicates that P(S # P(S)[S)), or that the source factor (a) generally exerts an effect on the target factor (b), within the context of dataset D. However, this test identifies general relationships instead of interactions it determines two factors are related overall, but not the specific event when they relate an interaction. In the next section, we utilize the active and passive models identify interactions.
4.3 Detecting Interactions
Again, an MG-causal test descríbes whether source a could be useful when predicting target general, but does not detect the event where the source state S, interacts with the target state S at particulor transition (s., sp, s²,). This distinction distinguishes general causality (the former) from actual causality (the latter). Actual cause is useful in many domains where an MG-causal source and target factors interact in only a few states. For example, in Breakout the paddle's influence on the ball is MG-causal but the paddle only affects certain transitions ball-paddle bounces. In a robot block-pushing domain, while the gripper to block is MGGcausal, the gripper only interacts when it is directly pushing the block. We are often particularly interested in those particular states where the ball bounces off of the paddle, and the gripper pushes the block. In this work, "interaction" is directional from source to target.
To detect these interactions extend the intuition of Granger-causality: an interaction is when the source factor is specifically useful for predicting the dynamics of the target factor. The interaction detector ha, compares the active model met and passive model mass log- ligeliisss in state transition Sa, S6, s to detect interactions. With as defined in Equations 5 5
(a, Sb, s/f 0) = (Cact Sa, s/1 v) > Eact) A (epas s/; < Epas)
(8)
Here, Caet and Cpas are hyperparameters basod on the environment's 's inherent stochasticity; additional discus- sion is in Appendix 0 The first condition ensures mctt predicts the next state with sufficiently high accuracy (log- likeliiel higher than Exet), and the second ensures that mas predicts with low accuracy (log-liket) ligeliiel lower than (pas). This infers interactions using the following logic: states with low passive log-likelihood (St, s/,) < fpas could arise from three cases: (1) when state transitions are highly stochastic, or (2) some other factor # a is interacting with b or (3) the source factor interacts with the target factor. In cases (1) and (2), f(ss. Sa, 1)) will be low. Figure includes qualitative illustration in Breakout.
When searching for MG-causal relationships, the MG score (Equation 7 can often be small when inter- actions are extremely rare. In Breakout, paddle-ball interactions only oceur once every 1000 time steps

---

Published in Transactions on Machine Learning Research (03/2024)
Paddle-Ban Ball Granger-causal Models
Paddle-Ball Interactions
Legend
Prediction:
Factored State
Inputs
Outputs
Active Prediction:
Actions
Passive Model
Shared Prediction:
Case
Case 2
Case 3
Paddle
Active Madel
Ball
Blocks
No Interaction 8oth Accurate
Interaction Passive inaccurate,
No Interaction
Both Inaccurate
Active accurate
Figure 2: Left: An illustration of the active and passive model imputs and outputs for the paddle-ball Granger models. Right: Three possible-inerraction states. In Case 1, COInS predicts interaction because both the passive and active models predict accuralely. In Case 2, the active model predicts accurately using paddle information, but the passive model does not, indicating a paddle-bal ball interaction. Case 3 is not a paddle-ball interaction since both the passive and active models predict poorly.
when taking random actions, summary statistics can fail to efficiently capture this relationship. Sim- ilarly, gripper-block interactions in Robot Pushing occur every ~5500 lime steps. To account for this, we adjust the MG-score by modulating it with the interactions, giving the interaction score Sey. For mm = E S, ED ha,b(Sa. Sb, shi 0) as the number of detected interactions in D:
Scr(Daa,b)
Σ
ha, Sb. s/: 1/, 0)) (fact 8b, s/s; 1)) Epas (86, s/5; 0))
(9)
Niat
4.4 Using Interactions in HRL
Now that we have the means to detect interactions between two factors, our next stop is to train a new skill that leverages existing source factor skills to learn to control a target factor. This requires three design choices: selecting the source factor, identifying the target factor, and learning control over the target factor. For the first choice, COInS uses the most recently acquired skill's corresponding factor as the source factor, creating chain, but future work can extend this to tree or DAG hierarchies.
4.4.1 Determining Target Factor Control
COInS selects the target factor for learning by evaluating Sey Scy(D, a, for all uncontrolled 8. a indexes the most recently controllable factor, and indexes factor yet. to be controlled by COInS. The target factor b is then chosen based on the highest interaction score, as long as it is higher than minimum threshold esi:
b min Sc; Sc, s.t. Sc, ess
(10)
However, just because one factor (a) can affect the dynamics of another (b) does not mean that every feature of Sb is controllable. Certain features St [k] of the larget factor Sa might not be directly controllable. For example Breakout, the paddle affects the ball dynamics, but within one timestep of an interaction, the ball position cannot be directly controllod. Trying to control both the ball position and velocity simultancously would result in many infeasible goals. To address this, COInS alters the goals by determining control mask 1h (0, K where K is the number of features in factor St-
Controllable features are identified based on this intuition: if the feature value differs significantly from the passive prediction after an interaction with the source factor when evaluated over the dataset D, the source factor likely caused the change. u(mpas(s6; ; f)) denotes the mean of the passive prediction and Nua as defined in Section 4.3 the masks (mL) are defined with:
[1
Nint En Em. ha (Sa. Sb. s/6 2.00) [||[[k] p( 0))]1 cn otherwise,
176 [k]
(11)

---

Published in Transactions on Machine Learning Research (03/2024)
For examples of 76-masks and post-interaction states in evaluated domains, see Section In abuse of notation, we define C. == Th,S,, to be the goal space for our options, even though it is space of masked target states Sb- If the size of the unique masked post-interaction state set C. = unique(() is sufficiently small (IC&) < ndise) then the goal space is treated as discrcte, otherwise, is treated as continuous.
4.4.2 Training Factor Control Skills
With the target factor and space of goals over that target factor determined, COInS can now perform goal-based reinforcement lcarning with factored interaction goals. A specific goal from the masked target factor space G₁ S C. parameterizes the skill. The termination function of the skill's temporal extension is represented with binary indicator function. This function indicates when the goal is reached 17,St Cb co-occurring with a detected interaction (S, Sb, s/3 v, 0). With mask T)b, source and target states Say Sb s/ and goal target features Cg:
(Sa, s/y cb) = (1 ha,b Sh, s/3 th 0)) A I1uSs - -coll < co
0 otherwise.
(12)
The associated reward function is R(s, s') = (s Sb, S/: Ca) rew, i.e. 0 at the goal and -e new elsewhere. By applying goal-based RL to this reward, an iteration of COInS learns to produce particular pairwise interaction (ex. ball bounce), that achieves a goal (ex. particular ball angle). The skill uses the goals of the last learned skill as a temporally extended action space.
The goal-based RL used train our goal-reaching skills utilizes hindsight experience replay (Andrychowicz et al. 2017), with Rainbow (Hessel et al. 2018) for discrete action spaces, and Soft Actor-Critic (Haarnoja et al.
2018b for contimoous action spaces. COInS uses only Sa: St instead of all of as input to the policy to accelerate learning by having the agent attend only to the minimum necessary aspects of the state. A skill is considered learned when the goal-reaching reward remains unchanged after Ncomplete training steps.
4.5 Building the Skill Chain
The previous sections detail the procedure for learning single link in the chain of skills. COInS iteratively adds links one skill at time until no new links are found through Equation 10 COInS starts with primitive actions as the only source factor and takes random actions. Each iteration: 1) learns active and passive models for each uncontrolled factor where the source is the last factor COInS learned control over. 2) Finds
the next. larget. factor using Equation 10 3) Learns goal-based policy with the reward function described in Section 4.4.2 based on Equation 12 Adds factor to the chain and restarts the loop.
Each step of the loop is executed autonomously, and the data used to evaluate the interaction test is collected from the RL training process and thus self- gelf-generated. The loop ends when there are more high-score factors to control: Sc, < ESI, where csi is hyperparamcter. Algorithm box 9 describes the algorithm. The benefits of COInS come from three sources: 1) COInS breaks down complex enviromments into series of transferable, intuitive skills automatically. 2) each skill is individually simpler than the overall task, resulting in improved sample efficiency. 3) By incorporating the causal test for interactions, the skills exploit sparse dynamic relationships for control.
Input: FMDP Environment E Initialize Chain of Interaction Skills W with single option set wprim- Assign primitive actions Data D = DU random policy data
repeat
Optimize likelihoods (Equations) 5 6 on D to get the passive and active models mass m met and interaction detector ha,b (Equation g for all uncontrolled
Interaction Detector:
Interaction Test: Find the candidate target object with Equation 10 or terminate (max, Sc, < eST) Option Learning: Determine п/ (Equation [T] and learn interaction skills using reward db Eyow Update Add wa to W. Progress a = b. Append dataset. D Poption learning data until COInS no longer finds any interaction test pairs

---

Published in Transactions on Machine Learning Research (03/2024)
5 Experiments
We systematically evaluate COInS in two domains: 1) an adapted version of the common Atari baseline Breakout [Bellemare et al. 2013) (Figure [ and Appendix [A1] and 2) a simulated Robot pushing domain in robosuite (Zhu et al. 2020 with randomly generated negative reward regions (Figure[aand Appendix In these domains, we compare COInS against several baselines: a. non-hierarchical bascline that attempts to learn each task from scratch, fin--tunigg-bseed transfer with 5M step pre-trained neural model, an adapted version of Hindsiggtt Actor Critic (HAC) (Levy et al. 2019b), model based causal RL algorithm Causal Dynamics Learning (CDL) (Wang et al. 2022), a curiosity and count-based exploration method, Rewarding Impact- Driven Exploration (RIDE) (Raileanu & Rocktäschel, 2020) and a hierarchical option chain algorithm Hypothesis Proposal and Evaluation (HyPE) (Chuck et al. 2020 Our results show that COInS is more sample efficient, achieves higher overall performance, and also learns skills that generalize well to variety of in-domain tasks. COInS' learning procedure does not require human intervention- it progresses automatically based on the performance of the interaction score and the goal-reaching policies.
Details about the skill learning procedure, including the number of time steps used for learning each level of the hierarchical chain, the interaction masks, and interaction rates are found in Appendix for Breakout, Appendix for Robot pushing and Table _ Hyperparameter discussion is also in Appendix In this section, we discuss the performance comparison with basclines.
We represent all the policies, including the baselines, with an order-invariant Neint Qi et al. (2017) style architecture and use factored states as inputs. When there are only two factors, this is similar to a multi-layer perceptron. Details about architectures are found in Appendix F
5.1 Overview of Baselines
Before describing the empirical results, we briefly detail how the baselines compare with the Granger-causal and hierarchical elements of COInS. Even though prior work has applied RL to Breakout and robotic manipulation, not all basclines have been applied to these domains. We modify and tune the basclines to improve their performance. The Breakout variants and Robot pushing with negative reward regions are novel tasks to demonstrate how reward-free Granger-causal skill discovery can improve RL performance.
HAC (Levy et al. 2019bb is a reward-based HRL method that uses goal-reaching chain of skills, where hierarchical levels each have a horizon of H. This assesses COInS against a multilevel reward- based hier- archical method to illustrate how interaction-guided skills compare against hicrarchical method without interactions. Default HAC is infeasible in both the Breakout and Robot pushing tasks because of the high dimensionality of the state spaces. To improve performance, we introduce domain knowledge through hand- chosen state factors as goal spaces. In Breakout, the lower-level skills use the ball velocity as goals. In Robot pushing, they use the block position. Additional skill hicrarchy results are in Appendix J
CDL (Wang et al., 2022 a state-offthee-rt causal method that learns causal edges in the factored dynamics for based reinforcement learning. It constructs general causal graph to represent the counterfactual dynamics between state factors. Without identifying interactions, CDL struggles in these domains because CDL fails to capture infrequent interactions even when it picks up general relationships. In addition, it must learn models over all the objects, of which there are many. CDL illustrates how general causal reasoning without interaction can struggle in rare-interaction domains with many factors.
RIDE (Raileanu & Rocktäschel 2020): Rewarding impact-driven exploration (RIDE) is an intrinsic reward mothod that combines curiosity with neural state counts to perform efficient exploration. This method disambiguates COInS skill learning from state-covering exploration bonuses and the advantage of directed behavior through interactions. While RIDE has been applied successfully to visual agent maze navigation settings, these domains often lack the combinatorial complexity of Breakout or Robot pushing. Our imple- mentation of RIDE utilizes the Pointnet-based architectures for the forward model with neural hashing, with Rainbow and SAC for RT. Additional details and discussion of RIDE can be found in Appendix J4
HyPE Chuck et al.) 2020) constructs a skill chain reward-free through interactions and causal reasoning, akin to COInS. Unlike COInS, which learns goal-oriented skills, HyPE uses clustering to acquire a discrete skill set, guided by physical heuristics such proximity and quasi-static dynamics. Despite these differ-
9

---

Published in Transactions on Machine Learning Research (03/2024)
Vanilla (5M)
HAC (5M)
R-HyPE (5M)
CDL (5M)
RIDE (5M) COInS (<0.7M)
Break
85.6 + 8.3
74.3 + 8.2
95.3 +
-50 + 1.7
92.5 + 8
90.7 = 15
Push
-30 0.01
-43 1 24
-90 +
-30 +
0.1
-31 2
-21.6 4.6
Table 1: Sample efficiency for Breakout (Break) and Robot Pushing (Push) with negative reward regions. Note that. -30 in Robot Pushing (vanilla, CDL) reflects the performance of policy that never touches the block. Baselines are evaluated after 5M time steps of training, while COINS gets equivalent or better performance in 300k/700k 700k timesteps for Breakout/Robol Pushing, respectively. The final performance after COINS trained for 1-2M time steps is shown in Table 5 in the Appendix.
ences, HyPE is the closest comparison for COInS since it learns skill chain based on interactions. HyPE performance illustrates how the Granger-causal interaction models and goal-based hierarchies allow more powerful and expressive skill learning. HyPE is designed for pixel spaces, but we adapted it to use the true factor states. We evaluate two variants of HyPE: R-HyPE uses the RL. algorithm Rainbow (Hossel et al. 2018) to learn policies, while C-HyPE uses CMA-ES (Hansen et al. 2003). When evaluating C-HyPE sample efficiency, we add together the cost of policy evaluation for the policies and graph the performance of the best policy after each update, following the methods used in HyPE.
Vanilla RL uses Rainbow (Hessel et al. 2018) for discrete action spaces and soft actor-critic (Haarnoja et al.] 2018b) for continuous action spaces. These baselines are chosen the most stable RL met for achieving high rewards on new tasks. In Breakout variants, we evaluate transfer using pre-train and fine-tune strategy by pretraining on the source task and then fine-tuning on the variant task.
We chose not to compare with diversity-based skill learning methods for Breakout and Robot pushing due to the considerable challenge these methods face in such high-dimensional spaces these methods are typically applied to low-dimensional locomotion or navigation. Breakout and Robot Pushing both have stale spaces that are quite large: Breakout has 104 objects, and Robot Pushing has 18. Reaching a large number of these states is often infeasible. Designing a suitable diversity-bascd method for this would roquire significant reengineering of existing methods.
5.2 Sample Efficiency
In this section, we discuss how COInS achieves improved sample efficiency by learning interaction--cnttolling skills. Specific learning details can be found in Appendix A.1 and A.2, and training curves in Figure 3
In Breakout, COINS learns high performance 4x faster than most of the baselines. This comes from utilizing the goal-based paddle control to shorten the time horizon of the ball bouncing task, and shortening credit assignment between bounces by the duration of paddle goal-reaching. COInS achieves high rewards in Breakout without optimizing extrinsic reward since good performance only requires bouncing the ball. COTnS skills exceed this by learning to control the ball to desired bounce angles.
HAC can learn only after the changes in Section 5.1 Even then, it performs poorly- worse than the vanilla RL baseline. We hypothesize this is because it is difficult to apply credit assignments through a hierarchy effectively. HyPE performs comparably to COInS in sample efficiency. C-HyPE even outperforms COInS, because the evolutionary search can very quickly identify the paddle-ball renationship teihiip---ihhhhhh hypothesis that interactions are a good guide for agent behavior. However, while C-HyPE can control bounces, it struggles to identify and control bounce angles, resulting in poor transfer (Figure
whiic supports the
The sample efficiency of COTnS highlights the advantage offered by interaction-guiddel skill acquisition, allowing the breakdown of complex behaviors using interaction signals and reducing the time horizon through skill learning. The comparatively similar performance of HyPE supports the efficacy of interactions since HyPE also uses interaction signals to achieve high performance. COInS and other interaction based methods are likely to perform well in domains where sparse interactions have a significant impact on performance, such domains with sparse contact large numbers of distant objects.
5.3 Overall Performance
Many of the baselines can achieve near-optimal performance in Breakout (Table 1.. The same is not true in Robot block pushing with negative reward regions task (see Appendix A.2 for details). In this task,
10

---

Published in Transactions on Machine Learning Research (03/2024)
Base Task Training
Average Total Reward Per Episode
Breakout
Robot Pushing
COInS begins bal!
COInS begins
100-
(skill training
optimization
80
-10
60-
40-
20
30-
-20
-40-
-50
0100k
300k
600k
1M
1.5M
2M
300k
600k
1M
1.5M
2M
Number of Timesteps
Chain of Intaraction Skills
Hierarchical Actior Gritic
Dynamies
Causal
Driven Exploration
Reward Impact
SAC (right)
Rainbow (left)
Hypothesiz Proposal and Evaluation with Rainbow
Hypothesis Proposal and Evaluation with CMA ES
Learning
Vanilia RL
Figure 3: Each algorithm is evaluated over 10 trials with the shaded region representing standard deviation, Training performance of COInS (bluc) against basclines (sce legend) Breakout (left) and negative reward regions Robot Pushing (right). The vertical pink lines indicate when COInS starts learning a high-reward policy. In the pushing domain the return for not moving the block is 330 Most algorithms do not touch the block. The minimum total reward -600 by spending every time step in a negative region.
with R- He's performance falls within this lowest bracket. Final evaluation of COInS against the baselines after 2M steps is found in Appendix Table 5
Algo
single
hard
big
neg
center
Vanilla
-5.8
0.74 0.1 0.81
2.9 0.3
-1
0.1 + 0.2
Tune-Tuned
0.9
-..1
0.79 1 0.1
-7.0 19
-42
0.2 0.1
R-HyPE
-5.6.10.45
-5.1 0.49
0.67 1 0.06
2.9 0.46
-20 1.5
-0.3 0.3
COInS
1.2
0.9
0.85 + 0.06
3.6 + 0.3
-12
0.5
Table 2: Transfer evaluation of trained policies on Breakout variants. "Fine-tuned" fine-tunes a model pre-trained on the base task, and "Vanilla" trains from scratch. The baselines are evaluated after 5M time steps, while COInS achieves superior performance in 500k (hard, neg) and 2M (center). Figure contains descriptions of the variants, and additional baseline transfer results in Appendix Table
method, even COInS, achieves perfect performance. This is because the tabletop is densely populated with negative reward regions in highly variable configurations between episodes, (sce Figure []. The task is also long horizon, taking many as 300-Lime steps to complete, and reward shaping is difficult because pushing the block to the goal must be mediated by the negative regions.
COInS discovers block control through the interaction test and optimizes reward using block movements as actions. This results in a complex policy that pushes the block between the negative regions. None of the baselines can achieve this policy: CDL learned dynamics models fail capture the rare dynamics of the gripper-block interaction. HAC and Vanilla RL use rewards, which trap the learned policies in the local minima of never touching the block. While HyPE policies can learn to move the block, they lack fine-grained control when pushing the block to desired locations. See Figure 3 for details.
COInS block pushing results demonstrate how interaction-ccotrolliin skills reframe challenging temporally extended task into one that is feasible for RL. Without using interactions, these skills are difficult to discover from reward, as seen with policies learned with HAC. The failure exploration methods demonstrates that without interactions, state-covering struggles as well. Interaction-bar basod methods offer tool for dirccted pretraining of factorized control before optimizing reward.
11

---

Published in Transactions on Machine Learning Research (03/2024)
Skill Transfer in Breakout Variants
Average Reward
Big Block
Proximity
Single Block,, Hard Obstacles c
Center Obstacles
Negative Blocks
axle
200
500
7006
Number of Timesteps
Chain Rainbow (vanilla RL)
Rainbow fine-tuning
Hypothesis Proposal and Evaluation w/Rainbow
Hypothesis Proposal and Evaluation w/CMAES
Interaction Skills
transfer
Skill transfer with COInS (blue), training from scratch (orange), pre-train and fine-tune to the variant (green), and HyPE skills (C-HyPE in cyan, R-HyPE in purple). C-HyPE has nothing to optimize transfer (it only learned single "bounce" action), so average performance is visualized. Below the performance curves, the variants are visualized with the red (light) blocks negative roward/unbreakable blocks, and the blue (dark) positive reward blocks. Details in Appendix J and Section 5.4
Figure 4:
5.4 Transfer
We evaluate skill transfer in variants of Breakout. These variants have complex reward structures that make them hard to learn from ccratch-negatee reward for bouncing the ball off the paddle, or for hitting certain blocks. These penalties make credit. assignments difficult in long horizon task like Breakout.
Single block (single): A domain with single, randomly located block, where the agent receives
-1 penalty for bouncing the ball, and the episode ends when the block is hit.
Hard obstacles block (hard): The single block domain, but with 10 randomly initialized un- breakable obstacle blocks.
Big block (big): A domain where a very large block can spawn in one of 4 locations, and the agent
must hit the block in one bounce or incur a -10 reward.
Negative blocks (neg): There are ten blocks the top in a row. Half are randomly assigned
+1 reward, the other half -1 reward, and one episode is 5 block hits.
Center obstacles (center): The agent is penalized (-1) for bouncing the ball, and the center
blocks are unbreakable, forcing the agent to learn to target the sides for reward.
Proximity (prox): domain with randomly seleeted target block, with reward scaling between
1 and -1 based on how close the ball is to hitting that particular block.
COINS learned skills that bounce the ball at particular angles, which transfer to learning these difficult Breakout variants. By reasoning in the space of ball bounces, policies trained in the variants can exploit temporal abstraction and simplified credit assignment to handle difficult reward structures.
CDL and HAC perform poorly the original task, and it is unclear how transfer their learned components. RIDE learned the same kind of policy as vanilla RL. Thus we do not evaluate transfer using these baselines. The fine-tuning strategy for transfer did show some success, as it transfers bouneing behavior to these new domains. However, it rarely learns past this initial behavior, and performance can even slightly decline. R-HyPE can transfer ball angle bouncing skills, but since the skills have many failure modes bouncing the ball, the overall performance on the variants is poor. C-HyPE does not learn bounce the balls at different angles (it only learns single discrete option for ball control), it only has action to take, and cannot transfer. It shows the baseline performance of ball-bouncing policy the different variants.
The results in Siguosdddmosstrate how the skills learned with COInS are agnostic to factors that are not part of the skill chain, i.e. the blocks. Like with robot pushing, using these skills can exceed the performance of vanilla RL trained directly because they reduce the time horizon for complex reward assignments. In settings where at least some of the same objects and dynamics are present, interaction-bused skills can rapidly transfer and achieve superior final performance training from scratch, especially on difficult reward tasks.
12

---

Published in Transactions on Machine Learning Research (03/2024)
6 Conclusion
This work introduces a novel approach to HRL in factored environments. The key idea is use adapted Granger-cauue cangel interaction detectors to build hierarchy of factor-controlling skills. The COInS algorithm presents one practical method that performs well on robotic pushing domain with negative reward regions and variants of the video game Breakout. In these domains, it shows improvement in sample efficiency, overall performance, and transfer to difficult. tasks that cannot. be trained from scratch with Vanilla RT. Future work can explore the current limitations of COInS, particularly 1) its reliance on the core assumptions of a single chain of skills being sufficient to represent the task, 2) pairwise interactions capturing the object dynamics, and 3) given state factorization.
To extend COInS beyond chain hierarchies, DAG structure of skills can be learned by selecting from any of the controllable factors and addressing the design question of multiple- skills as inputs. To scale to numerous interactions, future work can investigate multiple-interaction extensions of Granger Causality, such as with gradient-based mcasures (Hu ct al. 2023). Taking this further, interactions could be differ- entiated between different kinds of behavior within a pair of objects, such as differentiating pushing, and picking. Finally, vision models such as (Kirillov al. 2023) can address factorization and correspondence. Despite these challenges, COInS represents significant initial step in leveraging interactions among factors, opening avenues for future work utilizing human demonstrations to handle tasks like grasping, simplifying hyperparameter complexity, tuned stopping conditions, and reducing computational cost. Overall, COInS shows evidence that controlling interactions in state factors is a promising direction for skill discovery.
7 Acknowledgements
This work has taken place in part in the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at The University of Massachusetts Amherst and the University of Texas at Austin. SCALAR research is supported in part by the NSF (TIS-2323384), AFOSR (FA9550-20-1-0077)) ARO (78372-CS), and the Center for AI Safety (CAIS). The work was supported by the National Defense Science & Engineering Graduate (NDSEG) Fellowship sponsored by the Air Force Office of Science and Research (AFOSR). Special thanks to my collaborators Stephen Guigere, Yuchen Cui, Akanksha Saran, Wonjoon Goo, Daniel Brown, Prasoon Goyal, Harshit Sikchi, Christina Yuan, and Ajinkya Jain for their fruitful conversations and timely help.
References
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schncider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight. experience replay. Advances in neural information processing systems, 30, 2017.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinfrrcment learning. Discrete event dynamic systems, 13(1):41-77, 2003.
Mare G Bellemare, Joel Veness, and Michael Bowling. Investigating contingency awareness using atari 2600
games. In Twenty-Sirth AAAI Conference on Arlificial Intelligence, 2012.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Arlificial Intelligence Research, 47:253 17:253-279, 2013.
Mare G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using rein- forcement lcarning. Nature, 588(7836):77-82, 2020.
Glen Berseth, Daniel Geng, Coline Devin, Nicholas Rhinchart, Chelsca Finn, Dinesh Jayaraman, and Sergey Levine. Smirl: Surprise minimizing reinforcement lcarning in unstable environments. International Con- ference for Learning Representations, 2021.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Craig Boutilier, Thomas Dean, and Steve Hanks, Decision-theoretic planning: Structural assumptions and computational leverage. Journal of Artificial Intelligence Research, 11:1-94, 1999.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint 18880008080455 2018.
Caleb Chuck, Supawit Chockchowwat, and Scott Niekum. Hypothesis-driven skill discovery for hierarchi- cal deep reinforcement learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (TROS), pp- 1572-5578 TEFE, 2020.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representations, 2019.
Clive WJ Granger. Investigating causal relations by econometric models and cross-spectral methods. Econo- metrica: journal of the Econmetricc Society, pp. 424-438, 1969.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. Conference on Robot Learning, 2019.
Tuomas Haarnoja, Kristian Hartikainen, Picter Abbeel, and Sergey Levine. Latent space policics for hicrar- chical reinforcement learning. In International Conference Machine Learning, pp. 1851-1860. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICML), 2018b.
Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the deran- domized evolution strategy with covariance matrix adaptation (ema-es). Evolutionary computation, 11(1): 1 18, 2003.
Jean Harb, Picrre-Lue Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option: Learning options with a deliberation cost. In Thirty-Second AAAI Conference on Arlificial Intelligence, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Jiaheng Hu, Zizhao Wang, Peter Stone, and Roberto Martin-Martin. Elden: Exploration via local depen- dencies. 37th Conference on Neural Information Processing Systems, 2023.
Tobias Jung, Daniel Polani, and Peter Stone. Empowerment for continuous envirommen systems. Adaptive Behavior, 19(1):16 39, 2011.
Ken Kansky, Tom Silver, David A Mély, Mohamed Eldawy, Miguel Seearo-Grrdillaa Xinghua Lou, Nimrod Dorfman, Skymon Sidor, Scott Phoenix, and Dileep George. Schema networks: Zero-shot transfer with a generative causal model of intuitive physics. In International Conference on Machine Learning, pp. 1809-1818. PMLR, 2017.
Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learn- ing. International Conference on Machine Learning, 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehoad, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint 1eXiv:2304.026.4 2023.
George Konidaris and Andrew G Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. In A Advances in neural information processing systems, pp. 1025-1023, 2009.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Jannik Kossen, Karl Stelzner, Marcel Hussing, Claas Voelcker, and Kristian Kersting. Structured object- aware physies prediction for video modeling and planning. International Conference for Learning Repre- sentations, 2020.
Oliver Kroemer, Scott Niekum, and George Konidaris. A review of robot learning for manipulation: Chal- lenges, representations, and algorithms. Journal of Machine Learning Research, 2021.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. International Conference for Learning Representations, 2019a.
Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical actor-critic. International Conference on Learning Representations, 2019b.
Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems, 33:9180 9192, 2020.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decom- position. Tn International Conference on Learning Representations, 2020. URL https://openrevier. het/foorumidd=x103yyyy)
Marlos C Machado, Mare G Bellemare, and Michael Bowling. laplacian framework for option discovery in reinforcement learning. In International Conference on Machine Learning, pp. 2295 2304. PMLR, 2017.
Hai Nguyen and Hung La. Review of deep reinforcement learning for robot manipulation. In 2019 Third
IEEE International Conference on Robotic Computing (IRC), pp. 590-595. IEEE, 2019.
Judea Pearl. Causal inference in statistics: An overview. Statistics surveys, 3:96-146, 2009.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652 660, 2017.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. International Conference for Learning Representations, 2020.
Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Advances in Neural Information Processing Systems, 32, 2019.
Philippe Rochat. Object manipulation and exploration in 2-to 5-month-old infants. Developmental Psychol- ogy, 2716):871, 1989.
Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillierap, and Sylvain Gelly. Episodic curiosity through reachability. International Conference for Learning Repre- sentations, 2019.
Maximilian Seitzer, Bernhard Schölkopf, and Georg Martius. Causal influence detection for improving officioncy in reinforcement learning. Advances in Neural Information Processing Systems, 04229955-2211 2021.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsu- pervised discovery of skills. International Conference for Learning Represenlations, 2020.
Olivier Sigaud and Olivier Buffet. Markov decision processes in artificial intelligence. John Wiley & Sons, 2013.
Özgür Şimşek and Andrew Barto. Skill characterization based on betweenness. Advances in neural infor- mation processing syslems, 21, 2008
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181 111-11,, 1999.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Alex Tank, Ian Covert, Nicholas Foti, Ali Shojaic, and Emily B Fox. Neural granger causality. 1EEE Transactions on Pallern Analysis and Machine Intelligence, 44(8):4267-4277 2021.
Sam Toyer, Felipe Trevizan, Sylvie Thiébaux, and Lexing Xie. Action schema networks: Generalised policies with deep learning. In Thirty-Second AAAT Conference Artificial Intelligence, 2018.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua
Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. Conference on Robot Learning, pp. 1439-1456. PMLR, 2020.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, :75(7752)3350 354, 2019.
Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, An- dreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Paltern Recognition, pp. 7942 7951, 2019.
Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for task- independent stale abstraction. Proceedings of Machine Leurning Research, 2022.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint X0v9 12293, 2020.
16

---

Published in Transactions on Machine Learning Research (03/2024)
A Environments
A.1 Breakout
The Breakout domain is not the same as Atari Breakout. The Breakout domain contains as factors the actions, ball, paddle, and blocks, shown in Figure 1 It contains 100 blocks, and the agent receives +1 reward whenever it strikes a block. An end-of-episode signal and -10 reward are given whenever the agent lets the ball fall past the paddle. All other rewards for the task are 0. The ball moves at #1 x velocity and +1, +2 velocity, depending on how it is struck by the block. Like in Atari Breakout, the ball angle is determined by where on the paddle the ball strikes, with four different angles.
We use an adapted Breakout enviroument for three reasons. First, we want the domain to have stationary dynamics so that learning dynamics models is relevant. However, the Atari Breakout domaín has the velocity of the ball change basod on the number of bounces the agent has taken, where the number of bounces is a hidden variable. Rather than overcomplicate the learning procedure, we opted to use a domain where this feature was absent. Second, to investigate long-horizon tasks wanted the domain to have sufficiently long episodes, which we accomplished by slowing the speed of the ball. Long horizons are often mitigated in RL algorithrss through frame skipping to improve efficiency, but we are interested in investigating the granular features of the Environment like the transition dynamics. Third, since we are working in a factored space we wanted a domain where we could have easy access to the factors and related statistics like which objects are interacting with which others a given time step. This is casier to accomplish by having a custom version rather than trying to extract that information from the RAM state of Atari Breakout.
A.2 Robot Pushing
The Robot pushing domain uses as factors the actions, gripper, block, and negative reward "obstacles", The gripper is 7-DOF simulated Panda arm generated in robosuite (Zhu et al., 2020) with 3-DOF position control using Operational Space Control (OSC). The OSC controller generates joint torques to achieve the desired displacement of the end effector specified by the action. The objective of the domain is to push a 5cm2 block to a 5em diameter circular goal location randomly placed in a 30cmx30em spawn area. The area is broken into a 5x5 grid, where 15 of the 25 6em x 6cm grids are negative reward regions (NRR), which implies almost 300M possible configurations
the obstacles, block, and target, forcing the agent to learn a highly generalizable policy. If the block enters an NRR or leaves the spawn area, the agent receives a -2 reward. The negative regions are generated such that there will always exist an NRR-free path the goal. The domain also has constant -0.1 reward per timestep for goal reaching, with episodes of 300 time steps. An image of the domain is shown in Figure
The negative reward region locations are generated such that there will always be trajectory to the goal location, so the actual number of possible configurations is limited by this. However, this is still more than enough complexity for most problems, and in fact, this could be framed as a generalization problem, since it is entirely possible in millions of timesteps that the agent has never seen the provided configuration. In other words, the agent must learn reason about the negative reward regions, something that is only possible with current RL algorithms if it reasons from the space of the block positions, and not from the gripper, hence our choice.
the RL
We would have liked to use a pick-and-place task, but these tasks become infeasibly difficult to perform from purely random actions, often requiring clever hacking of the data to get the gripper to grasp the block. This would undermine the causal nature of the tests since now the actions are chosen without being agnostic to the objects. However, we are looking for adaptations
B Details on Active and Passive Model Training for Interactions
The interaction model often requires data balancing to train meaning weighting the frequency of interaction to con-interaction states. For one thing, the policy used to gather data agnostic the interactions since if it not this can induce correlation between objects that come from the policy, not the dynamics. Thus,
17

---

Published in Transactions on Machine Learning Research (03/2024)
the passive model is trained on states where the actions are taken from policy agnostic to b, the target factor.
In practice, this is often sufficient to train the passive model for good prediction on states where interactions do not occur, though we have found that using a heuristic such as proximity to determine which states are unlikely to have interactions can greatly smoothen the learning process. This is because the model can often fixate on the states that it cannot predict, resulting in misprediction states where it could otherwise perform well. This also helps to widen the gap between the passive and active model log- likelihoods, which makes I more accurate
While perfect data and modeling give an that captures most interactions, interactions can vary from being extremely rare (the ball bouncing off of a randomly moving paddle), or extremely common (the actions affecting the paddle everywhere). In practice, this makes data balancing an issue when training the active model S(), where a network will end up struggling to predict the states where S2 useful because it is overwhelmed by the volume of passive stales. Simply adding model complexity can cause the active model to memorize states that might be hard to predict involve interaction with a different factor c, resulting in spurious interactions.
To combat this, COInS uses the following strategies: first, up-sample states with high passive error (low log-likelihood) when training the active model. This causes the active model to favor predicting states where the passive model is already performing poorly. Second, downweight states with low a b proximity to prevent the active model from overfitting to all states with high passive error, including ones where a does not interact with b. This keeps the active model from memorizing any state. Future work could consider other, general strategics for controlling data imbalances.
These two data balancing methods can be combined in an unnormalized weight w for any state that has the following passive error and proximity, with hyperparameter A:
[x+1 1
log m 0)[s)] 0 ||s sall Éclose
Wb
(13)
otherwise
We also propose a possible way to tune the active model with interactions, by weighting the loss of the active model by the interaction model. This has to be balanced from overfitting, however, this can make the interaction model boost the active model. This tuning would make the forward loss with weighted dataset Db:
min [(mact D'.)) = M -0, (he log (m S6: ()(b)))
+ (1 T(a,, Sb, s/)) log (m ( sb; x) [s,])]
(14)
Where \ is the mixing parameter. However, combined with the proximity this can end up boing too much boosting and result overfitting in the active model.
We also tried using a learned interaction model instead of decision test: 1(s,, ss; 0); S, x S, - 00, 1], the probability of an interaction at given state, and learns to predict the interaction detections I(sa, Sb. s()) This could have the benefit of generalizing the characteristics of an interaction to states where S might be hard to predict. However, because we continuously train the active model with new data from the policy, this turns out not to perform well because it complicates the learning process.
C Object Centric Actions and State Augmentations
When using the goal space of the current option as actions, there are two important design questions: should the action space be inntis or discrete, and should the actions be in a fixed space, or relative to the state of the target object?
In general based RL both discrete and continuous action spaces are possible, but when the goal space is small, such as the ball velocity, discrete spaces are preferable. This is because the space of states seen
18

---

Published in Transactions on Machine Learning Research (03/2024)
after an interaction Cy, after being further reduced by the controllable elements mask no: can be quite small. When this space is small, the locality that is present in normal state spaces might not be present, so selecting continuously could add bias to 'nearby" states. Thus, if C. Th <= 10, or there are less than 10 seen interaction states, then we use discrete actions.
For the second question, continuous actions are generally better when relative to the agent/object being controlled. This is because continuous spaces often have a sense of locality eg. the paddle in Breakout is close to states with close values. We encode relative state in the action space by having policy actions ar between -1 and 1, and then remapping that into relative factor space, Og d 17 + Sb, where d is the relative distance single action can go, which is typically 0.2 of the normalized space St.
D Causality and Object Interactions
While this work introduces a novel motric for identifying entity interactions, similar idcas in mechanisms have been incorporated in model-based and planning- based
like learning causal relationships (Pearl 2009]
methods to explain dynamics (Li et al.
2020; Wang et al.) 2022). Schema networks (Kansky et al.) (0177) and
Action Schema Networks (Toyer et al. 2018) extend causal models to planning. Alternatively, variational
methods such as (Lin ct al. 2020) learn to identify objects in visual scene for planning (Kossen et al. 2020
or reinforcement learning /Veeraaanen et al. 2020). By contrast, COInS limits the modeling burden by only capturing forward dynamics models good enough to identify interactions and using them for HRL, drawing ideas from empowerment (Jung et al., 2011] and contingency (Bellemare et al. 2012) to improve exploration. The model-disagreemen technique used by COInS builds on neural methods for Granger Causality (Granger 1969 Tank et al. 2021), though this a novel application to HRI.
In this work we differ from general causality in two ways: first, we acknowledge that the Granger-causal test does not demonstrate causality, but rather is predictive hypothesis test designed for time series. However, in the case of forward time (s' cannot cause something in or any prior state), fully observable (there are no unobserved confounders) and Markov (s' depends only on s) dynamics, our MG-cansal test (Equation D this matches a causal discovery test under two conditions: 1) There is not a "true cause" X, of X2 with lesser information than S. about X 2) The selection of do = 5b) is decorrelated from So, Here, we use Xe to denote the set of causal variables denoting a cause, and X the set of causal variables denoting an outcome.
In the first case, this describes the case where some factor confounds on b that is, P(s Sb, Sa, se) # P(s s.)) Note that because of the Markov assumption s, is NOT a common cause of S,, s/c Instead, this occurs when factor a shares some information with factor c, that is P(S((Sa) # P(S)), and S. is the true cause of the transition dynamics of s + s// In our case, because we choose the factor pair with the highest MG-causal test score, meaning that S, must be a better predictor of S% than S., despite not being the true cause. As an example of this, imagine that in Breakout factor a describes the paddle which rarely interacts with the ball, while factor is another paddle with noisy observation whose location is determined by factor a. In this case, where factor b is the ball and we search for c-> b relation, then factor may confound this information because information about paddle bounces is subsumed in paddle a. This condition is rare but possible in many real-world or real-world- woa--worddinsppreed domains, and it is a limitation of using pairwise scope relationships.
In the second case, if do(a) is not selected randomil but based on 8p, and this difference is not accounted for, then the Granger-causal relationship not captured. This is the case if the input data for s, has randomly assigned Sa, for example, if the goals for the policy are sampled randomly and the policy has no dependency Sb (which we use in this work). An alternative way to correct this would be to correct values with the importance sampling weights P(s. ((sulladreesd)) where Wairected the policy with imbalanced behavior.
P(s,) Frando))
We describe interactions using the language of actual causality because we are implicitly assuming that an interaction occurs when one object is the cause of one behavior in another object in a particular state. In actual causality, the causal relationship is not described in the general case (could cause b), but in the specific case between factors (did cause b in particular state s). The active and passive models also capture
19

---

Published in Transactions on Machine Learning Research (03/2024)
Hlerarchy Interaction Skill Chain
Legend:
Factor
larget Factor
Primitive Actions
Padile
Ball
Reward Optimization
Paddle (Saurce)-Ball (target) Skill Interface
Pedole-Ball
[ (3r (7::%)
Interaction Detection
Ball Goal Reached
Upper Hiwerrhy Levil (cudl relsed))
Heraachy
Lovel (action paddio)
Ball Action Space:
Space (Reward policy Actions)
(paddlle goals)
Figure 5: An illustration of the COInS hierarchy. At each edge, a policy controls the target factor to reach interaction goal, and takes interaction goals as temporally extended actions. The child of the primitive actions passes primitive actions pseudogoals. (lower) Learning single layer of the HinTS hierarchy, the pairwise interaction between the paddle and the ball. The interaction detector and ball skill policy both use the paddle (parent) and ball (target) state as imputs and indicate if a goal is reached and new temporally extended action needed respectively. The ball skill policy uses paddle goals as actions and receives ball bounces as goals from the reward-optimizing policy.
this, because the passive model is trying to capture "what would have happened", and the active model is capturing "but for in a particular state Sa, resulting in s
E
Transferring Skills
When we say that we transfer the learned hierarchy from COINS, this means that we take the final skill, whether ball control by hitting the ball with high accuracy at one of four desired angles in Breakout, or block control by moving the block to specified location in Robot pushing, and use this goal space as the action space for a high-level policy that takes in all of the enviromment states. Thus, the action space for this new domain is temporally extended according to the length of time it takes to reach the desired goal or cutoff. These actions will still call lower level actions (paddle or gripper control), as necessary. In Breakout, the temporally extended skills on the order of ball bounces, which is around 70x less dense, simplify credit assignment, causing performance improvement. We replaced the hierarchy with a known perfect ball- bouncing policy with slightly better results to COInS, supporting this premise. In robot pushing, we suspect that the benefit comes somewhat less from temporal extension, and more from the fact that the action space is much more likely to move the block (compared with primitive actions that move the gripper), which allows for better exploration.
In Breakout, we could have learned an additional layer for block control. However, this would have required added complications to the algorithm: first, it is not always possible to hit a block, either because there are other blocks in the way, or because the block simply does not cxist. This would require us to either hard-code a sampler or train one that learns these properties. Next, we were to transfer this policy, it would then hit, either a discrete selection or a continuous location, which
require policy that chose a block location would put the omus the learner to figure out how to hit it. Finally, learning the block policy is sample inefficient, and is likely not to give much benefit, and we can stop learning at any level of the hierarchy. As it is, a block targeting policy is learnable, since the proximity variant essentially captures this.
F Network architectures
We use I-d convolutional network with five layers: 128, 128, 128, 256, 1024 which is then aggregated with max pooling and Rectified lincar unit activations followed by 256 lincar layer before cither actor, critic, passive distribution oulputs, and active distribution outputs. In every case, the inputs are the 8, and sa object(s), where we use only a single parent class a. In the final evaluation training, is the set of all factors that are not multiple, except for primitive actions in policy training. We then append s, to Sy and treat each of these as point. There are also well-known issues Breakout that without a relative state between the paddle and the ball, the policy fails, so augment the state with s, S, and Ss e when appropriate (the
20

---

Published in Transactions on Machine Learning Research (03/2024)
dimensions match). Combined, we call this architecture the Pair-net architecture (a Pointnet architecture for pairs of factors)
We can use the same architecture for all of the policies. Note that when there are not multiple instances of objects (ie. cases without the blocks in Breakout or the Negative reward regions in Robot Pushing), the Pairnet architecture reduces to a simple multi-layer perceptron. For the sake of consistency, we used the same network for every choice, though this was overparameterized in some cases.
G Breakout Training
This section walks through step by step the learning process for COInS in Breakout. Note that the algorithm uses automatic cutoffs to decide which factors to learn a policy over using the interaction score in Equation g and comparing it against ésI, minimum cutoff weighted log-likelihood. In practice, we found that 5 worked for all edges in Breakout and Robot Pushing, though this can probably be automatically learned based on estimate of environment stochasticity. The training curves can be seen in Figure G and visualizations of the skill spaces in Figure 7
G.0.1 Paddle Skill
COInS collects random samples (10000 sample intervals) until it can detect. the action-paddle connection through the interaction score (Equation D). Interaction tests between other factor pairs aetssn-btinballl and action-block, have low scores, with details of the comparative performance in Table The paddle interaction detector detects every state as an "interaction" since the actions control the paddle dynamics at every state. /paddle masks out all components except the x-coordinate of the paddle: [0, 1, since it can only move horizontally, and 7paddle Cpaddle' is the x-range of paddle locations. The paddle skill uses Spaddle as input and learns to perfectly move the paddle a randomly chosen target x-coordinate in roughly 10k time steps. Paddle training ends when the success rate over 100 updates no longer decreases.
G.0.2 Ball Skill
COInS continues to gather samples in 10k increments using the paddle policy. For every 10k additional samples the interaction detectors between the paddle and other factors are updated, and if the performance exceods Cedge, a skill for that factor automatically learned. COInS discovers the paddle-ball interaction with the interaction score reported in Table | after 80k additional samples. It takes this many time steps because of the infrequency of ball interactions. Using Thall; COInS discovers Crall 7/aal with four velocities [-1,-1] [-2, -1,, [-2, 1], [-1, 1], the four directions the ball can be struck in Breakout, with Thall [0, 0, 1, 1]. Since Mhall < Adise; we sample discretely from this set.
The discovered interaction describes a COInS skill trained with hindsight to hit the ball at the randomly sampled angle. Notice that this problem subsumes the one of just playing Breakout, which just requires bouncing the ball. The resulting skill run with random ball velocities plays Breakout well after < 100k time steps as seen in Figure 3 and Table [ though at that point it only has 30% accuracy at hitting the ball the desired angle. Since the success rate has not converged over 100 updates, training continues for Im time steps to achieve 99.5% accuracy for striking the desired angle. At this point, COInS terminates, though future skills such as block targeting were trained as variant. Since blocks are separate factors, their individual interaction scores are low due to data, infrequency. A class-based extension would allow COInS to handle this.
Robot Pushing with NRR Training
The negative reward regions pushing lask is difficult, and all these baselines fail as seen in Table Even intrinsic reward methods, like RIDE, struggle this domain, possibly because the intrinsic reward signal is washed out before the agent can learn to manipulate the block, and sufficient intrinsic reward can be acquired simply by manipulating the gripper. Only HyPE has a non-trivial reward because the options it learns will force it to move the block. However, the skills navigate it to negative reward regions out of
21

---

Published in Transactions on Machine Learning Research (03/2024)
HIntS Skill Training Breakout
Paddle Training
Ball Training
Average goal reaching rate /Episode
10-
0.8-
0.6-
0.6
0.4
0.2-
0.2
0.-
2.5k
5k
7.5k
10k
250k 500k
750k
1M
Number of Timesteps
Figure 6: This graph shows the training curves for COInS learning the Breakout skills, where rate of 1.0 means that the goal reached on 100% of episodes. We averaged performance over 10 runs. Note the difference in scale between the paddle skill learning (10k time steps), and ball training (10M time steps). Training terminates automatically when performance converges (train goal reaching rate does not increase).
bounds most of the time instead of the goal, resulting in lower trajectory reward than just not moving. In most cases, the agent is highly disincentivized to push the block at all since the likelihood of reaching the goal is low, but the likelihood of entering negative region is high. However, using smaller penalty would make directly pushing to the goal optimal-it can take dozens of time steps to push the block around an obstacle. We tried a modified baseline, which pre-trains the agent to push the block to the larget, and then fine-tunes the policy to avoid obstacles, but that baseline policy still ends up quickly regressing to inaction. Only COInS achieves non-trivial success, with an average performance can reach the goal consistently, though occasionally incurs a penalty for slipping into the negative reward zones because the block skill is agnostic to the NRR. Future work could consider training the policies end to end that NRR information can be used when lcarning lower-level skills.
22.66 Importantly, the agent
We illustrate the COInS training curves in the Robot Pushing domain in Figure N and visualize the goal spaces in Figure g We describe the learning procedure here: COInS first gathers 10k random actions according to the same stratogy described in Breakout training- it automatically discovers edges using the Interaction test. In this domain, the majority of the benefit arises from based on the amount needed to discover useful correlations with action. It enumerates edges between the primitive actions and the different objects in the domain and learns object models according to the method described in Section 4
The action-gripper model passes the interaction test with the active model having weighted performance of 4em better than the passive model, while the spurious action-block edge fails with less than 0.1cm difference, as enumerated in Table 1 Even though designed for sparse interactions, the model can pick up that the action affects the gripper at almost every time step. The controllable set operation from Section 4.3 returns a mask of [1, 1, since all the gripper components are sensitive to actions. Since the parameter set Cgripper is > 10, the goal state is sampled continuously not discretely.
The gripper policy is then trained to reach randomly sampled gripper positions. The gripper model takes state the state of the gripper and the last action. The gripper policy converges to 100% accuracy at moving within 0.5em of target positions within 50k time steps. However, again following the 10k sampling procedure for gripper-block interaction, we continue to gather gripper movement behavior until gathering a dataset. of 100k time steps.
COInS then samples the edge between the gripper and the block, appended with action information as a "velocity" for the gripper. With the combined input state, the forward model passes the interaction test
22

---

Published in Transactions on Machine Learning Research (03/2024)
Figure 7: Visualization of the Breakout skills. The first two rows illustrate the block, where the green dot is the desired goal position. The last three rows show the target ball angles, with each row illustrating how the agent manipulates the paddle to produce the desired post-interaction angle. There are four possible angles.
with about 1.5em better-weighted performance. This difference smaller because block interactions from random actions are rarc, and can be low magnitude- the robot only mudges the block. The mask trained over the controllable block policy learns to move the block to within lem of accuracy, and the block- skill converges within 500k time steps.
is [1, 1, 0) since the block height. does not change (the gripper cannot grasp). The
Learning the final policy with the COInS skills takes around 700%: time steps. This is because the goal-based movement the block allows the agent to navigate carefully around objects. However, the task is sufficiently difficult that even with that, the agent still plateaus in performance even though a human using the learned options could get a better reward. This is probably because there are many local minima. This is where model that predicts block movement could be incredibly useful for learning because it would allow imagining multiple trajectories without getting collapsed into single low-reward one.
Hyperparameter Sensitivity
In Sections A.1 and A.2 we briefly describe some the hyperparameter details related to the COInS training process, but in this soction, we provide additional context for the sensitivity of hyperparameters described in Table 0 and throughout the work. In general, the volume of hyperparameters comes from the reality that hierarchical causal algorithm complex system. RI, causal learning, and hierarchy all contribute
23

---

Published in Transactions on Machine Learning Research (03/2024)
HIntS Skill Training Robot Pushing
Gripper Training
Block Training
Average goal reaching rate /Episode
0.8-
0.6-
0.6-
0.4-
0.2-
0.2-
0.0
0.0
25k
50k
75k
100k
125k
250k
375k
500k
Number of Timesteps
Figure 8: Training curves of COInS learning the Robot pushing skills. Left: Gripper reaching. Right: Block pushing. 1.0 means goal is reached (interaction and within epsilon of the desired state) on 100% of episodes, averaged performance over 10 runs. Note the difference in scale between the gripper skill learning (100k time steps), and block training (500k time steps). Training terminates automatically when performance converges (train goal reaching rate does not increase).
hyperparameters, and though the overall algorithm may not be sensitive to most of them, some value must be assigned to each. In this section, we aim to provide some clarity hyperparameter choices.
To construct the skeleton and skills of the skill chain, this algorithm introduces the minimum set size didsse: minimum test score est, success convergence cutoff 0.01, and success convergence timesteps Ncomplete- These terms can be chosen without any tuning since they generally capture details about the environment that often have significant effects. For example, the minimum test score simply needs to be chosen based on how difficult is to provide accurate predictions of the state. More randomness would suggest a lower value, but this value is just to determine when the algorithm should give up. As result, the csı + 2 without causing COInS to exit early. A simple way to generally select this is to train the passive model on the data, and then select the minimum test score to be one minus the average performance of the passive model. Similar to Epas, in rare-interaction environments this will generally indicate one order of magnitude less likelihood on average prediction for that state factor. Note that a spurious edge such as action-tball in Breakout could never occur regardless of the choice of eST, since COInS always chooses the highest likelihood edge, which would be action-ppaddle. Typically, after the learnable edges have been exhaustive, the difference in likelihood is as much as order of magnitude for any new edges. Adisc follows similar reasoning- -vido games can have discrete outcomes, and so want capture that possibility, but the value for cutoff can vary significantly. Since there are usually a limited number of these kinds of choices for a single factor, a choice of 10 is reasonable across many domains.
For the interaction learning hyperparameters: Epas, Eact, Emins these values are generally environment specific. In particular, since the passive and active epsilons utilize environment information, they should be selected based on what "good" or "bad" prediction is. We convert some of the mean errors into the relevant units for each environment in Table 4 In relevant units, the difference between the predictions of the passive and active models is often significant (6 pixels 2cm for Breakout and Robot pushing respectively). However, this property does not necessarily hold for all domains. To set Cpas, the simplest solution is simply to train the passive model over the dataset first, then set the épas to be one less than the average value, since this is an order of magnitude lower likelihood event. Alternatively, we have found that 0 appears to work well out of the box. To set fact; we can similarly use the average performance of the passive model on all the data.
When interactions are rare, this will be indicative of good prediction. Overall, the choice of Epas, Cact has limited effect on the overall structure (which skills to connect together), but must be chosen so that the
24

---

Published in Transactions on Machine Learning Research (03/2024)
Figure 9: Visualization of the Robot Pushing Block skill, where the upper row renders the robot environment, and the lower row illustrates the state on a grid. The block goal represented as an x,y position, since the block z coordinate not controllable, represented on the grid as a black dot (the green circle is the task goal, which relates only extrinsic reward.
number of inaccurately classified interactions is low. If the number of inaccurate interactions is sufficiently high, skill learning can fail to converge. In Breakout and Robot pushing, choice of Eact + and 6pas + 2 would not affect the outcome of the connectivity (the edge between the paddle and ball in Breakout, for example). Since these are log-likelihoods, this dernonstrates there a sizable gap between the predictions from the two models on true interaction states. However, in certain states changing the e might result in more misclassified interactions, which would degrade RL performance. Emin the minimum variance of the Gaussian distributions output by matt pas and setting this 0.01 seems to work in most domains, but the choice affects what the average log-likelihood will be.
25

---

Published in Transactions on Machine Learning Research (03/2024)
With skill learning parameters: Celose: Crews Eyy hindsight selection rate, and relative action ratio, these are chosen using intuition from the environment except for frew, which does require tuning. The environment- specific parameters are often not particularly sensitive since the margins for the environment are often quite substantial. For example, Celose just needs to be set to a value that is low enough to be a reasonable definition of "reaching a goal." frew on the other hand could have wildly different results because a setting too low would result in no incentive for the agent to search out reward, and a setting too high would result in exploding losses. It might have also been possible to perturb the discount rate Y in the RL losses for the same effect, but we used y = 0.99 in all skills and reward optimization. En is robust in similar way to values like fact and 6pas, with an effective range around + of normalized units (it would have been around #1 in log likelihood, but we used the mean difference). A simple way of identifying the appropriate value for this is to use a fixed percentage of the possible range for that feature. If the difference in the range of that feature is greater than 10%, then it is likely to be the result of an interaction, rather than some spurious co-occurrence. Alternatively, a more robust strategy might analyze the crrors in the passive model, and select some quantile range for the errors. As an example of how we set the mask in Breakout the ball velocity mask used changes of 0.5 of the possible range (-2 to +2). from the position changes 0.05% of the possible range (at most or 2 pixels difference of a range of size 84).
In general, regarding hyperparameters, we believe that many of the values can be set automatically, especially based on the average values of the active and passive values, and plan to investigate this in future work. Since the focus of this work is to analyze the performance of Granger causal models when used to construct a skill chain, we primarily focused on finding a usable set of hyperparameters, rather than devising adaptive strategies that can apply across many domains. The intuition for the hyperparameters is often based on inherent stochasticity in the domain, and finding ways to identify the propertics of domains is the subject of ongoing research, both in future work related to this project and in the machine learning community.
The RL and network parameters often result in the most difficulty tuning, since they often do not have clear intuitions. However, in general choices such as the continuous and discrete lcarning algorithms, the learning rale and network archilecture did not have a significant effect on the outcome based on the search of prior work. That is, without borrowing parameter choices from previous work related to learning rate and algorithm choices, COInS and other baselines, would struggle. However, when building on top of an existing set of RL hyperparameters, performance was stable even when some of these parameters were then perturbed.
As a whole, the majority of hyperparameters have wide margins for their settings, because the difference between Granger models for interactions, or goal-reaching for skills is relatively broad. As a result, the majority of the same parameters apply to both Breakout and Robot pushing, and may also apply to many other domains. It seems likely these could also be automatically identified, though we did not explore this possibility in this work.
J Additional Baseline Details
For all the baselines, mitigate the effect of COInS possibly having an advantage in the state representation, we augmented the state space with relative state features such as the paddle-ball information in Breakout, and the gripper-block and block goal information in robot pushing. Combined with the Pointnet architecture, this makes the vanilla algorithms object-centric, though they do not have the hicrarchical and interaction advantages that COInS is demonstrating.
J.1 Hypothesis Proposal and Evaluation (HyPE)
The HyPE algorithm relies on discrete action spaces which requires changing the Robot pushing environment by constructing a discrete action space for HyPE to use. This space consists of actions in the cardinal directions x,y, and z, where the agent moves 1/10 of the workspace for each action. This was tested by having a human perform demonstrations with this workspace. HyPE also performs much better in Breakout, where the quasi-static assumptions capture the relationship between the paddle and ball especially well since changepoints capture the instantaneous motion easily. However, in the Robot pushing domain, struggles
26

---

Published in Transactions on Machine Learning Research (03/2024)
because the block movement can vary in magnitude, but the nature of the actions is such that any movement will get assigned to an action completion. As result, learning the block-pushing options is only somewhat effective, able to move the block in some regions of space, but getting stuck in others. HyPE is the only baseline that performs meaningful learning on the base task, starting from a -400 reward to only -90. Tronically, it also has the lowest. performance because the other baselines just never touch the block.
HyPE has two varieties, one where the agent is trained using Covariance matrix adaptation evolution strategy (CMA-ES) Hansen et al. (2003) (C-HyPE), and another where it is trained with Rainbow Hessel et al. (2018) (R-HyPE), In the CMA-ES case, while the evolutionary algorithm is good at picking up easy relationships like the ball to the paddle, it cannot learn multiple policies to hit the ball at each desired angle, and only trains to create bounce changepoint (the next layer that would be used for transfer has only one action, which is why C-HyPE is not trained at transfer). Thus, the CMA-ES version performs well at the main task and poorly at the overall task. On the other hand, because HyPE learns a separate policy for each ball angle, it takes much longer to learn. These policies struggle to keep the ball from dropping (letting the ball go past. the paddle), which is the main reason for the poor transfer performance when compared with even the pretrained-fine fine tuned baseline.
C-HyPE is trained with a 128-activation hidden layer multi-layer perceptron, because it needs an n² compu- lation in the number of parameters, so using the larger NeintNet architecture is infeasible. This is another reason why it struggles to learn complex policies.
By comparison, HyPE and COINS learn similar object-option chains, but because the interaction detector is more robust and the goal-based option is more gemeral, COInS can perform when HyPE cannot. HyPE also makes stronger assumptions about the environment, including the quasi-static assumption and discrete actions. The reliance on discrete modes for high-level skills also limits HyPE.
J.2 Causal Dyanamics Learning
We ran CDL in every domain (Breakout and Robot Pushing) and all the Breakout variants. It failed in every domain. We hypothesize this is because when the data that it gathers is not particularly meaningful, which is true of random actions in both Breakout and Robot Pushing, struggles to construct a model that is useful to the agent. Without temporal abstraction, the policies struggle, and this lack of temporal extension is exacerbated by a model that makes inaccurate predictions about key prodictions (object interactions). Even with a good model of the system, both these domains have very long time horizons, making them very challenging for on-policy algorithms like PPO (which CDL uses for the model-based component). Random shooting only helps for certain special states (before an interaction), but detecting those states can be challeuging, and the rest of the time, it gives back sparse signals. On top of the fact that the rewards are sparse, this is why the domains end up being challenging for CDL. We include table of performance for CDL in Table 5
We tested the model-learning component of CDL and found that it can detect relationships between difficult- to-model objects like the paddle and the ball or the gripper and the block. However, this takes a large number of time steps (>-500000), and just because a relationship is detected does not mean that the model is generating useful samples for the policy. In particular, this suggests that having a general model to capture general causality is only so useful in RT tasks. Since COTnS creates a model to capture certain rare, but incredibly task-important states, and to identify those well, it ends up being more useful for performance, even though the overall model is probably worse predicting the next state.
Finally, pairwise interaction tests are attractive in comparison to CDL because in domains with many factors (there are 100 blocks in Breakout) and thus a large state space, this can become prohibitively expensive for CDL. Since CDL uses cut from the overall state, this results in several models that grow in the number of objects, where each network must include all the objects. We change the state space of Brcakout so that each block state only has one value (whether that block exists or not), because otherwise, the cost of this computation would be prohibitively expensive.
to detect causal relationships, where s/x; denotes that the factor æ; is
27

---

Published in Transactions on Machine Learning Research (03/2024)
J.3 HAC variants
HAC lypically uses the full stale of the environment actions for the higher-level policies and goals for the lower-level policies. However, this means that the HAC higher-level policies need to select complete states to reach, and the HAC low-lovel policics have to reach those states. This is next to impossible, and that means the hierarchy ends up learning nothing: the low-level states learn nothing because they cannot reach any of the goals, and the high-level states learn nothing because their actions are meaningless.
We mitigate this issue by assigning HAC layers to the objects, essentially emulating what COInS does by having each layer control an entity. In Breakout, this means that HÁC has one layer that has block goals/rewards and outputs ball goals, one layer with ball goals and then outputs paddle goals, and one layer to convert paddle goals to action goals. However, this full hierarchy also fails because HAC relies on hindsight that is based on a fixed duration of time. Unfortunately, this often means that the ball interactions appear very rarely in the replay buffer, and the top level using actual rewards has no hindsight targets. The best we could find was with a 2-layer hicrarchy that had extrinsic reward as the top-level signal, and output ball velocities to low-level policy that outputs primitive actions. In this case, out upward velocities were somewhat learnable and would give some reward. A similar object-level hierarchy was used for robot pushing, though in this case, there was no clear reason why the layers should fail except that the reward structure is difficult enough that HAC probably needed to pre-lcarn the block moving policy before doing anything else. However, that would make it the same as COInS in terms of the final structure.
J.4 RIDE details
Rewarding impact-driven exploration (RIDE) Raileanu & Rocktäschel 2020 uses a learned forward and inverse dynamics model, combined with count- based regularization to provide an intrinsic reward for explo- ration. We ablated over the RIDE tradeoff parameter (the amount of weight for the intrinsic versus extrinsic reward), the RIDE learning rate (rate to train the ing the number of hidden layers, and Pointnet or MLP implementation) and the count-based scaling (rate to reduce the reward for similarly hashed components). We implemented hashing by taking the learned embedding from the forward and inverse models, applying a sigmoid, and then taking the values greater than 0.5 as 1 and less as 0.
models) the network architecture (chang-
RIDE has been applied to visual navigation domains with some success, and in this work, we apply what appears to be the first use of this to a factorized dynamical environment. However, RIDE struggles to perform in both environments. We speculate this is bocause it relies on the inverse dynamics to provide a significant signal to learn meaningful embedding. Unlike in pixel-based environments, where the embedding often retains much of the information about the state through the convolutional layers, in the factorized format much of the information can be lost, even with pointnet, because of the use of fully connected layers. As a result, the RIDE-learned reprosentation quickly converges to the minimal information needed to recover the action. In this case, the forward model is easy to predict, and since the loss is based on the 12 error in the prediction of the next state, without much of the information the agent quickly loses much of the intrinsic motivation. This is true in both Breakout and Robot Pushing, where there are a significant number of objects whose state does not change much (the blocks and obstacles), and whose state is not closely correlated with actions.
As result, the RIDE loss struggles to provide much exploration bonus. The performance of RIDE mirrors the performance of vanilla RL for the simple reason that the best choice of RIDE reward scaling (after searching through several orders of magnitude of scalings), is the one that has the least negative impact. We think that the slight negative impact might be from the intrinsic reward: in Breakout, once the good behavior has been found (bouncing the ball), the lask is closer to exploitation than exploration and overt exploration often reduces performance. For computational reasons, and because there is no reason that intrinsic reward method such as RIDE should show significantly different transfer than pretraining-fino tuning, we did not run RIDE on the Breakout variants.
28

---

Published in Transactions on Machine Learning Research (03/2024)
METHOD
No PROXIMITY
PROXIMITY
2.4 10-3
2.2 10
FN
0.07
0.003
Table 3: Table breakout interaction predictive ability. A false positive (FP) is when the interaction model incorrectly predicts ball bounce, measured per state, and a false negative (FN) is when the interaction model fails to identify ball bounce, mcasured per bounce. We get these bounces from the simulator, so these comparisons
ground truth. These ball bounces training we just use them for this evaluation. While both FP, FN are undesirable, FN is greater issue for learning because they result in missed ball bounces.
not given the interaction model during
PARENT
ACT
ACT
PAD
Acr
ACt
GRP
TARGET
PAD
BALL
BALL
GRIP
BLK
BLK
PE
1.03
1.21
6.81
4.8
0.22
2.2
AE
3e-3
1.69
0.30
0.17
0.18
0.6
Table 4: Table of interaction test scores (Equation 9 for interaction training in breakout and Robosuite pushing between pairs of objects. "Act" is actions, "Pad" is the paddle, "Grip" is the gripper. "PE" is passive error in pixels for breakout (left) and em in Robosuite pushing (right), "AE" is the active error in pixels/em (weighted by interaction). In this table, we convert likelihood comparisons to 12 distance in cm using the mean of the normal distribution, which is proportional but not equivalent. We do this because distances are more interpretable.
ALGO
BREAK
PUSH
SINGLE
HARD
BIG
NEG
CENTER
PROX
BASE
85.6 8.3
30 +
-5.8 + 0.5
-7.1 0.8
.74 0.1
2.9 + 0.3
-37 + 11
0.1 + 0.2
FT
NA
NA
-5.4 0.9
-6.1 0.4
+
-7.0 19
42 9
0.2 0.1
HAC
74.3 8.2
89
NA
NA
NA
NA
NA
NA
RIDE
92.5
8.2
31 + 2.2
NA
NA
NA
NA
NA
NA
CDL
-49 + 1.6
-33 + 1.0
100.0 + 0.0
-9.7 + 1.5
-9.26 + 2.7
-49.9 + 2.5
-49.8 = 0.6
-10.0 + 0.0
C-HyPE
76 1 15
221 1.5
6.4 3.4
-7.3 1 1.0
.35 - 0.05
0.72 1 0.9
39 1 10
-0.6 1 2.5
R-HyPE
95 3
90 19
-5.6 0.45
-5.1 0.49
.67 0.06
2.9 + 0.46
-2 1.5
-0.3 1 0.3
COINS
09 + 0.2
-21 +
-3.2 + 1.2
-4.2 + 0.9
.85 0.06
3.6 + 0.3
-12
0.5 0.05
Table 5: Table of the final evaluation of trained policies on Breakout variants COInS trains high-level controller using the learned options, 'FT" fine-tunes a pretrained model, and "Base" trains from scratch. The baselines are evaluated after 5m time steps, while COInS is trained for 2m time steps. The abbreviations are the ones used Section 5.1
29

---

similar significant if too high significant if too small significant if too large Published in Transactions on Machine Learning Research (03/2024)
Parameter
Values
Relative performance
Continuous Learning Algorithm
SAC, DDPG
similar
Discrete Learning algorithm
DQN, Rainbow
Learning rate
.0001--0007
Interaction Proximity
5-7px, 0.5-1cm
fclose, parameter proximity
1px, lem-2cm
Network layers
3-5
Fixed across skills Fixed across skills moderate sensitivity sensitive if too low moderate sensitivity sensitive low sensitivity moderate sensitivity low sensilivity moderate sensitivity low sensitivity low sensitivity
Network width
128-1024
hindsight selection rate
0.1-0.5
inline training rate
100-1000
relative action ratio
0.05-0.3
Constant negative reward frew Maximum passive log-likelihood cpas Minimum active log-likelihood Feature Mask cutoff e, (normalized units)
-0.1--1.0
2
0.1
Minitmum next state distribution variance (Σ)
0.01
Minimum Set size ndise Minimum test esi (in log-likelihood)
10
Success Convergence cutoff Skill Convergence timesteps (Neomplete)
0.01
Affects sample efficiency vs final performance Affects sample efficiency vs final performance
10000
Table 6: Table of hyperparameters
30


Reviews:
[{"review_id": "OK1w3GuNPx", "reviewer": "Reviewer_xrRe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths \n-\tThe idea of using Granger Causality for skill discovery seems novel and the authors show that the proposed method has an advantage over baselines.\n-\tThe conceptual idea behind the method is well-motivated by a simple example (Breakout).  \n-\tI appreciate the discussion around baselines, and it helps position the proposed algorithm in relation to already existing ones.\n-\tThe transferability experiments are convincing.\n\n# Weaknesses:\n-\tThe main weakness of this paper is the presentation. I get the feeling that the paper was written in a hurry. Some variables are never defined, such as $\\mathcal{X}_b$. Plots and tables contain several typos, e.g. the algorithm HIntS is never defined? I assume this is supposed to be COInS? \n-\tIt is unclear how robust the algorithm is to perturbations of the many hyperparameters, e.g. $\\epsilon_s, \\epsilon_{close}, \\epsilon_{SI}, \\epsilon_\\eta, n_{disc}, Sc_\\min$. Many of these parameters are thresholds where certain parts of the algorithm depends on quantities being above/below the corresponding threshold.  I’m worried that finding the right skills is very sensitive to the choice of thresholds. Some discussion about the robustness w.r.t. changes in these parameters and how hard it is to find the right ones would be nice to see. Also, is it possible to tune them during training?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of skill discovery in factorized MDPs and suggests a novel method based on Granger Causality. The proposed CoInS algorithm  identifies causal links between source and target factors and then learns how to control these target factors. CoInS improves over baselines on two different control tasks both w.r.t. sample complexity and performance. The authors also illustrate that the skills learned by CoInS can transfer to new tasks.  \n\nI think the main contributions of the paper are:\n -  Using Granger Causality to find interactions between factors.\n-  Introducing the algorithm CoIns which identifies interactions between factors and then learn goal-conditioned policies to control these factors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Critical:\n-\tImprove writing and fix the typos, missing definitions etc.\n-\tDiscussion regarding hyperparameters (See weaknesses).\n\n# Other\n-\tI suggest that the authors cite the conference or journal version of articles and not the arXiv versions. Some examples are Sharma 2019 and Wang 2022.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sbn45hxP91", "reviewer": "Reviewer_TzJM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method COInS is able to automatically discover useful relationships in the two test domains, with performance that is roughly equal or better than other hierarchical methods, either in training time or reward.\n\nGiven free choice of environments, it would be nice to see a slightly larger difference in performance: rainbow and SAC were strong contenders in all of the experiments. I also wondered how much success comes from the combinations of hand-generated factors, and parameter choices governing the causality tests. Could the tuning of epsilon_act, epsilon_pas, and epsilon_SI (and other parameters) be inadvertently acting as a selection of the skill chain?\n\nThat said, it is good that COInS does at least do better than baseline RL in the chosen domains, given a number of existing HRL methods did not.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces COInS, a hierarchical reinforcement learning method for factored state spaces. COInS uses a test based on the Granger causality test to build a chain of controllable factors, where the goals of a child factor are the actions of the parent factor.\n\nThe authors provide experimental results for the proposed method in Atari Breakout and a custom block-pushing problem in the physics-based Robosuite, compared to a number of baseline and basic RL methods. The authors also look at transfer learning in a direct and fine-tuned setup to modified versions of the training domain.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "All the issues / questions below are relatively small concerns:\n\n\n“Unlike reward-based skills, humans, even as young as infants, exhibit exploratory behavior without top-down rewards.”\nPossibly weaken this statement very slightly, maybe something like “without obvious top-down rewards”.  Otherwise, I’m not sure that’s inarguably true: there are arguments made for humans as an agent having a complicated reward function, with some preference for novelty for its own sake in some more abstract features.\n\neq’n 3 and 4 – should script N and N be the same?\n\nEq 5 and 6, log likelihood, and negative log likelihood as loss.\nAs the text stands, it actually seems to be saying the eq’s 5 and 6 are negative log likelihood. It might be more clear to say “we train m^pas and m^act to maximise the log likelihood of the observed data: eq 5 and 6”.  If you really want to be clear you’re minimising negative log-likelihood add something like “(by using \\negative log-likelihood as the loss function.)”\nRelated, the following paragraph about a high Markov-Granger score would be more clear without flipping back and forth between loss.\n\nsection 4.3: “With l as define is Equations 5 6”  5 and 6\n\n1 - epsilon_act:  Why the choice of 1-epsilon for a log likelihood, rather than just -epsilon or even epsilon?  This seems like it’s pointing at probabilities instead of log likelihood, where there’s a natural sense where 1-p is meaningful.\n\n“states with high passive error log f”\nwhat is f?\n\n“In cases (1) and (2), l(s_b,s_a,s’_b;psi) will be high”.\nAs earlier, with log-likelihood and loss=negative log likelihood, it would be more clear if the text avoided flipping between views with different signs: l_pas where low is a low probability prediction, “predicts with low accuracy” where low is a low probability prediction, and “high passive error” where low is a high probability prediction.\nThis leads to “In cases (1) and (2) l_act(...) will be high”.  The probability l_act / accuracy of m^act will be low when the environment is highly stochastic or a is not a relevant causal factor. The loss – which is not l – will be high.\n\n“When searching for general MG-causal relationships, the score can often be”\nWhat score? A causal score hasn’t been introduced yet, only the non-causal score of eq’7.\n\n“Where N_int represents is the number of detected interactions in D”    (remove “is”)\nDefine N_int. It’s presumably sum_D h(), but the reader shouldn’t need to presume.\n\nEq 11\nIs this work assuming features are in [0,1]? Section 3.2 only mentions “a fixed length vector of real values we call features”.  However, the use of ||s-u|| > 1-epsilon seems to suggest that the features are in a [0,1] range.\n\n“then the goal space is discrete, otherwise, it is continuous”\n“... the goal space is treated as if it is discrete, otherwise it is treated as continuous\" ?\n\nCheck the use of citations throughout: there are a number of sentences like “The goal-based RL used to train our goal-reaching skills utilizes hindsight experience replay Andrychowicz et al. (2017)” The citet citation should either be in the sentence like “.. experience replay of \\citet{Andrychowicz2017}” or a parenthetical like “... experience replay (\\citep{Andrychowicz2017})”\n\nHIntS – earlier choice of name?  There are a few occurrences of this name.\n\nWhat are the choices of epsilon_act and epsilon_pas used in the experiments?\n\nCenter obstacles: what is the penalty? Is there a reward (+1?) for breaking a non-obstacle block?", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PDRrZVKIDD", "reviewer": "Reviewer_JuHF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The problem is well-motivated. The idea of using casualty mask to devise unsupervised reward signal is interesting.\n\nWeakness: Many of the notations are not clear. The technical contribution lacks clarity which hinders reading. Please see the requested changes for more questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes COInS, that learns a chain of skills using detection between state factors. This provides unsupervised reward signals. The method first detect target and source goal state factors, based on which a reward function is defined at any state. Then the algorithm optimize for an RL policy and construct a chain of skills. The method gives competitive performance on two environments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The idea of using Granger causality for HRL is interesting. However, I find it difficult to understand the techniques developed in this paper as many of the notations are confusing. There also lack detailed explanation of how each component works and their implication. I request the authors to rewrite the methodology part. Some examples:\n\n1.  In section 3.2, should it be $S=S_1\\times...,\\times S_n$? It took me a while to realize that $\\bf{s}_a$ refers to the factor $a$ of the state.\n\n2. In section 3.3 the definition of $\\pi$, what does it mean by this policy if the main argument $A_\\omega$ is an action set?\n\n3. $C_{\\omega_i}$ always in $S$?\n\n4. Equations (3) and (4) are confusing. What does the notation $S\\rightarrow \\mathcal{N}(\\mu, \\Sigma)$ even mean? What is $\\mu$ and $\\sigma$\n\n5. What exactly is the meaning of a passive test? I suggest the authors spend more text on explaining passive test and active set as they are key to your algorithm. How is these function trained?\n\n6. What exactly is \"target to control\"?\n\n7. In equation (7) why the first term is scaled by $1/D$ while the second is not?\n\n8. How is data collected? And how does the data distribution affect the quality of the algorithm?\n\n9. It is still unclear to me how a chain of skills is learned by the reward function (12). And what exactly is the benefit of your method? Is it learning a better representation for the neural nets that can facilitate downstream tasks?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a hierarchical RL algorithm that discovers a chain structure in decision making using Granger causality. The paper was rejected by TMLR in 2023 because it was not written clearly. The notation was highly ambiguous even when I read the paper. The revision was reviewed by the same reviewers. The reviewers still pointed out errors and typos, which were promptly corrected by the authors. This was mostly because the authors changed notation to make the paper clearer. All three reviewers support the acceptance of this paper and so do I.
Decision: Accept (poster)