Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning
Authors: Caleb Chuck
Fan Feng
Carl Qi
Chang Shi
Siddhant Agarwal
Amy Zhang
Scott Niekum
Venue: ICLR 2025 Poster (2025)
Keywords: Goal Conditioned Reinforcement Learning
Factor Interactions
Factored State
Hindsight Experience Replay
Counterfactual
Primary area: reinforcement learning
TL;DR: First, introduces a method for inferring general factor interactions using a counterfactual test on learned models, then integrates interactions into hindsight relabeling to improve the sample efficiency of GCRL

Abstract: Hindsight relabeling is a powerful tool for overcoming sparsity in goal-conditioned reinforcement learning (GCRL), especially in certain domains such as navigation and locomotion. However, hindsight relabeling can struggle in object-centric domains. For example, suppose that the goal space consists of a robotic arm pushing a particular target block to a goal location. In this case, hindsight relabeling will give high rewards to any trajectory that does not interact with the block. However, these behaviors are only useful when the object is already at the goal---an extremely rare case in practice. A dataset dominated by these kinds of trajectories can complicate learning and lead to failures. In object-centric domains, one key intuition is that meaningful trajectories are often characterized by object-object interactions such as pushing the block with the gripper. To leverage this intuition, we introduce Hindsight Relabeling using Interactions (HInt), which combines interactions with hindsight relabeling to improve the sample efficiency of downstream RL. However, interactions do not have a consensus statistical definition that is tractable for downstream GCRL. Therefore, we propose a definition of interactions based on the concept of _null counterfactual_: a cause object is interacting with a target object if, in a world where the cause object did not exist, the target object would have different transition dynamics. We leverage this definition to infer interactions in Null Counterfactual Interaction Inference (NCII), which uses a ``nulling'' operation with a learned model to simulate absences and infer interactions. We demonstrate that NCII is able to achieve significantly improved interaction inference accuracy in both simple linear dynamics domains and dynamic robotic domains in Robosuite, Robot Air Hockey, and Franka Kitchen. Furthermore, we demonstrate that HInt improves sample efficiency by up to $4\times$ in these domains as goal-conditioned tasks.

Published as a conference paper at ICLR 2025
NULL COUNTERFACTUAL FACTOR INTERACTIONS FOR GOAL-CONDITIONED REINFORCEMENT LEARNING
Caleb Chuck' : Fan F3ng2.3, Carl Qi', Chang Shi', Siddhant Agarwal', Amy Zhang', Scott Niekum
1 The University of Texas at Austin University of California San Diego
3 MBZUAI University of Massachusetts Amherst
ABSTRACT
Hindsight relabeling is powerful tool for overcoming sparsity in goal-conditioned reinforcement learning (GCRL), especially certain domains such as navigation and locomotion. However, hindsight relabeling can struggle in object-centric domains. For example, suppose that the goal space consists of robotic arm pushing a particular target block to a goal location. In this case, hindsight relabeling will give high rewards to any trajectory that does not interact with the block. However, these behaviors are only useful when the object is already the goal-an extremely rare case in practice. A dataset dominated by these kinds of trajectories can complicate learning and lead to failures. In object-centric domains, one key intuition is that meaningful trajectories are often characterized by object-object interactions such as pushing the block with the gripper. To leverage this intuition, we introduce Hindsight Relabeling using Interactions (HInt), which combines interactions with hindsight relabeling to improve the sample efficiency of downstream RI. However, interactions do not have a consensus statistical definition that is tractable for downstream GCRL. Therefore, we propose a definition of interactions based on the concept of null counterfactual: cause object is interacting with target object if, in a world where the cause object did not exist, the target object would have different transition dynamics. We leverage this definition to infer interactions in Null Counterfactual Interaction Inference (NCII), which uses a "nulling" operation with a learned model to simulate absences and infer interactions. We demonstrate that NCII is able to achieve significantly improved interaction inference accuracy in both simple linear dynamies domains and dynamic robotic domains in Robosuite, Robot Air Hockey, and Franka Kitchen. Furthermore, we demonstrate that HInt improves sample efficiency by up 4x in these domains as goal-conditioned tasks.
INTRODUCTION
Reinforcement Learning (RI.) has made great strides when applied to specific tasks with clear, well-designed rewards (Silver et al., 2018; Wurman et al., 2022; Trinh et al., 2024) but learning generalist policies remains open problem. This is because vanilla RL is formulated as the optimization of a single reward function. Goal-conditioned Reinforcement Learning (GCRL) offers powerful mechanism of generalization by conditioning the learned policy
variety of goals. These goals can be parameterized by a particular setting of an object, such as hitting a puck to goal location in air hockey or moving target block to goal position robotics task. One challenge of goal-conditioned rewards is their sparsity. Without additional inductive biases, making this reward dense is challenging and often requires significant exploration (Andrychowicz al., 2017: Fang et al., 2019). This sparsity is especially significant in combinatorially complex domains, such as room with many objects or a scene with numerous state elements. the desired behavior requires the policy to induce a chain of interactions to achieve the goal, the agent often lacks the opportunity to observe goals and, as a result, does not receive meaningful feedback.
ddeotts equal authorship.

---

Published as a conference paper at ICLR 2025
Actual Outcome
Null Counterfactual
Actual Outcome
Counterfactual
Target Object
Target Object
Target Object
Terget Object
Cause Object
Cause Object "Nulled"
Cause Object
Current state
Current valocity
Next velocity
Next state
(a) Implies Null Interaction
(b) Null Interaction
Figure 1: Figure (a) shows case when null counterfactual interaction occurs between the cause object and the target object, by comparing the actual event (left) with the null counterfactual ("nulled") event (right) and observing difference in the target velocity. Figure (b) shows when interaction does not occur since the actual event matches the null counterfactual for the target object.
Hindsight (Andrychowicz et al., 2017) offers a promising direction for receiving rich feedback by relabeling the achieved object state as its intended goal. This allows an agent who may not be achieving many or any of the desired goals to receive learning feedback. Recent methods have added further nuance to hindsight relabeling, allowing for smooth rewards based on distribution matching between the goal and observed state (Ma et al., 2022; Sikchi et al., 2024a). However, while hindsight provides impressive improvements in settings when the goal space is relatively uniform, hindsight still struggles in combinatorial domains. This is because two fundamental distribution mismatches: (i). the distribution of desired goals often does not match the states reached by the agent early in training; (ii). the distribution of high reward behaviors, i.e. action sequences, under the hindsight distribution often differs significantly from those that induce high rewards under the true test distribution.
To see how this is a challenge for hindsight, consider the example of a robot in a large room. Imagine robot agent tasked with moving block to goal location in a large room, where the starting position of the block and goal are randomly initialized anywhere in that room. Hindsight relabeling will then give high rewards to trajectories where the block does not move at all. However, the desired goals are unlikely to be initialized with the block already inside the goal, so there is a mismatch between the actual and hindsight distribution. Furthermore, when the object is already the goal, the optimal behavior is to avoid interacting with the block so that the agent does not push it out of the goal. This behavior is highly specific to states where the block is already the goal- other relative goal positions, it is necessary to interact with the block. These intuitions suggest that filling hindsight buffer with mismatched trajectories will limit sample efficiency.
In this work, we focus on a simple and promising intuition: filtering the hindsight buffer to contain action-depenie deiiedeet interactions. In our example, this will entail only keeping hindsight trajectories where the agent actually pushes the block. These trajectories better match the desired goal distribution, since the agent must have shifted the object from the initial position. They also contain more meaningful behaviors, because by definition the agent must have exerted some control on the target object. However, compared to Noomann-speciti heuristics, such as checking the target object has moved, interactions can apply any relationship between primitive agent actions, and effects. For example, consider dynamic domain such as robot air hockey, where the target object (the puck) is moving, but there are often still many trajectories with low-scoring goals (letting the puck fall the floor) that would have significantly different goal distributions and desired action distributions. By contrast, an interaction-baas -based filtering method requires principled model of interactions.
Identifying interactions through causal relationships has shown recent headway in several applica- tions (Chuck et al., 2020; 2023; Wang et al., 2023; Hwang et al., 2023), and recently incorporated formalization inspired by actual causality (Chuck et al., 2024b). Under these formulations, an interac- tion between one object (the cause) and another (the target) is when the cause object induces an effect in the target. In this work, we incorporate a general inductive bias we describe as null counterfactuals. Null counterfactuals describe the counterfactual states where an object does not exist (i.e. nulled) and everything else is the same. Then, an interaction between the cause and the target occurs when the outcome changes when the cause is nullled. For example, consider Figure where two balls set to bounce off each other. If removing the red ball changes the outcome, then the red ball is interacting with the blue ball. In this work, we take this intuition, formalize it, and learn models to identify interactions.
2

---

Published as a conference paper at ICLR 2025
To identify the interactions, we introduce the Null Counterfactual Interaction Inference (NCII) Algorithm, which uses a masked dynamics model using trajectories with varying subsets of the causal variables in the environment, and then queries that model to identify the interactions. Then, we introduce the Hindsight Relabeling using Interactions (HInt) algorithm, which uses interaction inference to filter the trajectories added to the hindsight buffer, prioritizing action-induced interactions. Empirically, we demonstrate that NCHI matches the performance achieved by prior work in the Random Vectors domain (Hwang al., 2023; Chuck et al., 2024b), demonstrating the efficacy of the method, and performs well in simulated robotics using Robosuite (Zhu et al., 2020), Robot Air Hockey (Chuck et al., 2024a), and Franka Kitchen (Gupta et al., 2019). Next, we demonstrate that Hlnt improves RL performance compared to recent baselines of GCRL and Causal RL baselines, even when using interactions it identifies, through evaluations on variations of goal-conditioned Spriteworld, Robot Pushing, Robot Air Hockey, and Franka Kitchen.
To summarize the contributions of this work:
We introduce the Null Counterfactual Interaction Inference (NCII) for inferring interactions, and the Hindsight Relabeling using Interactions (HInt) algorithm for improving hindsight in GCRL. We provide empirical evaluation of NCII compared existing actual cause inference methods in Random Vectors, Spriteworld, Robosuite, Robot Air Hockey, and Franka Kitchen domains when using ground truth variable state We evaluate the efficiency of Hint applied to GCRL in Spriteworld, Robosuite, Robot Air Hockey, and Franka Kitchen.
RELATED WORK
Interactions have been applied in variety of ways for reinforcement learning (Buesing et al., 2018), particularly for understanding compositional relationships between objects and causal relationships among object interaction events. Several analogous ideas have been proposed in the RL and robotics literature for detecting interactions between objects, including local causality (Pitis et al., 2020) and controllability (Seitzer et al., 2021). Other related approaches include identifying causes using changepoint detection (Chuck et al., 2020), Granger-causality based tests (Chuck et al., 2023), point- wise conditional mutual information (Seitzer et al., 2021), model gradients (Wang et al., 2023), graph networks (Feng & Magliacane, 2024), context-specific invariance (Hwang et al., 2023), contacting inference for rigid bodies (Manuelli & Tedrake, 2016; Liu al., 2024) and using interventional data (Baradel et al., 2019; Lippe et al., 2023). These methods then use interactions in the context of hierarchical reinforcement learning (Chuck et al., 2023), exploration (Seitzer et al., 2021; Wang et al., 2023), model-based RL (Wang et al., 2022) and data augmentation in RL. (Pitis et al., 2020; 2022; Urpi et al., 2024). The HInt algorithm is a novel application of interactions to hindsight relabeling in goal-conditioned reinforcement learning, while NCII is a novel counterfactual inference algorithm for identifying factor interactions.
Recent work by Chuck et al. (2024b) proposes unification of the correlational and heuristic definitions of state-specific cause-effect relationships with those described by actual cause. The actual cause problem identifies the variables that are the cause of an effect in a particular state, which is a causal interpretation of factor interactions. Our work builds upon prior definitions of actual causation (Pearl, 2000; Halpern, 2016), which have often used contrastive necessity (Beckers, 2021) to define actual causes. However, these methods are not typically applied outside the context of discrete states with a small state space. Unlike these methods, NCII includes an assumption about the "null counterfactual state" of objects, which makes the problem more tractable, though it assumes an inductive bias about the existence of a null state.
Goal-conditions Reinforcement Learning (GCRL) (Puterman, 1990; Kaelbling, 1993) has been investigated as a way to learn multiple behaviors from sparse geal-nea reward (Liu et al., 2022). GCRL has seen significant success learning to achieve complex behaviors (Chane-Sane et al., 2021) through image-based goals (Nair et al., 2018) 8) or goals in a learned latent space (Khazatsky et al., 2021). !). The last work is particularly relevant, since it generates interaction goals, though from prior experience. Hindsight experience replay (HER) (Andrychowicz et al., 2017) or generalized HER (Li et al., 2020) reduces the challenge of learning from sparse rewards by relabeling failures with the achieved state. However, this biases the distribution of goals to those visited by the agent (Lanka & Wu, 201 8), and induces bias in the visited states in the replay buffer (Bai al., 2021). Both
3

---

Published as a conference paper at ICLR 2025
theoretical analysis (Zheng al., 2024) and empirical methods have been applied to modify the hindsight resampling strategy, including curriculum-bast based sampling (Fang et al., 2019), curiosity- based sampling (Zhao & Tresp, 2019) and maximum-entrop regularized sampling (Zhao et al., 2019). Hint offers an alternative and complementary method by using interactions as an inductive bias on useful states for resampling. By modifying the distribution of hindsight goals to those with interactions, HInt draws parallels with distributional representations of GCRL (Ma et al., 2022; Sikchi et al., 2024b) In this context, modifying hindsight an alternate way of providing dense signal (Agarwal et al., 2023) while still matching the desired goal distribution.
3 PROBLEM FORMULATION
3.1 GOAL-CONDITIONED REINFORCEMENT LEARNING
A Markov Decision Process (MDP) is formalized with the tuple AS r. p). where S is a state in a state space. In this work, we focus on Factored MDPs (FMDP) (Kearns & Koller, 1999; Boutilier al., 2013), where the state space is factored into n factors: S = Si x ... Sn. a E A is an action in the action space, and p( ls. a) is the transition probability. The reward function is S x A - R. In the goal-conditioned RL setting (Kaelbling, 1993). g E G is a goal in the space of goals, and r: S x G + R is sparse, goal-conditional reward. The policy x (alss, g) outputs a distribution over actions conditioned on the state and goal. The objective of RL is to maximize the expected return. For trajectory defined of trajectories induced by a policy π as d and goal g. the objective of goal-conditioned RL is to maximize the expected return (over transition dynamics, policy and goal distributions). The expected return of a state and goal is: ret(s,g) == Fgg-G.r-da,7 3/=oh'r(stti
sequence (So, 0o, ...,, ST, @T). space of trajectories T, distribution
880 = s]-
One key insight in practically applying goal-conditioned RL is hindsight, where the desired goal is replaced in hindsight with the actual goal reached by the agent. Note that in this setting, we operate under the formulation where the goal space does not need to be the same as the state space. For example, the goal space the state of a single state factor. Formally, describe gs dā,g as the desired goal, sampled from the distribution of desired goals dá,g. This distribution is assigned by the goal-conditioned RL setting and assigned at the start of the trajectory. By contrast, hindsight uses goals sampled from the space of states induced by the policy. P(s) m is the probability of a state given a policy, we can describe this distribution of hindsight goals induced by a as dx.g = (() Then, Hindsight relabeling induces the distribution of goals daind Mix (ds,g, da,g), where Mix. denotes mixture distribution with coefficient or: Mix.(M) (21,, asa + (1 a)
3.2 INTERACTION INFERENCE
In order to identify interactions, this work builds on formulations from Chuck et al. (2024b) to describe an interaction a directed connection between set of state factors X drawn from S Sn, and particular outcome S = s/. In previous work, identifying this edge requires identifying a globally minimal invariant set of the cause state, which is intractable. This work replaces that with the following null assumption:
Definition 3.1 (Null Counterfactual Interaction Assumption). Suppose that for any state factor S,, there exists a null state Sg, which is when that factor is not present. Define s so as the counterfactual state where is exactly the same except for S, = Si,o. Then if the observed transition has a different probability under this state:
r(S, = s/|S = s) + p(S) = $/15 = s 8t),
S, considered one of the state factors interacting with S, in state transition s, a, s'. We describe this comparison as St being "nulled."
This definition describes the intuition in Figure 1, where mull-counterfactual interaction is defined when the transition is modified by replacing the state of a "cause" object with its null state. Leveraging this definition, NCII takes advantage of the null assumption to learn a model that can infer which state factors are interacting through counterfactual test.
We add the probability mass/density functions, scaling them appropriately to ensure they still sum/integrate
4

---

Published as a conference paper at ICLR 2025
4
METHODS
This section is separated first into the NCII algorithm, which infers interactions by null counterfactual interaction assumption, followed by the Hint method, which uses inferred interactions for hindsight.
4.1 NULL COUNTERFACTUAL INTERACTION INFERENCE ALGORITHM
The objective of the NCII algorithm is to identify which state factors are interacting in given transition. Formally, take a factored state transition s, a, s' and identify an n x (n + 1) matrix B E [0, 1 [0, 1] == B, where B,, 1 indicates that S, is interacting with Sj in the state transition, and IB, the row of all state factors interacting with Sy. The n + 1 column corresponds to the actions.
Null-counterfactual interactions are defined by the difference between the next couttrruntrralt state 8/ and the actual next state s/.. Since cannot actually observe the counterfactual outcome, we instead query a learned forward dynamics model. NCII learns a masked forward model f X A x B P(-SS, A), which predicts the distribution over the next state. The prediction of sj is masked by B. so 1B,, = O, this is equivalent to factor S; not being present in that state state-taking on the null state. We can then query this model to identify whether an interaction occurs in a state nulling a variable significantly changes the probability of the observed outcome S.
To train the masked forward model / (s, a, O), assume that the null state exists in the dataset. We include this in two ways: first, we assume that in each trajectory Tke there is subset of factors V*, represented with a binary vector. Then, each state tuple augmented with Obviously, not all tasks exhibit this property, and we discuss an alternative method that relies on passive modeling to simulate this null effect in Appendix D. Given dataset D of state-valid-action next state tuples, we can construct valid-dependen matrix B(v) where the column i is zero if V; = O. This corresponds to a trajectory where the object i is not present, which is only possible in settings where each trajectory can contain different subset of the state factors. Then we learn by maximizing the log-likeldoood of the outcome of the dataset, where f(s, a, IB (v)) 0)(s)] denotes the probability of s':
give (s, V, a, s').
max
Σ (s.va,,"))>D
log f(s, a, a,B(():)[[s)
The null operation for state factor S, is then comparison between the forward model with and without nulling of the ith causal variable. This is performed by defining IB (vv) S₁ as the row with index BB BVv);: - O. Then identify an interaction if the difference is greater than eull
compare the predicted log-likelihoods of the actual state 8j. and
Null a, a, = 1(108 (log (J a, a
/(s. a, B (,;0)[(s)))
Enall
observe null states can result in limited distribution. This is because, even the null state is equivalent, the duitrf-diittiuution nature of the prediction can result in a low predicted log-likelihood, falsely suggesting an interaction when none exists. To combat this, we utilize an iterative joint optimization process where the forward model is retrained with B(v) replaced by the null counterfactuals, as estimated by an inference function
However, relying only
x A B.
To learn h(s, a; Φ) B. which we call the interaction model, we take Null a, 0)5 every state under the current forward model as the targets for h. and optimize h using binary cross entropy:
min
Σ
Null(s, a, 0) log(h(s; Φ)) (1 Null(s, a, 0)) log h(s; Ø)).
(4)
We use the outputs of h instead of Null(s, 0),; directly because 1) trained model may extrapolate better as opposed to a statistical test utilizing a possibly inaccurate forward model: 2) inference using the null test is ((n?) evaluate each B,; using the forward model, but ((1) as inference model; and 3) h can output soft predictions rather than binary values, which make the optimization more smooth. To jointly optimize the forward and null models iterate between the following steps:
5

---

Published as a conference paper at ICLR 2025
A
A
A
A
A
A
S
S
S
S
S
S
S
S
S
S
S₂
S₂
S tar
S Star
S tar
S tar
S tar
S
tar
t=
t= 2
= 3
t 4
5
t
6
S Control
S Control
Figure 2: An example of the unrolled dynamic interaction graph, where an edge indicates an interaction B 1 from to L 1. HInt identifies trajectories where the agent exerts control on the target object, as measured by path in the unrolled interaction graph. The colors indicate the timesteps during which each state factor is controlled: S₁ is controlled from t controlled, and Sur is controlled from / = 4 to t = 6.
6, S2 not
1. Train f with fixed h to maximize the log probability of the dataset using Equation 2.
2. Train h using Equation 4 to predict the null operation outputs of computed using Equa- tion 3.
In practice, learning to identify interactions is challenging when interactions are rare events, such robotic manipulation domain where the robot rarely touches the object it should manipulate. These challenges have been approached in prior work Chuck et al. (2023; 2024b) by reweighting and augmenting the dataset using statistical error. We include description of how those methods adapted and used in this algorithm in Appendix D.
4.2 INTERACTIONS FOR HINDSIGHTT
Next, we use interactions select hindsight trajectories (Andrychowicz et al., 2017) for GCRL to improve sample efficiency and performance. As intuition, consider Robot Air Hockey example where the objective is to hit the puck to goal position, where the environment will reset when the puck hits the agent's 's side of the table. Vanilla relahiingg relabeling will reassign hindsight goals to wherever the puck hits the agent end of the table, which would incentivize missing the puck. This scenario and the block- block-pushing scenario in Section 1, illustrate situations where hindsight relabeling can hurt performance. Keeping only relabeled trajectories where the agent exerted control on the target object can mitigate this issue. HInt identifies these trajectories using interactions.
Formally, assume a goal-conditioned setting where the goal space consists of single target factor i,2 such that 9ach S + Sy. Then for any particular trajectory of length T and desired goal gdes- the hindsight (relabeled) goal is giel = Each (s((()()))) where vanilla hindsight selects f, = T. : {IB(1) B(T)} is the sequence of interaction graphs for the trajectory. Hint adds the trajectory if passes filtering function x(B) + i, Lr. where is a binary decision reject from the hindsight buffer, and 4, E (1,..., T} is the timestep of a suitable hindsight goal.
Intuitively, Hint should keep trajectories where the agent's behavior is responsible for the outcome. Interactions give us factored way of identifying this responsibility. Each edge (i, in the interaction graph (t) E B denotes that factor exerts an effect on factor j. In the time unrolled graph, where B()) are the edges at time / (see Figure 2), a path in the unrolled graph between actions and the target object implies sequence of control where the agent exerted effect the target object.
To identify this path, construct the temporal graph corresponding to the trajectory, where there are n + 1 nodes for each timestep, and (31) describes the edges from timestep to timestep + 1. Note that the "passive" edge represented by B,j is always 1. Then, identify the set of timesteps Timer as
without loss of generality to multiple factor goals

---

Published as a conference paper at ICLR 2025
(a) Spriteworld
(b) Robosuite
(c) Air Hockey
(d) Franka Kitchen
Figure 3: Visualizations domains used for evaluation. Goals are in green and target objects are in red (except for Franka Kitchen domain).
those for which there is path from the action node n. + 1 to the target variable S.. The states are visualized an example in Figure Then, t, is sampled from Timer. If this set is empty, then = 0, suffer
Lc. the trajectory is rejected for hindsight relabeling. In practice, identifying this path from compounding model errors, since it relies on accurate interaction inference for every factor, so propose several simpler alternatives in Appendix E. In particular, limit the length 1 - 3 of chain in the graph including actions, which we call the "control-target interaction" criteria.
Putting it all together, the Hindsight Relabeling using Interactions algorithm is summarized in Algorithm and Appendix Figure 6.
Algorithm Hindsight relabeling using Interactions (HInt)
Input: Goal-conditioned MDP E with target variable S, Learned Null Model f(s, a, 13; 0), (sss; o) Initialize Dataset D, hindsight dataset H. empty sets of state transitions. Off-policy Goal- conditioned RL algorithm A x D + tb which outputs policy T(, a, (5).
repeat
Policy Rollout: Utilize Goal-conditioned RL policy π (s, a, gitss vb) and null model / to collect trajectory and interactions B 1 (..r) and add D. Hindsight Relabeling: Test (B) ) - 1, tr and update H with the trajectory T if = 1, selecting hindsight goal Imch (s)+))) sampled uniformly from Timer- Policy Update: update policy vi + A(v, DU H) until di converges
EXPERIMENTS
In this section, we aim to answer the following questions: (1) How does the NCII perform compared with existing interaction inference algorithms? (2) Does Goal-conditioned RL benefit from hindsight relabeling using interactions? (3) How does the performance of HInt compare when using inferred interactions from NCII vs ground truth interactions? The first question is evaluated in Section 5. and the other two in Section 5.2.
Before going to the results, we provide some details on the domains visualized Figure 3. Each domain has several variations that change the number of factors (an "obstacles" variant) and the size of the factors (a "small" variant) in physical domains (making interactions less frequent). These changes further discussed in Appendix H.
Random DAG Null (Hwang et al., 2023; Chuck et al., 2024b) is a simple domain to test inter- actions represented by arbitrary functional relationships between state factors. For the parents of S, represented with X, normally distributed noise variable Vi A(, 1) and [s,,sy] denotes the concatenation the two vectors, the next state S is determined by:
8/ = X Ex,ex [(c [s., s, > 0) A.(s,, sy] Es, s.t.
oj [s, s,] >
(5)
Ays;
otherwise

---

Published as a conference paper at ICLR 2025
To support nulling, each length 50 trajectory randomly samples subset of the factors. This domain assesses whether inference algorithms can recover arbitrary linear interaction relationships in the null setting. n-in indicates random DAG domain with n factors.
Spriteworld Null (sprite) (Watters et al., 2019) is a suite of environments based on Spriteworld, where 2D polygons and circles collide in frictionless O-gravity environment. To make this domain goal-conditioned, a "target" sprite is used to define the goal space, and the agent controls a single sprite manipulate the target to the goal. This domain assesses challenging control of striking and securing the frictionless target in the goal. Sprite-n indicates a Spriteworld with n objects.
Robosuite (robo) (Zhu al., 2020) is 3D robotics domain where the agent controls 9-DOF robot arm through end effector control to move a block to a target location in the midst of obstacles. This domain assesses a 3D tabletop pushing domain with quasistatic dynamics (the objects do not move unless manipulated).
Air Hockey (air) (Chuck et al., 2024a) is a dynamic robotics domain where the agent controls a paddle to strike puck to desired target location with constant gravity pulling the puck towards the bottom of the table. The domain resets the puck hits the bottom the screen. This domain assesses 2D dynamic domain where some achieved goals (the puck at the bottom) are never desired goals (all goals are initialized on the upper half of the table).
Franka Kitchen (kitchen) (Gupta et al., 2019) is a 9-DoF Franka robot arm to complete series of tasks a simulated kitchen environment, including opening the microwave, turning on the light switch, and opening the sliding cabinet. This domain asses the control over a 3D simulated kitchen where the robot needs to interact with various objects to achieve the desired goal configuration.
In Spriteworld. Robosuite, Air Hockey, and Franka Kitchen, instead of using explicit null trajectories we use simulated nulling, as described in Appendix D.
5.1 INFERENCE EXPERIMENTS
When assessing inference, we compare against several previous methods that cover wide range of techniques for identifying interactions: JACI (Chuck et al., 2024b), gradients (Wang et al., 2023), attention weights (Pitis et al., 2020), and Neural Contextual Decomposition (NCD) (Hwang et al., 2023). We test NCII with two architectures, Pointnet (Qi et al., 2017) and Graph Neural Network (GNN) (Scarselli et al., 2008; Kipf & Welling, 2017) based architecture, which we include additional details for in Appendix G.
To answer question (1), if the null assumption achieves statistically significant reduction in misprediction rate when compared with the baselines, we perform a comparison on the above domains. We collect a fixed dataset of 1M states for random DAG and 2M states for Spriteworld, Robosuite Air Hockey, and Franka Kitchen. Then we compare accuracy at recovering the ground truth interactions, described by contacts in the physical domains, and 1 [(, [s; sj] > 0) in Random DAG Null. Each method is trained over 5 seeds until convergence. As we observe in Table 1 provided with null data or simulated nulling, this inference method achieves statistically significant reduction in misprediction rate compared to existing baselines.
Null Inference:
5.2 HINDSIGHT RELABELING USING INTERACTIONS
Before describing the empirical evaluation of GCRL, we briefly describe the RL baselines we compare against for this work:
Vanilla: Basic offline goal conditioned RL using Deep Deterministic Policy Gradient (DDPG) (Lilli- crap, 2015) without hindsight.
Hindsight (Andrychowicz et al., 2017): Vanilla hindsight gives a baseline for adding hindsight in the context of rare-interaction domains.
Prioritized Replay (Schaul et al., 2015): Prioritized replay with hindsight uses TD error to prioritize high emrrr states, assessing if policy error is sufficient to emphasize desirable states.
f-policy gradients (f-pg) (Agarwal et al., 2023): f-divergence policy gradients use distributional perspective on goal reaching. From that perspective, Hlnt be seen modifying the hindsight
8

---

Published as a conference paper at ICLR 2025
Method
NCII w/ Point
NCII w/Graph
JACI
Gradient
Attention
NCD
1-in
0.8 + 0.2
1.0 + 0.1
1.6
4.5
4.8 1.6
30.3
2-in
1.4 1 0.2
1.4
2.0
33.9
27.3 5.2
22.1 3.4
6-in
25.1 1..1:58 5.8
26.3 5.2
35.4 4.8
31.1
Sprite-2
1.511. 1.2
3.9
1..3
13.7
28.6
25.3 1.7
Sprite
6.5
3.4
15.5
1.
301
23.2 2 2.5
Sprite-6
4.3
2.9
1.3
2.3
1.3
16.0
Air Hockey
11.3 11.31.
2.8
2.3
4.6
3.5
14.3 1.4
Robosuite
5.1+0.3 0.3
7.2
6.2
1.2
43.7
49.5 +9.5
Kitchen
10.0
19.7
3.3
21.8
39.2 + 6.4
Table I: Misprediction rate (lower is better) with standard error of inference in evaluated domains from state. Interactions are reweighted to be 50% of the test dataset. Boldface indicates within ~ 1 combined standard error of the best result.
Epicode Remward
%
0
(a) Sprite default
(b) Sprite small
(c) Sprite obstacles
(d) Sprite velocity
82-
50
3.0
Epiooda Reward
Success
83
25.
23
180
100
0
4H
080
12M
7.5
20m
08
6.5
37M
100
20M
0.0g
036
18
1M
214
14
(c) Air default
(f) Air small
(g) Air obstacles
(h) Kitchen default
Episode Reward
Success
0
(i) Robo default
(j) Robo small
(k) Robo ohstacles
()) Kitchen obstacles
w/ (ours)
(ours)
Hindsight
Replay
Vanilla
Figure 4: Comparison of Hint and HInt with NCII against baselines, 5 trials for each. Hint with NCII used Spriteworld default, small, Robosuite default, Air Hockey default, Kitchen default, and obstacles. The shading indicates standard error. X axis is the number of timesteps.
distribution to better match the distribution of desired goals since desired goals will tend to be gated by an interaction. f-pg explicitly directs behavior towards the desired goal distribution, but without using interactions.
Exploration via Local Dependencies (ELDEN) (Wang et al., 2023): This method uses gradient- defined caused-based interactions for exploration with the ensemble of learned forward models. This compares interactions with hindsight against interactions for exploration.
Causal Action Inffuence (CAT) (Seitzer al., 2021): This method uses pointwise mutual informa- tion with actions to incentivize exploration of controllable states. This compares chains of factor interactions with pure action controllability. Results of CAI are shown in Appendix Figure 12.
5.2.1 HINDSIGHT EVALUATION
Now, we can provide empirical evidence towards (2) Does GCRL benefit from filtering resampling with actual cause graphs? We evaluate Goal-conditioned RL in of the modified Spriteworld domains,

---

Published as a conference paper at ICLR 2025
(a) Desired Goal
(b) Hindsight
(c) After Hint
(d) Filtered Out
Figure 5: Relative position heatmap between initial state and a) sampled "desired" goal, hindsight goals.
<) goals after HInt filtering, d) goals removed by Hint, over 3000 goals in Spriteworld default.
3 modified air hockey domains, and modified Robosuite domains and Franka Kitchen. This variety of domains allows us to assess wide range of interaction rates, kinds of dynamics (collision, quasistatic, articulated, etc.), and numbers of state factors. In this work we applied Hlnt using the action graph filtering strategy in all the domains except the obstacles variants, where found the control-target graph filtering strategy was more stable. Training details and hyperparameters can be found in Appendix I. As we see through the training curves in Figure HInt with ground truth interactions or learned interactions achieves as-good-as or better sample efficiency than existing algorithms, demonstrating an up 4x performance improvement in sample efficiency. While inference is not perfectly accurate at detecting contact, we hypothesize that filtering can occasionally improve accuracy by rejecting contact that only produces minute changes in the target object as result of control. We also visualize in Figure 5 the proposed source of this performance benefit: filtering out trivial goals, such as those where the target object is initialized (in hindsight) at the goal. This clearly visualizes how this distribution is better matched after HInt filtering compared to the hindsight distribution. Lastly, we also visualize some learned policy rollouts in Appendix M.
5.2.2 HINDSIGHT FILTERING USING NCII
Finally, provide empirical evidence for (3) How does the performance of HInt compare when using inferred interactions from NCII vs ground truth interactions? We use a trained NCII model in four domains: Spriteworld default, Spriteworld small, Robosuite default, Air hockey default and Franka Kitchen Default. In Figure 4, we see that interactions identified by NCII are equivalent or sometimes exceed ground truth interactions indicated by contact between objects. We also plot just HInt with ground truth interactions and Hint with NCII interactions in Appendix Figure 7.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
Many real-world tasks, from object pushing to dynamic striking, involve sparse interactions in large spaces. Learning a goal-conditioned policy to achieve arbitrary goals in these tasks often challenging, especially without injecting specific domain information such distance, and intelligent design of the reward function. Furthermore, because hindsight can fall prey to adding trivial rewards, also struggle in these domains. This work demonstrates the effectiveness of applying interactions to improve sample efficiency in GCRL through the HInt algorithm. As interactions become more rare, the additional information provided by HInt through interactions becomes more useful. HInt relies on fundamentally different assumptions from those used in other GCRL methods, future work can investigate integrating it with existing techniques. The effectiveness of HInt is contingent on how the interactions are inferred, and this work also demonstrates powerful physical inductive bias for interaction identification: the null assumption. Leveraging this assumption, NCII uses learned counterfactual models to improve interaction inference. While this leaves some open questions, NCII sufficiently improves the accuracy of inference compared with existing methods, so that it can be integrated with Hint. Altogether, this work offers promising intuitive tool for improving the performance of GCRL real-world environments.
Limitations. While Hlnt is designed to model interactions effectively, it may have limited utility in domains where interactions are less critical, such as certain locomotion tasks. In some cases, interactions might even be detrimental, in driving or drone navigation. However, in these scenarios, a model that effectively captures interactions can still be highly beneficial, as helps to understand and avoid potential collisions conflicts.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work has taken place in part in the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) The University of Massachusetts Amherst. SCALAR research is supported in part by the NSF (IIS-2323384), the Center for Al Safety (CAIS), and the Long-Term Future Fund. This work also took place in the Machine Intelligence through Decisionmaking and Interaction Lab at The University of Texas at Austin, supported in part by NSF 2340651, NSF 2402650, DARPA HR001 12490431, and ARO W911NF-24-1-019. The work was supported by the National Defense Science & Engineering Graduate (NDSEG) Fellowship sponsored by the Air Force Office of Science and Research (AFOSR).
REFERENCES
Siddhant Agarwal, Ishan Durugkar, Peter Stone, and Amy Zhang. f-policy gradients: general framework for goal-conditioned using f-divergences, Advances in Neural Information Processing Systems, 36, 2023.
Marein Andrychowicz, Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Chenjia Bai, Lingxiao Wang, Yixin Wang, Zhaoran Wang, Rui Zhao Chenyao Bai, and Peng Liu. Addressing hindsight bias multigoal reinforcement learning. IEEE Transactions on Cybernetics, 53(1):392-405, 2021.
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Cophy: Counterfac- tual learning of physical dynamics. arXiv preprint arXiv:1909. 2019.
Sander Beckers. Causal sufficiency and actual causation. Journal of Philosophical Logic, 50(6): 1341-1374, June 2021. doi: 10.1007/110992-021-099000
URL https://doi.coo http///oi.ooro../////
1007/s10992 021 121-099 09601- 2.
Craig Boutilier, Nir Friedman, Moises Goldszmidt, and Daphne Koller. Context-specific independence in bayesian networks. arXiv preprint
2013.
Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean-Baptiste Lespiau, and Nicolas Heess. Woulda, coulda, shoulda: Counterfactuulll--giddee policy search. arXiv preprint arXiv: 11111111112101 1.6222, 2018
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International conference machine learning, pp. 1430-1440. PMLR, 2021.
Caleb Chuck, Supawit Chockchowwat, and Scott Niekum. Hypothesis-driven skill discovery for hierarchical deep reinforcement learning. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5572-5579. IEEE, 2020.
Caleb Chuck, Kevin Black, Aditya Arjun, Yuke Zhu, and Scott Niekum. Granger-causal hierarchical skill discovery. arXiv preprint arXiv:2306. 09509, 2023.
Caleb Chuck, Carl Qi, Michael J Munje, Shuozhe Li, Max Rudolph, Chang Shi, Siddhant Agarwal, Harshit Sikchi, Abhinav Peri, Sarthak Dayal, et al. Robot air hockey: A manipulation testbed for robot learning with reinforcement learning. arXiv preprint arXiv:2405.0311. 2024a.
Caleb Chuck, Sankaran Vaidyanathan, Stephen Giguere, Amy Zhang, David Jensen, and Scott Niekum. Automated discovery of functional actual causes complex environments. arXiv preprint arXiv:2404.10%3 2024b.
Meng Fang, Tianyi Zhou, Yali Du Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay. Advances neural information processing systems, 32, 2019.

---

Published as a conference paper at ICLR 2025
Fan Feng and Sara Magliacane. Learning dynamic attribute-factore world models for efficient multi-object reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning:
Solving long-horizon tasks via imitation and reinforcement learning. Conference on Robot Learning (2019), 2019.
Joseph Y Halpemn Actual causality. MIT Press, 2016.
Inwoo Hwang, Yunhyeok Kwak, Yeon-Ji Song, Byoung-Tak Zhang, and Sanghack Lee. On dis- covery of local independence over continuous variables via neural contextual decomposition, In Conference on Causal Learning and Reasoning, pp. 448-472. PMLR, 2023.
Leslie Pack Kaelbling. Learning achieve goals. In IJCAI, volume 2, pp. 1094-8, Citeseer, 1993.
Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In IJCAI, volume 16, pp. 740-747, 1999.
Alexander Khazatsky, Ashvin Nair. Daniel Jing, and Sergey Levine. What can i do here? learning new skills by imagining visual affordances. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pp. 14291-14299) IEEE, 2021.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint
2013.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. networs. In International Conference on Learning Representations, 2017. URL /ttptttts////peerreviie net/ / htre//orrummeoo/eeo (aerrumiidsSSUAAYYyy
Sameera Lanka and Tianfu Wu. Archer: Aggressive rewards to counter bias in hindsight experience replay, arXiv preprint
02070, 2018
Alexander Li, Lerrel Pinto, and Picter Abbeel. Generalized hindsight for reinforcement learning. Advances in neural information processing systems, 33:7754-7767. 2020.
TP Lillicrap. Continuous control with deep reinforcement learning. arXiv preprint 109.0.150,.02971, 2015.
Phillip Lippe. Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Biscuit: Causal representation learning from binary interactions. In Uncertainty in Artificial Intelligence, pp. 1263-1273, PMLR, 2023.
Minghuan Liu, Menghui Zhu, and Weinan Zhang, Goal-conditioned reinforcement learning; Problems and solutions. International Joint Conference on Artificial Intelligence (ICCAI). 2022.
Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics- grounded image-tor to-ieooo generation. In European Conference on Computer Vision, pp. 360-378. Springer, 2024.
Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline conl-conditioned reinforcement learning via f-advantage regression. Advances in neural information processing systems, 35:310-323, 2022.
Lucas Manuelli and Russ Tedrake. Localizing external contact using proprioceptive sensors: The contact particle filter. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5062-5069. IEEE, 2016.
Ashvin V Nair, Vitchyr Pong. Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018
Judea Pearl. Causality. Cambridge University Press, New York, 2000.
Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems, 33:3976-3990, 2020.
12

---

Published as a conference paper at ICLR 2025
Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model-based counterfac- tual data augmentation. Advances in Neural Information Processing Systems, 35:18143-1811 2022.
Martin L Puterman. Markov decision processes. Handbooks in operations research and management
science, 2:331-434, 1990.
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 1rXi::551.0059555 2015.
Maximilian Seitzer, Bernhard Schölkopf, and Georg Martius. Causal influence detection for improv- ing efficiency in reinforcement learning. Advances in Neural Information Processing Systems, 34: 22905-222 22918, 2021.
Harshit Sikehi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Nickum. Smore: Score models for offline goal-conditioned reinforcement learning. International Conference on Learning Representations, 2024a.
Harshit Sikehi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Nickum. Score models for offline condiioddt reinforcement learning. International Conference Learning Representations, 2024b.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-1144, 2018.
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476-482, 2024.
Núria Armengol Urpí, Marco Bagatella, Marin Vlastelica, and Georg Martius. Causal action influence aware counterfactual data augmentation. arXiv preprint arXiv:2405.1 28917 18917, 2024.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for task-independent state abstraction. arXiv preprint arXiv:2206.1344 13452, 2022.
Zizhao Wang, Jiaheng Hu, Peter Stone, and Roberto Martin-Martin. Elden: Exploration via local dependencies. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P Burgess, and Alexander Lerchner. Cobra: Data-efficient model-base based rl through unsupervised object discovery and curiosity-driven exploration. arXiv preprint arXiv: 10XXvv9055.0990 1905.09275..... 2019.
Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896): 223-228, 2022.
Rui Zhao and Volker Tresp. Curiosity-driver experience prioritization via density estimation. arXiv preprint 1902:
2019.
Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-regularizzed multi-goal reinforcement learning. In International Conference on Machine Learning, pp. 7553-7562. PMLR, 2019.
Sirui Zheng, Chenjia Bai, Zhuoran Yang, and Zhaoran Wang. How does goal relabeling improve sample efficiency? In Forty-first International Conference on Machine Learning, 2024.
13

---

Published as a conference paper at ICLR 2025
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12299 2020.
14

---

Published as a conference paper at ICLR 2025
APPENDIX NULL COUNTERFACTUAL FACTOR INTERACTIONS FOR GOAL-CONDITIONED REINFORCEMENT LEARNING
TABLE OF CONTENTS
Reproducibility Statement
16
Flow Diagram
16
C Visualization of Hint using Ground Truth or NCII interactions
16
D Simulated Nulling
16
Filter Criteria
17
F Investigating Cuull
18
G Network Architectures
19
G.I Pointnet Architectures
19
G.2 Graph Architectures
19
G.3 Transformer Architectures
20
H Environment Details
20
H.1 Spriteworld
20
H.2 Robosuite
20
H.3 Air Hockey
H.4 Franka Kitchen
21
Training Details
J Hindsight Sampling Ablation
22
K Vision Experiments
23
Additional Comparisons
24
M Rollout Visualizations
25
15

---

Published as a conference paper at ICLR 2025
A REPRODUCIBILITY STATEMENT
We provide the implementation of NCII, Hint, and other baselines in the appendix code. The installation instructions, detailed settings, and configurations for the data generation process for the benchmarks and datasets can be found in Appendix H and the appendix code. The training details, including hyperparameter settings, experimental setups, and platforms, are provided in Appendix
B FLOW DIAGRAM
Interaction Model
Hindsight Filter
Policy
Action
Environment
State
Interactions
Interaction Identified
Next State
Hindsight Trajectory
Replay
Trajectory
Policy Update
Learned Model
State Voctor
Sequence
Sequence Filter
Figure 6: The data flow for the HInt method. An interaction matrix IB is generated at each time step, and at the end each trajectory used to filter non-interacting trajectories.
C VISUALIZATION OF HINT USING GROUND TRUTH OR NCII INTERACTIONS
Epicode Reward
10H
200
4014
10M
Sprite Default
(b) Sprite Small
(c) Robo Default
Air Default
Succass
0.5
282
135
184
2N
3M
(c) Franka default
(1) Franka obstacles
Hint w/ GT
Hint w/ NCII (ours)
Figure 7: Comparison of HInt and Hint with NCII 5 trials for each. The shading indicates standard
еттог.
D SIMULATED NULLING
Nulling is powerful tool for lifelong learning settings, where the agent will experience a large number of trajectories in wide variety settings. However, many RI. domains often only contain few, specific factors and have a sparse number of interactions. It can be expensive observe every distribution of possible states in this setting. However, we can often leverage the fact that interactions rare identify high-priority states using passive error, technique introduced by Chuck et al. (2023).
Intuitively, estimate states where there is unlikely be interaction affecting the target object by observing if the target object exhibits unexpected behavior. If does not, then there likely not interaction. For example in object-based domain, object probably not interacting
16

---

Published as a conference paper at ICLR 2025
with any other object if it maintains its velocity. More specifically, if autoregressive model f : S + 1S)) is a good predictor of the next target object state s/ using only the current target object state sj, then there is most likely not an interaction. The passive signal is then just the log-likelihodd of the observed outcome under the distribution induced by the forward model:
e ejacsive J s¹) == log f [s,)]s
(6)
Where [(s])]s}]] is the probability sf under the distribution from [(s)). When the passive signal is high, this suggests non-interaction.
We use the passive signal in two ways, first, as substitute for the null signal. This because, assuming that there is not any interaction with the target factor in state state under the null assumption (if there is no interaction, then the outcome distribution should equal the null distribution). The passive signal can then be used as a proxy for this circumstance since low passive error implies non-interacttin. In practice, this means taking the null vector v, which again is 1 vector indicating which objects are present, and randomly zeroing certain indices with some probability. This gives the passive-adjusted null vector V:
null out any other
v Bernolli(1 Esim mull)
passive (s). Gpassive
otherwise
Where esim-null is the probability of nulling out a value in high passive likelihood states, and Epassive is the log likehood threshold to be considered high likelihood. We can then train using NII using v instead of v.
The second way during training. In many of the domains, interactions are very infreeuent, ranging from 0.5% to 0.01%. A model learned with these is likely to simply ignore interactions entirely. Because interactions are low-likelihood, however, they also tend to have low likelihood under the passive model f. Consequently, we upweight low passive likelihood states so that they are sampled on average 20% of the time. This is a technique used in both Chuck et al. (2023; 2024b).
the passive enror is to modify the frequency of interaction states observed
E FILTER CRITERIA
The action path criteria for defined in section 4.2 will capture any control exerted by the agent but has three weaknesses. First, because incorporates information about all edges, it can be susceptible to false positives, where a false positive edge in different object could induce path. As an example, imagine the Spriteworld domain with obstacles. If there is a false positive between an object that was controlled and an obstacle that happened to collide with the target object, this would result in a trajectory misclassified as passing the criteria. Second, again because identifying a path requires all edges, this means using the null counterfactual to generate interactions requires an O n²)) operation. Learning these models accurately can be expensive. Finally, while using a binary signal indicates if any action influence was exerted, it does not give good measure of the amount of influence. In practice, this is a challenging problem, but we provide a few heuristic strategies in this section.
First, we define 3 simplifying filtering strategies for X-
1. Non-Passive:
This is a permissive strategy that rejects any trajectory that does not have a non-trivial interaction, and assigns a random state after the first non-trivial interaction as the hindsight goal. Formally, construct (t)) using some subset of the interaction graph and assigning the remaining edges O. Ignore the edges between (i, i), and (action, i) (because action edges are typically dense). If there is not a nonzero edge remaining, reject the trajectory.
EBBE E/= E" B(!) -0.
This formulation allows us to utilize a subgraph while still identifying interactions. However, since not all non-pas -assiise graphs indicate control, can be overly permissive.
2. Mon-passive target: This strategy is simplification of the general non-passive graph by taking only the row j corresponding to the target object to test for interactions. Formally, for target index j. EzwEs Σ B
0.
17

---

Published as a conference paper at ICLR 2025
0.1
Foull
0.3
Gaal
Faall
Caall
faill
1.5
Guall
Salll
Cull
I-in Speite-1
+
1.8
0.5
1.1 1.3
3.1
1.6
1.8
1840044
2.2 0.8
2.4 + 0.9
2.6 4:
5.2
3.3
6.4 + 5.0
21.6
12.6
19.6
27.4 +
39.5
17.9
Table 2: Ablation on Caull parameter (horizontal) for NCII. Note that because Emlll indicates a difference in normalized log-likelihood, the parameter is quite robust. Each is over seeds.
This has the advantage of reducing the computation to a single row of the graph, though it is still overly permissive. However, since non-passivy interaction must occur for control interaction to occur, it will not filter out any interacting trajectories.
3. Control-target interaction: This strategy is the most stringent, and identifies a control factor based on the state factor with the most frequent action edge. Call this factor Then, the criteria simply checking the control factor has an edge with the target factor j. and rejecting trajectories where the control factor does not interact with the target factor. Formally, 4 == Ep B(4) O.
This requires, after identifying the control factor, a single interaction test. However, is stringent, since is not required that the control factor directly connects the target object to exert control. In practice, we identify the control factor by learning 2n models, one for each state factor, frussive (st) P(S) SS sx) and faction (Sk, a) + P(S Sk. A a). Then we identify the factor for which adding helps the most.
The above strategies can reduce some sensitivity to false positives since they only require inference on a reduced subset of edges. For the same reason, they reduce the computational cost. However, they do identify the degree of control. We utilize simple strategy of only keeping a trajectory if the number of interactions according to the testing strategy exceeds certain count. In other words, there should be least some amount interactions if is to keep trajectory. The meaning of frequent based on an interaction count b, which is defined differently for each method.
1. Action Graph: define the interaction count as the number of unique paths:
b == #unique paths from actions to j
A unique path is any path that contains at least one unique edge.
2. Non-Passive: define the interaction count the number of non-passive edges:
b= Σ Σ7
E
f=0
3. Non-Passive Target: define the interaction count as the number of non-passive edges the
target:
b
Σ
Σ
B
4. Control-target:
define the interaction count as the number of control-target edges:
Σ
B
B
Then T'nin interaction number
In practice, we use action- -rag-g interactions in all domains except Spriteworld Obstacles, Robosuite Pushing Obstacles, and Air Hockey Obstacles. In these domains, we use control-target interactions for experimental inference.
F INVESTIGATING ÉXULL
The Emull parameter can have significant effects on the success the algorithm, since selected to be small, this will overestimate the prevalence of interactions, and if too large, interactions will fail
18

---

Published as a conference paper at ICLR 2025
to be identified. While in the experiments for this work found that fixed Coull is sufficient, see Table 2, we provide the following strategies for identifying this parameter using information from learning the null model.
First, learn a null forward model f S A + P(-|S, A) that uses information about all input states predict distribution over S,, in Section 4. Then compute the difference in value from the null operation on the dataset:
as O) = log f(s B(v): f(s, B(v) 0);0){{s)
(8)
Now, objective is to identify the differences that most likely correspond interactions. Non- interactions will generally have low error, since nulling should have no effect on the outcome diff(s, a, 8j:0) 0). Thus, there should be cluster of low likelihood difference outcomes. Since interactions mean that the non-nulled evaluation will be higher likelihood, diff() will be higher for these values, corresponding algorithm to get the interaction and non-interaction clusters, and take the midpoint (or some other in-between point) the mean the two clusters.
the second cluster. In practice, can apply 2-mode clustering
G NETWORK ARCHITECTURES
In this section, we describe the network architectures used this work. The same architecture is used for h, f, the interaction and forward dynamics networks.
G.1 POINTNET ARCHITECTURES
Pointnet (Qi al., 2017) utilizes ID convolution-basee architecture and an order-invariance reduce function. In this work, we utilize a multilayer pointnet, which can re-append an embedding output by the final layer as an input. Our Pointnet uses ID convolution embed the states and action ..... S,, a, masks them with and then follows that with second 1D convolution. This architecture visualized Figure 8.
Input State
Embed
Embed net
Pointnet
S
Embed net
Pointnet
Reduce
Output net
S
Embed net
Pointnet
Figure 8: The Pointnet-based architecture used for interactions. Multilayer methods repeatedly re-append the output the embeddings. Shared colors (green, yellow) denote weight sharing through 1D convolution.
G.2 GRAPH ARCHITECTURES
We use graph neural networks (GNNs) to model the dynamics, while 1D convolutions are applied to embed the state and actions. For message passing GNN, utilize GCNConv layers (Kipf & Welling, 2017), and when nulling out objects, we directly remove the corresponding nodes and its edges from the graph. The architecture visualized in Figure 9.
19

---

Published as a conference paper at ICLR 2025
Input
Embed not
Embed
Embed
GNN
Message Passing
Output
Embed net
V
Figure 9: The GNN architecture used for interactions. Similarly, shared colors (green, ycllow) denote weight sharing through ID convolution.
G.3 TRANSFORMER ARCHITECTURES
To adapt the transformer backbone (Vaswani, 2017), we model the dynamics using transformer and nullify the effects of objects by setting their interactions to zero the cross-attention maps. However, challenge occurs when using multiple cross-attention layers: nulling out attention in one layer does not prevent information exchange subsequent layers. Results from Random DAG domains (Table 3) also indicate that transformers perform worse than PointNet or GNN, which will address in future work.
Method NCII w/ Point
NCII w/ Graph NCII w/ Transformer
1-in
0.8 + 0.2
1.0 0.1
2.6 0.3
2-in
1.4 0.2
1.4 0.2
4.9 1.7
6-in
25.1 5.8
26.3 5.2
41.3 + 8.2
Table 3: Accuracy inference in evaluated domains from states using NCII with different backbones. Interactions reweighted be 50% of the test dataset.
H ENVIRONMENT DETAILS
H.1 SPRITEWORLD
The three 2D Spriteworld domains consist of target ball being pushed goal region by control ball low friction, gravity-free 7x7 meter region. The action space consists of forces upon the control ball, and the goal space is Im diameter circle. The reward function is sparse, with reward outside and reward inside the goal region. Because controlling the control ball is already challenging, and when struck the target ball is likely pass through rather than stay in the goal region, this task can take a very substantial number of interactions to train. In the variants, we add 2 obstacles of similar mass the target ball which impede it from reaching the goal. Ground truth interactions in this domain consist of contacts between the different objects.
The three variants differ according the radius of the goal and target objects, and the presence of obstacles. In the Default domain, the radius of the control and target objects are 0.5 meters, while in the small and Obstacles domains they are 0.3 meters. In the obstacles domain are two additional obstacles, a triangle and a circle, of similar mass to the target object and where the radius of the polygon (distance from centroid to further vertex) and circle 0.5 meters. Each domain uses 100 time steps for trajectory before timing out.
H.2 ROBOSUITE
The three Robosuite domains involve pushing a block to desired target location using end effector control of PANDAs robotic arm. The workspace is a 0.6 x 0.6 x 0.3 meter region in length, width height respectively. The action space consists of desired end effector deltas, using an OSC controller achieve the desired delta position. The goal space is 0.05m diameter circle. Again, the reward function is sparse, with reward outside and I reward inside the goal region. Increasing the dimensionality moving the gripper in 3D space means that interactions even less frequent,
20

---

Published as a conference paper at ICLR 2025
though the task is over much smaller area. In the obstacle variant we add immovable 0.05m blocks to the domain, initialized randomly such that they do not lie on top of the goal. Once again, Ground truth interactions in this domain consist of contacts between the different objects.
The three variants differ according to the size of the target block, and the presence of obstacles. In the Default domain, the cube side length of the target block is 0.015 meters, while in the small and obstacles domain it is 0.007 meters, where the obstacle domain has the two additional 0.05m obstacles. Each domain uses 100 time steps for trajectory before timing out.
H.3 AIR HOCKEY
The three 2D Air Hockey domains consist of a puck being struck into a goal region using a paddle in low friction 2x1 I meter region, where there is gravity pulling the puck down towards the paddle. The action space consists of forces upon the paddle, and the goal space is 0.2m diameter circle. In this domain only,
shaped reward instead of sparse one, although hindsight still proves to be useful because of the complexity of the dynamics. The reward function is densified with a 12 distance, which was necessary for any policy to learn in this domain. The agent receives - 1 Spuck gl reward outside and reward inside the goal region. The challenge in this domain is to use sparse interaction to achieve a downstream effect, with the puck constantly moving in and out of the goal. In the variants, we add 2 blocks of high mass to the target ball which can impede the puck from reaching goal. Ground truth interactions in this domain consist of contacts between the different objects.
The three variants differ according to the radius of the puck and paddle, and the presence of obstacles. In the Default domain, the radius of the puck is 0.03m and puck is 0.05 meters, while in the small and obstacle domain the puck is 0.02 meters and the puck is 0.03 meters. The obstacle domain additionally adds three random obstacles. The goal in all domains 0.15m radius. Each domain uses 100 time steps for trajectory before timing out.
H.4 FRANKA KITCHEN
In the Franka Kitchen domain, a robot with 9 degrees of freedom tasked with controlling various kitchen objects, including the top and bottom burners, light switches, sliding and hinged cabinets, kettle, and microwave. Each object corresponds to a specific sub-task, and the robot must perform a sequence of tasks, where each is associated with the goal positions of its joints. The sparse reward is computed based on the number of completed tasks. Here in the Default domain, the robot's task is to open the microwave (three objects: desk and microwave). In the Obstacle domain, the robot's ss task is to open the microwave and the sliding cabinet, where the additional objects such as the kettle and burner switches added as obstacles.
I TRAINING DETAILS
In this section we describe the hyperparameters and training details for NCII and HInt.
All null experiments were collected with 10 seeds between 0-9. All RL experiments used seeds between 0-4. The experiments were conducted on machines of the following configurations:
4xNvidia A40 GPU; InIeel) Xeon(R) Gold 6342 CPU @2.80GHz
4x Quadro RTX 6000 GPU; 4x Intel()) Xeon(R) Gold 6342 CPU @2.80GHz
4xNvidia 4090 GPU; 8xInteI(R) Xcon(R) Gold 6342 CPU @2.80GHz
2xNvidia A100 GPU; Intee))(R) Xeon(R) Gold 6342 CPU @2.80GHz
Encoding Dim
512
Hidden
3 x 512
Activation
Leaky ReLu
Table 4: Forward/inference Model
21

---

Published as a conference paper at ICLR 2025
Parameter
Value
'pall
(log-likelhhod space)
Minimum Normalized distribution variance
0.001
Distribution
Diagonal Gaussian
Learning Rate
1 x 10
Table 5: Null Parameters
Parameter
Value
Algorithm
DDPG
Batch Size
1024
Optimizer
Adam
Actor/critic learning rate
1 x 10
Exploration Noise
0.1
0.9
Hidden Layers
2 x 512
0.005
Table 6: Reinforcement Learning Parameters
Domain
Timeout Normalized Goal Epsilon Null Train Steps
RI Train Steps
Spriteworld Default
100
0.1
1M
50M
Spriteworld Small
100
0.15
1M
50M
Spriteworld Obstacles
100
0.2
50M
Spriteworld Velocity
100
0.2
50M
Robosuite default
100
0.15
1M
10M
Robosuite small
100
0.15
10M
Robosuite obstacles
100
0.15
15M
Air Hockey default
400
0.2
1M
10M
Air Hockey small
400
0.2
20M
Air Hockey obstacles
400
0.2
20M
Franka Kitchen default
200
0.2
2M
3M
Table 7: Domain Specific Parameters
Domain
HInt learned
HInt
Hind
Prio
FPG
Vanilla
ELDEN
CAI
Sprites (50M)
62.21
22.10
23.27
26.02
17.15
22.57
40.12
69.34
Air (10M)
12.50
4.33
4.59
6.05
5.24
3.31
9.75
17.23
Robo (10M)
21.17
11.52
15.32
11.28
8.59
8.21
18.28
25.20
Franka (3M)
16.68
9.26
8.45
8.92
10.39
9.16
13.25
22.48
Table Wall Clock Compute Time Hours
J HINDSIGHT SAMPLING ABLATION
In this work, we focused on the introduction of interactions into Hindsight Filtering using the "final" sampling scheme, which takes the last state of trajectory as the hindsight goal. In practice, different sampling schemes can be used for hindsight Andrychowicz et al. (2017), including sampling any state after the observed "future" any state from the trajectory "episode." We provide some analysis
22

---

Published as a conference paper at ICLR 2025
Method
NCII w/ Point
JACI
Gradient
Attention
NCD
liin-nonlincar
0.9 + 0.2
2.4 0.8
32.5 6.1
37.4 2.1
2-in-nonlinear
2.3 1 0.1
2.5 0.2
36.4
0.4 21.8 1 0.9
40-dim
1.410.1
2.4
34.7 4.4 26.4 6.9 12.5 1 0.8
Table 9: Misprediction rate (lower is better) of inference in additional domains from state, similar
to Table 1. Interactions are reweighted to be 50% of the test dataset. Boldface indicates within ~- combined standard deviation of the best result. k-in-nonlinear uses nonlinearities in the random DAG instead of linear relationship. 40-dimensional uses a 40 dimensional state.
suggesting that hindsight filtering is applicable in any sampling scheme in Figure 10. Note again that general, we used the "final" sampling strategy for all other experiments (Figure 4, Figure Figure 12).
10M
10M
S0M
(a) Sprite Final
(b) Sprite Future
(c) Sprite Episode
100
0
2M
4M
6M
8M
10M
0
2M
4M
6M
8M
10M
0
2M
4M
68
8M
10M
(d) Robo Final
(c) Robo Future
(f) Robo Episode
80
40
0
2M
4M
6M4
8M
10M
0
2M
4M
6M4
BM
10M
0
2M
4M
6M
8M
10M
(g) Air Final
(h) Air Future
(i) Air Episode
Hint w/ GT
Hindsight
Figure 10: Comparison of HInt and Hindsight using different HER sampling schemes "final", "future" and "episode", with 3 trials for each sampling scheme. In these experiments, modified the sampling scheme for both Hint and Hindsight to the sampling strategy indicated the caption. The shading indicates standard error. Sprite,, Air- and Robo- domains each use the default variant. The Y axis average reward per episode.
VISION EXPERIMENTS K
While the objective of this work is to demonstrate a generally applicable method for utilizing counterfactual nulls for interaction inference (NCII) and interaction filtering for GCRL (HInt), briefly explore the scaling capabilities of NCIland HInt. In both interaction inference and GCRL, performance in higher dimensional states remains challenging problem. While results certainly
23

---

Published as a conference paper at ICLR 2025
Method
NCII Point
JACI
Gradient
Attention
NCD
Sprite-I-visiie
13.9 0.5
18.1 0.2 26.4 + 39.6 4.5 21.2 + 1.1
Table 10: Misprediction rate (lower is better) of inference Spriteworld-1 domain. Interactions are reweighted be 50% of the test dataset. Boldface indicates within combined standard deviation of the best result.
suggest that the NCIlis competitive with baselines in both domains, these scaling questions remain unsolved.
Empirically, we take a segmentation of each object in the frame (given from the simulator) in the Sprite default domain, and train variational autoencoder Kingma (2013) frame stack of frames of each segmented object with a latent dimension of 128. We then append object-centric features pixel position and velocity, duplicated to 128 dimensions. This is then used as the input for both NCHland RL using HInt. We first compare NCIIto interaction inference baselines in the Spriteworld Default domain Table 10. Then we provide a performance curve for Hlntwhen compared against the Hindsight and Vanilla RL baselines in Figure 11.
10
20M
40M
60M
80M
100M
(a) Sprite Default 256-encoded
Hint w/ GT
Hindsight
Vanilla
Figure 11: Comparison of HInt, Hindsight and Vanilla RL, 3 trials for each, Spriteworld Default using 256-dimension image encodings. The shading indicates standard error. The Y-axis average reward per episode.
22
0.5
35M
00
0.2
(a) Sprite Obstacles
(b) Robo Default
(c) Air Default
(d) Kitchen Default
Hint w/ GT (ours)
Hint w/ Nill lours)
Hindsight
Prioritized Replay
PPG
ELDEN
Vanilla
CAI
Figure 12: Addition of Causal Action Influence (CAD) Seitzer et al. (2021) baseline in selected domains. This baseline similar to ELDEN, but has more inductive bias towards actions. The axis the average reward per episode.
L ADDITIONAL COMPARISONS
We also compare with Causal action influence (CAI) (Seitzer al., 2021). CAI uses conditional mutual information to infer the local causal relations, detecting when and what the agent can influence the state variables with its actions. Then they employ the influence the intrinsic motivation for
24

---

Published as a conference paper at ICLR 2025
exploration that benefits sample efficiency. The results in selected domains given in Figure 12, where CAI performs similarly to ELDEN empirically.
M ROLLOUT VISUALIZATIONS
Figure 13: Selected frames from a successful policy rollout for Spriteworld Obstacles
Figure 14: Selected frames from successful policy rollout for Robosuite Obstacles
Figure 15: Selected frames from successful policy rollout for Air Hockey Default
Figure 16: Selected frames from successful policy rollout for Franka Kitchen
25


Reviews:
[{"review_id": "rrNyKYEbAC", "reviewer": "Reviewer_rrvS", "summary": "This paper proposes to leverage the null assumption to filter out states without interaction between the agent and object, improving the sample efficiency of GCRL. The approach begins by using a learned dynamics model to identify null states—where the next state remains the same in the absence of a specific state. It then keeps those trajectories where the agent directly interacts with the object, training the agent with hindsight relabeling. This approach shows comparable or superior sample efficiency in both 2D dynamic and 3D manipulation environments.", "questions": "- While the authors use a learning-based dynamics model to infer the interaction, it can be clearly distinguished from existing work that utilizes other approaches. For example, [1] utilizes proprioceptive state changes to distinguish contact.\n- The explanation of mixture distribution on L189 wasn't clear. How could it mix two distributions with a multiplication factor?\n- The discussion on the limitation of this work can make readers better understand of the method. For example, authors can mention the domain where interaction is actually prohibitive (e.g., drone navigation)\n\n#### Minor typo\n- I believe L187 should be $d_{\\pi}$\n\n### References\n[1] Manuelli and Tedrake, \"Localizing external contact using proprioceptive sensors: The Contact Particle Filter\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The introduction of the *Null counterfactual interaction assumption* could be a important contribution, improving sample efficiency across various domains, particularly in manipulation tasks where interaction is minimal.\n- The method details engineering practices to make the approach both manageable and efficient.\n  + This includes null state inference with a dynamics model and predicting null operation.\n- The paper presents a rich set of environments, and design choices for these environments, etc.", "weaknesses": "- **Scalability to high-dimensional state**\n  + How is the state space defined across all environments? Assuming the entire environment has a low-dimensional state space, I’m curious how it computes the difference between states (Eq. 3) and infers the null state (Eq. 4) in a high-dimensional case (e.g., image).\n  + From my understanding, inferring the null state should have a complexity of $O(n^2)$ based on state dimensionality, which may limit scalability in high-dimensional state spaces. However, L263 mentions a time complexity of $O(1)$. Could the authors clarify this?\n  \n- **Dependence on hyperparameters**\n  + The method distinguishes null states based on prediction error (Eq. 3), but setting this hyperparameter could vary depending on environments and tasks.\n  + Moreover, certain states, even within an environment or task, may have more complex dynamics than others. In such cases, how does the method define a single $\\epsilon_{null}$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fm05rjS2Kd", "reviewer": "Reviewer_EVkv", "summary": "This paper considers a notion of null counterfactual: a cause object is interacting with a target object if in a world where the cause object did not exist, the target object would have different transition dynamics. Using this definition, the paper proposes ways to simulate virtual rollouts and perform data augmentation by leveraging null counterfactual. Hindsight Experience Replay is playing a key role in the algorithm, and the algorithm seems to inject some compositionality into hindsight replay. Toy tasks and robotic tasks are considered for evaluation.", "questions": "See the weakness sections.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The notion of null counterfactual is interesting.\n- The paper manages to devise a full-algorithm from this notion, and showed practical gain in object-centric robotic domains.\n- Goal-conditioned RL is an important area of research, and using null counterfactual for data augmentation is a promising direction.", "weaknesses": "- This method seems relatively limited to object-centric domains, where the dynamics between objects is relatively simple.\n- Certain set-based architecture (such as PointNet and some version of GNN) might not work in general domains to model dynamics.\n- The simulated nulling procedure and the filter criterion feel very heuristic and specific to the considered domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YNnJfb3lqN", "reviewer": "Reviewer_xepm", "summary": "In the context of goal-conditioned RL, building on top of Hindsight Experience Replay, the paper proposes a filtering method that aims to improve the efficiency of learning. Under the proposed definition of interaction that is based on the change of the transition probabilities under null counterfactuals, a masked forward dynamic model is learned to identify interaction (NCII). Then the method filters the trajectory to be relabeled and only keeps those that the agent interacted with the target (NInt). The effectiveness of NCII and the improvements of NInt are verified by empirical analysis on simulated environments compared with established methods.", "questions": "1. On page 5 around line 221, how exactly does the extra action column work with the core \\( n \\times n \\) submatrix corresponding to the states? It appears that the interaction is defined around a pair of states. I also have the same confusion with Figure 2.\n\n2. On page 5 around line 232, the mentioning of the vector \\( \\mathbf{V}^k \\) would need more context. It seems to be a vector to zero out a column of the interaction matrix \\( \\mathbb{B} \\), but it is not very clear. How is it related to, and what exactly is the property that not all tasks exhibit on line 233?\n\n3. How should we deal with cases when there are very few trajectories satisfying the interaction criterion?\n\n4. In Table 1, it is listed as accuracy, but it seems like lower values are better, which is a bit confusing.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem setup is well motivated and the proposed algorithm extends , HER, an important technique in goal conditioned RL, to settings where it doesn’t work well and is effective. The presentation from the background of HER to the proposed method is smooth and well thought out, except a few minor places that can use some polish.", "weaknesses": "1. The null operation in Equation 3 depends on the threshold \\( \\epsilon_{\\text{null}} \\). This is an important part of the algorithm. Discussion on how to choose it and an ablation on the sensitivity of this threshold would make the analysis more comprehensive. More specifically we are interested in answering the following questions (actionable feedback below):\n   - How sensitive is NCII to the choice of the threshold?\n   - Does one threshold work across different environments in Table 1, or does each environment variant require a different threshold?\n   \n Figures showing how the accuracy of HCII varies corresponding to a range of thresholds for environments, or one variant from each environment the authors already considered Table 1, would be compelling. Additionally, for a few selective environments that are sensitive to thresholds in the previous ablation, how does the episode reward change when HCII with different thresholds is used in HInt? This second ablation may not be necessary if HCII is shown to be robust across a range of thresholds in the previous one. The range of thresholds should be selected by the authors to show if there are values on the left or right tail where the algorithm starts to break down and success rates start to fall off. Success rate is the metric. \n\n2. Hindsight Experience Replay (HER) is an important baseline here. HER has several variants for how the goal is chosen, including “future,” “final,” and “episode.” It seems that, but it’s not clear, the HER implementation here refers to the default “final” variant. Expanding the baseline in Figure 4 to include other variants of HER, especially both the “final” and “future” variants, would make the comparison more comprehensive. This is particularly relevant as the performance difference between HInt and HER is small in a few environments in Figure 4, and a stronger variant of HER might change the gap here. This would entail running on the environments in Figure 4 and reporting on the already established metric, only this time under the alternative versions of HER goal selection strategies. \n\n3. In Equation 3, it appears that the logarithm is intended to apply to the entire subtraction; however, the syntax suggests otherwise.\n\n4. There is a typo on line 268, page 5: “using using.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aDyE4Di4dJ", "reviewer": "Reviewer_SPH4", "summary": "The paper proposes Hindsight relabeling using Interactions (HInt), a variant of Hindsight Experience Replay (HER) that leverages interactions to filter candidate trajectories for relabeling. Drawing inspiration from causality, an interaction is defined as an instance where removing (or nulling) an object would have an impact on the next state of another object (Null Counterfactual Interaction Inference or NCII). Given object-centric representations, the paper proposes to learn a masked dynamics model which can predict the next state of an object conditioned on what other objects are present. An influence of object A on object B is then detected by comparing the difference of the predictions for B with and without A against a threshold. During training, interaction detection is amortized in an interaction classifier. The main proposed criterion for using a trajectory for HER is the detection of a path in the unrolled graph corresponding to interactions, leading from an action to a target object (hence, an influence of the action on the target object). Experiments in two abstract and three robotics-inspired continuous control domains show increased sample efficiency when using HInt. An analysis suggests that HInt filters out trajectories in which the target object does not move (in the Spriteworld domain).", "questions": "* It would be interesting to see wall-clock time comparisons with the baselines as HInt adds quite a bit of complexity to them.\n\n* I would have expected an expectation over the goal in the RL objective in like 181.\n\n* The next paragraph (starting from line 183) is written like the goal space is equal to the state space. However, in the rest of the paper this is not the case.\n\n* ‘min’ in equation (2) should be ‘max’.\n\n* Why is no absolute value taken in equation (3) when thresholding the difference of log probabilities.\n\n* In line 303, the filtering function is defined is defined as a decision to reject a trajectory while in appendix D it seems to be the decision to accept a trajectory.\n\n* I think (left) and (right) are switched in the caption of Figure 1.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The argument for an interaction-based inductive bias in HER is well motivated. Moreover, the interpretation of a deviating transition probability under a null counterfactual as an interaction between objects is intuitive and concise. The existence of a path from an action to the target state as a filtering criterion for HER is well founded in causality and illustrated well by figure 2.\n\nThe domains considered for the experimental evaluation are relevant and sufficiently established. Table 1 indicates that NCII is more accurate in detecting interactions than the considered baselines. The RL performance is demonstrated to benefit significantly from using HInt.\n\nThe writing in the main text is clear and the presentation is well structured.", "weaknesses": "In my opinion, moving too much crucial algorithm components to the appendix is a main weakness of the paper. The main text conveys the impression that it presents the whole algorithm except for some unimportant implementation details, and that this algorithm achieves good performance in practice. However, the content of Appendix C and especially Appendix D seem to be quite important, and are probably crucial for obtaining the empirical results that were reported.\n\nIn particular the filtering criteria presented in Appendix D deviate from the intuitive path-from-action-to-target definition in the main text. Moreover, a somewhat simplified and engineered criterion is then used for the experiments. Yet, it is only referred to as one of several “alternatives” in the main text (line 318). In my opinion, it should be made more clear what form of the algorithm is actually used for the experiments and which components are crucial for obtaining good performance. An ablation study with different filtering criteria would be interesting, for example.\n\nMy understanding, based on the last paragraph of appendix D, is furthermore that for the experiments, only interactions between the controlled object and the target object were detected and used as a criterion. This is a much simpler algorithm than what is presented in the main text and effectively uses domain knowledge (as it happens to be sufficient to consider such interactions in the chosen domains). Moreover, another hyperparameter thresholding the interaction frequency in a trajectory is introduced. Combined, this makes me question the claim that NCII is really more general than using heuristics (line 87). \nAs the algorithm used in the experiments is considerably simplified, it seems like running CAI [1] as a baseline is required. CAI simply estimates the influence of actions on the target object. It would be interesting to see how much HInt can add in terms of sample efficiency to this approach.\n\nThe content of Appendix C reads like quite a few tricks were needed to get HInt to work well. In particular the reweighing based on the log probability of a transition seems important and should therefore be mentioned in the main text.\n\nThe writing in Appendix D is sometimes a bit dense and hard to understand, for example the enumeration point “1. Non-passive”. I think there is potential for illustrating these strategies better.\n\n[1] Seitzer, Maximilian, Bernhard Schölkopf, and Georg Martius. \"Causal influence detection for improving efficiency in reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 22905-22918.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)