ELDEN: Exploration via Local Dependencies
Authors: Zizhao Wang
Jiaheng Hu
Peter Stone
Roberto Martín-Martín
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning; intrinsic motivation; exploration
TL;DR: An exploration method for reinforcement learning based on novel local dependencies.

Abstract: Tasks with large state space and sparse rewards present a longstanding challenge to reinforcement learning. In these tasks, an agent needs to explore the state space efficiently until it finds a reward. To deal with this problem, the community has proposed to augment the reward function with intrinsic reward, a bonus signal that encourages the agent to visit interesting states. In this work, we propose a new way of defining interesting states for environments with factored state spaces and complex chained dependencies, where an agent's actions may change the value of one entity that, in order, may affect the value of another entity. Our insight is that, in these environments, interesting states for exploration are states where the agent is uncertain whether (as opposed to how) entities such as the agent or objects have some influence on each other. We present ELDEN, Exploration via Local DepENdencies, a novel intrinsic reward that encourages the discovery of new interactions between entities. ELDEN utilizes a novel scheme --- the partial derivative of the learned dynamics to model the local dependencies between entities accurately and computationally efficiently. The uncertainty of the predicted dependencies is then used as an intrinsic reward to encourage exploration toward new interactions. We evaluate the performance of ELDEN on four different domains with complex dependencies, ranging from 2D grid worlds to 3D robotic tasks. In all domains, ELDEN correctly identifies local dependencies and learns successful policies, significantly outperforming previous state-of-the-art exploration methods.

ELDEN: Exploration via Local Dependencies
Zizhao Wang"
Jiaheng Hu" University Texas at Austin jhu wtw.us. edu
University of Texas Austin zizhao. wang@utexas......... edu
Peter Stone
Roberto Martín-Martee University of Texas at Austin robertommêcs. utexas edu
University of Texas at Austin, Sony AI pstoneêcs. utexas. edu
Abstract
Tasks with large state space and sparse rewards present a longstanding challenge reinforcement learning. these tasks, an agent needs to explore the state space efficiently until it finds reward. To deal with this problem, the community has proposed to augment the reward function with intrinsic reward, a bonus signal that encourages the agent to visit interesting states. In this work, we propose new way defining interesting states for environments with factored state spaces and complex chained dependencies, where agent's actions may change the value one entity that, in order, may affect the value of another entity. Our insight is that, in these environments, interesting states for exploration the agent is uncertain whether (as opposed to how) entities such as the agent or objects have some influence each other. We present ELDEN, Exploration via Local DepENdencies, a novel intrinsic reward that encourages the discovery of new interactions between entities. ELDEN utilizes novel scheme the partial derivative of the learned dynamics to model the local dependencies between entities accurately and computationally efficiently. The uncertainty of the predicted depen- dencies is then used as an intrinsic reward to encourage exploration toward new interactions. We evaluate the performance ELDEN on four different domains with complex dependencies, ranging from 2D grid worlds to 3D robotic tasks. In all domains, ELDEN correctly identifies local dependencies and learns successful policies, significantly outperforming previous state-of -ftthe-art exploration methods.
states where
Introduction
learning (RL) has achieved remarkable success in recent years in tasks where well-shaped dense reward function is easy to define, such as playing video games [33 [8] 4 and controlling robots 99 2 15 [6]. However, for many real-world tasks, defining dense reward function non-trivial, yet a sparse reward function based on success or failure directly available. For such reward functions, learning good policies is often challenging, it requires efficient exploration of the state space.
Reinforcement
To address this challenge, RL researchers proposed the use of an intrinsic reward, an additional task-agnostic signal given to the agent for visiting interesting states, Intrinsic reward methods can be
roughly classified into two main paradigms: curiosity (20) 25 6 and empowerment 229 27 14)
where the agent rewarded either for visiting novel states or for obtaining maximal control over environment, respectively.
Eqqual contribution
I "qqual supervision
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

behaviors of different agents different learning stages
the task requires
turning on the stove de cook the meatball O the pot
Curiosity: interested high prediction errors in entlyy movements
ELDEN (ours): interested novel local dependencies belween entilie
meatball
pot
stove switch
Empowerment interested in maximizing the action's influence environment entities
stove
Figure 1: (Left) In kitchen task with multiple potential agent-object and object-object interactions, (Middle) for curiosity-based agent interested in hard-to-predt preedccc entity motion, it will initially facus on exploring arm movement, then on pot and meatball manipulation, and finally keep rolling the meatball whose outcomes are challenging predict. On the other hand, for an empowerment srapdeeeeeet-bbeeddd agent interested in maximizing the action's s influence, it begins with controlling the and then learning move the pot and meatball simultaneously, but ignores the potential interaction between the stove and the meatball. (Right) ELDEN avoids those issues by identifying whether dependencies between entities happen and focusing the exploration on novel oncs. After the agent learns that it can control the pot and meatball, it will move on to explore other potential interactions, c.g., whether the stove can influence the meatball. Hence it has larger opportunity to learn this task, compared with curiosity empowerment-basg agent.
While these methods significantly improve exploration in some domains, there are cases where the aforementioned methods fail. Consider, for example, kitchen environment with several objects where there are multiple potential agent-object and object-object interactions, and an agent is tasked with putting a meatball in a pot and cooking the stove (Fig. D On the one hand, curiosity-driven methods will encourage the agent to explore the environment by visiting states where the exact outcome of an action is uncertain. Consequently, for each interactable object, the agent will exhaust any possible interaction until it can accurately predict every change in the object's 's state. As result, such an exploration strategy can be inefficient, especially for environments with many objects. In the kitchen example, it is hard to predict how the meatball rolls in the pot, and thus the curiosity-driven On the other hand, for empowerment methods, the agent encouraged
agent would keep rolling to remain in states where it can influence as many states (objects) simultaneously as possible (e.g. holding the pot with the meatball inside). By doing so, however, it ignores object-object interactions that the action cannot directly control but indirectly induce, which can be the key to task completion. In the kitchen case, an empowermennt-driver agent will therefore not be interested placing the pot and the meatball on the stove, as it forfeits control of them by doing so, even though this action enables the stove to heat the meatball. Our main insight is that, in this type of environment, an intelligently exploring agent should be able to learn that it can use the pot to move the meatball after few trials. Then, instead of spending time learning the complex meatball movement or different styles to manipulate the pot, it would move to explore other modes of interacting with other objects, e.g.. putting the pot on the stove and switching on the stove.
Following this motivation, we propose new definition of interesting states focusing whether the environment entities (consisting of the agents and objects) can interact, rather than how exactly they interact. We present ELDEN, Exploration via Local DepENdencies, novel intrinsic reward mechanism that models the local dependencies between entities in the scene (agent-object, object- object) and uses the uncertainty about the dependencies to guide exploration. By relaxing the curiosity signal from dynamics prediction to dependencies prediction, ELDEN implicitly biases the exploration toward states where novel interaction modes happen rather than states where the state value is novel but dependencies remain the same. Specifically, ELDEN trains an ensemble of dynamics models. In each model, the local dependencies between objects are modeled by the partial derivatives of state predictions w.r.t. the current state and action. Then, the local dependency uncertainty is measured as the variance across all dynamic models.
We evaluate ELDEN on discrete and continuous domains with multiple objects leading to many interaction modes and tasks with chained dependencies. Our results show that using partial

---

derivative- based extractor on dynamic models training allows to accurately identify local connec- tivities between environment entities. Furthermore, the intrinsic reward derived from the identified local connectivities allows ELDEN to outperform state-of-the-art exploration methods (curiosity and empowerment-drien on the tested domains.
Related Work
The idea of using intrinsic reward to facilitate exploration reinforcement learning a long-studied topic that can be dated back to Schmidhuber (25) In this section, first discuss two main classes of intrinsic reward related to ELDEN. Since ELDEN requires reasoning about the local dependencies between environment entities, we also discuss works that involve utilizing dependenncies/causaliyy in reinforcement learning.
2.1 -urivsity-Driven Exploration
Curiosity-driven exploration rewards an agent for visiting "novel" states, where different methods define the "novelness" of a state in different ways. For methods that utilize visit count to define state novelty, Bellemare al. [3] utilized density model to estimate pseudo-count for each state, which is then used to derive intrinsic reward for the agent; Tang et al. [30] uses locality-sensitive hashing code to convert nigh-dimensional states hash codes such that the visit count for each hash code can be explicitly kept track of. For methods that utilize predictiveness to define state novelty, Stadie al. 29 learns a forward dynamics model that operates on a learned latent space of the observation, and uses the prediction error as the intrinsic reward for exploration: Burda et al. [o] uses randomly initialized networks to extract state features, where the agent encouraged to visit states where predictions about the state features are inaccurate; Pathak et al. [21] utilizes disagreements within an ensemble of dynamic models as a signal for intrinsic reward, and directly backpropagates from the reward signal to the policy parameters to improve exploration. Pathak et al. (20 incorporates empowerment into the curiosity-bel bassed method by learning an inverse dynamics model that maps from state to action. The inverse dynamics model defines feature space for the states, where the prediction error of a forward dynamic model is measured in this feature space.
However, learning accurate dynamics can be difficult and require a significant coverage of (possibly large) state-action space; e.g., when a robot manipulates block, would have experience multiple action-reaction pairs to be able to make accurate predictions [3] Furthermore, prior curiosity-driven exploration methods can derailed by the stochasticity in the dynamics, i.e., when the outcome of an action has large entropy (e.g., tossing a coin), so the agent would keep repeating it. ELDEN can be considered type of curiosity-driven exploration method. However, unlike previous works which only consider improving knowledge about the dynamics, ELDEN explicitly considers knowledge about local dependencies between environment entities, and utilizes it to encourage exploration. Thus, it avoids the need to learn an accurate dynamic model and is less sensitive environmental noise and stochasticity.
2.2 Empowermennt-Drieen Exploration
Empowerment--asee exploration methods based on a different understanding of what states
should be encouraged for task-agnostic exploration (35 27 g 19. Their main idea is that the most interesting states explore for any task are states where the agent has the most controllable diversity about what the next state will be, i.e., states where there are multiple possible next states that can be chosen by the agent. From those states, it casier fulfill any downstream task that requires purposefully changing the state. Empowerment-be based exploration methods reason about the controllable elements in the environment (states that can be influenced by agent's actions), and encourage the agent to find states where this controllability is large, typically through some form of mutual information maximization between the agent's 's actions and the next states. In particular, Zhao et al. 35 uses domain knowledge to divide up the state space into internal state and exteral state, and maximize the mutual information between them. Seitzer et al. (27 measures the local dependencies between action and environment entities by estimating their conditional mutual information (CMI) and using as intrinsic reward signal encourage the agent to maximize the influence of action over environment entities.
3

---

However, due to the difficulty in measuring the mutual information across a multi-step trajectory. existing empowerment- based methods only measure 1-step empowerment, e.g., how much the agent directly influences variables. Thus they cannot detect dependencies between objects (e.g., indirect tool use such as using the stove to cook meals) and the environment's influence on the agent (e.g.. the elevator can take the agent to the desired floor). Furthermore, since the objective of empowerment- based methods is to maximize controllability, they tend to only control the easiest-to- -anniiiullee object to maximize its empowerment when there are multiple controllable variables.
ELDEN is closely related Seitzer al. [27], but differs from it in three main aspects: first, unlike Seitzer et al. [27], which only considers the interaction between action and environment entities, ELDEN also considers the interaction between environment entities that are not locally dependant on the action, which allows curiosity about indirect interaction between the agent and the environment entities to propagate through time during the RL. training. Second, unlike Seitzer et 227. ELDEN tries to visit states with novel interactions, instead of maximizing controllability, thus avoiding the tendency to only interact with easy-to-control objects. Lastly, EL.DEN estimates local dependencies through reasoning about the partial derivatives of learned dynamic models, which we empirically show to be more accurate and computationally efficient compared to the CMI-based estimation in
Seitzer et al. 27
2.3
Causality in Reinforcement Learning
The concept of incorporating causality in the training of reinforcement learning agent has been utilized in many different forms. Wang et al. [31] demonstrates that incorporating causal dependencies between environment entities can greatly improve generalization to out-of-distribttion states. Hu et al.
[00 exploited causal dependencies between action dimensions and reward terms to reduce variance in the gradients and facilitate policy learning of mobile manipulation tasks. Pitis et al. [24] shows that knowing the local causal dependencies between objects can facilitate policies generalize to unseen states and actions. Pitis et al. [23] uses local dependencies to generate counterfactual samples in order to facilitate sample efficiency. Sontakke et al. (28) discovers causal factors in the dynamics of a given environment through causal curiosity reward term.
Like in Pitis et al. 23 ELDEN learns to predict the local connectivities between environment entities depending on the state and the action values. However, we do not generate counterfactuals but create intrinsic reward based on the local dependency that facilitates exploration with RL in sparse-reward setups.
ELDEN: Exploration via Local Dependencies
In the section, introduce ELDEN, which infers the local dependencies between environment entities and uses the uncertainty of dependencies as an intrinsic reward for tackling hard-exploration problems. In Sec. we formally define the problem setup of ELDEN, In Sec. 3.2 diseuss how ELDEN uncovers local dependencies. In Sec. 3.3 we describe how ELDEN improves exploration with the intrinsic reward.
3.1 Problem Statement
We consider decision- makinn as a discrete-tims Markov Decision Process (S, A, P. R), where S is a state space which we assume can factored as S S' x ... X SNN A is an action space, P is a Markovian transition model, and R is reward function. The goal of the RL. agent is optimize the parameters of a policy wo such that the total expected return under wo is maximized. Specifically, focus cases where R is sparse, and therefore intelligent exploration is crucial the discovery of optimal policies.
Local Causal Graph Model We can model the transition at time step /. Causal Graphical Model (CGM) 22 consisting of (1) nodes AS, S,41), (2) a directed graph G describing global dependencies between nodes, and (3) a conditional distribution p for each state variable the next time step, ST. We assume the transition can be factorized as P( 1 |8e, ax) = II p( i (P }). where the Pa(v) are parents of a node in the causal graph G. For many environments, G can be dense even fully connected, because whenever it is possible for 8 to depend on S no matter how

---

Algorithm Training of ELDEN (on-policy)
1: Initialize the dynamics ensemble [ policy ne. and replay buffer D.
2: for number of training iterations do
3: Collect environment transitions (St, ot, Ptauak; 81+1) with current policy n
4:
for k = 1... K do
5:
G f compute_ G( a (Sec. 3.2
Compute the local dependency graph
6:
A = variance (g /m=1)
7:
r( =Tr rraukk + 8. T..iirinss
8:
end for
9:
Update the policy ne with (st, de, rr, $141 (Sec. 3.3
Add {st: ae: *t+1) into the replay buffer Sample mini-batchh B {st, da, a,, 8641 Jurr from D
11:
12: Train the dynamics ensemble {f] with B (Sec. 3.2
13:
end for
unlikely, it is necessary to include the edge S SV. However, in the real world, even if possible to interact, most entities are independent of each other most of the time. Following this observation, we are interested inferring local dependencies that are specific to (s), represented by local causal graph Gr that is minimal by removing inactive edges in G.
3.2 Identifying Local Dependencies with Dynamics Partial Derivatives
Based on the definition in Sec. 3.1 the key component of ELDEN is to accurately evaluate which potential dependencies between environment entities are locally active, i.e., identify the local causal graph G. given (st, dt). This identification requires answering a challenging question - whether entity i's value, S st is the cause of entity to have value 8/41 8/11. ELDEN approaches with the inspiration from the but-for test: the local dependency exists if S/41 s would not happen but for S st In other words, we assess whether S1+1 would change if S has different value. Notice that, since focus local dependencies, we only want to vary St near its actual value s rather than trying all its possible values.
To this end, ELDEN utilizes partial derivatives to identify local dependencies, they naturally capture the extent of change in S with respect St. Specifically, assuming the access of ground truth transition probability p (which we will relax later), ELDEN considers 8/ = $/41 locally depends on S of if
Öp( 82 1
26
Ös:
where is a predefined threshold. A large partial derivative indicates that a slight change in Sj will lead to substantial change in S/41. thus satisfying the but-for test.
To evaluate partial derivatives without the ground truth transition probability, ELDEN approximates p with dynamics model f parameterized by neural network ((s{41) (st, ar) and trains by maximizing the log-likelihood of (($(1)) (for notational simplicity, we omit the conditionals 8t, a in p this section). Due limited data and training errors, even when environment entities not locally dependent, there occasionally exists a large partial derivative between them. To reduce such false positives, ELDEN further applies regularization suppress partial derivatives w.r.t. inputs The overall loss of dynamics training is
that are not necessary for predicting 8441-
ap(s( ask
L, - log blg Σ
(2)
where À is the regularization coefficient.
5

---

butter
meatball
(a) Thawing
(b) CarWash
(c) Minecraft 2D
(d) Kitchen
Figure 2: We test ELDEN three domains and four environments. (a) (b) Mini-behavior and (c) Minecraft 2D with discrete state spaces, where the agent has to achieve series of temporally extended tasks with complex object interactions. (d) Robosuite, robot table-top manipulation simulation environment with continuous state spaces, where the robot needs to perform multiple interdependent subtasks to finish the cooking task.
3.3 ELDEN Policy Learning
ELDEN utilizes the local dependency identification described in Sec. 3.2 to improve exploration for model-free RL. The key idea behind ELDEN encourage agent to visit states where new local dependencies are likely to emerge. When the ground truth local dependencies are available, the novelty of local dependencies between state variables can be measured by the magnitude of eπгor of the dependencies identified by our method. Unfortunately. in many cases, it is hard to manually specify the ground truth local dependencies. Instead, ELDDEN trains an ensemble of dynamics models, and measures dependency novelty by the variance of the local dependency graphs extracted independently from each of the dynamics models in the ensemble. Specifically, ELDEN first computes the variance of each edge in the graph and then uses the mean of the edge variance as the graph variance. Finally, the calculated variance used as the intrinsic reward and is scaled with a coefficient 3 that controls the magnitude of the exploration bonus, and added to the task reward Tusk-
ELDEN is applicable both on-policy and off-policy settings. We show the pseudo-code for on- policy ELDEN in [Igorithm[] The off off-policy version of ELDEN can be easily derived by updating Importantly, any
the policy with transitions sampled from the replay buffer line on Algorithm [ model-free RI algorithm can be used for the policy update step.
4 Experiments
In our experiments, we aim to answer two main questions: Q Is ELDEN able accurately detect local dependencies between environment entities in factored state spaces (Sec. 4.1 Q2: Does ELDEN improve the performance of RL algorithms in sparse-reward environments with chained dependencies (Sec. 4.2
we evaluate ELDEN in four simulated environments with different objects that have complex and chained dependencies: (1) CARWASHI (2) THAWING, (4) 2D MINECRAFT and (3) KITCHEN. Both CARWASH and THAWING are long-horizon household tasks in discrete gridworld from the Mini- Hini-EEAVIOR Benchmark 122 MINECRAFT 2D is environment modified from the one used by Andreas et al. I. where the agent needs to master a complex continuous robot table-top manipulation domain
Environments As shown in Fig. 2
technology tree to finish the task. KITCHEN implemented in RoboSuite [36]. To complete tasks in these environments and receive the sparse reward, the agent has conduct series of actions that change not only the state the interacted entities but also induce further interaction between interacted entities and others (e.g.. interacting with the stove switch that enact interaction between the stove and the cooking the meatball). The agents in all environments can select between set of action primitives, set of discrete actions that can be applied on each object, e.g., goTo (obj) or pick (obj). Notice that even with the action primitives, these domains are still very hard to solve due to the presence of many interaction modes and the difficulty in finding the correct (potentially long) sequence of interactions among many options that will lead to task success. We provide further descriptions of each environment in Appendix See

---

Table Mean + std. enror of ROC AUC (1) and F1 (1) of local dependency prediction
THAWING
CARWASH
KITCHEN
ROC AUC
F1
ROC AUC
F1
ROC AUC
F1
ELDEN
0.71 + 0.01
0.57 0.00
0.78 = 0.02
0.66
0.02
0.66
0.01
0.25 = 0.01
PCMI
0.55 + 0.01
0.60 + 0.00
0.73 + 0.02
0.78
0.01
0.60
0.00
0.28 + 0.00
Attn
0.65 + 0.04
0.63 + 0.01
0.01
0.66 0.01
0.55 0.03
0.51
0.22 + 0.02
Input Mask
0.50 + 0.00
0.40 + 0.00
0.50 0.00
0.32 1 0.01
0.50
0.00
0.08 + 0.00
Attn Mask
0.45 0.03
0.47 + 0.02
0.47 0.07
0.43 - 0.03
0.52 0.01
0.13 = 0.01
Implementation Details For discrete state or action spaces, the partial derivatives w.r.t. 8t/ / are undefined. To address this issue, we use Mixup [34] to create synthetic inputs and labels by linearly combining pairs of inputs and labels, thus approximately changing the input space to be continuous. Compared learning from discrete inputs only, dynamics models trained on such data generate partial derivatives that better reflect local dependencies, as shown in Sec 4.3.1 For the 2D MINECRAFT and KITCHEN environments where some local dependencies have complex preconditions and thus are hard to induce, we apply sample prioritization to dynamics learning, where the priority measured as prediction enror. In this way, the dynamics model gets aware of unknown interactions faster and guides the exploration more efficiently than not using prioritization. Further details are provided in the Appendix.
4.1 Evaluating the Detection of Local Dependencies
We compare the local dependencies extracted by ELDEN with the following baselines (see imple- mentation details in Appendix Sec. C
p( p(s(e15500d) It
PCMI (point-wise conditional mutual information) 27 31 defined log plot.pbbr/ef.or)
quantifies how likely it is that s/ا depends on s).
Attn (attention): Use the score between each entity pair computed by the attention modules inside the dynamics model to quantify local dependencies.
we implement a learnable binary mask to the dynamics model that can zero out some inputs conditioned on ag): f[[s), a,] ] (st, az)). During training. the mask is regularized to use as few inputs as possible with L1 regularization, leading to quantification of minimal local dependencies.
Input Mask:
Attn Mask: we implement a learnable mask to the dynamics model similar to the one in Input Mask, but in this case, the mask is applied to the attention scores. The mask is regularized following the method by Weiss et al. [32]
We train the dynamics model of each method with three random seeds on pre-collected transition data and evaluate their performance by predicting the local causal graph G, for 50 unseen episodes based on the state-action pair (*x, dz). We compare their predictions with the ground truth local dependencies extracted from the simulator. inactive most of the time, and thus only small portion (< 3%) of the ground truth labels indicate the existence of local dependencies for a given entity pair. To account for such imbalance, we use the area under the receiver operating characteristic curve (ROC-AUC) and the best achievable F-score (FI) as evaluation metrics.
the three environments, many potential local dependencies are
The results of the evaluation on the detection of local dependencies are summarized in Table ELDEN outperforms all baselines in terms of ROC-AUC consistently across all environments (QI). For the Fl score, PCMI performs best in most environments (especially in the more complex CARWASH and KITCHEN), but ELDEN performs comparably or achieves the second-best Fl scores with much less computation: pCMI computation cost is N times higher than ELDEN, where N is the number of environment entities, and thus PCMI scales badly to environments with large number of objects. Further evaluation details can be found in the Appendix.
7

---

Disagreement
Dynamics Curiosity
CAI
Vanilla PPO
ELDEN (ours)
1.0
1.
1.3
Nomostee Stgje
0.55
0.5
0.3
0.5
0.0g
756 Episodes
150k
0.00
1Dc Episedes
80K
92K
0.0
10K Epsodes
30x
Episddss
(a) Thawing
(b) CarWash
(c) Craft
(d) Kitchen
Figure 3: Learning curve of ELDEN (ours) compared to bascline approaches. Each method uses three random seeds, and show the mean + std dev of the number of stages completed toward task success. The stage count normalized to [0, 1], where corresponds to task completion. ELDEN learns successful policies in all four test environments, and is the only method that succeeds in the CarWash, 2D Minecraft, and Kitchen environments with complex chained dependencies.
4.2 Evaluating Exploration in Sparse-Reward RL Tasks
The ultimate goal of our method is to improve exploration for RL in sparse-reward setups. In the second evaluation, we compare the performance ELDEN against several state-of-the-an exploration algorithms in reinforcement learning, including:
the intrinsic reward is computed based the variance of the predictions from an ensemble of forward dynamics models.
Disagreement [2
Dynamies Curiosity [5]: intrinsic reward is computed based on the prediction error of a trained forward dynamics model.
CAI (Causal Influence Detection) 2 an empowerment-basee method, where the agent given intrinsic reward for maximizing the number of state entities that depend on its action.
Vanilla PPO 26 baseline without intrinsic reward that serves as control signal.
While ELDEN can be used with any RL algorithm, in our experiments we use proximal policy optimization (PPO) (26), as well as with the baselines. To facilitate the introspection of the results, we define manually a set of semantic stages representing internal progress toward task completion. described in detail in the Appendix. Notice
The stage definitions for each of the environments that these stages are not used by the agents during training and do not provide any additional reward; they are only used to facilitate the analysis of the results.
Fig. 3 depicts the count of reached stages per episode during training for each task, normalized by the number of stages complete the task. In the normalized stage count, value of 1 corresponds to successfully completing the task and it is the only stage where the learning agents receive sparse task reward (not intrinsic). Fig. 3 indicates that ELDEN is able to learn successful policies in all four environments. Importantly, in CARWASH, 2D MINECRAFT and KITCHEN, ELDEN is the only method that successfully learns to complete the task, demonstrating the advantage of ELDEN over the baseline algorithms in tackling tasks with complex chained dependencies (Q2).
The normalized stage count of ELDEN in CARWASH, 2D MINECRAFT and KITCHEN does not converge to 1 (completing
entire task in all episodes) mainly due to two reasons: First, both tasks, the locations of the objects are randomly initialized at the start of each episode. For some initialization (c.g. a target object is blocked by unmovable obstacles), the task is impossible to solve. Second, in both tasks, two out of the three ELDEN training procedures with different random seeds converge to succeeding most of the time, but the training process with one seed fails to find a good policy, dragging down the mean value of the normalized stage count. This large variance in success a current limitation of ELDEN.
In the relatively simple THAWING environment, we found ELDEN does not provide a significant advantage over the other baseline methods. The Dynamics Curiosity baseline learns faster to achieve the task indicating better sample efficiency. This was rather expected: as with any exploration heuristic, ELDEN is not universally better than previous intrinsic reward methods instead, better suited for specific type of environment, where there are many complex and chained object

---

Table Ablation ELDEN on local dependency prediction (mean + std. error ROC AUC and FI)
THAWING
CARWASH
KITCHEN
ROC AUC (t)
F1 (1)
ROC AUC (1)
F1
ROC AUC (1)
F1 F1
no Mixup &
0.48 1 0.01
0.42 1 0.01
0.44 + 0.00
0.27 0.01
N/A
N/A
Reg
Reg,
0.57 0.01
0.52 0.01
0.54 0.01
0,42
0.64 + 0.01
0.24 0.01
i.c., 0
A 10
0.68 0.00
0.57 0.00
0.73 0.01
0.58 + 0.02
0.55 + 0.00
0.14 0.00
10
0.71 0.01
0.57
0.76
0.00
0.21
À = 10
0.64 + 0.01
0.01
0.78
0.66
0.65 0.00
0.24
10-4
0.65 1 0.02
0.55
0.75
0.60
0.66 0.00
= 10
0.63 1 0.00
0.53 0.01
0.72 0.00
0.57 0.00
0.65 1.01
0.24
dependencies, not the case for THAWING. We provide additional experimental evaluations of the failure cases of ELDEN in Appendix See
4.3
Ablation Studies
We ablate different components of ELDEN to examine their importance to the overall methods.
4.3.1
Ablations for Local Dependency Detection
In our ablation study on ELDEN for local dependency detection, we investigate the impact of each component with the following variations:
No Mixup & No Reg: We disable the use of Mixup for discrete space prediction, and no partial derivative regularization is applied in this case.
Different partial derivative regularization coefficients: we test with different À values in {0,10-) 1,10- 1,,1000110-1
10
10 1,000,100) 10.
As shown in Table. 2 Thawing and CarWash environments, partial derivative regularization with appropriate coefficients significantly improves ELDEN'S detection of local dependencies, compared to no regularization (i.e., 0) or inappropriate À values. Furthermore, in discrete-state environments, Mixup smooths the landscape of partial derivatives by providing synthesized continuous inputs as exemplified in Fig. 1(b) of Zhang et al. (34), thus facilitating local dependency prediction - even when compared to using Mixup without any regularization, not using Mixup leads to noticeable degradation the prediction performance.
4.3.2 Ablations for Task Learning
Next, examine how different components and hyperparameters of ELDEN affect task learning:
Ablation of Local Dependency Metrics We compare the exploration performance when using different local dependency detection methods. Specifically, we compare with pCMI as it achieves the best local dependency detection in Sec. 4.1 We present the comparison results between ELDEN and pCMI in the Kitchen environment in Fig. 4a) where both methods successfully learn to solve the task. However, it is important to notice that the computation cost of pCMI N times more than that of ELDEN (where N is the number of environment entities), and thus may not scale to environments with large number of entities.
Ablation of Dynamics Sample Prioritization We study the effectiveness of applying sample prioritization in dynamics model training. Specifically, we test ELDEN with and without prioritization in the Kitchen environment, and show the result in Fig. 4b.. We can see that ELDEN without prioritization fails to learn a useful policy. The reason is that some key entity interactions oceur rather rarely before the agent masters them, e.g., frying meatball with butter. In such cases, the dynamics model needs to quickly learn that unknown dependencies appear so that can bias the exploration toward reproducing such dependencies. Sample prioritization helps the dynamics model learn such

---

1.8
10
1.0
14
Momakoee Stage
coefi=10
0.5
0.5
10-3
0.5
co0
3
0.00
0.0
15c Number Epitedes
33C
0.0
6DC Number Editedas
0.0
BFK
Namberof Estodes
Number Epitodes
(a) Dependency detection (b) Sample prioritization
(c) Gradient thresholds
(d) Reward coefficients
Figure 4: Ablation of ELDEN on task learning. Each curve uses three random seeds and shows the mean + std dev of the normalized stages. We found ELDEN to have moderate tolerance towards hyperparameters. We found sample prioritization in dynamics learning be crucial the
performance of ELDEN.
infrequent dependencies quickly, making it critical environments with novel and hard-to-induce local dependencies.
Ablation of Partial Derivative Threshold: The partial derivative threshold c determines the depen- dency predictions. A threshold that is too large too small will make all dependency predictions negative/ positive respectively, leading to deteriorated performance. whether our method is sensitive to the choice of threshold in the CarWash environment, where the results are presented in Fig. c). We observe that our method is relatively sensitive to the choice of threshold, and an inappropriate threshold could cause catastrophic failure. A potential next step for ELDEN is to automatically determine the partial derivative threshold.
this section, we examine
Ablation of Intrinsic Reward Coefficient: The intrinsic reward coefficient controls the scale of the intrinsic reward relative to the task reward. We examine the effect of this coefficient by experimenting with different values in the CarWash environment, where the results are presented in Fig. 4 d). We find that our methods work well in large range of the intrinsic reward coefficients (1 10), since the task only gives sparse rewards and the intrinsic rewards are the only learning signal most of the time. The only exceptions are (1) when the intrinsic reward coefficient too large (e.g.. 100), the intrinsic reward significantly surpasses the task reward, and (2) when the coefficient is too small (e.g., 0.1), the episode intrinsic reward is too small (c.g., 0.03) for PPO to learn any useful policy.
Limitations and Conclusion
We introduce ELDEN, method for improving exploration in sparse reward reinforcement learning tasks. ELDEN identifies local dependencies between environment entities and uses the uncertainty about such dependencies as intrinsic reward improve exploration. Experiments demonstrate that ELDEN uncovers local dependencies more accurately compared to related methods, and significantly outperforms previous exploration methods in tasks with complex chained dependencies.
However, ELDEN is not without limitations. First, ELDEN intentionally bias exploration towards "covering up the possible interactions between objects" rather than "becoming expert at manipu- lating a particular object". While such an inductive bias works well in many practical domains, may fail when facing tasks that require precise object interaction (e.g. rotating the meatball in the pot to a specific orientation). solvable tasks is to combine ELDEN with dynamics curiosity and formulate composite intrinsic reward. Second, as noted in the experiment section, the variance of ELDEN across different random seeds can be large, While the high variance a general problem Reinforcement Learning, finding ways to further stabilize ELDEN can be an important direction for future work.
future direction to alleviate this problem and expand the scope of
10

---

This work has taken place in the Robot Interactive Intelligence Lab (Robin) and Learning Agents Research Group (LARG) the Artificial Intelligence Laboratory, The University of Texas at Austin. LARG research is supported in part by the National Science Foundation (FAIN- 2019844, NRT-21258588 the Office of Naval Research AX00014-18-2243) Army Research Office (E2061621), Bosch, Lockheed Martin. Both LARG and RobIn are supported by Good Systems, research grand challenge at the University of Texas at Austin. The views and conclusions contained in this document are those of the authors alone. Peter Stone serves the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas Austin in accordance with its policy on objectivity in research. We thank Bo Liu and Caleb Chuck for their valuable feedback on the manuscript.
Acknowledgements
References
[1] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, pages 166-175. PMLR, 2017.
[2] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[3] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[4] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint
2019.
[5] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint 1808.08800033555 2018.
[6] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:/810) /2894, 2018
[7] Maxime Chevalier-Boisvert,, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprini 181.:028 2018
[8] Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-awann exploration in reinforcement learning. arXiv preprint arXiv:1811.01483. 2018.
[9] Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine Learning to walk via deep reinforcement learning. RSS, 2018
[10] Jiaheng Hu, Peter Stone, and Roberto Martín-Martin. Causal policy gradient for whole- mobile manipulation. arXiv preprini arXiv:2305.0488 044666...6........... 2023.
[11] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 1xXiv:161.00114. 2016.
[12] Emily Jin, Jiaheng Hu, Zhuoyi Huang, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, and Roberto Martin-Martin. Mehivoee A procedurally generated benchmark for long-horizon decision- making in embodied ai. arXiv preprint 2310.01824, 2023.
[13] Alina Kloss, Stefan Schaal, and Jeannette Bohg. Combining learned and analytical models for predicting action effects. arXiv preprint 170.441710.04102, 11, 2017.
[14] Alexander S Klyubin, Daniel Polani, and Chrystopher L. Nehaniv. All else being equal be empowered. Advances in Artificial Life: 8th European Conference, ECAL 2005, Canterbury, UK. September 5-9, 2005. Proceedings 8, pages 744-753. Springer, 2005.
11

---

[15] Jens Kober. J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, (2((1): 2013.
[16] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-3375 2016.
[17] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael T'schanen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348 6389. PMLR, 2020.
[18] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015,
[19] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. Advances in neural information processing systems, 28, 2015.
[20] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity exploration by self-supervised prediction. In International conference machine learning. pages 2778- 2787. PMLR, 2017.
[21] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. ment. In International conference machine learning, pages 5062-5071, PMLR, 2019.
Self-supervised exploration via disagree-
[22] Judea Pearl. Causal inference in statistics: An overview, 2009.
[23] Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems, 33:3976-3990, 2020.
[24] Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model-based counterfactual data augmentation. arXiv preprint vrXiv:2210.11287, 2022.
[25] Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pages 222-227. 1991.
[26] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[27] Maximilian Seitzer, Bernhard Schölkopf. and Georg Martius. Causal influence detection for improving efficiency in reinforcement learning. Advances Neural Information Processing Systems, 34:22905-22911 2021.
[28] Sumedh A Sontakke, Arash Mchrjou, Laurent Itti, and Bernhard Schölkopf. Causal curios- ity: RI agents discovering self-supervised experiments for causal representation learning. In International conference machine learning, pages
PMLR, 2021.
[29] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arxivv 2015.
[30] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[31] Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for task-independent state abstraction, 2022. httpsssss://arxi org/:/.rg....///. httpg//bb//220.... 13452
[32] Martin Weiss, Nasim Rahaman, Francesco Locatello, Chris Pal, Yoshua Bengio, Bernhard Schölkopf, Erran Li Li, and Nicolas Ballas. Neural attentive circuits. Advances in Neural Information Processing Systems, 35:7741-7754, 2022.
12

---

[33] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896): 223-228, 2022.
[34] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
[35] Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, and Wei Xu. Mutual information state intrinsic control. arXiv preprini arXiv:2103.0// 2021.
[36] Yuke Zhu, Josiah Wong, Ajay Mandickar, and Roberto Martín-Martin. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 12293, 2020.
13

---

A ELDEN details
Assumptions We summarize our assumptions on the MDP as follows:
1. The state space can be factored S = S' x.... SN.
2. The transition of each state factor is independent, i.e., the dynamics can be represented P(s a.) II== p( Pa)S)+1))
3. There is no instantaneous dependency between state factors at the same time step I, i.e., no dependency such sf + sy for any i,j.
For assumption 1, factored state space is commonly employed in causality literature and applies to many simulated or robotics environments. In cases where low-level observations or partial observability are present, disentangled representation or causal representation methods can be utilized to learn a factored state space [17]. When a factored state space is available, assumptions 2 and generally hold.
Network Architecture In Figure 5 a), the architecture of ELDEN for predicting each state factor 441 is illustrated. The process consists of the following steps:
1. Feature Extraction: For each input state factor st, ELDEN utilizes a separate multi-layer perception (MLP) extract its corresponding feature g².
2. Entity Interaction: ELDEN employs multi-head self-attention module model entity interactions and generates a set of transformed features that incorporate information from other state factors.
With hJ as the query, ELDEN utilizes a multi-head attention module to compute the prediction p( ( |st, ae) for each state factor. For continuous state factor, 1) is modeled as normal distribution with the mean computed by the network and a fixed variance equal to 1. For discrete factor, p(s{+1) is a categorical distribution with network outputs class probabilities.
Prediction using Multi-Heaa Attention:
Throughout the prediction process, there are a total of N such networks in ELDEN, with each network responsible for predicting a separate state factor 8/41-
The training loss for the dynamics model is:
N L = - log I
0p(5(+11
((s + 2
(3)
ast
j=1
where À is the coefficient for partial derivative regularization.
B Environment Details
In this section, we provide detailed description of the environment, including its semantic stages representing internal progress toward task completion, state space, and action space. We also highlight that while each task consists of multiple semantic stages, agents do not have access to this information. The learning signal for agents is solely based on a sparse reward or 1, indicating whether the task has been completed or not. Additionally, in each environment, the poses of all environment entities are randomly initialized for each episode.
Meanwhile, as ELDEN focuses on exploring novel local dependencies between environment entities, all environments, the action space consists hard-coded skills increase the probability of entity interactions and bypass navigation challenges under sparse rewards. Extending ELDEN to explore local dependency and learn such skills simultaneously would be an important direction for future work.
Thawing As shown Fig. ((a), the Thawing environment consists of a sink, a refrigerator, and a frozen fish. The task requires the agent to complete the following stages: (1) open the refrigerator, (2) take the frozen fish out of the refrigerator, and (3) put the fish into the sink thaw it. The discrete state space consists of (i) the agent's position and direction, (ii) the positions of all environment entities,
14

---

ELDEN (ours)
f f 18t, (x)
query
MLP
Multt-Head
Multt-Head
MLP
SelF Attentioo
Attention
MIP
xN
(b) PCMI
M
query
MLP
Muls--Haad
Multi-Head
4
MLP'
Self-Attemtion
Atsention
MLP
Multi-Head
(c) Input Mask
Self AateAtttentonn
M
query
MLP!
Multi-Head
Multi-Heed
MLP
Self Attention
Attention
MLP
Multi-Head
M
(d)
Attn Mask
Salf- Astemnittn
query
MLP
Multi Head
Multi Head
MLP
Self Self-Attettion
Astention
MLP
The dynamics model of each local dependency detection method. (a) The dynamics model of ELDEN for predicting & Notice that each network predicts s441 only, and there are N such networks in total, each responsible for predicting one state factor in 8141- For visual simplicity. the N symbol is only shown in (a). (b) pCMI computes
Figure 5:
a.) and p(5/+1ls4 \ st ax) by manually setting the binary mask M to different values, where & represents element-wiss multiplication. (c) For Input Mask, M is learned to condition (st, as) and is regularized to use as few inputs as possible. (d) For Attn Mask, M also conditions on (st, a,) but is applied to the attention score in the self-attention module.
(iii) the thawing status of the fish, and (iv) whether the refrigerator door is opened. The discrete action space consists of (i) moving to specified environment entity, (ii) picking up / dropping down the fish, and (iii) opening / closing the refrigerator door.
Car Wash As shown in Fig. (b). the CarWash environment consists of a car, sink, a bucket, shelf, a rag, and a piece of soap. The task requires the agent to complete the following stages: (I) take the rag off the shelf, (2) put it in the sink, (3) toggle the sink to soak the rag up, (4) clean the car with the soaked rag, (5) take the soap off the self, and (6) clean the rag with the soap inside the bucket. The discrete state space consists of (i) the agent's 's position and direction, (ii) the positions of all environment entities, (iii) the soak status of the rag, (iv) the cleanness of the rag and the car, specified and (iv) whether the sink is toggled. The discrete action space consists of (i) moving environment entity, (ii) picking up / dropping down the rag, (iii) toggling the sink, and (iii) picking up / dropping down the soap.
2D Minecraft As shown in Fig. ((c), the environment has complex chained dependencies get the gem, the agent needs to
15

---

butter
mealball
Initial State
Goal State
(a) Thawing
(b) CarWash
(c) Kitchen
craft @
apply to
>
(d) Minecraft 2D
Figure 6: Environments.
get across the river to reach the gem by
(a) collecting a unit of grass and crafting a rope,
(b) collecting a unit of wood and crafting bridge with the rope,
(c) building the bridge on top of the river;
2. collect the gem
(a) collecting a unit of wood to craft wood stick
(b) collecting another unit of wood and combining it with the stick to craft wood pickaxe that is required for collecting the stone,
(c) collecting unit of wood and unit of stone to craft stick and then stone pickaxe that is required for collecting the gem,
(d) collecting the gem with the stone pickaxe.
Notice that all crafting must be conducted the crafting table. The discrete state space consists of (i) the agent's 's position and direction, (ii) an inventory tracking the number of materials and tools that the agent has, and (iii) the positions of all environment entities. The discrete action consists of (i) picking up applying tools to (only effective when the agent faces an environment entity and has the necessary tools interact with it). (ii) crafting specified tool (only effective when the agent has enough materials and faces the crafting table), and (iii) moving specified environment entity.
Kitchen As shown in Fig. (d), in the kitchen environment, there are robot arm (i.e., the agent), piece of butter, a meatball, a pot, and a stove with its switch. The task requires the agent to complete the following stages: (1) pick and place the butter into the pot, (2) pick and place the pot onto the stove, (3) turn on the stove to melt the butter in the pot, (4) pick and place the meatball into the pot cook it, and turn off the stove. Notice that melting the butter is a prerequisite for cooking the meatball, otherwise, it will result in the meatball being overcooked and the task failing. The state space is continuous, consisting of the pose of all objects, the melting status of the butter, and the cooking status of the meatball (whether is raw, cooked, or overcooked). The action space is discrete, consisting of hard-coded skills: moving to [butter, meatball, pot, pot handle, stove, stove switch], grasping, dropping, and toggling the switch. Grasping and toggling are only applicable when the end-effector is close the corresponding environment entities.
C Implementation of Local Dependency Detection
Baselines
We give detailed description of each bascline as follows:
16

---

PCMI (point-wise conditional mutual information): it considers that the local dependency 1/11 exists if their wisnt-wise conditional mutual information is greater than a predefined threshold, i.e. PCMT := log plss4alse 2 E. As shown Fig. b)),, to compute PCMT
Wang et al. [3 uses manually defined binary mask M [0, 10 to ignore some imputs when predicting 8/41: (1) to compute p(s(41ist St, at), M uses all inputs (all its entries are set to 1), and (2) to compute p(s \ ss
the entry for g¹ is set to O. When evaluating the local sf. ae) for every i. and thus its computation
dependency, PCMI needs to compute p cost is N times larger than ELDEN. We also computes pCMI following Seitzer et al. [27]. which yields similar performance but is even more computationally expensive compared to the method proposed by Wang et al. [31].
Atin (attention): it uses the same architecture as ELDEN that is shown in Fig. 5(a). When computing the overall attention score, it averages the attention score across all heads in each module, then computes the likelihood of dependency s - 5/41 as Ek=1 ca 'n . .844 where ea is the averaged score between the input a and the output b.
Input Mask: as shown in Fig- [jc), it also uses a binary mask M except that M computed from (8), During training, to only use necessary inputs for o/+, prediction, M is regularized with the LI norm on its number of non-zero entries. The Gumbel reparameterization is used to compute the gradient for the binary M [1]].
Attn Mask: as shown in Fig. d(d), similar to Input Mask, a mask M of size N x N is computed from (st, a.). but it is applied to the attention score. The mask is regularized with Stochastic Kernel Modulated Dot- not-Product (SKMDP) proposed by Weiss al. [32].
For modules that are shared by all methods, we use the same architecture for fair comparison.
Data For fair comparison, when training each method, a scripted policy, rather than let each method collect its own data, to avoid potential performance differences caused by data discrepancies. Specifically, use scripted policy to expose all potential local dependencies and collect 500K transitions in each environment.
the same dataset collected by
Notice that, in exploration with sparse reward experiments, the dynamics models are still trained online, using the transition data collected on its own.
The hyperparameters used for evaluating local dependency detection of each method are provided in Table 3 Unless specified otherwise, the parameters are shared across all environments.
Hyperparameters
D Evaluating Exploration in Sparse-Reward RL Tasks
D.1
Implementation
During policy learning, all methods share the same PPO and training hyperparameters, provided in The hyperparameters for dynamics model setup during policy learning are the same in Table unless specified otherwise.
Table 4
D.2 Success Rate Plots
As supplementary to the normalized stage metric used in the main paper, we provide the success rate as an additional metric. The success rate learning curves of methods the three environments are shown in in Fig. 7 Again, ELDEN outperforms and performs comparably with all baselines. Notice that, in the Car Wash and Kitchen environments, all baselines never succeed throughout the training (i.c., success rate 0 for all episodes), leading to training curves that overlap with the axis.
17

---

Table 3: Parameters of the dynamics model training for local dependency detection experiments. Parameters shared if not specified.
Name
Tasks CarWash
Thawing
Kitchen
episode length
20
100
100
environment
grid size
10
N/A
optimizer learning rate batch size of training batches
Adam
3 x 10
32
training
500k
# random seeds mixup Beta parameter
N/A
activation functions
ReLU
(MLP) i=
[64, 64]
[64, 64]
[128, 128]
annealing starts
50k
100k
annealing ends
100k
100k
200k
ELDEN
# of heads
4
use bias
False
attention
key, query. value size
32
output size post attn MI.P
128
[64, 64]
[64, 64]
[128, 128)
attention parameters
same as ELDEN
Input Mask
M regularization coefficient
x 10
M regularization annealing starts
50k
50k
100k
M regularization annealing ends
100k
100k
200k
attention parameters
same as ELDEN
Attn Mask
signature size Icarnable bandwidth
64
SKPMD
True
bandwidth initialization
Table 4: Parameters of the Policy Learning. Parameters shared if not specified.
Name
Tasks
Thawing
Car CarWash
Kitchen
optimizer
Adam Tanh
activation functions
learning rate
x 10
batch size clip ratio
32
0.1
PPO
MI.P size
[128, 128]
GAF À target steps n steps of environments
0.98
250
600
100
20
20
80
of random seeds intrinsic reward coefficient 8
# of dynamics update per policy step
dynamics learning rate
x 10-5
training
ensemble size
level of sample prioritization
N/A 0.1
N/A
0.5
mixup Beta parameter partial derivative threshold
0.1
N/A
x 10
18

---

Disagreement
Dynamics Curiosity
CAI
Vanilla PPO
ELDEN (ours)
1.0
1.0
1.3
1.0
Succres Rate
Suncees Bate
TFT
0.5
65
0.3
6.0g
756 Episodes
150K
0.0
10c Episodes
B0K
45A
92K
00
15c Episodes
30x
(a) Thawing
(b) CarWash
(c) Craft
(d) Kitchen
Figure 7: Learning curve of ELDEN (ours) compared to baseline approaches (mean + std dev of success rate across three random seeds). For CarWash, Craft, and Kitchen, the success rates for all baselines are zero throughout the training, overlapping with the axis.
1.0
Success
Dynamics Curiosity
Vanilla (eurs)
50K Number Episodes
100K
(a) navigation task with the goal in green
(b) ELDEN performs worse than Dynamics Curiosity
Figure 8: We demonstrate failure mode of our method on navigation task.
E
Failure Modes of ELDEN
As mentioned in the main paper, ELDEN may have limited advantages for tasks that require precise control of specific environment entity. One such example is navigation, where the agent needs to reach a very specific point space that has no particular semantic meaning. We empirically examine this statement in the Minigrid environment [7]. where the agent needs to navigate to the green goal point in an empty room through primitive actions (turn left, turn right, and move forward), as shown in Fig. B(a). We compare ELDEN against Dynamics Curiosity and Vanilla PPO, and present the result in Fig. B). Since this environment is relatively simple, all three methods are eventually able to solve the task. However, the Dynamics Curiosity converges faster than ELDEN, showing that ELDEN is indeed not as capable as curiosity-drive
explorations tasks that focus on precise control rather than exploring dependencies between environment entities. The Vanilla PPO converges slowest, indicating that even in the Empty environment, ELDEN still has advantages over purely random exploration.
Compute Architecture
The experiments were conducted on machines of the following configurations:
Nvidia 2080 Ti GPU; AMD Ryzen Threadripper 3970X 32-Core Processor
Nvidia A40 GPU; Intel(R) Xeon(R) Gold 6342 CPU @2.80GHz
Nvidia A100 GPU; Intel(R) Xeon(R) Gold 6342 CPU @2.80GHz
19


Reviews:
[{"review_id": "Qysl5Xxhiv", "reviewer": "Reviewer_kab1", "summary": "This paper proposes ELDEN, a method for intrinsic reward based on local dynamics dependencies between factored state variables. It learns an ensemble of factored dynamics models, and uses the magnitude of the partial derivatives to detect local dependencies between state variables, and uses the ensemble variance as an exploratory intrinsic reward. This method is tested across a variety of environments such as gridworlds and robotic control tasks, showing great accuracy in capturing local dependencies and exploratory performance.", "questions": "It is mentioned that ELDEN can help with more complex, chained dependencies, however the basic method is still only looking at 1-step local dependencies. Why would one expect ELDEN to do better than other 1-step methods?", "limitations": "Are addressed appropriately in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper presents a very clear and thorough test of the main ELDEN method. ELDEN itself is straightforward to understand, and section 4.1 does a convincing job of evaluating how well it detects local dependencies compared to reasonable baselines. Section 4.2 also has reasonable baselines, and clearly demonstrates the effectiveness of ELDEN. The ablations on mixup and the regularization are also well done.\n\nOverall, many interesting questions around ELDEN are covered by the paper, and the results are very clear.\n", "weaknesses": "A common weakness, not specific to ELDEN, is of course the assumption of the factored state space. While this can be overcome with object-identifying representations, there is still the question of how effective ELDEN may be when paired with learned factored representations as opposed to ground truth. Hopefully future work can look in this direction.\n\nWhile it is mentioned that ELDEN is not meant to excel in all domains, an additional experiment in a more common domain such as the sparse reward tasks of DM Control Suite would be enlightening to see. Perhaps ELDEN would also work well, or it may not compared with the baselines; either way it would provide some additional insight into the limitations of ELDEN.\n\n---- After Author Rebuttals ----\n\nAfter reading other reviews and author rebuttals, I maintain my score. I think this paper has clearly shown when and where ELDEN can excel, and the limitations when ELDEN is not very effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following questions about our work.\n\n> A common weakness, not specific to ELDEN, is of course the assumption of the factored state space. Hopefully future work can look in this direction.\n\nWe agree that testing the effectiveness of ELDEN when the factored representations are learned would be interesting next steps.\n\n> While it is mentioned that ELDEN is not meant to excel in all domains, an additional experiment in a more common domain such as the sparse reward tasks of DM Control Suite would be enlightening to see. Perhaps ELDEN would also work well, or it may not compared with the baselines; either way it would provide some additional insight into the limitations of ELDEN.\n\nIn the global response, we add the results of the DMC Cheetah domain with sparse rewards in Fig 2. As expected by the reviewer, ELDEN and empowerment method (CAI) performs worse than the curiosity-based method. We analyze the reasons as follows:\n\n- With regard to ELDEN, DMC with sparse rewards is challenging in the aspect of precise low-level sensorimotor control, which is orthogonal to the focus of ELDEN on selecting the correct interaction among many options. Hence, it is not surprising that ELDEN does not help with the exploration in the cheetah domain.\n- With regard to CAI: the cheetah task requires precise control of each joint. In contrast, CAI is motivated to maximize its control over all joints (rotating them between the joint limits), and thus it fails to learn the tasks and harms the exploration. \n\n> It is mentioned that ELDEN can help with more complex, chained dependencies, however, the basic method is still only looking at 1-step local dependencies. Why would one expect ELDEN to do better than other 1-step methods?\n\nTake the Minecraft 2D environment in Appendix G as an example, where it has the chained dependencies of “collect wood” $\\rightarrow$ “craft wood axe” $\\rightarrow$ “collect stone” $\\rightarrow$ “craft stone axe” $\\rightarrow$ “collect gem”. As shown in Appendix G, our method solves this problem better compared to baselines (dynamics curiosity, dynamics uncertainty, CAI, and RND) because:\n- Our method prioritizes novel 1-step interactions (local dependencies). In the initial learning stage, compared to baselines that are preoccupied with repetitive movements for seeking novel state values (e.g., new agent positions after movements), our approach prioritizes novel interactions (e.g., collecting wood, grass, or other objects). This proactive exploration strategy increases the agent's chances of successfully collecting wood, which, in turn, **opens up new possibilities for further interactions**, such as crafting the wood axe.\n- Though the reward is based on 1-step local dependencies, as RL optimizes the return, the agent will seek both current and future novel dependencies. For example, as the agent undergoes training, it may become familiar with former interactions in the chain (i.e., up to “collect stone”). However, it remains motivated to tackle the latter, less familiar dependencies that follow the “craft stone axe”. Hence, even if the former dependencies do not give any 1-step reward, the agent will still finish them to continue exploring the latter dependencies and maximize its return.\n\nWe will include this discussion in the next version of the paper.\n\n"}, {"review_id": "4qA2DV5UeQ", "reviewer": "Reviewer_FZmh", "summary": "This paper proposes a novel intrinsic reward called ELDEN for reinforcement learning. ELDEN encourages the discovery of new local dependencies between entities. Experiments on some robotic tasks are carried out to valid the idea.", "questions": "1. What are the state-space and action-space of the used tasks (including the meanings and number of dimensions)? How long is the horizon for each task?\n\n2. How does proposed method perform in image-based tasks? \n", "limitations": "The authors discussed their limitations and potential negative societal impact properly.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea is novel and interesting.\n\n2. The usage of dynamics models to approximate causal dependency is interesting.\n\n3. The method does not make any assumption of knowing groundtruth dependency.\n\n4. The paper is well-written. The authors discussed their method are different from existing works.", "weaknesses": "1. My concern is about the significance of proposed method. This work focuses on solving semantic-level tasks and assumes knowing underlying objects. Then, why should we use RL in this case? A better option is probably using large language models (LLM) to do the semantic reasoning directly. Another option is to use LLM to design exploration reward like [1]. I would like to see authors opnion on this.\n\n2. In the experiments, two of the considered tasks are not solved (reach normalize score 1.0). The performance is not strong enough. \n\n3. The authors can compare to more baselines. One important baseline can be RND.\n\n[1] Yuqing Du, Olivia Watkins, Zihan Wang, Cedric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, Jacob Andreas. Guiding Pretraining in Reinforcement Learning with Large Language Models. In ICML 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following concerns regarding the significance and evaluation of our work.\n\n> Weakness 1: compare ELDEN (and RL in general) with LLMs\n\nThank you for making this point. While we are as excited as everyone else about the possibilities of leveraging LLMs in robotics research, we also think it’s important that the community does not lose interest in research outside the scope of LLMs.  While it may indeed be possible to study a similar problem through the lens of LLMs, please note that \n**ELDEN does not rely at all on the semantic meaning of state factors and actions**. It only requires that the state space is factored. As pointed out by R1, this is itself an important assumption to examine.  But it is a completely separate assumption than those made by LLM-based research.Specifically, ELDEN differs from research using LLMs (including Du et.al 2023 [1]) in at least the following ways:\n- LLMs require the semantic meaning of each state factor and action for effective reasoning.\n- Using LLMs for reasoning assumes that LLMs know how the environment transitions, but many environments are naturally hard to describe in language and are thus not well learned by LLMs. In contrast, our method (and RL in general) does not assume pre-existing knowledge of the transition probabilities and can learn from the environment interactions.\n- Using LLMs constrains the performance to be at the human level, while our method (and RL in general) has the potential to surpass humans and find novel strategies for solving the task, like how AlphaGo finds new Go strategies.\n\n>Weakness 2: two tasks are not solved.\n\nPlease note that in those two tasks, our method was able to solve them for 2 out of 3 random seeds. The other seed only achieves some intermediate stages, rendering the average score smaller than 1.0. \nFor the unsuccessful seed, notice that under our problem setup, the agent gets a reward only if it successfully completes the task. **For such tasks, it is very common for successful exploration methods to have high variance.** (for example, in Fig 5 (c) of Pathak et al [2]). Intuitively, this is because the intrinsic reward can at its best provide some sort of state-coverage guidance, and the agent often has to rely on luck to obtain its first successful run, which naturally introduces lots of stochasticity.\n\n[2] Deepak Pathak, et al. Curiosity-driven Exploration by Self-supervised Prediction. In ICML 2017.\n\n>comparison with RND.\n\nIn the global response, we present new experiments comparing against RND on the challenging Minecraft 2D domain, where our method significantly outperforms all baselines including RND.\n\n>Environment specifications (state and action space, their meanings, and horizon)\n\nWe list the environment specifications below and will include the extra specifications in the next version of the paper (note that the **MEANINGS** of state variables and actions are not given to our method). Given the space limits, we put the specification of Thawing and CarWash environments in the second part of the global response:\n- Kitchen:\n  - state space (43d), consisting of robot end-effector position (3d) and velocity (3d), robot gripper angles (2d) and angular velocity (2d), butter position (3d) and quaternion (4d), butter melting state (1d), meatball position (3d) and quaternion (4d), meatball cook state (1d), pot position (3d) and quaternion (4d), stove position (3d), target position (3d),  stove switch position (3d) and state (1d), \n  - action space (12d), consisting of\n    - move the hand to 1) the butter, 2) the meatball, 3) above the pot, 4) the pot handle, 5) the stove, 6) the stove switch. \n    - grasp 7) the butter, 8) the meatball, 9) the pot handle, if the hand is close to it\n    - 10) turn on/off the stove switch, if the hand is close to it\n    - 11) open the gripper\n    - 12) no action, which is useful for waiting for the meatball to be cooked\n  - given horizon: 200\n  - minimal # of time steps required to finish the task: 25\n- Minecraft 2D:\n  - state space (40d), consisting of \n    - agent position (2d) and direction (1d)\n    - positions (2d) of the 1 gem block, 5 water blocks, 2 grass blocks, 5 wood blocks, and 1 stone block (28d in total).\n    - the number of materials (grass, wood, stone, block) and tools (axe stick, wood axe, stone axe, rope, bridge) in the inventory (9d).\n  - action space (21d):\n    - move next to each of the environment objects (gem, water, grass, wood, stone blocks, and the workbench), if there is a path (15d in total)\n    - craft a tool among the choice of (axe stick, wood axe, stone axe, rope, bridge) if the agent has enough materials and is next to the workbench (5d)\ncollect the material that faced by the agent, applicable only if the agent has the tool necessary for collection (1d)\n  - given horizon: 200\n  - minimal # of time steps required to finish the task: 23\n\nIn these environments, the exploration challenges are caused by the large action space and the chained dependencies — the agent needs to efficiently explore throughout many interaction modes and find the correct order of interactions that leads to task success.\nFor all environments, the actions only have effects when their preconditions are met, i.e., in the kitchen environment, grasping the butter will execute only if the robot hand is empty and close to the butter.\n\n>How does the proposed method perform in image-based tasks?\n\nSolving image-based tasks is beyond the scope of this paper. However, a simple potential way to extend our method to image-based tasks is to extract disentangled features from an image (e.g. Schölkopf, Bernhard, et al. [1]) and use the extracted features as the state space.  We will add this as an interesting direction for future work.\n\n[1] Schölkopf, Bernhard, et al. \"Toward causal representation learning.\" Proceedings of the IEEE 109.5 (2021): 612-634.\n"}, {"review_id": "UqMEeSdoQn", "reviewer": "Reviewer_G2vP", "summary": "The paper proposes an intrinsic exploration reward that explicitly takes dependencies between different entities in the environment into account. Such an intrinsic reward helps sparse-reward RL in a few skill-based domains.", "questions": "- How is the variance of the dependency graph computed (Algorithm 1)?\n- Is there a domain with low-level control in which ELDEN can be benchmarked?", "limitations": "Some limitations are addressed in the Conclusion, and more interesting ones are also discussed in Appendix F. In particular Appendix F makes the point that ELDEN is suboptimal in navigation environments, and thus that it is particularly scoped for settings with a big dependency graph.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- I find the idea of modeling dependency graphs between entities interesting and worth exploring.\n- The related work section is written especially well, making it very easy to understand how the proposed work is contextualized.\n- Explanation of the method is clearly written and easy to understand.\n- Results on given domains are promising.", "weaknesses": "- The major weakness of the paper is in the evaluation. In particular, the domains that are used for evaluation all use high-level primitives (like goto object and pickup object) which results in relatively few steps in order to accomplish a goal. It's hard to argue that this setting is comparable to the difficulty of something like Franka Kitchen (low-level control in a kitchen environment with sparse reward). And especially given many moving parts are required to make this work, it would limit the applicability in my reading. It's even more unfortunate that this is not justified and only discussed in detail in Appendix C, given that it transforms a long-horizon sparse-reward problem (the typical understanding) into a relatively short-horizon problem.\n- With regards to the above, ELDEN is proposed as a standalone method, but in the most complex domain tested (Kitchen), an additional exploration method, PER, is required in order to make it work.\n- Too much is pushed to the appendix, for instance architectural details that are important to understand for evaluation (Attn Mask in section 4.1) are only discussed in Figure 4. Ablations for a method with many moving parts only show up in Appendix E.\n- All domains are discrete action, and the most convincing experiments (no PER required) are discrete state. The use of partial derivatives through Mixup is a little bit strange and I don't understand where the claim \"dynamics models trained on such data... reflect local dependencies\" is justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following concerns regarding the evaluation of our work.\n\n> The experiment domains use high-level primitives which results in relatively few steps in order to accomplish a goal. It's hard to argue that this setting is comparable to the difficulty of something like Franka Kitchen (low-level control in a kitchen environment with sparse reward). \n(from the questions section: Is there a domain with low-level control in which ELDEN can be benchmarked?)\n\nWe agree with the reviewer that the Franka Kitchen environment is interesting and challenging. However, we believe environments with high-level primitives that focus on a different level of exploration challenges are not inherently less interesting or easier to solve. Specifically,\n - As mentioned in the introduction, our method aims to solve tasks with many interaction modes, where the challenge is to **find the correct sequence of interactions among many options** that leads to task success. For example, in our crafter (Minecraft 2D) environment described in appendix G, the agent has > 20 objects to interact with and 5 tools that can craft, and the agent has to efficiently explore through these interactions. As demonstrated by our experiments, even with high-level primitives, all of the baseline methods fail to solve the task (except for thawing), establishing that dealing with many interaction modes is indeed challenging.\n - By comparison, the original Franka Kitchen environment [1] has almost no object dependencies, where each object can be toggled independently and the task objective is just to achieve multiple independent goals, one for each object. As such, low-level control in the Franka Kitchen environment is challenging in the aspect of **goal-reaching** (moving the end-effector to the object, similar to navigation) and **precise sensorimotor control** (grasping the object), but not in the aspect of selecting the correct interaction among many options, making it orthogonal to the focus of our method. \n\nThat being said, we provide new experimental results examining the performance of ELDEN on low-level, continuous domains in the global response. Specifically, we test replacing the action primitives in our kitchen environment with low-level position control of the gripper.\nAs shown in Fig 3 in the global response, the problem becomes so hard that none of the methods can make meaningful progress. To the best of our knowledge, most methods that can learn with low-level actions and sparse rewards require some form of human priors (like offline data) [2].\n\nWe agree with the reviewer that an exciting future direction of this work would be to combine ELDEN with more fine-grained exploration methods to solve domains that entail both rich interaction and low-level controls. \n\n[1] Gupta, Abhishek, et al. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. CoRL, 2019.\n\n[2] Alakuijala, Minttu, et al. Learning reward functions for robotic manipulation by observing humans. ICRA 2023\n\n>And especially given many moving parts are required to make this work, it would limit the applicability in my reading. With regards to the above, ELDEN is proposed as a standalone method, but in Kitchen domain, an additional exploration method, PER, is required in order to make it work.\n\nWe want to clarify, to generate the intrinsic reward, our method only needs to learn an ensemble of dynamics models. \n\nPlease note that prioritized experience replay (PER), or sample prioritization, is applied to **dynamics training** only, rather than an exploration method for policy learning. As the data buffer grows with collected transitions, PER is a common technique to let the dynamics model learn from new data efficiently, and it is widely used in other intrinsic reward methods, like Lobel et al [3].\n\nFor all methods, we use PPO to learn policy. PPO is an on-policy algorithm that does not learn from the replay buffer and **PER is not applicable**.\n\n[3] Lobel, Sam, Akhil Bagaria, and George Konidaris. Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning. ICML 2023\n\n>Too much is pushed to the appendix, for instance architectural details that are important to understand for evaluation (Attn Mask in section 4.1) are only discussed in Figure 4. Ablations for a method with many moving parts only show up in Appendix E.\n\nWe agree with the reviewer that details and justifications of the environments, as well as some ablation experiments should be moved to the main text. We will prioritize making space for that in the next version of our paper.\nHowever, with regards to Attn Mask, it is a baseline that we describe clearly in the text of Section 4.1. The architecture is not our own contribution, so we consider it appropriate to appear only in the appendix.\n\n>The use of partial derivatives through Mixup is a little bit strange and I don't understand where the claim \"dynamics models trained on such data... reflect local dependencies\" is justified.\n\nMixup has been shown to smooth the partial derivatives of the model output w.r.t. inputs (Fig 1 (b) and Fig 2 (b) in [4]), and thus is suitable to improve partial derivative estimation in our method.\nFor the justification that mixup improves local dependency identification, the results are shown in Table 3 (the “no Mixup” row vs others).\n\n[4] Zhang, Hongyi, et al. \"mixup: Beyond empirical risk minimization.\" arXiv preprint arXiv:1710.09412 (2017).\n\n>How is the variance of the dependency graph computed?\n\nThanks for pointing this out. Each dependency graph is represented as a binary adjacency matrix. Given a set of graphs, we calculated their per-entry variances and used the mean acorss all entries in the graph as the eventual variance. We will include this specification in the next version of the paper.\n"}, {"review_id": "gLGCg534N6", "reviewer": "Reviewer_xY6d", "summary": "This work proposes `ELDEN, Exploration via Local DepENdencies`, a framework with an intrinsic reward that encourages the discovery of new interactions between entities such as the agent or objects that have some influence on each other. The method uses partial derivative of the learned dynamics to model the local dependencies between entities. The uncertainty of the predicted dependencies is used as an intrinsic reward. What distinguishes this from other related work, is that the focus for defining what makes a state interesting. Traditional approaches may focus on the specifics of how entities in an environment interact. In contrast, here interesting states are defined\nbased on whether the entities *can* interact, without focusing on the specifics of how the interaction occurs. Specifically, the algorithm is biased towards exploring states where the relationships between entities are not well understood.\n\nTo implement this, the authors train an ensemble of dynamics models. Within each model in the ensemble, local dependencies between objects are modeled using partial derivatives of state predictions with respect to the current state and action. Partial derivatives can capture how changes in state or action affect changes in predictions, which can be thought of as capturing local interactions. The uncertainty in local dependencies is then quantified as the variance across all the models in the ensemble where a high variance suggests high uncertainty.\n\nFor learning the object interactions themselves, they propose using a `Causal Graphical Model` to represent the transition at a given time step 't'. CGM consists of three parts: `1. Nodes`: These represent the state at time step 't' (S_t), the action at time step 't' (A_t), and the state at the next time step 't+1' (S_t+1). `2. Directed Graph G`: This graph illustrates the global dependencies between the nodes. \n`3. Conditional Distribution 'p'`: For each state variable at the next time step (S_n_t+1),  there is a conditional distribution 'p' which represents the probability of reaching that state given the current state and action.\n\nThe assumption is that the transition probability `P(s_t+1 | st, at)` can be factorized into a product of conditional probabilities for each state variable at the next time step. Here, `Pa(v)` refers to the parent nodes of a node `v` in the causal graph `G`. This means that the probability of transitioning to a new state depends on the current state and action, and can be represented as a product of several smaller probabilities based on the causal relationships.\n\nInstead of using these global graph `G` which includes all possible dependencies, the authors use a local causal graph `G_t`.\nThis Local Causal Graph Model is aimed at more accurately and efficiently representing the real-world scenarios by focusing on relevant \ndependencies at a given time rather than considering all possible interactions. This can potentially lead to more efficient learning and better generalization in reinforcement learning environments.\n\nELDEN's main goal is to identify the dependencies that are locally active among the entities in the environment. \nIn other words, for a specific state 'st' and action 'at', \nELDEN tries to determine which entities in the environment have an active causal relationship with each other.\n\nThis involves constructing a graph where the nodes represent entities, and the edges represent active causal relationships between them. \nThis graph is specific to the given state and action (st, at), and is called the local causal graph 'Gt'.\n\n **Experiments**\n\nExperiments are conducted with the following baselines: `pCMI (point-wise conditional mutual information)`,  `Attn (attention)`: Use the score between each entity pair computed by the attention modules inside the dynamics model to quantify local dependencies. `Vanilla PPO`.\n\nPresented tasks : `Kitchen Task` has a very high variance! ` Thawing Dynamics` curiosoty outperorms ELDEN. `CarWash` results also has high variance.\n\n", "questions": "- For the PPO baseline this serves as a baseline with sparse reward? So it is not surprising that this baseline doesnt really work ?", "limitations": "- Underwhelming experiments.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The ideas presented here are very interesting and novel. Creating an intrinsic reward based on local dependency, which aids exploration in RL, especially in environments with sparse rewards is novel.", "weaknesses": "- Weak Experimental results. The results as they stand are not very convincing.\nThat being said, these are hard problems so I understand the challenges in making these environments to work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and for the very constructive suggestions! We hope our responses adequately address the following concerns regarding the evaluation of our work.\n\n>The results as they stand are not very convincing. That being said, these are hard problems so I understand the challenges in making these environments work.\n\nWhile we appreciate where the reviewer is coming from, we submit that the experimental results are stronger than they may appear to be on the surface. Specifically,\n\n- High variance is normal: Notice that under our problem setup, the agent gets a reward only if it successfully completes the task. For such tasks, it is very common for successful exploration methods to have high variance. (for example, in Fig 5 (c) of Pathak et al [1]). Intuitively, this is because the intrinsic reward can at its best provide some sort of state-coverage guidance, and the agent often has to rely on luck to obtain its first successful run, which may never happen for some random seeds.\n- Extra experiment domains: Also notice that we provide extra experiments on the crafter (Minecraft 2D) domain in appendix Sec G. In this challenging environment, the agent has > 20 objects to interact with, 5 tools that it can craft, and a complex technology tree to follow, where the agent has to efficiently explore through these interactions.\n- Additional baseline: Additionally, as part of the rebuttal, we added new experiments that compare against a new baseline, RND, on the crafter domain. Results can be seen in the global response in Fig 1.\n\n[1] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros and Trevor Darrell. Curiosity-driven Exploration by Self-supervised Prediction. In ICML 2017.\n\n> For the PPO baseline this serves as a baseline with sparse reward? So it is not surprising that this baseline doesnt really work?\n\nYes, as described in Sec 4.2 Vanilla PPO is a baseline that learns from sparse reward only.  So as the reviewer points out, it is not expected to work in the challenging settings that we consider.\n\n"}]

Author rebuttal:
We thank all reviewers for the detailed reading of our paper and constructive suggestions! In the global response, we would like to describe the setup of additional experiments and the results can be found in the attached pdf.
- Comparison with RND: Using the challenging crafter (Minecraft 2D) domain described in Appendix G (featuring > 20 objects for the agent to interact with and 5 tools to craft, with a complex technology tree), we compare our method against RND. As shown in Fig 1, our method ELDEN significantly outperforms all baselines, which again demonstrates the strength of ELDEN in solving tasks with many interaction modes and complex preconditions.
- Results on a deepmind control suite (DMC) domain, asked by R1 kab1: In Fig 2, we show the performance in the Cheetah environment with sparse reward. Following the "no free lunch theorem", no intrinsic reward method can perform best in all environments. As discussed in Sec 5 (and above), our method ELDEN aims to solve tasks with many interaction modes, where the challenge is to find the correct interaction leading to the task success. However, DMC with sparse rewards is challenging in the aspect of precise low-level sensorimotor control. Hence, unsurprisingly, ELDEN performs worse than curiosity-based methods.
- Results on using low-level actions in the Kitchen domain, asked by R3 G2vP: In Fig 3, in the kitchen environment, we use low-level actions consisting of end-effector x, y, z movements (in the range of [-5, 5] cm) and gripper control (in the range of [-1, 1], where > 0 for closing the gripper and < 0 for opening), which is one of the low-level action formats provided in the robosuite [1]. Similar to DMC domains, using low-level actions is challenging in the aspect of goal-reaching (moving the end-effector to the object, similar to navigation) and precise sensorimotor control (grasping the object). To the best of our knowledge, in such manipulation domains, most methods that can learn from sparse rewards with low-level actions require some form of human priors (like offline data) [2][3]. As a result, none of the methods make meaningful progress.

With regard to DMC domains and manipulation domains with low-level actions, we agree with the reviewer that an exciting future direction of this work would be to combine ELDEN with more fine-grained exploration methods to solve such domains that entail both rich interaction and low-level controls.

[1] Zhu, Yuke, et al. "robosuite: A modular simulation framework and benchmark for robot learning." arXiv 2020.
[2] Gupta, Abhishek, et al. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In CoRL, 2019.
[3] Alakuijala, Minttu, et al. Learning reward functions for robotic manipulation by observing humans. ICRA 2023


### In the remaining global response, we would like to continue the descriptions of Thawing and CarWash environments, asked by R2 FZmh.

> Environment specifications (state and action space, their meanings, and horizon)

- Thawing:  
  - state space (12d), consisting of agent position (2d), agent direction (1d), fish position (2d), fish frozen state (1d), sink position (2d), refrigerator position (2d), refrigerator state (1d), timestamp (1d)
  - action space (7d), consisting of 
    - move towards 1) the refrigerator, 2) the sink, 3) the fish,
    - 4) open, or 5) close the refrigerator,
    - 6) pickup, or 7) drop the fish
  - given horizon: 100
  - minimal # of time steps required to finish the task: 11
- CarWash: 
  - state space (18d), consisting of agent position (2d), agent direction (1d), rag position (2d), rag state (2d), sink position (2d), sink state (1d), bucket position (2d), soap state (2d), car position (2d), car state (1d), timestamp (1d).
  - action space (10d), consisting of 
    - move towards 1) the rag, 2) the sink, 3) the soap, 4) the bucket, 5) the car
    - pick up 6) the rag, 7) the soap
    - drop 8) the rag, 9) the soap
    - toggle the sink 10) 
  - given horizon: 300
  - minimal # of time steps required to finish the task: 31


Meta-review: The reviewers have laregly formed a positive consensus, while suggesting a number of suggestions and in particular requested further comparison and experimentation. This appears to be an already high-quality paper that will be improved by incorporation of the author's rebuttal replies and additional results (1-pager). I recommend acceptance, while requesting the authors take extra care to fully integrate the updated(improved) story into all aspects of the final paper.
Decision: Accept (poster)