SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CarWash As shown in Fig. 6(b), the CarWash environment consists of a car, a sink, a bucket, a shelf, a rag, and a piece of soap. The task requires the agent to complete the following stages: (1) take the rag off the shelf, (2) put it in the sink, (3) toggle the sink to soak the rag up, (4) clean the car with the soaked rag, (5) take the soap off the self, and (6) clean the rag with the soap inside the bucket. The discrete state space consists of (i) the agent's position and direction, (ii) the positions of all environment entities, (iii) the soak status of the rag, (iv) the cleanness of the rag and the car, and (iv) whether the sink is toggled. The discrete action space consists of (i) moving to a specified environment entity, (ii) picking up / dropping down the rag, (iii) toggling the sink, and (iii) picking up / dropping down the soap.
2D Minecraft As shown in Fig. 6(c), the environment has complex chained dependencies — to get the gem, the agent needs to
(c) Kitchen
(a) Thawing
Figure 6: Environments.
(d) Minecraft 2D
1. get across the river to reach the gem by
(a) collecting a unit of grass and crafting a rope,
(b) collecting a unit of wood and crafting a bridge with the rope,
(c) building the bridge on top of the river;
2. collect the gem by
(a) collecting a unit of wood to craft a wood stick
(b) collecting another unit of wood and combining it with the stick to craft a wood pickaxe that is required for collecting the stone,
(c) collecting a unit of wood and a unit of stone to craft a stick and then a stone pickaxe that is required for collecting the gem,
(d) collecting the gem with the stone pickaxe.
Notice that all crafting must be conducted at the crafting table. The discrete state space consists of (i) the agent's position and direction, (ii) an inventory tracking the number of materials and tools that the agent has, and (iii) the positions of all environment entities. The discrete action consists of (i) picking up / applying tools to (only effective when the agent faces an environment entity and has the necessary tools to interact with it), (ii) crafting a specified tool (only effective when the agent has enough materials and faces the crafting table), and (iii) moving to a specified environment entity.
Kitchen As shown in Fig. 6(d), in the kitchen environment, there are a robot arm (i.e., the agent), a piece of butter, a meatball, a pot, and a stove with its switch. The task requires the agent to complete the following stages: (1) pick and place the butter into the pot, (2) pick and place the pot onto the stove, (3) turn on the stove to melt the butter in the pot, (4) pick and place the meatball into the pot to cook it, and (5) turn off the stove. Notice that melting the butter is a prerequisite for cooking the meatball, otherwise, it will result in the meatball being overcooked and the task failing. The state space is continuous, consisting of the pose of all objects, the melting status of the butter, and the cooking status of the meatball (whether it is raw, cooked, or overcooked). The action space is discrete, consisting of hard-coded skills: moving to [butter, meatball, pot, pot handle, stove, stove switch], grasping, dropping, and toggling the switch. Grasping and toggling are only applicable when the end-effector is close to the corresponding environment entities.
C Implementation of Local Dependency Detection
Baselines We give a detailed description of each baseline as follows:
- pCMI (point-wise conditional mutual information): it considers that the local dependency  $s_t^i \to s_{t+1}^j$  exists if their point-wise conditional mutual information is greater than a predefined threshold, i.e.,  $\mathrm{pCMI}^{i,j} := \log \frac{p(s_{t+1}^j | s_t, a_t)}{p(s_{t+1}^j | s_t \setminus s_t^i, a_t)} \geq \epsilon$ . As shown in Fig. 5(b), to compute  $\mathrm{pCMI}^{i,j}$ , Wang et al. [31] uses a manually defined binary mask  $M \in [0,1]^N$  to ignore some inputs when predicting  $s_{t+1}^j$ : (1) to compute  $p(s_{t+1}^j | s_t, a_t)$ ,  $M$  uses all inputs (all its entries are set to 1), and (2) to compute  $p(s_{t+1}^j | s_t \setminus s_t^i, a_t)$ , the entry for  $g^i$  is set to 0. When evaluating the local dependency, pCMI needs to compute  $p(s_{t+1}^j | s_t \setminus s_t^i, a_t)$  for every  $i$ , and thus its computation cost is  $N$  times larger than ELDEN. We also computes pCMI following Seitzer et al. [27], which yields similar performance but is even more computationally expensive compared to the method proposed by Wang et al. [31].
- Attn (attention): it uses the same architecture as ELDEN that is shown in Fig. 5(a). When computing the overall attention score, it averages the attention score across all heads in each module, then computes the likelihood of dependency  $s_t^i \rightarrow s_{t+1}^j$  as  $\sum_{k=1}^{N} c^{g^i, h^k} \cdot c^{h^k, s_{t+1}^j}$  where  $c^{a,b}$  is the averaged score between the input  $a$  and the output  $b$ .
- Input Mask: as shown in Fig. 5(c), it also uses a binary mask  $M$  except that  $M$  is computed from  $(s_t, a_t)$ . During training, to only use necessary inputs for  $s_{t+1}^j$  prediction,  $M$  is regularized with the L1 norm on its number of non-zero entries. The Gumbel reparameterization is used to compute the gradient for the binary  $M$  [11].
- Attn Mask: as shown in Fig. 5(d), similar to Input Mask, a mask  $M$  of size  $N \times N$  is computed from  $(s_t, a_t)$ , but it is applied to the attention score. The mask is regularized with Stochastic Kernel Modulated Dot-Product (SKMDP) proposed by Weiss et al. [32].
For modules that are shared by all methods, we use the same architecture for a fair comparison.
Data For a fair comparison, when training each method, we use the same dataset collected by a scripted policy, rather than let each method collect its own data, to avoid potential performance differences caused by data discrepancies. Specifically, we use a scripted policy to expose all potential local dependencies and collect 500K transitions in each environment.
Notice that, in exploration with sparse reward experiments, the dynamics models are still trained online, using the transition data collected on its own.
Hyperparameters The hyperparameters used for evaluating local dependency detection of each method are provided in Table 3. Unless specified otherwise, the parameters are shared across all environments.
D Evaluating Exploration in Sparse-Reward RL Tasks
D.1 Implementation
During policy learning, all methods share the same PPO and training hyperparameters, provided in Table 4. The hyperparameters for dynamics model setup during policy learning are the same as in Table 3 unless specified otherwise.
D.2 Success Rate Plots
As a supplementary to the normalized stage metric used in the main paper, we provide the success rate as an additional metric. The success rate learning curves of all methods in the three environments are shown in in Fig. 7. Again, ELDEN outperforms and performs comparably with all baselines. Notice that, in the CarWash and Kitchen environments, all baselines never succeed throughout the training (i.e., success rate  $= 0$  for all episodes), leading to training curves that overlap with the x axis.
Table 3: Parameters of the dynamics model training for local dependency detection experiments. Parameters shared if not specified.
Name	Tasks
Thawing	CarWash	Kitchen
environment	episode length	20	100	100
grid size	10	10	N/A
training	optimizer	Adam
learning rate	3 × 10-4
batch size	32
# of training batches	500k
# of random seeds	3
mixup Beta parameter	1	1	N/A
ELDEN	activation functions	ReLU
{MLP}N i=1	[64, 64]	[64, 64]	[128, 128]
λ annealing starts	50k	50k	100k
λ annealing ends	100k	100k	200k
# of heads	4
use bias	False
attention	key, query, value size	16	16	32
output size	64	64	128
post attn MLP	[64, 64]	[64, 64]	[128, 128]
Input	attention parameters	same as ELDEN
M regularization coefficient	1 × 10-2
Mask	M regularization annealing starts	50k	50k	100k
M regularization annealing ends	100k	100k	200k
Attn Mask	attention parameters	same as ELDEN
signature size	64
skPMD	learnable bandwidth	True
bandwidth initialization	1
Table 4: Parameters of the Policy Learning. Parameters shared if not specified.
Name	Tasks
Thawing	CarWash	Kitchen
PPO	optimizer	Adam
activation functions	Tanh
learning rate	\( 1 \times 10^{-4} \)
batch size	32
clip ratio	0.1
MLP size	[128, 128]
GAE λ	0.98
target steps	250
n steps	60	600	100
# of environments	20	20	80
training	# of random seeds	3
intrinsic reward coefficient β	1
# of dynamics update per policy step	1
dynamics learning rate	\( 1 \times 10^{-5} \)
ensemble size	5
level of sample prioritization	N/A	N/A	0.5
mixup Beta parameter	0.1	0.1	N/A
partial derivative threshold \( \epsilon \)	\( 3 \times 10^{-4} \)
(a) a navigation task with the goal in green
Figure 8: We demonstrate a failure mode of our method on a navigation task.
Figure 7: Learning curve of ELDEN (ours) compared to baseline approaches (mean  $\pm$  std dev of success rate across three random seeds). For CarWash, Craft, and Kitchen, the success rates for all baselines are zero throughout the training, overlapping with the x axis.
(b) ELDEN performs worse than Dynamics Curiosity
E Failure Modes of ELDEN
As mentioned in the main paper, ELDEN may have limited advantages for tasks that require precise control of a specific environment entity. One such example is navigation, where the agent needs to reach a very specific point in space that has no particular semantic meaning. We empirically examine this statement in the Minigrid environment [7], where the agent needs to navigate to the green goal point in an empty room through primitive actions (turn left, turn right, and move forward), as shown in Fig. 8(a). We compare ELDEN against Dynamics Curiosity and Vanilla PPO, and present the result in Fig. 8(b). Since this environment is relatively simple, all three methods are eventually able to solve the task. However, the Dynamics Curiosity converges faster than ELDEN, showing that ELDEN is indeed not as capable as curiosity-driven explorations in tasks that focus on precise control rather than exploring dependencies between environment entities. The Vanilla PPO converges slowest, indicating that even in the Empty environment, ELDEN still has advantages over purely random exploration.
F Compute Architecture
The experiments were conducted on machines of the following configurations:
- Nvidia 2080 Ti GPU; AMD Ryzen Threadripper 3970X 32-Core Processor
- Nvidia A40 GPU; Intel(R) Xeon(R) Gold 6342 CPU @2.80GHz
- Nvidia A100 GPU; Intel(R) Xeon(R) Gold 6342 CPU @2.80GHz