SlowGuess/NeurIPS_2023_OCR huggingface dataset:
configurations (solutions to the inverse kinematics problem). Explicitly, this relation is governed by the manipulator's forward kinematics  $f: \mathcal{C} \to \mathbb{S}\mathbb{E}(3)$  which maps (typically) 7-DoF configurations of the arm  $q \in \mathcal{C}$  to poses in Cartesian space. In practice, only a subset of these configurations for a given grasp pose enable successful manipulation [Schiavi et al., 2022]. As a result, we task the initiation set learning algorithm with choosing start states directly from the space of arm configurations  $\mathcal{C}$ .
We generate collision-free grasp poses on each object using off-the-shelf grasp generation method GPG [Ten Pas et al., 2017] and corresponding arm poses using IKFLOW[Ames et al., 2022]. We chose to generate 50 grasp poses with 5 random inverse kinematics solutions each yielding a total of 250 starting configurations for each task.
Reward function. The reward functions are implemented as progress toward 1-DoF object joint position goals. The agent receives reward when the current joint position exceeds its previous maximum in a given episode.
Parameterization. As described in Section A, the agent receives proprioceptive and object-state observations and controls the manipulator's end-effector pose and impedance. The learning algorithm employed is TD3 [Fujimoto et al., 2018]. Goal-conditioning is omitted in these experiments as they have a single goal and a single option.
C.3 Deep Skill Chaining
Deep skill chaining (DSC) [Konidaris and Barto, 2009, Bagaria and Konidaris, 2020] proceeds recursively backward from the goal: the termination region of the first option is the task goal (Line 2, Algorithm 3); the agent first learns an option policy and initiation set for that option. Then, it learns another option whose termination region is the initiation set of the first option; this process continues until there is some option whose initiation set covers the start state. The result is a collection of options that can be sequenced to reliably reach the goal.
Since the initiation sets of options are the subgoals for other options, the entire process is sensitive to the way in which the initiation sets are learned: poorly estimated initiation sets can lead to subgoals that do not improve the agent's ability to reach the task goal.
Details about line 12 of Algorithm 2 differ based on the method used to learn the initiation set. When using the pure IVF approach, we perform as many minibatch gradient updates as the length of the option rollout; when using weighted classification, we recompute weights using Eq 1 for all training examples and then proceed to minimize weighted cross-entropy loss (3 epochs, batch size 128). When using classification (weighted or unweighted), we boost the contribution of the minority class by the ratio of the size of the majority class to that of the minority class.
Algorithm 2 Robust DSC Rollout
Inputs. Skill Chain  $\mathcal{O}$    
Hyperparameters. Option horizon  $H$    
1: Initialize empty trajectory buffer  $\mathcal{B}$    
2: for each timestep t do   
3: Select option o using policy over options  $\pi_{\mathcal{O}}(s_t)$    
4: Sample a goal for selected option:  $g\sim \beta_o$    
5: Execute option policy  $\pi_o(\cdot |g)$  in the environment   
6: Add trajectory  $\tau = \bigcup_{i = 0}^{H - 1}(s_i,o,a_i,s_{i + 1},g)$  to  $\mathcal{B}$    
7: if final state  $s_H$  reached goal g then   
8: Add  $\tau$  to o's list of positive examples   
9: else   
10: Add  $\tau$  to o's list of negative examples   
11: end if   
12: Refit option o's initiation classifier   
13: Add  $\tau$  to replay buffer and update  $\pi_o$  using TD3   
14: end for   
15: return  $\mathcal{B} = \bigcup_{t}(s_{t},o_{t},a_{t},s_{t + 1},g_{t})$
Algorithm 3 Robust DSC Algorithm
Inputs. Start state  $s_0$ , Goal region  $g$ .  
1: Initialize global option  $o_G$  such that  $\mathcal{I}_{o_G}(\cdot) = 1$   
2: Initialize goal option  $o_g$  such that  $\beta_{o_g} = g$   
3: Initialize skill chain  $\mathcal{O}$  with  $\{o_g\}$   
4: for each episode do  
5: transitions = ROLLOUT(\mathcal{O})  
6: if  $s_0 \notin \mathcal{I}_o, \forall o \in \mathcal{O}$  then  
7: Create new option  $\omega$   
8: Add  $\omega$  to skill chain  $\mathcal{O}$   
9: end if  
10: end for
Picking a goal for option execution. Line 4 of Algorithm 2 samples a goal from the option's termination region. To implement this sampling procedure, we consider the option's parent  $\omega$  in the chain (the parent option is the one whose initiation set is being targeted by the current option  $o$ ). We enumerate the positive examples used to train  $\mathcal{I}_{\omega}$  and pick the goal with the highest initiation probability under the current option. This process is done iteratively backward from the goal: the first
goal is the task goal, the next one is the positive example closest (in terms of highest IVF value) to the task goal and so on.
D Additional Manipulation Experiments
Initiation set accuracy and true size are computed during training by performing an analogous procedure to Algorithm 1. Periodically, the manipulator was reset to each candidate start state and the initiation prediction was compared with the outcome of a policy rollout. Initiation set accuracy is visualized in Figure 8a. The methods generally converge to similar accuracy. True initiation set size is plotted in Figure 8b; size increases with optimism and correlates with success rates.
(a) Initiation set accuracy for manipulation domains.
(b) Initiation set size for manipulation domains (out of 250 start states).
Figure 8: (a) Accuracy of the learned initiation sets in the robot manipulation domains. (b) The size of the "true" initiation sets measured by performing Monte Carlo rollouts of the option policy.
E Ablation: Optimism
We ablate role of optimism in initiation set learning in the manipulation experiments of Section 4.2; the results are visualized in Figure 10. Count-based bonuses are computed as described in Section 3.3. The optimistic bias is critical for the success of the Weighted algorithm in the SLIDE task, in which only a small fraction of potential grasps are amenable to task success. For the Binary classification baseline, optimism dramatically deteriorates performance in the DOOR task, but yields a large improvement in the LEVER domain.
Figure 9 shows that the optimistic bonus slightly hurts in FOURROOMS, but is beneficial in MONTEZUMA'S REVENGE. This is likely because FourRooms is a much simpler problem with a smaller state-space, thereby not demanding an exploration bonus.
Figure 9: Optimism ablation in MINIGRID-FOURROOMS and MONTEZUMASREVENGE. All curves are averaged over all state-option pairs and 5 random seeds.
Figure 10: Optimism ablation in ROBOSUITE. Solid line represents mean success rate aggregated over four seeds; shaded region shows standard error.
F Initiation Sets In Four Rooms
Figure 11 shows the initiation set and IVF of the option targeting the center of the top-right room in MINIGRID-FOURROOMS. These figures show that the initiation set of the option expands over time to reflect the growing competence of the option policy.
Rainbow was used for policy learning in Section 4.1, TD3 was used in the other experiments. Their hyperparameters (Tables 2 and 5) were not tuned and are either identical to the original paper implementation or borrowed from Bagaria et al. [2021a]. The bonus scale  $c$  (described in Sec 3.3)
Figure 11: Initiation set (left) and Initiation Value Function (IVF) (right) of the option targeting the center of the top-right room (subgoal position is (14, 14)). Plots in the top row visualize the initiation set after the 1st episode of training, those in the bottom row are after 200 episodes of training.
Env	Horizon H0
FourRooms	50
Montezumas Revenge	200
Robosuite	250
Ant-Maze	200
Table 1: Option Horizon  ${H}_{o}$  for each environment.
was tuned over the set  $\{0.05, 0.1, 0.25, 0.5, 1.0\}$ , the best performing hyperparameters are listed in Table 3.
G.1 Option Horizon
Options operate over a fixed time horizon  $H_{o}$  (see Section 3.1). These values (Table 1) were picked based on overall time limit (max_steps per episode) for each domain and we did not tune them.
Parameter	Value
Replay buffer size	10^6
Critic Learning rate	3 · 10^-4
Actor Learning rate	3 · 10^-4
Optimizer	Adam
Target Update Rate	5 · 10^-3
Batch size	256
Iterations per time step	1
Discount Factor	0.99
Output Normalization	False
Policy Improvement Prior	False in ROBOSUIT; True otherwise
Table 2: TD3 Hyperparameters for ROBOSuite and DSC Experiments
Method	Bonus Scale
Optimistic Binary	0.1
Optimistic GVF	0.5
Optimistic Weighted	0.5
Table 3: Exploration Hyperparameters for Robosuite Experiments.
Parameter	Value
Replay buffer size	3·105
Replay start size	1024
Learning rate	10-4
Table 4: Rainbow Hyperparameters for Accuracy Experiments
Parameter	Value
Learning rate	10-4
Optimizer	Adam
Replay buffer size	105
Batch size	32
Threshold	0.5
Target network update rate	5 · 10-3
Table 5: IVF Hyperparameters