SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In our graph representation  $G_{\mathcal{B}}$  of a Büchi automaton  $\mathcal{B}$ , a "self-transition" on a node  $q$  describes the goal-related constraint  $\psi'$  that must be maintained until the agent can transition to a neighbor node of  $q$  with the underlying goal-conditioned policy  $\pi$ . If  $q$  is on the planned optimal path, the agent for  $B$  needs to use  $\pi$  to simultaneously ensure  $\psi'$  while transitioning to the target region. In the paper, we
primarily focus on reach avoidance, where  $\psi' = \bigwedge_{k} \neg g_{k}$  encodes regions in the goal space to avoid. At a current environment state  $s$ , when the value function  $V^{\pi}(s, g_{k})$  is greater than a threshold, we take a safe action  $\arg \min_{a} Q^{\pi}(s, g_{k}, a)$  that moves the agent away from the most dangerous zone  $g_{k}$ . Similarly, our strategy applies to cases where  $\psi' = \bigwedge_{k} g_{k}$  encodes goal regions that the agent must stay within before transitioning out from  $q$ . In such cases, if the value function  $V^{\pi}(s, g_{k})$  is less than a threshold, we could take an action  $\arg \max_{a} Q^{\pi}(s, g_{k}, a)$  that encourages the agent to remain in  $g_{k}$ . As our navigation benchmarks do not support the evaluation of this feature, we plan to explore it in our future work.
In the paper, we restrict the target region for a transition as  $\psi = \bigwedge_{j}g_{j}$ . Our tool supports more diverse target regions in the form of  $\psi = \bigwedge_{j}g_{j}\wedge \bigwedge_{l}\neg g_{l}$  to handle tasks such as  $\mathbf{F}g_1\wedge \neg g_2$ . The strategy is to reuse our avoidance strategy to avoid  $g_{2}$  when the agent is deemed close to the goal region  $g_{1}$  i.e. the value function output  $V^{\pi}(s,g_1)$  is above the threshold  $\sigma$ . Fig. 17 presents an example to solve tasks in the form of  $\mathbf{F}g_{1}\wedge \neg g_{2}$ .
In the handling avoidance in ZoneEnv and LetterEnv, finding appropriate value thresholds to define dangerous is quite necessary. As we discussed in Sec. G.2, we use sparse reward functions that only yield 0 or 1 in the training process, so the expected threshold to be deemed
as dangerous should be located somewhere close to (but less than) 1. We perform experiments to study success rates given different value thresholds for both environments. The results are summarized in Table 3. From the table, we can see that a relatively smaller  $\sigma$  works better with ZoneEnv, while a larger value threshold tends to be a suitable choice for LetterEnv. According to the results, we set the  $\sigma = 0.85$  for ZoneEnv, and  $\sigma = 0.92$  for LetterEnv.
(a)  $\mathbf{F}(j\wedge r)$
Figure 17: Fig (a) and (b) show our capability to handle overlapping goal regions. In (b), the agent takes a detour to avoid touching the red zone.
(b)  $\mathbf{F}(j\wedge \neg r)$
Environment	Value threshold σ
σ=0.80	σ=0.85	σ=0.87	σ=0.90	σ=0.92	σ=0.95
ZoneEnv	0.912	0.958	0.941	0.932	0.902	0.878
LetterEnv	0.635	0.788	0.812	0.804	0.845	0.820
Table 3: Performance of trained agents on avoidance tasks with different value thresholds  $\sigma$ . We report the success rates averaged over 1000 episodes.
G.4 Hyperparameters
The following hyperparameters are used to train the dynamic primitive policies for ZoneEnv with PPO (Schulman et al. [2017]).
- Discount factor  $\gamma = 0.998$ .
- SGD optimizer; actor learning rate 0.001; critic learning rate 0.001.
- Mini-batch size  $n = 256$ .
The following hyperparameters are used to train the dynamic primitive policies for LetterEnv with PPO.
Discount factor  $\gamma = 0.94$
- SGD optimizer; actor learning rate 0.001; critic learning rate 0.001.
- Mini-batch size  $n = 256$ .
The following hyperparameters are used to train the goal-conditioned agent for ZoneEnv with the goal-conditioned PPO algorithm shown in Algorithm 1.
- Discount factor  $\gamma = 0.998$ .
- SGD optimizer; actor learning rate 0.0003; critic learning rate 0.0003.
- Mini-batch size  $n = 1000$ .
The following hyperparameters are used to train the goal-conditioned agent for LetterEnv with the goal-conditioned PPO algorithm shown in Algorithm 1.
Discount factor  $\gamma = 0.94$
- Adam optimizer; actor learning rate 0.001; critic learning rate 0.001.
- Mini-batch size  $n = 256$ .
The following hyperparameters are used to train the dynamic primitive policies for Ant 16 rooms with the SAC (Haarnoja et al. [2018]) algorithm.
Discount factor  $\gamma = 0.99$
- SGD optimizer; actor learning rate 0.001; critic learning rate 0.001.
- Mini-batch size  $n = 100$ .
Replay buffer size 100000.
- Soft update targets  $\tau = 0.005$ .
- Target update interval and gradient step are set to be 1.
The following hyperparameters are used to train the goal value function  $\mathcal{V}$  and policy  $\pi$  with our extended goal-conditioned iterative supervised learning algorithm shown in Algorithm 2.
- For goal value function  $\mathcal{V}$ :
Discount factor  $\gamma = 0.99$
- Adam optimizer; learning rate 0.001.
- Mini-batch size  $n = 100$ .
- For policy  $\pi$ :
- Adam optimizer; learning rate 0.0005.
- Replay buffer size 20000000.
- Mini-batch size  $n = 512$ .
For all training with PPO algorithm, we use GAE  $\lambda = 0.97$ , clip range  $\epsilon = 0.2$ , and we set the number of iterations when optimizing the surrogate loss to be 10.
Table 4 summarizes the neural network architectures used by our reinforcement learning algorithms. All neural networks are multi-layer feed-forward neural networks.
Policies	Environment	Hidden Units
PPO dynamic	ZoenEnv	(64, 64)
PPO dynamic	LetterEnv	(64, 64)
PPO goal-reaching	ZoneEnv	(512, 1024, 256)
PPO goal-reaching	LetterEnv	(256, 512, 128)
SAC dynamic	Ant	(256, 256)
Goal-conditioned value	Ant16rooms	(256, 256,256)
GCSL	Ant16rooms	(256, 256,256)
Table 4: Neural Network Structures.