SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C.6 Important Hyperparameters
First, we introduce the structure of the networks used in our algorithm and the baselines as follows. We use  $s\_dim$ ,  $a\_dim$  to represent the dimension of the state space and action space respectively, and use  $c\_num$  to represent the number of options to learn at a time, which can be 10, 20, 40 or 60 in our experiments. Also, we use  $\tanh$  and  $\text{relu}$  to denote the hyperbolic tangent function and rectified linear unit used as the activation functions,  $FC(X,Y)$ ,  $BiLSTM(X,Y)$  to denote the fully-connected and bidirectional LSTM layer with the input size  $X$  and output size  $Y$ .
- The prior network  $P_{\omega}$  is used in all the algorithms other than DCO and its structure is [FC(s_dim, 64), tanh, FC(64, 64), tanh, FC(64, 64), tanh, FC(64, c_num)]. The value network corresponding to  $P_{\omega}$  has the same structure as  $P_{\omega}$ , except that the output is of size 1.
- The policy network  $\pi_{\theta}$  is used in all the algorithms, with the structure  $[FC(s\_dim + c\_num, 64), \tanh, FC(64, 64), \tanh, FC(64, 64), \tanh, FC(64, a\_dim), \tanh]$ . Its corresponding value network has the same structure except that the output is of size 1 and there is not  $tanh$  at the end.
- The decoder  $P_{\phi}$  used in ODPP and VALOR takes a sequence of states in the trajectory as input, so it uses bidirectional LSTM as part of the network, i.e., [BiLSTM(s_dim, 64), FC(2 * 64, c_num)]. While, the decoder of VIC and DIAYN takes one state as input rather than sequential data, so it uses the fully-connected layer instead, i.e., [FC(s_dim, 180), tanh, FC(180, 180), tanh, FC(180, 180), tanh, FC(180, c_num)].
- The option selector  $P_{\psi}$  has the same structure as  $P_{\omega}$  and is used in all the algorithms.
- The eigenfunction network introduced in Section C.3 is used in ODPP and DCO to estimate the Laplacian spectrum. Its structure is  $[FC(s\_dim, 256), relu, FC(256, 256), relu, FC(256, 256), relu, FC(256, 256), relu, FC(256, 30)]$ , where 30 denotes the dimension of the feature vector  $\vec{b_i}$  mentioned in Section 3.2.
As noted in the paper, the crucial hyperparameters are  $\beta$ ,  $\alpha_{1:3}$  in Eq. (6) and (10) which control the importance of each objective term, relating to diversity and coverage. Conducting a grid search on the set of parameters can be exhaustive. Therefore, we follow the process of the ablation study shown in Figure 2, add objective terms and adjust their corresponding weights one by one. In particular, in Figure 2(e), we retain only the  $L^{IB}$  objective and select its weight  $\beta = 10^{-3}$  from five possible choices:  $1, 10^{-1}, 10^{-2}, 10^{-3}, 10^{-4}$ , guided by the visualization results. Next, for Figure 2(f), we introduce  $\mathcal{L}_1^{DPP}$  and fine-tune the corresponding weight  $\alpha_1$  while keeping  $\beta$  fixed at  $10^{-3}$ . Last, we incorporate  $\mathcal{L}_2^{DPP}$  and  $\mathcal{L}_3^{DPP}$  and adjust  $\alpha_2$  and  $\alpha_3$  accordingly, while keeping  $\beta$  and  $\alpha_1$  fixed. Note that the final two terms must work in tandem to ensure that the discovered options exhibit diversity across different options and consistency for a specific option choice.
After fine-tuning, we set  $\beta = 10^{-3}$ ,  $\alpha_{1} = 10^{-4}$ ,  $\alpha_{2} = 10^{-2}$ ,  $\alpha_{3} = 10^{-2}$ . It is worthy noting that our evaluations across various challenging RL tasks utilize the same hyperparameter set, highlighting the robustness of our algorithm. This is because our proposed (DPP-based) coverage and diversity measures are task-agnostic and universally applicable to RL tasks.
D Additional Evaluation Results
D.1 Complementary Results on the Effect of the Prior Network
As discussed in Section 3, we learn a prior network  $P_{\omega}$  concurrently with the option policy network  $\pi_{\theta}$ . Figure 7 demonstrates how initializing the option selector  $P_{\psi}$  with  $P_{\omega}$  can lead to further performance improvement in the downstream task, using the Point Corridor goal-achieving task as an example. This is based on the fact that both networks share the same structure.
First, in (a), we sample 10,000 trajectories for each option and visualize the agent orientation distribution corresponding to different options. In (b), we present the coordinate system setup along with the four turning points for evaluation. The option choices at these turning points provided by the prior network are displayed in (c). It can be observed that  $P_{\omega}$  favors the most significant options at a given state. For instance, at Location #1, Option #5, which tends to go left or down as shown in (a), is preferred, while at Location #3, Option #3 is favored, guiding the agent to go up or right. Lastly, in (d), we demonstrate that initializing the option selector with the prior network can further enhance the agent's performance in the downstream goal-achieving task.
Previous variational methods choose to fix the prior distribution to avoid a collapse for the prior network to sample only a handful of skills. Our algorithm pretends that because our algorithm additionally introduces three DPP-based terms to explicitly maximize the coverage and diversity of the learned options. With  $\mathcal{L}_{1:3}^{DPP}$ , each option is expected to cover multiple state clusters with a long range and different options tend to visit different regions in the state space. In this case, the prior network would tend to select multiple diverse skills to improve its learning objective (i.e., Eq.
Figure 7: (a) Agent orientation distribution corresponding to different options. (b) Setup of the coordinate system and start points. (c) The output distribution of the prior network at different start points. (d) Performance improvement in the downstream task when applying the prior initialization. The trained prior gives preference to more useful options at corresponding states. At Location #1, Option #5, which tends to go left or down, is preferred; at Location #3, Option #3 is preferred which can lead the agent to go up or right. Moreover, Option #8 is preferred at Location #2 to lead the agent to go down, while Option #6 is preferred at Location #4 to lead the agent to go up.
(12)) rather than sampling only few of them. The collapse happens because the mutual information objective only implicitly measures option diversity as the difficulty of distinguishing them via the variational decoder and does not model the coverage, as noted in Section 3. This motivates us to introduce explicit diversity and coverage measures as regularizers and enhancement.
D.2 More Visualization of the Learned Ant Locomotion Behaviors
In Figure 8, we show more visualization results of the learned Ant locomotion behaviors without the supervision of task-specific rewards.
Figure 8: (a) The Ant agent learns to flip over first, then tries to flip back, and finally stands on its Leg 1. (b) The Ant agent walks to the right while rotating. It uses Leg 2&3 as the front legs at first and Leg 1&2 as the front leags at last.
D.3 Quantitative Ablation Study Results
Algorithm	Distance	Standard deviation in x	Standard deviation in y
ODPP (L1B)	9.600 ± 0.724	5.342 ± 0.521	9.239 ± 0.689
ODPP (L1B, L1DPP)	11.037 ± 0.627	7.067 ± 0.565	9.313 ± 0.731
ODPP (L1B, L1:3)	14.241 ± 0.653	7.493 ± 0.547	10.077 ± 0.799
Table 1: Numerical results on Mujoco Corridor. We compare our algorithm (the third row) with its ablated versions. ODPP  $(\mathcal{L}^{IB})$  represents a version using only the variational objective  $\mathcal{L}^{IB}$ , whereas ODPP  $(\mathcal{L}^{IB}, \mathcal{L}_1^{DPP})$  includes the additional coverage objective  $\mathcal{L}_1^{DPP}$ . For each algorithm, we collect ten different option trajectories and compute the mean distance (i.e.,  $\sqrt{x^2 + y^2}$ ) of the final states as the (single-option) coverage measure, and the standard deviation on  $x$  and  $y$  (of the final states) as the diversity measure. For goal-achieving tasks, these are reasonable metrics, as adopted in VALOR. The table displays the  $95\%$  confidence intervals for these metrics from ten repeated experiments. It can be observed that the performance in coverage and diversity improves with the introduction of DPP-based objectives, supporting our algorithm design.
Algorithm	Distance	Standard deviation in x	Standard deviation in y
ODPP (LIB)	11.737 ± 0.581	7.424 ± 0.450	9.637 ± 0.766
ODPP (L1IB, L1DPP)	13.008 ± 0.786	8.649 ± 0.887	9.768 ± 0.633
ODPP (L1IB, L1:3DPP)	15.349 ± 0.629	10.551 ± 0.558	11.356 ± 0.515
Table 2: Numerical results on Mujoco Room. These results are gathered in the same way as described above, and the same conclusion can be derived from this table.
D.4 Baseline Performance on OpenAI Gym and Atari Games
(c) AirRaid
Figure 9: Each algorithm is employed to learn skills for various tasks in an unsupervised manner. The  $95\%$  confidence intervals of the option trajectory rewards from repeated experiments with different random seeds are depicted. Comparisons between DADS (brown) and APS (purple) with other baselines illustrate their enhanced performance.