Question: Maybe you should include Termination Critic as a baseline in the large scale experiments?
Evidence: As presented in this section, we conducted two series of experiments to analyze the property of IMTC. First, we qualitatively evaluated the diversity of options learned by IMTC with intrinsic rewards, without any extrinsic rewards. Second, we quantitatively test the reusability of learned options by). Also, we can see that the magnitude of intra-option policies tends larger with constant rewards. Right: Options learned by other methods. OC produces a dead option 3 that terminates everywhere and never-ending options 0 and 1. About intra-option policies, all methods successfully avoided learning the same policy, but they only have two directions.
task adaptation on a specific task. As a baseline of termination learning method, we compared our method with OC. OC is trained with VIC rewards during pre-training. We did not compare IMTC with TC because our TC implementation failed to learn options with relatively small termination regions as reported in the paper, and there is no official public code for TC 1 . During pre-training without extrinsic rewards, IMTC receives intrinsic rewards when the current option terminates. We compare three IMTC variants with different intrinsic rewards: (i) VIC, (ii) RVIC, and (iii) constant value (R IMTC = 0.01). Note that R IMTC = 0.01 is chosen from [0.1, 0.05, 0.01] based on the task adaptation results. We also compare IMTC with vanilla VIC and RVIC with fixed termination probabilities. We used ∀ x β o (x) = 0.1 since it performed the best in task adaptation experiments, while 0.05 was used in. Note that RVIC's objective I(X s ; O|x f ) is different from ours, while IMTC and VIC share almost the same objective. Thus, the use of VIC is more natural, and the combination with RVIC is tested to show the applicability of IMTC. Further details of our VIC and RVIC implementation are found in Appendix B. In order to check only the effect of the different methods for learning beta β o , the rest of the implementation is the same for all these methods. That is, OC, vanilla VIC, and vanilla RVIC are also based on PPO and advantage estimation methods in Section 4.2. In this section, we fix the number of options as |O| = 4 for all option-learning methods. We further investigated the effect of the number of options Appendix C, where we confirmed that |O| = 4 is sufficient for most domains. All environments that we used for experiments are implemented on the MuJoCo physics simulator. We further describe the details in Appendix C.
Option Learning From Intrinsic Rewards We now qualitatively compare the options learned by IMTC with options of other methods. Learned options depend on the reward structure in the environment, which enables manually designing good reward functions for learning diverse options. Thus, we employed a reward-free RL setting where no reward is given to agents. Instead, each compared method uses some intrinsic rewards, as explained. We fix µ as µ(o|x) = 1 |O| in this experiment, since we assume that the future tasks are uniformly distributed. Intra-option policies are trained by PPO and independent GAE (8). We show network architectures and hyperparameters in Appendix C. We set the episode length to 1 × 10 4 , i.e., an agent is reset to its starting position after 1 × 10 4 steps. For all visualizations, we chose the best one from five independent runs with different random seeds.  We visualized learned options in PointReach environment shown in Figure. In this environment, an agent controls the ball initially placed at the center of the room. The state space consists of positions (x, y) and velocities (∆x, ∆y) of an agent, and the action space consists of acceralations ( ∆x ∆t , ∆y ∆t ). Figure shows the options learned in this environment after 4 × 10 6 steps. Each arrow represents the mean value of intra-option policies, and the heatmaps represent β o . In this experiment, we observed the effect of IMTC clearly, for both termination regions and intra-option policies. Interestingly, we don't see clear differences between options learned with VIC and RVIC rewards, while constant rewards tend to make options peaker. OC failed to learn meaningful termination regions: option 0 and 1 never terminate, and option 3 terminates almost everywhere. This result confirms that IMTC can certainly diversify options. Moreover, compared to vanilla VIC and RVIC, intra-option policies learned by IMTC with VIC or RVIC rewards are clearer, in terms of both the magnitude and directions of policies. We believe that this is because diversifying termination regions gives more biased samples to the option classifiers employed by VIC and RVIC. Figure Transferring skills via task adaptation Now we quantitatively test the reusability of learned options by task adaptation with specific reward functions. Specifically, we first trained agents with intrinsic rewards as per the previous section. Then we transferred agents to an environment with the same state and action space but with external rewards. We prepared multiple reward functions, which we call tasks, for each domain and evaluated the averaged performance over tasks. We compare IMTC with OC, vanilla VIC, vanilla RVIC, and PPO without pre-training. Also, we compare three variants of IMTC with different intrinsic rewards during pre-training. For a fair comparison, UGAE (9) and PPO are used for all options learning methods. Note that we found UGAE is very effective in this experiments, as we show the ablation study in Appendix C.6. For vanilla VIC and vanilla RVIC, termination probability is fixed to 0.1 through pre-training and task adaptation. -greedy based on Q O with = 0.1 is used as the option selection policy µ. We hypothesize that diverse options learned by IMTC can help quickly adapt to given tasks, supposing the diversity of tasks.
Figure shows all domains used for task adaptation experiments. For simplicity, all tasks have goalbased sparse reward functions. I.e., an agent receives R t = 1.0 when it satisfies a goal condition, and otherwise the control cost −0.0001 is given. Red circles show possible goal locations for each task. When the agent fails to reach the goal after 1000 steps, it is reset to a starting position. PointReach, SwimmerReach, and AntReach are simple navigation tasks where an agent aim to just navigate itself to the goal. We also prepared tasks with object manipulation: in PointBilliard and AntBilliard an agent aims to kick the blue ball to the goal position, and in PointPush and AntPush, it has to push the block out of the way to the goal. We pre-traine options learning agents for 4 × 10 6 environmental steps and additionally trained them for 1 × 10 6 steps for each task. Figure shows learning curves and scatter plots drawn from five independent runs with  different random seeds per domain. 2 Here, we observed that IMTC with VIC or RVIC rewards performed the best or was compatible with baselines. IMTC with VIC performed better than OC with VIC except for AntRearch, which backs up the effectiveness of diversifying termination regions for learning reusable options. Also, IMTC with VIC and IMTC with RCIC respectively performed better in most of the tasks than VIC and RVIC with fixed termination probabilities. This result suggests that IMTC can boost the performance of option learning methods based on option classifiers, even when the objective is different as with RVIC. On the other hand, IMTC with constant rewards (R IMTC = 0.01) performed worse than IMTC with VIC or RVIC rewards, although it also learned diverse options as we show in Figure, suggesting the importance of adjusting rewards. We further analyzed the evolution of intrinsic rewards of VIC and RVIC in Appendix C.5. In addition, we can observe that IMTC's performance is especially better than other methods in relatively complex PointBilliard, AntBilliard, and AntPush, where object manipulation is required. Considering that manipulated balls and boxes move faster than agents in these domains, a choice of options can lead to larger differences in the future state. IMTC is suitable to these domains since it maximizes the diversity of the resulting states, while PPO struggles to learn. Contrary, IMTC's performance is close to other methods in Reach tasks, where the goal states are relatively close to the starting states in terms of euclidian distances.
Gridworld experiments and limitation of the method Although IMTC successfully learned diverse options in MuJoCo experiments, our analysis in Figure shows the possibility of learning options that are not interesting but have large MI. We further investigated this possibility by visualizing options in a classical four rooms gridworld in Appendix C.8. Interestingly, we observed that IMTC could fall into diverse but unmeaningful options in that environment. We believe that IMTC is often sufficient in a large environment where a randomly-initialized agent rarely gets the same trajectory. However, when the number of possible trajectories is small, diversifying the destinations could be insufficient. In such cases, it can be necessary to extend IMTC to diversify trajectories as done in.
First, we qualitatively evaluated the diversity of options learned by IMTC with intrinsic rewards, without any extrinsic rewards.
Answer: Unfortunately, as we shortly mentioned at the beginning of section 5, we failed to reproduce options with small entropy by our implementation of termination critic. A major difficulty we had is in training the option transition model Po(xf|xs) and the marginalized Poμ(xf|xs). More precisely, in our implementation, Po produced close values for each option, leading to similar termination regions per option. This failure motivated our design decision of approximating pO(o|xs,xf) instead of Po. We will clarify this point in the revised manuscript.