Question: On eq. 9, what is the assumption on the reward function to take the maximum over 0 as optimistic?
Evidence: The original PPO implementation employs GAE for estimating the advantage, which is important for learning performance. Therefore, we employed two variants of GAE in experiments for option learning according to the experimental setup. In the following paragraphs, we let N denote the rollout length used for advantage estimation and let t + k denote the time step at which the current option o t terminates. Thus, we need to consider the effect of option-switching in advantage estimation When k < N . Furthermore, we use two variants of the option-specific TD errors δ
Independent GAE for Reward-Free RL For no-reward experiments with VIC, we used the following variant of GAE:
Here, we ignore the future rewards produced by other options after the current option o t terminates. This formulation enhances learning diverse intra-option policies per option.
Upgoing GAE for task adaptation For single-task learning, increasing the rollout step N often speeds up learning. However, future rewards after option termination heavily depend on the selected option and have high variance, especially when learning diverse options. This high variance of future rewards slows advantage learning and causes underestimation of Âo . Thus, to prevent underestimation, we introduce an upgoing GAE (UGAE) for estimating advantage with options:
Like the upgoing policy update, the idea is optimistic regarding future rewards after option termination by taking the maximum over 0. We use Âo upg for task adaptation experiments in Section 5, and confirmed its effectivity in the ablation study in Appendix C.6.
The original PPO implementation employs GAE for estimating the advantage, which is important for learning performance. Therefore, we employed two variants of GAE in experiments for option learning according to the experimental setup. In the following paragraphs, we let N denote the rollout length used for advantage estimation and let t + k denote the time step at which the current option o t terminates. Thus, we need to consider the effect of option-switching in advantage estimation When k < N . Furthermore, we use two variants of the option-specific TD errors δ
Independent GAE for Reward-Free RL For no-reward experiments with VIC, we used the following variant of GAE:
Here, we ignore the future rewards produced by other options after the current option o t terminates. This formulation enhances learning diverse intra-option policies per option.
Upgoing GAE for task adaptation For single-task learning, increasing the rollout step N often speeds up learning. However, future rewards after option termination heavily depend on the selected option and have high variance, especially when learning diverse options. This high variance of future rewards slows advantage learning and causes underestimation of Âo . Thus, to prevent underestimation, we introduce an upgoing GAE (UGAE) for estimating advantage with options:
Like the upgoing policy update, the idea is optimistic regarding future rewards after option termination by taking the maximum over 0. We use Âo upg for task adaptation experiments in Section 5, and confirmed its effectivity in the ablation study in Appendix C.6.
Thus, to prevent underestimation, we introduce an upgoing GAE (UGAE) for estimating advantage with options****Like the upgoing policy update (Vinyals et al., 2019), the idea is optimistic regarding future rewards after option termination by taking the maximum over 0.
Answer: There, we take the maximum of 0 and an estimated advantage. Since advantages should converge to 0 in theory, we called taking the maximum over 0 optimistic.