SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We present a series of evaluations of WT across tasks involving reward and goal-conditioning, with comparisons to prior offline RL methods. For this, we leverage D4RL, an open-source benchmark for offline RL, consisting of varying datasets for tasks from Gym-MuJoCo, AntMaze, and FrankaKitchen [Fu et al., 2020].
Tasks in the AntMaze and FrankaKitchen environments have presented a challenge for offline RL methods as they contain little to no optimal trajectories and perform critical evaluations of a model's stitching ability [Fu et al., 2020]. Specifically, in FrankaKitchen, the aim is to interact with a set of kitchen items to reach a target configuration, but the partial and mixed offline datasets consist of suboptimal, undirected data, where the demonstrations are unrelated to the target configuration. Similarly, AntMaze is a maze navigation environment with sparse rewards, where the play and diverse datasets contain target locations unaligned with the evaluation task. For our experiments on these environments, we use goal-conditioning on the target goal state (i.e.,  $\omega = s_{\mathrm{target}}$ ), constructing intermediate targets with the goal waypoint network.
Gym-MuJoCo serves as a popular benchmark for prior work in offline RL, consisting of environments such as Walker2D, HalfCheetah, and Hopper. Importantly, we evaluate across offline datasets with varying degrees of optimality by considering the medium, medium-replay, and medium-expert datasets [Fu et al., 2020]. For these tasks, we use reward-conditioning given a target return, constructing intermediate reward targets using the reward waypoint network; as noted in Emmons et al. [2021], we find that the notion of goals in undirected locomotion tasks is ill-defined.
Across all environments and tasks, we use the same set of hyperparameters, as reported in Appendix B. To measure the stability (i.e., variability) in our method across random initializations, we run each experiment across 5 seeds and report the mean and standard deviation.
6.1 Comparing WT with Prior Methods
To evaluate the performance of WT, we perform comparisons with prior offline RL methods, including conditional BC methods such as DT and RvS-R/G; value-based methods such as Onestep RL
[Brandfonbrener et al., 2021], TD3 + BC [Fujimoto and Gu, 2021], CQL, and IQL; and standard BC baselines. For all methods except DT, we use reported results from Emmons et al. [2021] and Kostrikov et al. [2021]. We evaluate DT using the official implementation provided by Chen et al. [2021] across 5 random initializations, though we are unable to reproduce some of their results.
Table 1 shows the results of our comparisons to prior methods. Aggregated across all tasks, WT (71.4 ± 2.8) improves upon the next best method, IQL (68.3 ± 6.9), with respect to average normalized score and achieves equal-best runtime. In terms of variability across seeds, there is a notable reduction compared to IQL and most other methods.
In the most challenging tasks requiring stitching, our method demonstrates performance far exceeding the next best method, IQL. On the AntMaze Large datasets, WT demonstrates a substantial relative percentage improvement of  $83.1\%$  (play) and  $51.6\%$  (diverse). On Kitchen Partial and Mixed, the improvement is  $37.8\%$  and  $39.0\%$  respectively. WT's standard deviation across seeds is reduced by a factor of more than 2x compared to IQL for these tasks.
Similarly, on reward-conditioned tasks with large performance gaps between BC and value-based methods such as hopper-medium-replay-v2, WT demonstrates increased average performance by  $105.3\%$  compared to DT and  $21.0\%$  compared to RvS-R, with standard deviation reduced by a factor of 10.0x and 5.3x respectively.
6.2 Utility of Waypoint Networks
To analyze the utility and behavior of waypoint networks, we qualitatively evaluate an agent's performance across rollouts of trained transformer policies on antmaze-large-play-v2. For this analysis, we consider a WT policy (using a goal waypoint network with  $K = 30$ ) and a global goal-conditioned transformer policy (i.e., no intermediate goals). Across both models, the architecture and hyperparameters for training are identical.
The ant's locations across 100 rollouts of a WT policy (Figure 4a) and a global goal-conditioned transformer policy (Figure 4b) demonstrate that WT shows notably higher ability and consistency in reaching the goal location. Specifically, without intermediate goals, the ant occasionally turns in the wrong direction and demonstrates a lesser ability to successfully complete a turn based on the reduction of density at each turn (Figure 4b). Consequently, the
WT achieves more than twice the evaluation return  $(72.5 \pm 2.8)$  compared to the global goal-conditioned policy  $(33.0 \pm 10.3)$  and completes the task more quickly on average (Figure 4d).
Based on Figure 4c, we observe that the goal waypoint network provides goals that correspond to the paths traversed in Figure 4a for the WT policy. This shows that the waypoint network successfully guides the model toward the target location, addressing the stitching problem proposed in Figure 2. While the global goal-conditioned policy is successful in passing beyond the stitching region into the target region in only  $45\%$  of the rollouts, accounting for  $82\%$  of its failures to reach the target, WT is successful in this respect for  $87\%$  of rollouts.
Figure 4: Shows the ant's location across 100 rollouts of (a) a WT policy and (b) a global goal-conditioned transformer policy; (c) generated intermediate goals by the waypoint network  $W_{\phi}$ , (d) the proportion of all successful runs completed by timestep  $t$ .
Goal-Conditioned Tasks On goal-conditioned tasks, we examine the behavior of the goal waypoint network as it relates to the performance of the policy at test time by ablating aspects of its configuration
and training. For this analysis, we consider antmaze-large-play-v2, a challenging task that critically evaluates the stitching capability of offline RL techniques.
To understand the effect of the configuration of the goal waypoint network on test performance, we ablate two variables relevant to generating effective intermediate goals: the temporal proximity of intermediate goals  $(K)$  and the validation loss of the goal waypoint network.
The normalized score attained by the agent is shown as a function of  $K$  and the validation loss of the goal waypoint network in Figure 5. For this environment and dataset, an ideal choice for  $K$  is around 30 timesteps. For all nonzero  $K$ , the per
Figure 5: Normalized score attained by WT on antmaze-large-play-v2 based on varying left: the temporal proximity of generated goals,  $K$ , and right: goal waypoint network RMSE on a held-out dataset.
formance is reduced at a reasonably consistent rate on either side of  $K = 30$ . Importantly, when  $K = 0$  (i.e., no intermediate goals), there is a notable reduction in performance compared to all other choices of  $K$ ; compared to the optimal  $K = 30$ , the score is reduced by a factor of  $2.2\mathrm{x}$ .
In Figure 5 (right), the normalized score shows the negligible change for values of held-out RMSE between 0.4 and 0.6, corresponding to at least 1,000 gradient steps or roughly 30 sec of training, with a sharper decrease henceforth. As the RMSE increases to over 1, we observe a relative plateau in performance near an average normalized score of 35-45, roughly corresponding to performance without using a waypoint network (i.e.,  $K = 0$  in Figure 5 (left)).
Additionally, we perform comparisons between the goal waypoint network and manually constructed waypoints as intermediate targets for WT, for which the methodology and results are shown in Appendix D. Based on that analysis, we show that manual waypoints statistically significantly improve upon no waypoints  $(44.5 \pm 2.8$  vs.  $33.0 \pm 10.3)$ , but they remain significantly worse than generated waypoints.
Reward-Conditioned Tasks On reward-conditioned tasks, we ablate the choice of different reward-conditioning techniques. Specifically, we examine the performance of WT and variance of the reward waypoint network in comparison to CRTG updated using the rewards obtained during rollouts and a static ARTG (i.e., as done in Chen et al. [2021] and Emmons et al. [2021]). We consider the hopper-medium-replay-v2 task for this analysis as there is (a) a large performance gap between RvS and value-based methods, and (b) high instability across seeds for RvS methods (e.g., DT) as shown in Table 1. For all examined reward-conditioned techniques, the transformer architecture
and training procedure are identical, and the target (normalized) return is 95, corresponding to SOTA performance.
Figure 6: Comparison of different reward-conditioning methods on hopper-medium-replay-v2. Left: Performance profiles for transformers using ARTG, CRTG, and WT across 5 random seeds. Right: Standard deviation in CRTG inputted to the model when updated with attained rewards (CRTG  $t = \omega - \sum_{t} \gamma^{t} r_{t}$ ) and using predictions from the reward waypoint network ( $W_{\phi}(s_{t}, \omega)_{2}$ ) when average return is approximately held constant.
To examine the distribution of normalized scores across different seeds produced by each of the described reward-conditioning techniques, we construct performance profiles, displaying the proportion of runs greater than a certain normalized score [Agarwal et al., 2021]. As shown in Figure 6 (left), WT demonstrates increased performance and stability across random initializations compared to the remaining reward-conditioning techniques.
Additionally, we perform an analysis to determine whether using a reward waypoint network to predict the CRTG as opposed to updating the CRTG using attained rewards as in Chen et al. [2021] affects the variability of the conditioning variable passed to the policy network (i.e., not of the
performance as that is examined in Figure 6). Importantly, to account for performance differences between the policies trained with either method that may influence the variability of the attained CRTG, we sample a subset of runs for both methods such that the average performance is constant. Based on Figure 6 (right), it is clear that as a function of the timestep, when accounting for difference in average performance, the standard deviation in the CRTG predicted by WT grows at a slower rate compared to updating CRTG with attained rewards.
Transformer Configuration Based on the work in Emmons et al. [2021], we balance between expressiveness and regularization to maximize policy performance. We ablate the probability of node dropout  $p_{\mathrm{drop}}$  and the number of transformer layers  $L$ . To further examine this balance, we experiment with conditioning on past actions  $a_{t - k..t - 1}$ , similarly to the DT, to characterize its impact on performance and computational efficiency. In this section, we consider antmaze-large-play-v2, hopper-medium-replay-v2 and kitchen-mixed-v0, one task from each category of environments.
Based on Table 2, we observe that the sensitivity to the various ablated hyperparameters is relatively low in terms of performance, and removing action conditioning results in reduced training time and increased performance, perhaps due to reduced distribution shift at evaluation. In context of prior RvS work where dropout  $(p_{\mathrm{drop}} = 0.1)$  decreased performance compared to no dropout by  $1.5 - 3x$  on AntMaze, the largest decrease in average performance on WT is only by a factor of  $1.1x$  [Emmons et al., 2021].
Table 2: Ablation of transformer configuration showing normalized score on MuJoCo (v2), AntMaze (v2) and Kitchen (v0), including dropout ( $p_{\mathrm{drop}}$ ), transformer layers ( $L$ ), and action conditioning ( $a_t$ ), where bolded hyperparameters (e.g., 0.150) are used for final models and bolded scores are optimal.
\(p_{drop}\)	hopper-medium-replay	antmaze-large-play	kitchen-mixed	Average
0.000	75.5 ± 8.3	68.3 ± 5.9	72.9 ± 0.5	72.2 ± 4.9
0.075	89.8 ± 2.8	70.8 ± 4.5	71.8 ± 1.2	77.5 ± 2.8
0.150	88.9 ± 2.4	72.5 ± 2.8	70.9 ± 2.1	77.4 ± 2.4
0.225	75.7 ± 9.4	72.2 ± 2.7	71.2 ± 1.0	73.0 ± 4.4
0.300	74.7 ± 10.2	73.5 ± 2.5	69.2 ± 2.0	72.5 ± 4.9
0.600	58.4 ± 7.5	73.8 ± 5.2	66.5 ± 2.7	66.2 ± 5.1
L	hopper-medium-replay	antmaze-large-play	kitchen-mixed	Average
1	82.1 ± 8.8	72.1 ± 5.7	71.6 ± 1.6	75.3 ± 5.4
2	88.9 ± 2.4	72.5 ± 2.8	70.9 ± 2.1	77.4 ± 2.4
3	89.9 ± 1.6	71.8 ± 3.0	70.3 ± 2.1	77.3 ± 2.2
4	91.1 ± 2.8	65.8 ± 3.8	69.7 ± 1.0	75.5 ± 2.5
5	88.8 ± 4.5	66.7 ± 4.7	70.0 ± 0.8	75.2 ± 3.3
\(a_t\)	hopper-medium-replay	antmaze-large-play	kitchen-mixed	Average
Yes	76.9 ± 9.0	66.5 ± 5.6	65.2 ± 2.8	69.5 ± 5.8
No	88.9 ± 2.4	72.5 ± 2.8	70.9 ± 2.1	77.4 ± 2.4
In this study, we address the issues with existing conditioning techniques used in RvS, such as the "stitching" problem associated with global goals and the high bias and variance of reward-to-go targets, through the automatic generation of intermediate targets. Based on empirical evaluations, we demonstrate significantly improved performance and stability compared to existing RvS methods, often on par with or outperforming TD learning methods. Especially on challenging tasks with suboptimal dataset composition, such as AntMaze Large and Kitchen Partial/Mixed, the guidance provided by the waypoint network through intermediate targets (e.g., as shown in Figure 4) significantly improves upon existing state-of-the-art performance.
We believe that this work can present a pathway forward to developing practical offline RL methods leveraging the simplicity of RvS and exploring more effective conditioning techniques, as formalized by Emmons et al. [2021]. In addition to state-of-the-art performance, we demonstrate several desirable practical qualities of the WT: it is less sensitive to changes in hyperparameters, significantly faster to train than prior RvS work, and more consistent across initialization seeds.
However, despite improvements across challenging tasks, WT's margin of improvement on AntMaze U-Maze and Kitchen Complete (i.e., easier tasks) is lower: its normalized scores are more comparable to DT and other RvS methods. We believe this is likely due to stitching being less necessary in such tasks compared to difficult tasks, rendering the impact of the waypoint network negligible. To further characterize the performance of the waypoint networks and WT on such tasks is an interesting direction for future work. In addition, there are several limitations inherited by the usage of the RvS framework, such as manual tuning of the target return at test time for reward-conditioned tasks using a grid search, issues with stochasticity, and an inability to learn from data with multimodal outcomes.
We propose a method for reinforcement learning via supervised learning, Waypoint Transformer, conditioned on generated intermediate targets for reward and goal-conditioned tasks. We show that RvS with waypoints significantly surpasses existing RvS methods and achieves on par with or surpasses popular state-of-the-art methods across a wide range of tasks from Gym-MuJoCo, AntMaze, and Kitchen. With improved stability across runs and competitive computational efficiency, we believe that our method advances the performance and applicability of RvS within the context of offline RL.
This work was supported in part by NSF grant #2112926.
We thank Scott Emmons and Ilya Kostrikov for their discussions on and contributions to providing results for prior offline RL methods.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. arXiv preprint arXiv:2008.05556, 2020.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems, 34:4933-4946, 2021.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? arXiv preprint arXiv:2206.01079, 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pages 4055-4064. PMLR, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204.12581, 2022.
Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards--just map them to actions. arXiv preprint arXiv:1912.02875, 2019.
Jian Shen, Mingcheng Chen, Zhicheng Zhang, Zhengyu Yang, Weinan Zhang, and Yong Yu. Model-based offline policy optimization with distribution correcting regularization. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part I 21, pages 174-189. Springer, 2021.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaskowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02877, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. *Robotica*, 17(2): 229-235, 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Conference on Machine Learning, pages 24631-24645. PMLR, 2022.
Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. arXiv preprint arXiv:2209.03993, 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. arXiv preprint arXiv:2105.07351, 2021.
Siyuan Zhang and Nan Jiang. Towards hyperparameter-free policy selection for offline reinforcement learning. Advances in Neural Information Processing Systems, 34:12864-12875, 2021.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In International Conference on Machine Learning, pages 27042-27059. PMLR, 2022.
A Derivation for Illustrative Example
We provide detailed derivations based on the simple deterministic MDP shown in Section 4.1, in the context of an offline dataset  $\mathcal{D}$  collected by a random behavioural policy  $\pi_{b}$ . We show that minimization of a maximum likelihood objective on  $\pi_{G}$  yields  $\pi_{b}$ , the behavioural policy. Note  $\pi_{G}(a_{t} \mid s_{t}, \omega) = \pi_{G}(a_{t} \mid s_{t})$  as  $\omega = s^{(H)}$  is a constant (and as a result,  $a_{t}$  is conditionally independent). To obtain the optimal policy  $\pi_{G}^{*}$ , we maximize the following objective:
$$
\arg \max _ {\pi_ {G}} \mathbb {E} _ {(s _ {t}, a _ {t}) \in \mathcal {D}} [ \log \pi_ {G} (a _ {t} \mid s _ {t}, \omega) ]
$$
We simplify an expectation over an infinitely large dataset  $\mathcal{D}$  collected by  $\pi_b$ :
$$
\mathbb {E} _ {(s _ {t}, a _ {t}) \in \mathcal {D}} [ \log \pi_ {G} (a _ {t} | s _ {t}, \omega) ] = \mathbb {E} _ {s _ {t} \in \mathcal {D}} [ \lambda \log \pi_ {G} (a _ {t} = a ^ {(1)} | s _ {t}, \omega) + (1 - \lambda) \log \pi_ {G} (a _ {t} = a ^ {(2)} | s _ {t}, \omega) ]
$$
Since the actions are conditionally independent of the states, let  $\hat{p} = \pi_G(a_t = a^{(1)}\mid s_t,\omega)$  for any state  $s_t$ . Then:
$$
\mathbb {E} _ {(s _ {t}, a _ {t}) \in \mathcal {D}} [ \log \pi_ {G} (a _ {t} \mid s _ {t}, \omega) ] = \lambda \log \hat {p} + (1 - \lambda) \log (1 - \hat {p})
$$
We can use calculus to maximize the above objective with respect to  $\hat{p}$ .
$$
\begin{array}{l} \frac {d}{d \hat {p}} \left[ \lambda \log \hat {p} + (1 - \lambda) \log (1 - \hat {p}) \right] = \frac {\lambda}{\hat {p}} - \frac {1 - \lambda}{1 - \hat {p}} \\ = \frac {\lambda (1 - \hat {p}) - (1 - \lambda) \hat {p}}{\hat {p} (1 - \hat {p})} \\ \end{array}
$$
Setting the derivative to 0:
$$
\lambda (1 - \hat {p}) - (1 - \lambda) \hat {p} = \lambda - \lambda \hat {p} - \hat {p} + \lambda \hat {p} = 0 \Rightarrow \boxed {\hat {p} = \lambda}
$$
This yields an identical policy to the behavioural policy  $\pi_b$ . Next, consider the derivation of the probability of taking action  $a^{(2)}$  conditioned on  $\Phi_t$  and  $s_t$  based on data  $\mathcal{D}$  from  $\pi_b$ :
$$
\begin{array}{l} \operatorname * {P r} _ {\pi_ {b}} [ a _ {t} = a ^ {(2)} \mid s _ {t} = s ^ {(h)}, s _ {t + K} = s ^ {(h + 1)} ] \\ = \frac {\Pr_ {\pi_ {b}} [ a _ {t} = a ^ {(2)} , s _ {t + K} = s ^ {(h + 1)} \mid s _ {t} = s ^ {(h)} ]}{\Pr_ {\pi_ {b}} [ s _ {t + K} = s ^ {(h + 1)} \mid s _ {t} = s ^ {(h)} ]} \\ \end{array}
$$
In the above step, we used the definition of conditional probability. To compute these probabilities, we recognize that to end up with  $s_{t + K} = s^{(h + 1)}$  from  $s_t = s^{(h)}$ , the agent must take action  $a^{(2)}$  exactly once between timestep  $t$  and  $t + K - 1$ ; any more implies the agent has moved beyond  $s^{(h + 1)}$  and any less implies the agent is still at  $s^{(h)}$ .
The probability in the numerator can be written as a product of taking action  $a^{(2)}$  at timestep  $t$ , followed by taking action  $a^{(1)}$  at timestep  $t + 1$  to  $t + K - 1$ :
$$
\begin{array}{l} \operatorname * {P r} _ {\pi_ {b}} [ a _ {t} = a ^ {(2)}, s _ {t + K} = s ^ {(h + 1)} \mid s _ {t} = s ^ {(h)} ] = (1 - \lambda) \prod_ {t ^ {\prime} = t + 1} ^ {t + K - 1} \lambda \\ = (1 - \lambda) \lambda^ {K - 1} \\ \end{array}
$$
The probability in the denominator can be written as a product of taking action  $a^{(2)}$  at exactly one timestep  $t \leq t' < t + K$ , followed by taking action  $a^{(1)}$  at the remaining timesteps. This can be modeled by a binomial probability where there are  $K$  slots to take action  $a^{(1)}$ , each with probability  $1 - \lambda$ . Hence:
$$
\begin{array}{l} \Pr_ {\pi_ {b}} \left[ s _ {t + K} = s ^ {(h + 1)} \mid s _ {t} = s ^ {(h)} = \binom {K} {1} (1 - \lambda) \lambda^ {K - 1} \right. \\ = K (1 - \lambda) \lambda^ {K - 1} \\ \end{array}
$$
The overall probability is computed as:
$$
\begin{array}{l} \frac {\operatorname * {P r} _ {\pi_ {b}} [ a _ {t} = a ^ {(2)} , s _ {t + K} = s ^ {(h + 1)} \mid s _ {t} = s ^ {(h)} ]}{\operatorname * {P r} _ {\pi_ {b}} [ s _ {t + K} = s ^ {(h + 1)} \mid s _ {t} = s ^ {(h)} ]} = \frac {(1 - \lambda) \lambda^ {K - 1}}{K (1 - \lambda) \lambda^ {K - 1}} \\ = \frac {1}{K} \\ \end{array}
$$