TASK DEFINITION: In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
PROBLEM: We investigate the use of temporally abstract actions, or macro-actions, in the solution of Markov decision processes. Unlike current mod­ els that combine both primitive actions and macro-actions and leave the state space un­ changed, we propose a hierarchical model (using an abstract MDP) that works with macro-actions only, and that significantly reduces the size of the state space. This is achieved by treating macro­ actions as local policies that act in certain regions of state space, and by restricting states in the ab­ stract MDP to those at the boundaries of regions. The abstract MDP approximates the original and can be solved more efficiently. We discuss sev­ eral ways in which macro-actions can be gen­ erated to ensure good solution quality. Finally, we consider ways in which macro-actions can be reused to solve multiple, related MDPs; and we show that this can justify the computational over­ head of macro-action generation.

SOLUTION: Hierarchical Solution of Markov Decision Processes using Macro-actions

PROBLEM: This is a companion note to our recent study of the weak convergence properties of constrained emphatic temporal-difference learning (ETD) algorithms from a theoretic perspective. It supplements the latter analysis with simulation results and illustrates the behavior of some of the ETD algorithms using three example problems.

SOLUTION: Some Simulation Results for Emphatic Temporal-Difference Learning Algorithms∗

PROBLEM: Hierarchical Reinforcement Learning (HRL) exploits temporal abstraction to solve large Markov Decision Processes (MDP) and provide transferable subtask policies. In this paper, we introduce an off-policy HRL algorithm: Hierarchical Q-value Iteration (HQI). We show that it is possible to effectively learn recursive optimal policies for any valid hierarchical decomposition of the original MDP, given a fixed dataset collected from a flat stochastic behavioral policy. We first formally prove the convergence of the algorithm for tabular MDP. Then our experiments on the Taxi domain show that HQI converges faster than a flat Q-value Iteration and enjoys easy state abstraction. Also, we demonstrate that our algorithm is able to learn optimal policies for different hierarchical structures from the same fixed dataset, which enables model comparison without recollecting data.

SOLUTION:
Algorithms for Batch Hierarchical Reinforcement Learning