Decision Tree Algorithms for MDP

1 INTRODUCTION . Deep neural network based Reinforcement Learning ( RL ) has seen recent success in tackling Markov Decision Problem ( MDP ) instances with large state-action space . For example , the MuZero model ( Schrittwieser et al . ( 2020 ) ) has been able to successfully beat human performance in previously computationally untractable problems such as Shogi , Chess and Atari games with a single modeling paradigm . The success of these methods can be traced to two fundamental ideas in RL : First , the iterative nature of the Q-learning algorithm ( see Watkins & Dayan ( 1992 ) ) where the state-action value function can be updated locally without having to handle the whole state-action space at once as in exact methods ; Second , the non-linear parameterization of the state-action value function and the policy function with deep neural networks , where in a sense , the state space model is “ learned ” and compressed without having to update and explore the whole state space . While successful methods , they are not devoid of drawbacks . Their strength in not taking into account the whole action-state space is also their weakness which manifests in the well-known worst case slow convergence of Q-learning ( see Szepesvári et al . ( 1998 ) ) as well as the relatively slow convergence of the gradient-based methods for training deep neural networks for regression . In this paper , we aim to apply the essence of these two ideas to develop an algorithm to solve a subset of instances of MDP , that while having large enough state-action space such that exact methods are computationally intractable , their solution policy admits a representation as a decision tree that can be computed by exact methods . In summary , we aim to do an improvement over these instances with two ideas : First , the use of exact methods , such as Dynamic Programming ( DP ) or Linear Programming ( LP ) , rather than iterative methods to compute the state-action value function ; Second , the use of decision trees to partition the state space and codify the optimal policy rather than using deep neural networks . In Figure 1 we show the subset of problems we aim to tackle in the gray area . That is , problems with large enough state space such that exact methods are not tractable , but the action space is still not too large . This is key , as the size of the policy function representation as a decision tree depends on the structure of the state space being partitionable in regions with the same optimal action . To illustrate these concepts we consider the cartpole balancing example , which consists of controlling a cart that can move either to right or left in a straight rail . Perpendicular to the axis of movement of the cart , there is a pole attached to a rotating axis at its center . With two actions A = { ← , → } the goal is to have the pole attached to the cart balanced in the upright position for the longest possible amount of time . The state space of the system is a 4-dimensional vector of real numbers S = { ( x , v , θ , w ) : x ∈ [ −ℓ1 , ℓ1 ] , v ∈ [ −ℓ2 , ℓ2 ] , θ ∈ [ −ℓ3 , ℓ3 ] , w ∈ [ −ℓ4 , ℓ4 ] } ( position , horizontal velocity , angle and angular velocity ) . Here , the state space is of infinite size |S| = ∞ . Instead of using deep Q-learning , we use a decision tree based algorithm that outputs the policy in Figure 2 . This decision tree serves two functions simultaneously : It defines a partition or “ aggregation ” of the state space S into disjoint regions that will be the states of a much smaller MDP ; and second and most importantly , it summarizes the optimal policy as a decision tree , giving a transparent and interpretable control policy . These two points are the same ideas that make deep Q-learning methods successful : Aggregation of the space and non-linear parameterization of the optimal policy . The MDP went from having a infinite cardinality to a reduce state space of only 6 states . 2 RELATED WORK . In the RL and MDP literature the idea of state aggregation starts as early as the discipline itself as an answer to the curse of dimensionality for large state-action spaces . One of such ideas is to make a large state space discrete by aggregating states into “ boxes ” ( see Fox ( 1973 ) ) . Our method takes ideas from the Variable Resolution idea where a given discretization is further refined by constructing finer boxes around regions of the state space where the discretization does not approximate well the value function . One of those methods is the “ Trie Model ” ( see Moore ( 1991 ) ) where a tree model with similar ideas to ours is used to identify regions of the state space that need to be further partitioned while still applying Dynamic Programming methods to obtain a policy . Their main criterion to make a split was to identify regions of the state space where the model was inaccurate ( in a value function sense ) and increase the resolution of the MDP by further partitioning these regions . In Singh et al . ( 1995 ) the convergence for the Q-learning algorithm is proved for the “ softaggregation ” scheme where states belong to disjoint clusters with a given probability ( including the hard cluster case where states belong to only one cluster ) , our work borrows a similar description of the aggregated MDP ( from a cluster level ) that we use to define our Linear Programming formulation . Bounds on the performance of value iteration over disjoint state aggregation are given in Van Roy ( 2006 ) . A large stream of literature then focuses on parameterization of the value function as means to defeat the curse of dimensionality ( which is an indirect way to aggregate the state space ) . Approximate Dynamic Programming ( see Bertsekas ( 2008 ) for a summary overview ) is a parameterization of the value function by a lower-dimensional linear basis of the state space . On the non-linear case , nearest neighbor approximations have been used in Shah & Xie ( 2018 ) . Kernel functions are used in Ormoneit & Sen ( 2002 ) and Ormoneit & Glynn ( 2002 ) . Decision Trees have been used in Ernst et al . ( 2005 ) to similarly approximate the Q-function of unseen samples . Much of the recent literature focus on Deep Neural Network based RL which can be seen as a non-linear representation of the state-action value function ( see Li ( 2017 ) for an overview ) as well as a non-linear parameterization of the policy as stated in the introduction . Actor-critic architectures ( see Haarnoja et al . ( 2018 ) ) have also seen recent success in tackling large scale problems . A characterization theory of state aggregations is studied in Li et al . ( 2006 ) , where the most common families of aggregations ( bisimulation , and equivalent value function aggregations ) is studied and a hierarchy between them is given . A related work to ours is Kim & Dean ( 2003 ) where a similar sequential partition is used for finding sequential state aggregations , our work differs in the criteria to make splits , whereas we approximate the behavior of the whole MDP , Kim & Dean ( 2003 ) chooses a split based on the maximum distance between value functions from one iteration to the next positioning their work closer to the variable resolution literature . Our method is new in the sense that instead of exclusively focusing on the resolution relative to the value function ( that is , instead of trying to approximate and interpolate the value function at unseen points ) we find a state aggregation abstraction of the original state space with a reduced state space with the same optimal policy as in the original state space , given that the original MDP admits such representation . We further discuss the theory and potential computational savings of MDP abstraction in Section A.3 . 3 METHODOLOGY . In this section we describe our methodology connecting the representation of policies as decision trees and the aggregation of the original state space resulting in a much smaller MDP with equivalent optimal policy . 3.1 MARKOV DECISION PROCESSES . A Markov Decision Process ( MDP ) is a modeling framework that in simple terms finds optimal decisions with a changing environment over time with the goal of finding the “ best ” actions given the state of the environment . Formally , a MDP is composed of an environment represented by a state space S , a set of actions A . An agent in state s ∈ S , takes an action a ∈ A . After this , the agent transitions to state s′ ∈ S and receives a reward r ∈ R with probability p ( s′ , r|s , a ) , which is independent of the previous history of states or actions , making the model a Markov chain . Given the state and action s , a the reward is a quantity R ( s , a , s′ ) . A policy π : S → A is a function that maps states s ∈ S to a single action a1 . For simplicity , assume that there is a terminal state that ends the subsequent transitions ( an absorbing state E ) . This state is reached in a random number of T transitions . The cumulative discounted reward an agent receives by interacting in the system using a policy π is given by : Eπ [ ∞∑ t=0 γtR ( st , π ( st ) , st+1 ) ] . ( 1 ) Given a complete description of the model , the policy that maximizes the cumulative reward is given by the Bellman optimality equation : Vπ ( s ) : = ∑ s′ , r P ( s′ , r|s , π ( s ) ) [ r + Vπ ( s′ ) ] , ( 2 ) V∗ ( s ) = max a ∑ s′ , r P ( s′ , r|s , a ) [ r + V∗ ( s′ ) ] . ( 3 ) 1The extension to stochastic policies is straightforward and not considered here . Where Vπ ( s ) is the expected reward starting at state s following policy π , the expression Vπ ( s ) can be intuitively derived by a first-step analysis on the fact that the system is memoryless by the Markov property of Markov chains , that is , the expected reward in one state is simply the one step expected reward obtained in that state r ( after taking action π ( s ) and transitioning to state s′ ) plus the expected reward of starting at state s′ . The optimal policy V∗ ( s ) is simply taking the greediest action a at every state s in the expected reward sense . 3.2 REPRESENTATION OF POLICIES AS DECISION TREES . In this subsection we show the connection between state aggregation and policy representation as decision trees . A policy π is a mapping between the state space S and the action setA . This mapping is similar to a classification problem , in the sense that in a classification problem the goal is to divide the input space into “ regions ” classifying them into different classes . In MDP , the goal is similar as the policy divides the state space S into disjoint regions corresponding to elements of the action set A . Similar to Deep Neural Networks ( DNN ) , decision trees are also good function approximators . An analogous function approximation theorem for decision trees is that for any policy π ( s ) , there exist a tree T with J leaves L = { Ej } Jj=1 such that the policy π can be approximated by the function π ( s ) = ∑J j=1 π̄ ( Ej ) 1 { s ∈ Ej } where { π̄ ( Ej ) } Jj=1 is a collection of piecewise constant actions in A . Given this relation between a policy and a decision tree π ( s ) = ∑J j=1 π̄ ( Ej ) 1 { s ∈ Ej } , it is natural to think of a related MDP where the states are the partitions of the state space rather than the larger ( and potentially infinite ) original state space . This approximated MDP could be far away from accurately approximate the reward of the original MDP but its resolution is good enough to identify the optimal policies . The challenge lies in identifying a growing strategy for the trees that is able to find the optimal policy π∗ by iteratively partitioning the state space and taking the leaves of the tree as meta-states of a reduced MDP approximating the behavior of this policy in the original state space . The growing strategy needs an objective value that is able the compare the overall quality of different trees with respect to finding optimal policies in the original state space . For example , as any tree algorithm normally starts by collapsing all states into a single one , any partition into two disjoint subsets needs to take into account the sizes of the partitions as well as the approximated behavior of the aggregated MDP with respect to the original state space and quantify its improvement with respect to the initial tree where all states are collapsed into one .

Summary: The authors propose a method to solve MDPs with large state-space and moderately sized action-space, where a decision tree can represent the optimal policy.  The decision tree is learned by iteratively partitioning the state space such that the approximation of the policy by the expanded decision tree improves the expected reward. Finding the best partition in each step involves solving a Linear Programm.  The authors provide an extensive analysis of the computational complexity of their method and demonstrate its strengths in a Gridworld example and the Cartpole problem.