Fast Approximate Dynamic Programming for Infinite-Horizon Markov Decision Processes

1 Introduction . Value iteration ( VI ) is one of the most basic and wide-spread algorithms employed for tackling problems in reinforcement learning ( RL ) and optimal control [ 9 , 28 ] formulated as Markov decision processes ( MDPs ) . The VI algorithm simply involves the consecutive applications of the dynamic programming ( DP ) operator T J ( xt ) = min ut { C ( xt , ut ) + γEJ ( xt+1 ) } , where C ( xt , ut ) is the cost of taking the control action ut at the state xt . This fixed point iteration is known to converge to the optimal value function for discount factors γ ∈ ( 0 , 1 ) . However , this algorithm suffers from a high computational cost for large-scale finite state spaces . For problems with a continuous state space , the DP operation becomes an infinite-dimensional optimization problem , rendering the exact implementation of VI impossible in most cases . A common solution is to incorporate function approximation techniques and compute the output of the DP operator for a finite sample ( i.e. , a discretization ) of the underlying continuous state space . This approximation again suffers from a high computational cost for fine discretizations of the state space , particularly in high-dimensional problems . We refer the reader to [ 9 , 26 ] for various approximations of VI . For some problems , however , it is possible to partially address this issue by using duality theory , i.e. , approaching the minimization problem in the conjugate domain . In particular , as we will see 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in Section 3 , the minimization in the primal domain in DP can be transformed to a simple addition in the dual domain , at the expense of three conjugate transforms . However , proper application of this transformation relies on efficient numerical algorithms for conjugation . Fortunately , such an algorithm , known as linear-time Legendre transform ( LLT ) , has been developed in late 90s [ 23 ] . Other than the classical application of LLT ( and other fast algorithms for conjugate transform ) in solving Hamilton-Jacobi equation [ 1 , 13 , 14 ] , these algorithms are used in image processing [ 24 ] , thermodynamics [ 12 ] , and optimal transport [ 17 ] . The application of conjugate duality for the DP problem is not new and actually goes back to Bellman [ 4 ] . Further applications of this idea for reducing the computational complexity were later explored in [ 15 , 18 ] . However , surprisingly , the application of LLT for solving discrete-time optimal control problems , has been limited . In particular , in [ 11 ] , the authors propose the “ fast value iteration ” algorithm ( without a rigorous analysis of the complexity and error of the proposed algorithm ) for a particular class of infinite-horizon optimal control problems with state-independent stage cost C ( x , u ) = C ( u ) and deterministic linear dynamics xt+1 = Axt + But , where A is a non-negative , monotone , invertible matrix . More recently , in [ 20 ] , we also considered the application of LLT for solving the DP operation in finite-horizon , optimal control of input-affine dynamics xt+1 = fs ( xt ) + But with separable cost C ( x , u ) = Cs ( x ) + Ci ( u ) . In particular , we introduced the “ discrete conjugate DP ” ( d-CDP ) operator , and provided a detailed analysis of its complexity and error . As we will discuss shortly , the current study is an extension of the corresponding d-CDP algorithm that , among other things , considers infinite horizon , discounted cost problems . We note that the algorithms developed in [ 16 , 24 ] for distance transform can also potentially tackle the optimal control problems similar to the ones of interest in the current study . In particular , these algorithms require the stage cost to be reformulated as a convex function of the “ distance ” between the current and next states . While this property might arise naturally , it can generally be restrictive , as it is in the problem class considered in this study . Another line of work that is closely related to ours invloves utilizing max-plus algebra in solving deterministic , continuous-state , continuous-time , optimal control problems ; see , e.g. , [ 2 , 25 ] . These works exploit the compatibility of the DP operation with maxplus operations , and approximate the value function as a max-plus linear combination . Recently , in [ 3 , 5 ] , the authors used this idea to propose an approximate VI algorithm for continuous-state , deterministic MDPs . In this regard , we note that the proposed approach in the current study also involves approximating the value function as a max-plus linear combination , namely , the maximum of affine functions . The key difference is however that by choosing a grid-like ( factorized ) set of slopes for the linear terms ( i.e. , the basis of the max-plus linear combination ) , we take advantage of linear time complexity of LLT in computing the constant terms ( i.e. , the coefficients of the max-plus linear combination ) . Main contribution . In this study , we focus on an approximate implementation of VI involving discretization of the state and input spaces for solving the optimal control problem of discrete-time systems , with continuous state-input space . Building upon our earlier work [ 20 ] , we employ conjugate duality to speed-up VI for problems with separable stage cost ( in state and input ) and input-affine dynamics . We propose the conjugate VI ( ConjVI ) algorithm based on a modified version of the d-CDP operator introduced in [ 20 ] , and extend the existing results in three directions : We consider infinite-horizon , discounted cost problems with stochastic dynamics , while incorporating a numerical scheme for approximation of the conjugate of input cost . The main contributions of this paper are then as follows : ( i ) we provide sufficient conditions for the convergence of ConjVI ( Theorem 3.9 ) ; ( ii ) we show that ConjVI can achieve a linear time complexity of O ( X+U ) in each iteration ( Theorem 3.10 ) , compared to the quadratic time complexity of O ( XU ) of the standard VI , where X and U are the cardinalities of the discrete state and input spaces , respectively ; ( iii ) we analyze the error of ConjVI ( Theorem 3.11 ) , and use that result to provide specific guidelines on the construction of the discrete dual domain ( Section 3.4 ) ; ( iv ) we provide a MATLAB package for the implementation of the proposed ConjVI algorithm . Notations . The standard inner product in Rn and the corresponding induced 2-norm are denoted by ⟨· , ·⟩ and ∥·∥2 , respectively . ∥·∥∞ denotes the infinity norm . We use the superscript d to denote finite ( discrete ) sets ( as in Xd ) and discrete functions ( as in hd : Xd → R ) . We use the superscript g to denote grid-like finite sets ( as in Xg = Πni=1X g i where X g i ⊂ R ) . We also use X g sub to denote the sub-grid of Xg derived by omitting the smallest and the largest elements of Xg in each dimension . The cardinality of the finite set Xd or Xg is denoted by X . We use h̃d : Rn → R = R ∪ { ∞ } to denote a generic extension of a discrete function hd , and hd : Rn → R to denote multilinear interpolation and extrapolation ( LERP ) extension of a discrete function with a grid-like domain . Let X , Y be two arbitrary sets in Rn . co ( X ) is the convex hull of X . We use d ( X , Y ) : = infx∈X , y∈Y ∥x− y∥2 to denote the distance between X and Y . The one-sided Hausdorff distance from X to Y is defined as dH ( X , Y ) : = supx∈X infy∈Y ∥x− y∥2 . Let h : Rn → R be an extended real-valued function with a non-empty effective domain dom ( h ) = X : = { x ∈ Rn : h ( x ) < ∞ } . The range of h is denoted by rng ( h ) : = maxx∈X h ( x ) −minx∈X h ( x ) , and the subdifferential of h at a point x ∈ X is defined as ∂h ( x ) : = { y ∈ Rn : h ( x̃ ) ≥ h ( x ) + ⟨y , x̃− x⟩ , ∀x̃ ∈ X } . We define L ( h ) : = Πni=1 [ L−i ( h ) , L + i ( h ) ] , where L+i ( h ) ( resp . L − i ( h ) ) is the maximum ( resp . minimum ) slope of the function h along the i-th dimension . Note that ∂h ( x ) ⊆ L ( h ) for all x ∈ X . We report the complexities using the standard big-O notations O and Õ , where the latter hides the logarithmic factors . In this study , we are mainly concerned with the dependence of the computational complexities on the size of the finite sets involved ( discretization of the primal and dual domains ) . In particular , we ignore the possible dependence of the computational complexities on the dimension of the variables , unless they appear in the power of the size of those discrete sets . We note that the extended version of this article , including the technical proofs , is available in the supplementary material . 2 VI in primal domain . We are concerned with the infinite-horizon , discounted cost , optimal control problems of the form J⋆ ( x ) =min Ewt [ ∞∑ t=0 γtC ( xt , ut ) ∣∣∣∣x0 = x ] s.t . xt+1 = g ( xt , ut , wt ) , xt ∈ X , ut ∈ U , wt ∼ P ( W ) , ∀t ∈ { 0 , 1 , . . . } , where xt ∈ Rn , ut ∈ Rm , and wt ∈ Rl are the state , input and disturbance variables at time t , respectively ; γ ∈ ( 0 , 1 ) is the discount factor ; C : X × U → R is the stage cost ; g : Rn × Rm × Rl → Rn describes the dynamics ; X ⊂ Rn and U ⊂ Rm describe the state and input constraints , respectively ; and , P ( · ) is the distribution of the disturbance over the support W ⊂ Rl . Assuming the stage cost C is bounded , the optimal value function solves the equation J⋆ = T J⋆ , where T is the DP operator ( C and J are extended to infinity outside their effective domains ) [ 7 , Prop . 1.2.2 ] T J ( x ) : = min u { C ( x , u ) + γ · EwJ ( g ( x , u , w ) ) } , ∀x ∈ X . ( 1 ) Indeed , T is γ-contractive in the infinity-norm [ 7 , Prop . 1.2.4 ] . This property then gives rise to the VI algorithm Jk+1 = T Jk , which converges to J⋆ as k → ∞ , for arbitrary initialization J0 . Moreover , assuming that the composition J ◦ g ( for each w ) and the cost C are jointly convex in the state and input variables , T also preserves convexity [ 8 , Prop . 3.3.1 ] . For numerical implementation of VI , we need to address three issues . First , we need to compute the expectation in ( 1 ) . In order to simplify the exposition and include the computational cost of this operation explicitly , we consider disturbances with finite support in this study : Assumption 2.1 ( Disturbance with finite support ) . The disturbance w has a finite support Wd ⊂ Rl with a given probability mass function ( p.m.f . ) p : Wd → [ 0 , 1 ] . Under the preceding assumption , we have EwJ ( g ( x , u , w ) ) = ∑ w∈Wd p ( w ) · J ( g ( x , u , w ) ) . The second and more important issue is that the optimization problem ( 1 ) is infinite-dimensional for the continuous state space X . This renders the exact implementation of VI impossible , except for a few cases with available closed-form solutions . A common solution to this problem is to deploy a sample-based approach , accompanied by a function approximation scheme . To be precise , for a finite subset Xd of X , at each iteration k = 0 , 1 , . . . , we take the discrete function Jdk : Xd → R as the input , and compute the discrete function Jdk+1 = [ T J̃dk ] d : Xd → R , where J̃dk : X → R is an extension of Jdk . Finally , for each x ∈ Xd , we have to solve the minimization problem in ( 1 ) over the control input . Since this minimization problem is often a difficult , non-convex problem , a common approximation involves enumeration over a discretization Ud ⊂ U . Incorporating these approximations , we end up with the approximate VI algorithm Jdk+1 = T dJdk , characterized by the discrete DP ( d-DP ) operator T dJd ( x ) : = min u∈Ud { C ( x , u ) + γ · ∑ w∈Wd p ( w ) · J̃d ( g ( x , u , w ) ) } , ∀x ∈ Xd . ( 2 ) The convergence of approximate VI described above depends on the properties of the extension operation [ ̃· ] . In particular , if the extension operation is non-expansive ( in the infinity-norm ) , then T d is also γ-contractive . The error of this approximation also depends on the extension operation and its representative power . We refer the interested reader to [ 7 , 10 , 26 ] for detailed discussions on the convergence and error of different approximation schemes for VI . The d-DP operator and the corresponding VI algorithm will be our benchmark for evaluating the performance of the alternative algorithm developed in this study . To this end , we finish this section with some remarks on the time complexity of the d-DP operation . Let the time complexity of a single evaluation of the extension operator [ ̃· ] in ( 2 ) be of O ( E ) . Then , the time complexity of the d-DP operation ( 2 ) is of O ( XUWE ) . In this regard , note that the scheme described above essentially involves approximating a continuous-state/action MDP with a finite-state/action MDP , and then applying the VI algorithm . This , in turn , implies the lower bound Ω ( XU ) for the time complexity ( corresponding to enumeration over u ∈ Ud for each x ∈ Xd ) . This lower bound is also compatible with the best existing time complexities in the literature for VI for finite MDPs ; see , e.g. , [ 3 , 27 ] . However , as we will see in the next section , for a particular class of problems , it is possible to exploit the structure of the underlying continuous system in order to achieve a better time complexity in the corresponding discretized problem .

Summary: The authors consider value iteration [TJ](x) = min_u [c(x,u) + gamma E J(x’)] for problems where the transition function and cost functions are additively separable: c(x,u)=c1(x)+c2(u) and x’=f1(x)+f2(u) + w (Assumptions 3.1). Furthermore, f1(x) is Lipschitz continuous, f2(u) is linear, c1(x) is convex and Lipschitz continuous, and c2(u) is linear in u. The disturbance w has a finite support (Assumption 2.1). The state space for x and decision space for u are continuous. Thus, the authors consider problems such that [TJ](x) = c1(x) + min_u [c2(u) + gamma sum_i  p_i J(f1(x)+Bu+w_i)]. A problem that fits this description is the stabilization of an inverted pendulum with discretized disturbances, studied in Section 4. With this choice of assumptions, and assuming the value function is convex (which holds true if f1(x) is linear or linearized, that is, if x’=Ax+Bu+w), convex duality can be applied to the minimization problem over (u,x”) given x and subject to x”=Ax+Bu, which gives another representation of the optimization problem, based on the convex conjugate of c1 and the convex conjugate of e(x”)=gamma E J(x”+w).  The authors propose and study an algorithm that relies on computing these convex conjugate functions numerically in order to provide an alternative way to the computation of the value function J. They establish error bounds (Theorem 3.13) in terms of theoretical quantities of the problem data.