Fast Approximate Dynamic Programming for Infinite-Horizon Markov Decision Processes

1 Introduction . Value iteration ( VI ) is one of the most basic and wide-spread algorithms employed for tackling problems in reinforcement learning ( RL ) and optimal control [ 9 , 28 ] formulated as Markov decision processes ( MDPs ) . The VI algorithm simply involves the consecutive applications of the dynamic programming ( DP ) operator T J ( xt ) = min ut { C ( xt , ut ) + γEJ ( xt+1 ) } , where C ( xt , ut ) is the cost of taking the control action ut at the state xt . This fixed point iteration is known to converge to the optimal value function for discount factors γ ∈ ( 0 , 1 ) . However , this algorithm suffers from a high computational cost for large-scale finite state spaces . For problems with a continuous state space , the DP operation becomes an infinite-dimensional optimization problem , rendering the exact implementation of VI impossible in most cases . A common solution is to incorporate function approximation techniques and compute the output of the DP operator for a finite sample ( i.e. , a discretization ) of the underlying continuous state space . This approximation again suffers from a high computational cost for fine discretizations of the state space , particularly in high-dimensional problems . We refer the reader to [ 9 , 26 ] for various approximations of VI . For some problems , however , it is possible to partially address this issue by using duality theory , i.e. , approaching the minimization problem in the conjugate domain . In particular , as we will see 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in Section 3 , the minimization in the primal domain in DP can be transformed to a simple addition in the dual domain , at the expense of three conjugate transforms . However , proper application of this transformation relies on efficient numerical algorithms for conjugation . Fortunately , such an algorithm , known as linear-time Legendre transform ( LLT ) , has been developed in late 90s [ 23 ] . Other than the classical application of LLT ( and other fast algorithms for conjugate transform ) in solving Hamilton-Jacobi equation [ 1 , 13 , 14 ] , these algorithms are used in image processing [ 24 ] , thermodynamics [ 12 ] , and optimal transport [ 17 ] . The application of conjugate duality for the DP problem is not new and actually goes back to Bellman [ 4 ] . Further applications of this idea for reducing the computational complexity were later explored in [ 15 , 18 ] . However , surprisingly , the application of LLT for solving discrete-time optimal control problems , has been limited . In particular , in [ 11 ] , the authors propose the “ fast value iteration ” algorithm ( without a rigorous analysis of the complexity and error of the proposed algorithm ) for a particular class of infinite-horizon optimal control problems with state-independent stage cost C ( x , u ) = C ( u ) and deterministic linear dynamics xt+1 = Axt + But , where A is a non-negative , monotone , invertible matrix . More recently , in [ 20 ] , we also considered the application of LLT for solving the DP operation in finite-horizon , optimal control of input-affine dynamics xt+1 = fs ( xt ) + But with separable cost C ( x , u ) = Cs ( x ) + Ci ( u ) . In particular , we introduced the “ discrete conjugate DP ” ( d-CDP ) operator , and provided a detailed analysis of its complexity and error . As we will discuss shortly , the current study is an extension of the corresponding d-CDP algorithm that , among other things , considers infinite horizon , discounted cost problems . We note that the algorithms developed in [ 16 , 24 ] for distance transform can also potentially tackle the optimal control problems similar to the ones of interest in the current study . In particular , these algorithms require the stage cost to be reformulated as a convex function of the “ distance ” between the current and next states . While this property might arise naturally , it can generally be restrictive , as it is in the problem class considered in this study . Another line of work that is closely related to ours invloves utilizing max-plus algebra in solving deterministic , continuous-state , continuous-time , optimal control problems ; see , e.g. , [ 2 , 25 ] . These works exploit the compatibility of the DP operation with maxplus operations , and approximate the value function as a max-plus linear combination . Recently , in [ 3 , 5 ] , the authors used this idea to propose an approximate VI algorithm for continuous-state , deterministic MDPs . In this regard , we note that the proposed approach in the current study also involves approximating the value function as a max-plus linear combination , namely , the maximum of affine functions . The key difference is however that by choosing a grid-like ( factorized ) set of slopes for the linear terms ( i.e. , the basis of the max-plus linear combination ) , we take advantage of linear time complexity of LLT in computing the constant terms ( i.e. , the coefficients of the max-plus linear combination ) . Main contribution . In this study , we focus on an approximate implementation of VI involving discretization of the state and input spaces for solving the optimal control problem of discrete-time systems , with continuous state-input space . Building upon our earlier work [ 20 ] , we employ conjugate duality to speed-up VI for problems with separable stage cost ( in state and input ) and input-affine dynamics . We propose the conjugate VI ( ConjVI ) algorithm based on a modified version of the d-CDP operator introduced in [ 20 ] , and extend the existing results in three directions : We consider infinite-horizon , discounted cost problems with stochastic dynamics , while incorporating a numerical scheme for approximation of the conjugate of input cost . The main contributions of this paper are then as follows : ( i ) we provide sufficient conditions for the convergence of ConjVI ( Theorem 3.9 ) ; ( ii ) we show that ConjVI can achieve a linear time complexity of O ( X+U ) in each iteration ( Theorem 3.10 ) , compared to the quadratic time complexity of O ( XU ) of the standard VI , where X and U are the cardinalities of the discrete state and input spaces , respectively ; ( iii ) we analyze the error of ConjVI ( Theorem 3.11 ) , and use that result to provide specific guidelines on the construction of the discrete dual domain ( Section 3.4 ) ; ( iv ) we provide a MATLAB package for the implementation of the proposed ConjVI algorithm . Notations . The standard inner product in Rn and the corresponding induced 2-norm are denoted by ⟨· , ·⟩ and ∥·∥2 , respectively . ∥·∥∞ denotes the infinity norm . We use the superscript d to denote finite ( discrete ) sets ( as in Xd ) and discrete functions ( as in hd : Xd → R ) . We use the superscript g to denote grid-like finite sets ( as in Xg = Πni=1X g i where X g i ⊂ R ) . We also use X g sub to denote the sub-grid of Xg derived by omitting the smallest and the largest elements of Xg in each dimension . The cardinality of the finite set Xd or Xg is denoted by X . We use h̃d : Rn → R = R ∪ { ∞ } to denote a generic extension of a discrete function hd , and hd : Rn → R to denote multilinear interpolation and extrapolation ( LERP ) extension of a discrete function with a grid-like domain . Let X , Y be two arbitrary sets in Rn . co ( X ) is the convex hull of X . We use d ( X , Y ) : = infx∈X , y∈Y ∥x− y∥2 to denote the distance between X and Y . The one-sided Hausdorff distance from X to Y is defined as dH ( X , Y ) : = supx∈X infy∈Y ∥x− y∥2 . Let h : Rn → R be an extended real-valued function with a non-empty effective domain dom ( h ) = X : = { x ∈ Rn : h ( x ) < ∞ } . The range of h is denoted by rng ( h ) : = maxx∈X h ( x ) −minx∈X h ( x ) , and the subdifferential of h at a point x ∈ X is defined as ∂h ( x ) : = { y ∈ Rn : h ( x̃ ) ≥ h ( x ) + ⟨y , x̃− x⟩ , ∀x̃ ∈ X } . We define L ( h ) : = Πni=1 [ L−i ( h ) , L + i ( h ) ] , where L+i ( h ) ( resp . L − i ( h ) ) is the maximum ( resp . minimum ) slope of the function h along the i-th dimension . Note that ∂h ( x ) ⊆ L ( h ) for all x ∈ X . We report the complexities using the standard big-O notations O and Õ , where the latter hides the logarithmic factors . In this study , we are mainly concerned with the dependence of the computational complexities on the size of the finite sets involved ( discretization of the primal and dual domains ) . In particular , we ignore the possible dependence of the computational complexities on the dimension of the variables , unless they appear in the power of the size of those discrete sets . We note that the extended version of this article , including the technical proofs , is available in the supplementary material . 2 VI in primal domain . We are concerned with the infinite-horizon , discounted cost , optimal control problems of the form J⋆ ( x ) =min Ewt [ ∞∑ t=0 γtC ( xt , ut ) ∣∣∣∣x0 = x ] s.t . xt+1 = g ( xt , ut , wt ) , xt ∈ X , ut ∈ U , wt ∼ P ( W ) , ∀t ∈ { 0 , 1 , . . . } , where xt ∈ Rn , ut ∈ Rm , and wt ∈ Rl are the state , input and disturbance variables at time t , respectively ; γ ∈ ( 0 , 1 ) is the discount factor ; C : X × U → R is the stage cost ; g : Rn × Rm × Rl → Rn describes the dynamics ; X ⊂ Rn and U ⊂ Rm describe the state and input constraints , respectively ; and , P ( · ) is the distribution of the disturbance over the support W ⊂ Rl . Assuming the stage cost C is bounded , the optimal value function solves the equation J⋆ = T J⋆ , where T is the DP operator ( C and J are extended to infinity outside their effective domains ) [ 7 , Prop . 1.2.2 ] T J ( x ) : = min u { C ( x , u ) + γ · EwJ ( g ( x , u , w ) ) } , ∀x ∈ X . ( 1 ) Indeed , T is γ-contractive in the infinity-norm [ 7 , Prop . 1.2.4 ] . This property then gives rise to the VI algorithm Jk+1 = T Jk , which converges to J⋆ as k → ∞ , for arbitrary initialization J0 . Moreover , assuming that the composition J ◦ g ( for each w ) and the cost C are jointly convex in the state and input variables , T also preserves convexity [ 8 , Prop . 3.3.1 ] . For numerical implementation of VI , we need to address three issues . First , we need to compute the expectation in ( 1 ) . In order to simplify the exposition and include the computational cost of this operation explicitly , we consider disturbances with finite support in this study : Assumption 2.1 ( Disturbance with finite support ) . The disturbance w has a finite support Wd ⊂ Rl with a given probability mass function ( p.m.f . ) p : Wd → [ 0 , 1 ] . Under the preceding assumption , we have EwJ ( g ( x , u , w ) ) = ∑ w∈Wd p ( w ) · J ( g ( x , u , w ) ) . The second and more important issue is that the optimization problem ( 1 ) is infinite-dimensional for the continuous state space X . This renders the exact implementation of VI impossible , except for a few cases with available closed-form solutions . A common solution to this problem is to deploy a sample-based approach , accompanied by a function approximation scheme . To be precise , for a finite subset Xd of X , at each iteration k = 0 , 1 , . . . , we take the discrete function Jdk : Xd → R as the input , and compute the discrete function Jdk+1 = [ T J̃dk ] d : Xd → R , where J̃dk : X → R is an extension of Jdk . Finally , for each x ∈ Xd , we have to solve the minimization problem in ( 1 ) over the control input . Since this minimization problem is often a difficult , non-convex problem , a common approximation involves enumeration over a discretization Ud ⊂ U . Incorporating these approximations , we end up with the approximate VI algorithm Jdk+1 = T dJdk , characterized by the discrete DP ( d-DP ) operator T dJd ( x ) : = min u∈Ud { C ( x , u ) + γ · ∑ w∈Wd p ( w ) · J̃d ( g ( x , u , w ) ) } , ∀x ∈ Xd . ( 2 ) The convergence of approximate VI described above depends on the properties of the extension operation [ ̃· ] . In particular , if the extension operation is non-expansive ( in the infinity-norm ) , then T d is also γ-contractive . The error of this approximation also depends on the extension operation and its representative power . We refer the interested reader to [ 7 , 10 , 26 ] for detailed discussions on the convergence and error of different approximation schemes for VI . The d-DP operator and the corresponding VI algorithm will be our benchmark for evaluating the performance of the alternative algorithm developed in this study . To this end , we finish this section with some remarks on the time complexity of the d-DP operation . Let the time complexity of a single evaluation of the extension operator [ ̃· ] in ( 2 ) be of O ( E ) . Then , the time complexity of the d-DP operation ( 2 ) is of O ( XUWE ) . In this regard , note that the scheme described above essentially involves approximating a continuous-state/action MDP with a finite-state/action MDP , and then applying the VI algorithm . This , in turn , implies the lower bound Ω ( XU ) for the time complexity ( corresponding to enumeration over u ∈ Ud for each x ∈ Xd ) . This lower bound is also compatible with the best existing time complexities in the literature for VI for finite MDPs ; see , e.g. , [ 3 , 27 ] . However , as we will see in the next section , for a particular class of problems , it is possible to exploit the structure of the underlying continuous system in order to achieve a better time complexity in the corresponding discretized problem .

Summary: This paper presents a fast method for solving certain classes of optimal control problems, by implementing a value iteration (VI) algorithm that takes advantage of convex duality.  The basic idea is to take advantage of the fact that for two functions $f_1, f_2$, we have $(f_1 \square f_2)^* = f_1^* + f_2^*$, where $f^*$ denotes the Legendre-Fenchel transform of $f$, and $(f_1 \square f_2)(x) = \inf \\{f_1(x_1) + f_2(x_2) : x_1 + x_2 = x \\} $ denotes the infimal convolution of $f_1, f_2$. Thus, the slow $\inf$ operation can be replaced with a fast addition operation. The paper provides analyses of the convergence, time complexity, and error of the algorithm. When compared to a "naive" VI algorithm in which each iteration takes $O(X U)$ (where $X$, $U$ are the grid sizes of the state and input spaces, respectively), the algorithm presented in this paper can achieve $O(X+U)$. Finally, the paper provides a numerical example implemented in MATLAB.