The Geometry of Memoryless Stochastic Policy Optimization in Infinite-Horizon POMDPs

1 INTRODUCTION . Markov decision processes ( MDPs ) were introduced by Bellman ( 1957 ) as a model for sequential decision making and optimal planning ( see , e.g. , Howard , 1960 ; Derman , 1970 ; Puterman , 2014 ) . Many algorithms in reinforcement learning rely on the ideas and methods developed in the context of MDPs ( see , e.g. , Sutton & Barto , 2018 ) . Often in practice , the decisions need to be made based only on incomplete information of the state of the system . This setting is modeled by partially observable Markov decision processes ( POMDPs ) introduced by Åström ( 1965 ) ( for a historical discussion see Monahan , 1982 ) , which have become an important model for planning under uncertainty . In this work we pursue a geometric characterization of the policy optimization problem in POMDPs over the class of stochastic memoryless policies and its dependence on the degree of partial observability . It is well known that acting optimally in POMDPs may require memory ( Åström , 1965 ) . A POMDP with unlimited memory policies can be modeled as a belief state MDP , where the states are replaced by probability distributions that serve as sufficient statistics for the previous observations ( Kaelbling et al. , 1998 ; Murphy , 2000 ) . Finding an optimal policy in this class is PSPACE-complete for finite horizons ( Papadimitriou & Tsitsiklis , 1987 ) and undecidable for infinite horizons ( Madani et al. , 2003 ; Chatterjee et al. , 2016 ) . Therefore , it is of interest to consider POMDPs with constrained policy classes . A natural class to consider are memoryless policies , also known as reactive or Markov policies , which select actions based solely on the current observations . In this case , it is useful to allow the actions to be selected stochastically , which not only allows for better solutions but also provides a continuous optimization domain ( Singh et al. , 1994 ) . Although they are more restrictive than policies with memory , memoryless policies are attractive as they are easier to optimize and are versatile enough for certain applications ( Tesauro , 1995 ; Loch & Singh , 1998 ; Williams & Singh , 1999 ; Kober et al. , 2013 ) . In fact , finite-memory policies can be modeled in terms of memoryless policies by supplementing the state of the system with an external memory ( Littman , 1993 ; Peshkin et al. , 1999 ; Icarte et al. , 2021 ) . Hence theoretical advances on memoryless policy optimization are also of interest to finite-memory policy optimization . Theoretical aspects and optimization strategies over the class of memoryless policies have been studied in numerous works ( see , e.g. , Littman , 1994 ; Singh et al. , 1994 ; Jaakkola et al. , 1995 ; Loch & Singh , 1998 ; Williams & Singh , 1999 ; Baxter et al. , 2000 ; Baxter & Bartlett , 2001 ; Li et al. , 2011 ; Azizzadenesheli et al. , 2018 ) . However , finding exact or approximate optimal memoryless stochastic policies for POMDPs is still considered an open problem ( Azizzadenesheli et al. , 2016 ) , which is NP-hard in general ( Vlassis et al. , 2012 ) . One reason for the difficulties in optimizing POMDPs is that , even in a tabular setting , the problem is non-convex and can exhibit suboptimal strict local optima ( Bhandari & Russo , 2019 ) . For memoryless policies the expected cumulative reward is a linear function of the corresponding ( discounted ) state-action frequencies . In the case of MDPs the feasible set of state-action frequencies is known to form a polytope , so that the optimization problem can be reduced to a linear program ( Manne , 1960 ; De Ghellinck , 1960 ; d ’ Epenoux , 1963 ; Hordijk & Kallenberg , 1981 ) . On the other hand , to the best of our knowledge , for POMDPs the specific structure of the feasibility constraints and the optimization problem have not been studied , at least not in the same level of detail ( see related works below ) . Related works In MDPs , the ( discounted ) state-action frequencies have been shown to form a polytope resp . a compact convex set in the finite resp . countable state-action cases , whereby the extreme points are given by the state-action frequencies of deterministic stationary policies ( Derman , 1970 ; Altman & Shwartz , 1991 ) . Further , Dadashi et al . ( 2019 ) showed that the set of state value functions in finite state-action MDPs is a finite union of polytopes . The set of stationary state-action distributions of POMDPs has been studied by Montúfar et al . ( 2015 ) highlighting a decomposition into infinitely many convex subsets whose dimensions depend on the degree of observability . Although this decomposition can be used to localize optimal policies to some extent , a description of the pieces in combination is still missing , needed to capture the properties of the optimization problem . We will obtain a detailed description in terms of finitely many polynomial constraints with closed form expressions and bound their degrees in terms of the observation mechanism . This yields the formulation of reward maximization in infinite-horizon POMDPs as a polynomial optimization problem , where Cohen & Parmentier ( 2018 ) obtained a description of finite-horizon POMDPs as a mixed integer linear program . Grinberg & Precup ( 2013 ) showed that the expected mean reward is a rational function and obtained bounds on the degree of this function . We generalize this to the setting of discounted rewards and refine the result by relating the rational degree to the degree of observability . For both MDPs and POMDPs the expected cumulative reward is known to be a non-convex function of the policy even for tabular policy models ( Bhandari & Russo , 2019 ) . Nonetheless , for MDPs critical points can be shown to be global maxima under mild conditions . In contrast , for POMDPs or MDPs with linearly restricted policy models , it is known that non-global local optimizers can exist ( Baxter et al. , 2000 ; Poupart et al. , 2011 ; Bhandari & Russo , 2019 ) . However , nothing is known about the number of local optimizers . We will present bounds on the number of critical points building on our computation of the rational degree and feasibility constraints . The structure of the expected cumulative reward has been studied in terms of the location of the global optimizers and the existence of local optimizers . Most notably , it is well known that in MDPs there always exist optimal policies which are memoryless and deterministic ( see Puterman , 2014 ) . In the case of POMDPs , optimal memoryless policies may need to be stochastic ( see Singh et al. , 1994 ) . Montúfar et al . ( 2015 ) ; Montúfar & Rauh ( 2017 ) ; Montúfar et al . ( 2019 ) obtained upper bounds on the number of actions that need to be randomized by these policies , which in the worst case is equal to the number of states that are compatible with the observation . Although we do not improve these results ( which are indeed tight in some cases ) , our description of the expected cumulative reward function leads to a simpler proof of the bounds obtained by Montúfar et al . ( 2015 ) . Neyman ( 2003 ) considered stochastic games as semialgebraic problems showing that the minmax and maxmin payoffs in an n-player game are semialgebraic functions of the discount factor . Although this is not directly related to our work , we take a similar philosophy . We pursue a semialgebraic description of the feasible set of discounted state-action frequencies in POMDPs , which is closely related to the general spirit of semialgebraic statistics ( Zwiernik , 2016 ) . Based on this we characterize the properties of the optimization problem by its algebraic degree , a concept that has been advanced in recent works on polynomial optimization ( Bajaj , 1988 ; Catanese et al. , 2006 ; Nie & Ranestad , 2009 ; Draisma et al. , 2016 ; Özlüm Çelik et al. , 2021 ) . Contributions We obtain results for infinite-horizon POMDPs with memoryless stochastic policies under the mean or discounted reward criteria which can be summarized as follows . 1 . We show that the state-action frequencies and the expected cumulative reward can be written as fractions of determinantal polynomials in the entries of the stochastic policy matrix . We show that the degree of these polynomials is directly related to the degree of observability ( see Theorem 4 ) . 2 . We describe the set of feasible state-action frequencies as a basic semialgebraic set , i.e. , as the solution set to a system of polynomial equations and inequalities , for which we also derive closed form expressions ( see Theorem 16 and Remark 18 ) . 3 . We reformulate the expected cumulative reward optimization problem as the optimization of a linear function subject to polynomial constraints ( see Remark 19 ) , which we use to solve a navigation problem in a grid world ( see Appendix F ) . This is a POMDP generalization of the dual linear programming formulation of MDPs ( Kallenberg , 1994 ; Puterman , 2014 ) . 4 . We present two methods for computing the number of critical points , which rely , respectively , on the rational degree of the expected cumulative reward function and the geometric description of the feasible set of state-action frequencies ( see Theorem 20 , Proposition 21 and Appendix D ) . 2 PRELIMINARIES . We use standard notation for probability distributions and Markov kernels . We denote the simplex of probability distributions on a finite set X by ∆X . An element µ ∈ ∆X is a vector in RX with non-negative entries µx = µ ( x ) , x ∈ X adding to one . We denote the set of Markov kernels from a finite set X to another finite set Y by ∆XY . An element Q ∈ ∆XY is a |X | × |Y| row stochastic matrix with entries Qxy = Q ( y|x ) , x ∈ X , y ∈ Y . Given Q ( 1 ) ∈ ∆XY and Q ( 2 ) ∈ ∆ Y Z we denote their composition into a kernel from X to Z by Q ( 2 ) ◦Q ( 1 ) ∈ ∆XZ . Given p ∈ ∆X and Q ∈ ∆XY we denote their composition into a joint probability distribution by p ∗ Q ∈ ∆X×Y , ( p ∗ Q ) ( x , y ) : = p ( x ) Q ( y|x ) . The support of a vector v ∈ RX is the set supp ( v ) = { x ∈ X : vx 6= 0 } . A partially observable Markov decision process or shortly POMDP is a tuple ( S , O , A , α , β , r ) . We assume that S , O and A are finite sets which we call the state , the observation and the action space respectively . We fix a Markov kernel α ∈ ∆S×AS which we call the transition mechanism and a kernel β ∈ ∆SO which we call the observation mechanism . Further , we consider an instantaneous reward vector r ∈ RS×A . We call the system fully observable if β = id1 , in which case the POMDP simplifies to a Markov decision process or shortly MDP . As policies we consider elements π ∈ ∆OA . Every policy π ∈ ∆OA defines a Markov kernel τ = τπ = π ◦ β ∈ ∆SA which we call the effective policy . Every policy defines transition kernels Pπ ∈ ∆ S×A S×A and pπ ∈ ∆SS through Pπ ( s ′ , a′|s , a ) : = α ( s′|s , a ) ( π ◦ β ) ( a′|s′ ) and pπ ( s′|s ) : = ∑ a∈A ( π ◦ β ) ( a|s ) α ( s′|s , a ) . For any initial state distribution µ ∈ ∆S , a policy π ∈ ∆OA defines a Markov process on S ×A with transition kernel Pπ which we denote by Pπ , µ . For a discount rate γ ∈ ( 0 , 1 ) and γ = 1 we define Rµγ ( π ) : = EPπ , µ [ ( 1− γ ) ∞∑ t=0 γtr ( st , at ) ] and Rµ1 ( π ) : = lim T→∞ EPπ , µ [ 1 T T−1∑ t=0 r ( st , at ) ] , called the expected discounted reward and the expected mean reward , respectively . For a policy π we define the value function V πγ ∈ RS via V πγ ( s ) : = Rδsγ ( π ) , s ∈ S , where δs is the Dirac distribution concentrated at s. A short calculation ( Zahavy et al. , 2021 ) shows that Rµγ ( π ) = ∑ s , a r ( s , a ) ηπ , µγ ( s , a ) = 〈r , ηπ , µγ 〉S×A , where ηπ , µγ ( s , a ) : = { ( 1− γ ) ∑∞ t=0 γ tPπ , µ ( st = s , at = a ) , if γ ∈ ( 0 , 1 ) limT→∞ 1 T ∑T−1 t=0 Pπ , µ ( st = s , at = a ) , if γ = 1 . ( 1 ) Here ηπ , µγ is an element of ∆S×A called the expected ( discounted ) state-action frequency ( Derman , 1970 ) , ( discounted ) visitation/occupancy measure or on-policy distribution ( Sutton & Barto , 2018 ) . Denoting the state marginal of ηπ , µγ by ρ π , µ γ ∈ ∆S , have ηπ , µγ ( s , a ) = ρπ , µγ ( s ) ( π ◦ β ) ( a|s ) . We recall the following well-known facts . 1More generally , the system is fully observable if the supports of { β ( ·|s ) } s∈S are disjoint subsets of O . Proposition 1 ( Existence of state-action frequencies and rewards ) . Let ( S , O , A , α , β , r ) be a POMDP , γ ∈ ( 0 , 1 ] and µ ∈ ∆S . Then ηπ , µγ , ρπ , µγ and Rµγ ( π ) exist for every π ∈ ∆OA and µ ∈ ∆S and are continuous in γ ∈ ( 0 , 1 ] for fixed π and µ . For γ = 1 we work under the following standard assumption in the ( PO ) MDP literature2 . Assumption 2 ( Uniqueness of stationary disitributions ) . If γ = 1 , we assume that for any policy π ∈ ∆OA there exists a unique stationary distribution η ∈ ∆S×A of Pπ . The following proposition shows in particular that for any initial distribution µ , the infinite time horizon state-action frequency ηπ , µγ is the unique discounted stationary distribution of Pπ . Proposition 3 ( State-action frequencies are discounted stationary ) . Let ( S , O , A , α , β , r ) be a POMDP , γ ∈ ( 0 , 1 ] and µ ∈ ∆S . Then ηπ , µγ is the unique element in ∆S×A satisfying ηπ , µγ = γP T π η π , µ γ + ( 1− γ ) ( µ ∗ ( π ◦ β ) ) . ( 2 ) Further , ρπ , µγ is the unique element in ∆S satisfying ρ π , µ γ = γp T π ρ π , µ γ + ( 1− γ ) µ . Since the state-action frequencies satisfy this generalized stationarity condition , we sometimes refer to them as discounted stationary distributions and denote the set of all state-action frequencies in the fully and in the partially observable case by N µγ : = { ηπ , µγ ∈ ∆S×A | π ∈ ∆SA } and N µ , βγ : = { ηπ , µγ ∈ ∆S×A | π ∈ ∆OA } . We have seen that the expected cumulative reward function Rµγ : ∆ O A → R factorises according to ∆OA fβ−→ ∆SA Ψµγ−−→ N µ , βγ → R , π 7→ π ◦ β 7→ ηπ , µγ 7→ 〈r , ηπ , µγ 〉S×A . This is illustrated in Figure 1 . We make use of this decomposition in two different ways . First , in Section 3 we study the algebraic properties of the parametrization Ψµγ of the set of state-action frequencies N µ , βγ . In Section 4 we derive a description of N µ , βγ via polynomial inequalities .

Summary: This paper studies several properties of the landscape of memoryless planning in infinite-horizon POMDPs. The primary focus of this paper is to provide formal expression of (a) state-action frequencies and (b) cumulative reward functions given a set of memoryless stochastic policies. Specifically, they show that both quantities can be expressed as rational functions of polynomial equations. The degree of polynomials used is proportional to the number of possible internal states given a set of observations (Theorem 4). Then, the authors describes the set of feasible state-action frequencies as the solution set to a system of polynomial equations and inequalities (Theorem 16). Finally, they provide the upper bound for the number of critical points in the landscape of policy optimization (Section 5).