The Geometry of Memoryless Stochastic Policy Optimization in Infinite-Horizon POMDPs

1 INTRODUCTION . Markov decision processes ( MDPs ) were introduced by Bellman ( 1957 ) as a model for sequential decision making and optimal planning ( see , e.g. , Howard , 1960 ; Derman , 1970 ; Puterman , 2014 ) . Many algorithms in reinforcement learning rely on the ideas and methods developed in the context of MDPs ( see , e.g. , Sutton & Barto , 2018 ) . Often in practice , the decisions need to be made based only on incomplete information of the state of the system . This setting is modeled by partially observable Markov decision processes ( POMDPs ) introduced by Åström ( 1965 ) ( for a historical discussion see Monahan , 1982 ) , which have become an important model for planning under uncertainty . In this work we pursue a geometric characterization of the policy optimization problem in POMDPs over the class of stochastic memoryless policies and its dependence on the degree of partial observability . It is well known that acting optimally in POMDPs may require memory ( Åström , 1965 ) . A POMDP with unlimited memory policies can be modeled as a belief state MDP , where the states are replaced by probability distributions that serve as sufficient statistics for the previous observations ( Kaelbling et al. , 1998 ; Murphy , 2000 ) . Finding an optimal policy in this class is PSPACE-complete for finite horizons ( Papadimitriou & Tsitsiklis , 1987 ) and undecidable for infinite horizons ( Madani et al. , 2003 ; Chatterjee et al. , 2016 ) . Therefore , it is of interest to consider POMDPs with constrained policy classes . A natural class to consider are memoryless policies , also known as reactive or Markov policies , which select actions based solely on the current observations . In this case , it is useful to allow the actions to be selected stochastically , which not only allows for better solutions but also provides a continuous optimization domain ( Singh et al. , 1994 ) . Although they are more restrictive than policies with memory , memoryless policies are attractive as they are easier to optimize and are versatile enough for certain applications ( Tesauro , 1995 ; Loch & Singh , 1998 ; Williams & Singh , 1999 ; Kober et al. , 2013 ) . In fact , finite-memory policies can be modeled in terms of memoryless policies by supplementing the state of the system with an external memory ( Littman , 1993 ; Peshkin et al. , 1999 ; Icarte et al. , 2021 ) . Hence theoretical advances on memoryless policy optimization are also of interest to finite-memory policy optimization . Theoretical aspects and optimization strategies over the class of memoryless policies have been studied in numerous works ( see , e.g. , Littman , 1994 ; Singh et al. , 1994 ; Jaakkola et al. , 1995 ; Loch & Singh , 1998 ; Williams & Singh , 1999 ; Baxter et al. , 2000 ; Baxter & Bartlett , 2001 ; Li et al. , 2011 ; Azizzadenesheli et al. , 2018 ) . However , finding exact or approximate optimal memoryless stochastic policies for POMDPs is still considered an open problem ( Azizzadenesheli et al. , 2016 ) , which is NP-hard in general ( Vlassis et al. , 2012 ) . One reason for the difficulties in optimizing POMDPs is that , even in a tabular setting , the problem is non-convex and can exhibit suboptimal strict local optima ( Bhandari & Russo , 2019 ) . For memoryless policies the expected cumulative reward is a linear function of the corresponding ( discounted ) state-action frequencies . In the case of MDPs the feasible set of state-action frequencies is known to form a polytope , so that the optimization problem can be reduced to a linear program ( Manne , 1960 ; De Ghellinck , 1960 ; d ’ Epenoux , 1963 ; Hordijk & Kallenberg , 1981 ) . On the other hand , to the best of our knowledge , for POMDPs the specific structure of the feasibility constraints and the optimization problem have not been studied , at least not in the same level of detail ( see related works below ) . Related works In MDPs , the ( discounted ) state-action frequencies have been shown to form a polytope resp . a compact convex set in the finite resp . countable state-action cases , whereby the extreme points are given by the state-action frequencies of deterministic stationary policies ( Derman , 1970 ; Altman & Shwartz , 1991 ) . Further , Dadashi et al . ( 2019 ) showed that the set of state value functions in finite state-action MDPs is a finite union of polytopes . The set of stationary state-action distributions of POMDPs has been studied by Montúfar et al . ( 2015 ) highlighting a decomposition into infinitely many convex subsets whose dimensions depend on the degree of observability . Although this decomposition can be used to localize optimal policies to some extent , a description of the pieces in combination is still missing , needed to capture the properties of the optimization problem . We will obtain a detailed description in terms of finitely many polynomial constraints with closed form expressions and bound their degrees in terms of the observation mechanism . This yields the formulation of reward maximization in infinite-horizon POMDPs as a polynomial optimization problem , where Cohen & Parmentier ( 2018 ) obtained a description of finite-horizon POMDPs as a mixed integer linear program . Grinberg & Precup ( 2013 ) showed that the expected mean reward is a rational function and obtained bounds on the degree of this function . We generalize this to the setting of discounted rewards and refine the result by relating the rational degree to the degree of observability . For both MDPs and POMDPs the expected cumulative reward is known to be a non-convex function of the policy even for tabular policy models ( Bhandari & Russo , 2019 ) . Nonetheless , for MDPs critical points can be shown to be global maxima under mild conditions . In contrast , for POMDPs or MDPs with linearly restricted policy models , it is known that non-global local optimizers can exist ( Baxter et al. , 2000 ; Poupart et al. , 2011 ; Bhandari & Russo , 2019 ) . However , nothing is known about the number of local optimizers . We will present bounds on the number of critical points building on our computation of the rational degree and feasibility constraints . The structure of the expected cumulative reward has been studied in terms of the location of the global optimizers and the existence of local optimizers . Most notably , it is well known that in MDPs there always exist optimal policies which are memoryless and deterministic ( see Puterman , 2014 ) . In the case of POMDPs , optimal memoryless policies may need to be stochastic ( see Singh et al. , 1994 ) . Montúfar et al . ( 2015 ) ; Montúfar & Rauh ( 2017 ) ; Montúfar et al . ( 2019 ) obtained upper bounds on the number of actions that need to be randomized by these policies , which in the worst case is equal to the number of states that are compatible with the observation . Although we do not improve these results ( which are indeed tight in some cases ) , our description of the expected cumulative reward function leads to a simpler proof of the bounds obtained by Montúfar et al . ( 2015 ) . Neyman ( 2003 ) considered stochastic games as semialgebraic problems showing that the minmax and maxmin payoffs in an n-player game are semialgebraic functions of the discount factor . Although this is not directly related to our work , we take a similar philosophy . We pursue a semialgebraic description of the feasible set of discounted state-action frequencies in POMDPs , which is closely related to the general spirit of semialgebraic statistics ( Zwiernik , 2016 ) . Based on this we characterize the properties of the optimization problem by its algebraic degree , a concept that has been advanced in recent works on polynomial optimization ( Bajaj , 1988 ; Catanese et al. , 2006 ; Nie & Ranestad , 2009 ; Draisma et al. , 2016 ; Özlüm Çelik et al. , 2021 ) . Contributions We obtain results for infinite-horizon POMDPs with memoryless stochastic policies under the mean or discounted reward criteria which can be summarized as follows . 1 . We show that the state-action frequencies and the expected cumulative reward can be written as fractions of determinantal polynomials in the entries of the stochastic policy matrix . We show that the degree of these polynomials is directly related to the degree of observability ( see Theorem 4 ) . 2 . We describe the set of feasible state-action frequencies as a basic semialgebraic set , i.e. , as the solution set to a system of polynomial equations and inequalities , for which we also derive closed form expressions ( see Theorem 16 and Remark 18 ) . 3 . We reformulate the expected cumulative reward optimization problem as the optimization of a linear function subject to polynomial constraints ( see Remark 19 ) , which we use to solve a navigation problem in a grid world ( see Appendix F ) . This is a POMDP generalization of the dual linear programming formulation of MDPs ( Kallenberg , 1994 ; Puterman , 2014 ) . 4 . We present two methods for computing the number of critical points , which rely , respectively , on the rational degree of the expected cumulative reward function and the geometric description of the feasible set of state-action frequencies ( see Theorem 20 , Proposition 21 and Appendix D ) . 2 PRELIMINARIES . We use standard notation for probability distributions and Markov kernels . We denote the simplex of probability distributions on a finite set X by ∆X . An element µ ∈ ∆X is a vector in RX with non-negative entries µx = µ ( x ) , x ∈ X adding to one . We denote the set of Markov kernels from a finite set X to another finite set Y by ∆XY . An element Q ∈ ∆XY is a |X | × |Y| row stochastic matrix with entries Qxy = Q ( y|x ) , x ∈ X , y ∈ Y . Given Q ( 1 ) ∈ ∆XY and Q ( 2 ) ∈ ∆ Y Z we denote their composition into a kernel from X to Z by Q ( 2 ) ◦Q ( 1 ) ∈ ∆XZ . Given p ∈ ∆X and Q ∈ ∆XY we denote their composition into a joint probability distribution by p ∗ Q ∈ ∆X×Y , ( p ∗ Q ) ( x , y ) : = p ( x ) Q ( y|x ) . The support of a vector v ∈ RX is the set supp ( v ) = { x ∈ X : vx 6= 0 } . A partially observable Markov decision process or shortly POMDP is a tuple ( S , O , A , α , β , r ) . We assume that S , O and A are finite sets which we call the state , the observation and the action space respectively . We fix a Markov kernel α ∈ ∆S×AS which we call the transition mechanism and a kernel β ∈ ∆SO which we call the observation mechanism . Further , we consider an instantaneous reward vector r ∈ RS×A . We call the system fully observable if β = id1 , in which case the POMDP simplifies to a Markov decision process or shortly MDP . As policies we consider elements π ∈ ∆OA . Every policy π ∈ ∆OA defines a Markov kernel τ = τπ = π ◦ β ∈ ∆SA which we call the effective policy . Every policy defines transition kernels Pπ ∈ ∆ S×A S×A and pπ ∈ ∆SS through Pπ ( s ′ , a′|s , a ) : = α ( s′|s , a ) ( π ◦ β ) ( a′|s′ ) and pπ ( s′|s ) : = ∑ a∈A ( π ◦ β ) ( a|s ) α ( s′|s , a ) . For any initial state distribution µ ∈ ∆S , a policy π ∈ ∆OA defines a Markov process on S ×A with transition kernel Pπ which we denote by Pπ , µ . For a discount rate γ ∈ ( 0 , 1 ) and γ = 1 we define Rµγ ( π ) : = EPπ , µ [ ( 1− γ ) ∞∑ t=0 γtr ( st , at ) ] and Rµ1 ( π ) : = lim T→∞ EPπ , µ [ 1 T T−1∑ t=0 r ( st , at ) ] , called the expected discounted reward and the expected mean reward , respectively . For a policy π we define the value function V πγ ∈ RS via V πγ ( s ) : = Rδsγ ( π ) , s ∈ S , where δs is the Dirac distribution concentrated at s. A short calculation ( Zahavy et al. , 2021 ) shows that Rµγ ( π ) = ∑ s , a r ( s , a ) ηπ , µγ ( s , a ) = 〈r , ηπ , µγ 〉S×A , where ηπ , µγ ( s , a ) : = { ( 1− γ ) ∑∞ t=0 γ tPπ , µ ( st = s , at = a ) , if γ ∈ ( 0 , 1 ) limT→∞ 1 T ∑T−1 t=0 Pπ , µ ( st = s , at = a ) , if γ = 1 . ( 1 ) Here ηπ , µγ is an element of ∆S×A called the expected ( discounted ) state-action frequency ( Derman , 1970 ) , ( discounted ) visitation/occupancy measure or on-policy distribution ( Sutton & Barto , 2018 ) . Denoting the state marginal of ηπ , µγ by ρ π , µ γ ∈ ∆S , have ηπ , µγ ( s , a ) = ρπ , µγ ( s ) ( π ◦ β ) ( a|s ) . We recall the following well-known facts . 1More generally , the system is fully observable if the supports of { β ( ·|s ) } s∈S are disjoint subsets of O . Proposition 1 ( Existence of state-action frequencies and rewards ) . Let ( S , O , A , α , β , r ) be a POMDP , γ ∈ ( 0 , 1 ] and µ ∈ ∆S . Then ηπ , µγ , ρπ , µγ and Rµγ ( π ) exist for every π ∈ ∆OA and µ ∈ ∆S and are continuous in γ ∈ ( 0 , 1 ] for fixed π and µ . For γ = 1 we work under the following standard assumption in the ( PO ) MDP literature2 . Assumption 2 ( Uniqueness of stationary disitributions ) . If γ = 1 , we assume that for any policy π ∈ ∆OA there exists a unique stationary distribution η ∈ ∆S×A of Pπ . The following proposition shows in particular that for any initial distribution µ , the infinite time horizon state-action frequency ηπ , µγ is the unique discounted stationary distribution of Pπ . Proposition 3 ( State-action frequencies are discounted stationary ) . Let ( S , O , A , α , β , r ) be a POMDP , γ ∈ ( 0 , 1 ] and µ ∈ ∆S . Then ηπ , µγ is the unique element in ∆S×A satisfying ηπ , µγ = γP T π η π , µ γ + ( 1− γ ) ( µ ∗ ( π ◦ β ) ) . ( 2 ) Further , ρπ , µγ is the unique element in ∆S satisfying ρ π , µ γ = γp T π ρ π , µ γ + ( 1− γ ) µ . Since the state-action frequencies satisfy this generalized stationarity condition , we sometimes refer to them as discounted stationary distributions and denote the set of all state-action frequencies in the fully and in the partially observable case by N µγ : = { ηπ , µγ ∈ ∆S×A | π ∈ ∆SA } and N µ , βγ : = { ηπ , µγ ∈ ∆S×A | π ∈ ∆OA } . We have seen that the expected cumulative reward function Rµγ : ∆ O A → R factorises according to ∆OA fβ−→ ∆SA Ψµγ−−→ N µ , βγ → R , π 7→ π ◦ β 7→ ηπ , µγ 7→ 〈r , ηπ , µγ 〉S×A . This is illustrated in Figure 1 . We make use of this decomposition in two different ways . First , in Section 3 we study the algebraic properties of the parametrization Ψµγ of the set of state-action frequencies N µ , βγ . In Section 4 we derive a description of N µ , βγ via polynomial inequalities .

Summary: This paper studies the problem of finding the best memoryless stochastic policy for an infinite-horizon partially observable Markov decision process (POMDP). It is shown that the (discounted) state-action frequencies and the expected cumulative reward are rational functions of the policy, and the degree is determined by the degree of partial observability. The problem is then formulated as a linear optimization with polynomial constraints. The authors then demonstrate how the partial observability constraints can lead to multiple smooth and non-smooth local optimizers and we estimate the number of critical points.