Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPs

1 Introduction . Many sequential decision-making problems in the real-world have objectives that can be naturally decomposed into a set of conditionally independent targets . Control of water reservoirs , energy consumption optimisation , market making , cloud computing allocation , sewage flow systems , and robotics are but a few examples [ 36 ] . While many optimisation methods have been proposed [ 25 , 34 ] — perhaps most prominently using Lagrangian scalarisation [ 46 ] — multi-agent learning has emerged as a promising new paradigm for sample-efficient learning [ 6 ] . In this class of algorithms , the multi-objective learning problem is cast into a centralised , co-operative stochastic game in which co-ordination is achieved through global coupling terms in each agent ’ s objective/reward functions . For example , a grocer who must manage their stock could be decomposed into a collection of subagents that each manage a single type of produce , but are subject to a global constraint on inventory . This approach has been shown to be very effective in a number of domains [ 20 , 50 , 30 , 24 , 52 ] , but presents both conceptual and technical issues . The transformation of a multi-objective Markov decision process ( MOMDP ) [ 36 ] into a stochastic game is a non-trivial design challenge . In many cases there is no clear delineation between agents in the new system , nor an established way of performing the decomposition . What ’ s more , it ’ s unclear in many domains that a multi-agent perspective is appropriate , even as a technical trick . For example , the concurrent problems studied by Silver et al . [ 39 ] exhibit great levels of homogeneity , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . lending themselves to the use of a shared policy which conditions on contextual information . The key challenge that we address in this paper is precisely how to scale these single-agent methods — specifically , policy gradients — in a principled way . As we shall see , this study reveals that existing methods in both single- and multi-agent multi-objective optimisation can be formulated as special cases of a wider family of algorithms we entitle factored policy gradients . The contributions of this paper are summarised below : 1 . We introduce influence networks as a framework for modelling probabilistic relationships between actions and objectives in an MOMDP , and show how they can be combined with policy factorisation via graph partitioning . 2 . We propose a new control variate — the factor baseline — that exploits independence structures within a ( factored ) influence network , and show how this gives rise to a novel class of algorithms to which we ascribe the name factored policy gradients . 3 . We show that FPGs generalise traditional policy gradient estimators and provide a common framework for analysing state-of-the-art algorithms in the literature including actiondependent baselines and counterfactual policy gradients . 4 . The variance properties of our family of algorithms are studied , and minimum factorisation is put forward as a principled way of applying FPGs , with theoretical results around the existence and uniqueness of the characterisation . 5 . The final contribution is to illustrate the effectiveness of our approach over traditional estimators on two high-dimensional benchmark domains . 1.1 Related Work . Policy gradients . Variance reduction techniques in the context of policy gradient methods have been studied for some time . The seminal work of Konda and Tsitsiklis [ 19 ] was one of the earliest works that identified the use of a critic as beneficial for learning . Since then , baselines ( or , control variates ) have received much attention . In 2001 , Weaver and Tao [ 53 ] presented the first formal analysis of their properties , and later Greensmith et al . [ 13 ] proved several key results around optimality . More recently , these techniques have been extended to include action-dependent baselines [ 47 , 22 , 12 , 57 , 9 ] , though the source of their apparent success has been questioned by some [ 49 ] who suggest that subtle implementation details were the true driver . It has also been shown that one can reduce variance by better accounting for the structure of the action-space , such as bounds [ 4 , 10 ] or more general topological properties [ 7 ] . The SVRPG approach of Papini et al . [ 29 ] also addresses variance concerns in policy gradients by leveraging advances in supervised learning , and the generalised advantage estimator of Schulman et al . [ 37 ] has been proposed as a method for reducing variance in actor-critic methods with fantastic empirical results ; both of these can be combined with baselines and the techniques we present in this work . Factorisation . In a related , but distinct line of work , factorisation has been proposed to better leverage the transition structure of MDPs ; see e.g . [ 2 , 15 , 44 ] . Indeed , the notion of causality has also been utilised in work by Jonsson and Barto [ 16 ] . Most recently , Oliehoek et al . [ 27 ] presented an elegant framework for harnessing the influence of other agents ( from the perspective of self ) in multi-agent systems . This approach is complementary to the work presented in this paper , and more recent extensions have significantly advanced the state-of-theart [ 41 , 5 , 28 ] ; we build upon these principles . There is also a long line of research on “ influence diagrams ” that is pertinent to this work . While the majority of this effort has been focused on dynamic programming , the ideas are very closely related to ours and indeed we see this work as a natural extension of these concepts [ 45 ] . Miscellaneous . Causal/graphical modelling has seen past applications in reinforcement learning [ 11 ] . Indeed , our proposed influence network is related to , but distinct from , the action influence models introduced by Madumal et al . [ 23 ] for explainability . There , the intention was to construct policies that can justify actions with respect to the observation space . Here , the intention was to exploit independence structure in MOMDPs for scalability and efficiency . 2 Background . A regular discrete-time Markov decision process ( MDP ) is a tupleM .= ( S , A , R , p , p0 ) , comprising : a state space S , action space A , and set of rewardsR ⊆ R. The dynamics of the MDP are driven by an initial state distribution such that s0 ∼ p0 ( · ) and a stationary transition kernel where ( rt , st+1 ) ∼ p ( · , · | st , at ) satisfies the Markov property , p ( rt , st+1 |ht ) = p ( rt , st+1 | st , at ) , for any history ht . = ( s0 , a0 , r0 , s1 , . . . , st , at ) . Given an MDP , a ( stochastic ) policy , parameterised by θ ∈ Rn , is a mapping πθ : S × Rn → P ( A ) from states and weights to the set of probability measures on A . The conditional probability density of an action a is denoted by πθ ( a | s ) . = P ( a ∈ da | s , θ ) and we assume throughout that πθ is continuously differentiable with respect to θ . For a given policy , the return starting from time t is defined as the discounted sum of future rewards , Gt . = ∑T k=0 γ krt+k+1 , where γ ∈ [ 0 , 1 ] is the discount rate and T is the terminal time [ 42 ] . Value functions express the expected value of returns generated from a given state or state-action pair under the MDP ’ s transition dynamics and policy π : that is , vπ ( s ) . = Eπ [ Gt | st = s ] and qπ ( s , a ) . = Eπ [ Gt | st = s , at = a ] . The objective in control is to find a policy that maximises vπ for all states with non-zero measure under p0 , denoted by the Lesbesgue integral J ( θ ) . = Ep0 [ vπθ ( s0 ) ] = ∫ S vπθ ( s0 ) dp0 ( s0 ) . 2.1 Policy Search . In this paper , we focus on policy gradient methods which optimise the parameters θ directly . This is achieved , in general , by performing gradient ascent on J ( θ ) , for which Sutton et al . [ 43 ] derived ∇θJ ( θ ) = Eπθ , ρπθ [ ( qπθ ( s , a ) − b ( s ) ) z ] , ( 1 ) where z .= ∇θ lnπθ ( a | s ) is the policy ’ s score vector , ρπθ ( s ) . = ∫ S ∑∞ t=0 γ tp ( st = s | ds0 , πθ ) denotes the ( improper ) discounted-ergodic occupancy measure , and b ( s ) is a state-dependent baseline ( or , control variate ) [ 33 ] . Here , p ( st = s | s0 , πθ ) is the probability of transitioning from s0 → s in t steps under πθ . Equation 1 is convenient for a number of reasons : 1. it is a score-based estimator [ 26 ] ; and 2. it falls under the class of stochastic approximation algorithms [ 1 ] . This is important as it means qπ ( s , a ) may be replaced by any unbiased quantity , say ψ : S × A → R , such that Eπ , ρπ [ ψ ( s , a ) ] = qπ ( s , a ) , while retaining convergence guarantees . It also implies that optimisation can be performed using stochastic gradient estimates , the standard variant of which is defined below . Definition 2.1 ( VPGs ) . The vanilla policy gradient estimator for target-baseline pair ( ψ , b ) is denoted gV ( s , a ) . = [ ψ ( s , a ) − b ( s ) ] z , ( 2 ) where ∇θJ ( θ ) = Eπθ , ρπθ [ gV ( s , a ) ] . 2.2 Factored ( Action-Space ) MDPs . In this paper , we consider the class of MDPs in which the action-space factors into a product , A .= ⊗n i=1Ai = A1 × · · · × An , for some n. This is satisfied trivially when n = 1 and A1 = A , but also holds in many common settings , such as A .= Rn , which factorises n times as ⊗n i=1 R. This is equivalent to requiring that actions , a ∈ A , admit a “ subscript ” operation ; without necessarily having A be a vector space . For example , one could have an action-space of the form A .= R× N such that , for any a ∈ A , a1 ∈ R and a2 ∈ N. To this end , we introduce the notion of partition maps which will feature throughout the paper . Definition 2.2 ( Partition Map ) . Define X .= ⊗n i=1 Xi and J ⊆ [ n ] with XJ . = ⊗ j∈J Xj such that a partition map ( PM ) for a pair ( X , J ) is a function σ : X → XJ with complement σ̄ : X → X [ n ] \J . Partition maps are an extension of the canonical projections of the product topology , and are equivalent to the scope operator used by Tian et al . [ 48 ] . For example , if ( a1 , a2 , a3 ) . = a ∈ A .= R3 denotes a three-dimensional real action-space , then one possible PM is given by σ ( a ) = ( a1 , a3 ) with complement σ̄ ( a ) = ( a2 ) . Note that there should always exist a unique inverse operation that recovers the original space ; in this case , it would be expressed as f ( ( a1 , a3 ) , ( a2 ) ) = ( a1 , a2 , a3 ) .

Summary: This paper proposes a policy gradient algorithm for MOMDPs with large factored action spaces, where the background knowledge on the MOMDP structure is given as an influence network. A new baseline for policy gradients is given that makes use of the MOMDP background knowledge and a factorisation of the RL policy. The paper analyses the favourable variance properties of the resulting policy gradient estimator. Empirical results are provided on a pedagogical bandit example, and a traffic control problem. The latter demonstrates applicability of the method even when the graphical assumptions only hold approximately.