Multi-agent Deep FBSDE Representation For Large Scale  Stochastic Differential Games

1 INTRODUCTION . Stochastic differential games represent a framework for investigating scenarios where multiple players make decisions while operating in a dynamic and stochastic environment . The theory of differential games dates back to the seminal work of Isaacs ( 1965 ) studying two-player zero-sum dynamic games , with a first stochastic extension appearing in Kushner & Chamberlain ( 1969 ) . A key step in the study of games is obtaining the Nash equilibrium among players ( Osborne & Rubinstein , 1994 ) . A Nash equilibrium represents the solution of non-cooperative game where two or more players are involved . Each player can not gain benefit by modifying his/her own strategy given opponents equilibrium strategy . In the context of adversarial multi-objective games , the Nash equilibrium can be represented as a system of coupled Hamilton-Jacobi-Bellman ( HJB ) equations when the system satisfies the Markovian property . Analytic solutions exist only for few special cases . Therefore , obtaining the Nash equilibrium solution is usually done numerically , and this can become challenging as the number of states/agents increases . Despite extensive theoretical work , the algorithmic part has received less attention and mainly addresses special cases of differential games ( e.g. , Duncan & Pasik-Duncan ( 2015 ) ) , or suffers from the curse of dimensionality ( Kushner , 2002 ) . Nevertheless , stochastic differential games have a variety of applications including in robotics and autonomy , economics and management . Relevant examples include Mataramvura & Øksendal ( 2008 ) , which formulate portfolio management as a stochastic differential game in order to obtain a market portfolio that minimizes the convex risk measure of a terminal wealth index value , as well as Prasad & Sethi ( 2004 ) , who investigate optimal advertising spending in duopolistic settings via stochastic differential games . Reinforcement Learning ( RL ) aims in obtaining a policy which can generate optimal sequential decisions while interacting with the environment . Commonly , the policy is trained by collecting histories of states , actions , and rewards , and updating the policy accordingly . Multi-agent Reinforcement Learning ( MARL ) is an extension of RL where several agents compete in a common environment , which is a more complex task due to the interaction between several agents and the environment , as well as between the agents . One approach is to assume agents to be part of environment ( Tan , 1993 ) , but this may lead to unstable learning during policy updates ( Matignon et al. , 2012 ) . On the other hand , a centralized approach considers MARL through an augmented state and action system , reducing its training to that of single agent RL problem . Because of the combinatorial complexity , the centralized learning method can not scale to more than 10 agents ( Yang et al. , 2019 ) . Another method is centralized training and decentralized execute ( CTDE ) , however the challenge therein lies on how to decompose value function in the execute phase for value-based MARL . Sunehag et al . ( 2018 ) and Zhou et al . ( 2019 ) decompose the joint value function into a summation of individual value functions . Rashid et al . ( 2018 ) keep the monotonic trends between centralized and decentralized value functions by augmenting the summation non-linearly and designing a mixing network ( QMIX ) . Further modifications on QMIX include Son et al . ( 2019 ) ; Mahajan et al . ( 2019 ) . The mathematical formulation of a differential game leads to a nonlinear PDE . This motivates algorithmic development for differential games that combine elements of PDE theory with deep learning . Recent encouraging results ( Han et al. , 2018 ; Raissi , 2018 ) in solving nonlinear PDEs within the deep learning community illustrate the scalability and numerical efficiency of neural networks . The transition from a PDE formulation to a trainable neural network is done via the concept of a system of Forward-Backward Stochastic Differential Equations ( FBSDEs ) . Specifically , certain PDE solutions are linked to solutions of FBSDEs , and the latter can be solved using a suitably defined neural network architecture . This is known in the literature as the deep FBSDE approach . Han et al . ( 2018 ) ; Pereira et al . ( 2019 ) ; Wang et al . ( 2019b ) utilize various deep neural network architectures to solve such stochastic systems . However , these algorithms address single agent dynamical systems . Two-player zero-sum games using FBSDEs were initially developed in Exarchos et al . ( 2019 ) and transferred to a deep learning setting in Wang et al . ( 2019a ) . Recently , Hu ( 2019 ) brought deep learning into fictitious play to solve multi-agent non-zero-sum game , Han & Hu ( 2019 ) introduced the deep FBSDEs to a multi-agent scenario and the concept of fictitious play , furthermore , Han et al . ( 2020 ) gives the convergence proof . In this work we propose an alternative deep FBSDE approach to multi-agent non-cooperative differential games , aiming on reducing complexity and increasing the number of agents the framework can handle . The main contribution of our work is threefold : 1 . We introduce an efficient Deep FBSDE framework for solving stochastic multi-agent games via fictitious play that outperforms the current state of the art in Relative Square Error ( RSE ) and runtime/memory efficiency on an inter-bank lending/borrowing example . 2 . We demonstrate that our approach scales to a much larger number of agents ( up to 1,000 agents , compared to 50 in existing work ) . To the best of our knowledge , this represents a new state of the art . 3 . We showcase the applicability of our framework to robotics on a belief-space autonomous racing problem which has larger individual control and state space . The experiments demonstrates that the decoupled BSDE provides the possibility of applications for competitive scenario . The rest of the paper is organized as follows : in Section 2 we present the mathematical preliminaries . In Section 3 we introduce the Deep Fictitious Play Belief FBSDE , with simulation results following in Section 4 . We conclude the paper and discuss some future directions in Section 5 . 2 MULTI-AGENT FICTITIOUS PLAY FBSDE . Fictitious play is a learning rule first introduced in Brown ( 1951 ) where each player presumes other players ’ strategies to be fixed . AnN -player game can then be decoupled intoN individual decisionmaking problems which can be solved iteratively over M stages . When each agent1 converges to a stationary strategy at stage m , this strategy will become the stationary strategy for other players at stage m+ 1 . We consider a N -player non-cooperative stochastic differential game with dynamics dX ( t ) = ( f ( X ( t ) , t ) +G ( X ( t ) , t ) U ( t ) ) dt+ Σ ( X ( t ) , t ) dW ( t ) , X ( 0 ) = X0 , ( 1 ) where X = ( x1 , x2 , . . . , xN ) is a vector containing the state process of all agents generated by their controls U = ( u1 , u2 , . . . , uN ) with xi ∈ Rnx and ui ∈ Rnu . Here , f : Rnx × [ 0 , T ] → Rnx represents the drift dynamics , G : Rnx × [ 0 , T ] → Rnx×nu represents the actuator dynamics , and Σ : [ 0 , T ] ×Rn → Rnx×nw represents the diffusion term . We assume that each agent is only driven by its own controls soG is a block diagonal matrix withGi corresponding to the actuation of agent i . 1Agent and player are used interchangeably in this paper Each agent is also driven by its own nw-dimensional independent Brownian motion Wi , and denote W = ( W1 , W2 , . . . , WN ) . Let Ui be the set of admissible strategies for agent i ∈ I : = { 1 , 2 , . . . , N } and U = ⊗Ni=1Ui as the Kronecker product space of Ui . Given the other agents ’ strategies , the stochastic optimal control problem for agent i under the fictitious play assumption is defined as minimizing the expectation of the cumulative cost functional J it J it ( X , ui , m ; u−i , m−1 ) = E [ g ( X ( T ) ) + ∫ T t Ci ( X ( τ ) , ui , m ( X ( τ ) , τ ) , τ ; u−i , m−1 ) dτ ] , ( 2 ) where g : Rnx → R+ is the terminal cost , and Ci : [ 0 , T ] × Rnx × U → R+ is the running cost for the i-th player . In this paper we assume that the running cost is of the form C ( X , ui , m , t ) = q ( X ) + 1 2u T i , mRui , m + X TQui , m . We use the double subscript ui , m to denote the control of agent i at stage m and the negative subscript−i as the strategies excluding player i , u−i = ( u1 , . . . , ui−1 , ui+1 , . . . , uN ) . We can define value function of each player as V i ( t , X ( t ) ) = inf ui , m∈Ui [ J it ( X , ui , m ; u−i , m−1 ) ] , V i ( T , X ( T ) ) = g ( X ( T ) ) . ( 3 ) Assume that the value function in eq . ( 3 ) is once differentiable w.r.t . t and twice differentiable w.r.t . x . Then , standard stochastic optimal control theory leads to the HJB PDE V i + h+ V iTx ( f +GU0 , −i ) + 1 2 tr ( V ixxΣΣ T ) = 0 , V i ( T , X ) = g ( X ( T ) ) , ( 4 ) where h = Ci∗ + GU∗,0 . The double subscript of U∗,0 denotes the augmentation of the optimal control u∗i , m = −R−1 ( GTi V ix + QTi x ) and zero control u−i , m−1 = 0 , and U0 , −i denotes the augmentation of ui , m = 0 and u−i , m−1 . Here we drop the functional dependencies in the HJB equation for simplicity . The detailed proof is in Appendix A . The value function in the HJB PDE can be related to a set of FBSDEs dX = ( f +GU∗ , −i ) dt+ ΣdW , X ( 0 ) = x0 dV i = − ( h+ V iTx GU∗,0 ) dt+ V Tx ΣdW , V ( T ) = g ( X ( T ) ) , ( 5 ) where the backward process corresponds to the value function . The detailed derivation can be found in Appendix B . Note that the FBSDEs here differ from that of Han & Hu ( 2019 ) in the optimal control of agent i , GU∗ , −i , in the forward process and compensation , V iTx GU∗,0 , in the backward process . This is known as the importance sampling for FBSDEs and allows for the FBSDEs to be guided to explore the state space more efficiently .

Summary: Till page 3 the paper was easy to follow, i.e., the analytical expressions in eq(5), and the basic idea of Algorithm 1 (which is same as prior works by Han et al. , Wang et al., Periera et al.) are clear.  However, after page 3 the paper is hard to follow. The specific points are as follows: