Towards General Function Approximation in Zero-Sum Markov Games

√ d factor in the regret when the reward function and transition kernel are parameterized with d-dimensional linear features . In the coordinated setting where both players are controlled by the agent , we propose a model-based algorithm and a model-free algorithm . In the model-based algorithm , we prove that sample complexity can be bounded by a generalization of Witness rank to Markov games . The model-free algorithm enjoys a √ K-regret upper bound where K is the number of episodes . 1 INTRODUCTION . In competitive reinforcement learning , there are two agents competing against each other by taking actions . Their actions together determine the state evolutions and rewards . Function approximation , especially deep neural networks ( LeCun et al. , 2015 ) , contributes to the success of RL in many real world applications , such as Atari ( Mnih et al. , 2013 ) , Go ( Silver et al. , 2015 ) , autonomous driving ( Shalev-Shwartz et al. , 2016 ) , Texas holdem poker ( Sandholm , 2017 ) , and Dota ( Berner et al. , 2019 ) . The goal of competitive reinforcement learning aims to learn the Nash Equilibrium ( NE ) policy in a trial-and-error fashion . In a NE , no agent can be better off by unilaterally deviating from her policy . Most of the existing sample efficient competitive RL algorithms focus on the tabular or linear function approximation cases ( Pérolat et al. , 2017 ; Bai & Jin. , 2020 ; Bai et al. , 2021 ; 2020 ; Xie et al. , 2020 ; Zhang et al. , 2020 ) . The huge empirical success of neural network based RL methods has remained largely unexplained . Thus , there exists a wide gap between theory and practice in competitive RL with general function approximation . One demanding question to ask is : Can we establish provably efficient competitive RL algorithms with general function approximation ? In this paper , we make a step towards answering this question by providing structural conditions and complexity measures of Markov Games and function classes that allow for efficient learning . We focus on episodic zero-sum Markov Game ( MG ) with simultaneous move , where each episode consists of H time steps and two players act simultaneously at each time step . The state space is arbitrarily large and can be infinite . We consider two function approximation settings : 1 ) use a general function class F to approximate the action-value function ( Q function ) ; 2 ) use a general function classM to approximate the environment . To this end , we introduce algorithms based on optimistic principles . Due to subtle game-theoretic issues , naive optimism is no longer working in Markov games where minimax optimization is performed . To deal with this issue , we use the algorithmic idea called ‘ alternate optimism ’ . Specifically , we develop algorithms for both coordinated and decoupled setting . In the decoupled setting , the agent controls one player and plays against an arbitrary and potentially adversarial opponent . The sample efficiency is measured by the gap between the learned value and value of the Nash Equilibrium . In the coordinated setting , the agent controls both players and the goal is to find the approximate Nash Equilibrium , i.e . with small duality gap . We identify key complexity measures of function classes to examine the effectiveness of elimination . By doing so , we prove upper bounds on sample complexity and regrets of the presented procedures that are independent of the number of states . Our contributions are summarized into the following two folds : • In the decoupled setting , we introduce Minimax Eluder dimension–a new complexity measure for competitive RL problems . We propose an algorithm that incurs at most Õ ( H √ dEK logNF ) 1 regret in K episodes where dE denotes the Minimax Eluder di- mension NF denotes the covering number of function class , with probability at least 1− p. As a special case , this result improves Xie et al . ( 2020 ) by a √ d multiplicative factor in their setting when the reward function and transition kernel are linearly parameterized and d is the dimension of feature mapping . • In coordinated settings , we propose both model-based and model free algorithms . In the model-based setting , we generalize the witness rank ( Sun et al. , 2019a ) to competitive RL . We prove that Õ ( H3W 2/ϵ2 ) samples are enough to learn a policy ϵ-close to the Nash Equilibrium , where W is witness rank . In the model-free setting , we develop algorithm for agnostic learning with a candidate policy class Π . The algorithm incurs at most Õ ( H √ dK log ( NFNΠ ) ) regret . Here d is a variant of Minimax Eluder dimension and NΠ denotes the covering number of policy class . 1.1 RELATED WORKS . There is a rich literature studying the learning and decision-making of Markov Games ( Littman & Szepesvari , 1996 ; Greenwald et al. , 2003 ; Grau-Moya et al. , 2018 ; Pérolat et al. , 2018 ; Srinivasan et al. , 2018 ; Sidford et al. , 2020 ; Wei et al. , 2017 ; Pérolat et al. , 2017 ; Bai & Jin. , 2020 ; Bai et al. , 2021 ; 2020 ; Zhang et al. , 2020 ; Zhao et al. , 2021 ) . The most related to us are perhaps ( Xie et al. , 2020 ; Chen et al. , 2021 ; Jin et al. , 2021b ) , where the authors address the challenge of explorationexploitation tradeoff in large state spaces . Due to space constraint , a detailed literature discussion is deferred to Appendix A . 1.2 TECHNICAL CHALLENGES . Previous work ( Xie et al. , 2020 ) imposes optimistic bonus on the action-value functions in every stateaction pairs and performs planning by the Coarse Correlated Equilibrium ( CCE ) on the optimistic value functions . To achieve improved rates , we leverage the idea of ‘ global optimism ’ ( Zanette et al. , 2020 ; Jin et al. , 2021a ; Du et al. , 2021 ) , which maintains a constraint set of candidate functions that do not deviate much from the empirical estimates and performs optimistic planning on the initial state . However , going beyond MDPs towards MGs , two problems arise . First , the concentration property of functions in constraint set is hard to characterize due to multi-agent interplay . For this , we use the concentration methods in Jin et al . ( 2021a ) and extend it from MDPs to MGs . The second and more prominent issue is the exploration and exploitation tradeoff . Since ‘ global optimism ’ only obtains optimism along the trajectories of behaviour policies , it may not guarantee optimism on the trajectories of target policies ( i.e . NE ) . As a result , directly using CCE to plan will cause the duality gaps to diverge . To deal with this problem , we apply ‘ alternate optimism ’ to guide explorations , which was previously used in Wei et al . ( 2017 ) for model-based methods . The ‘ alternate optimism ’ used in this work is slightly different for value-based methods . We prove two regret decomposition lemmata to support this optimism principle . 2 PRELIMINARIES . We consider two-player zero-sum simultaneous-moves episodic Markov game , defined by the tuple ( S , A1 , A2 , r , P , H ) 1We use Õ to hide logarithmic terms in H , 1/p and K. where S is the state space , Ai is a finite set of actions that player i ∈ { 1 , 2 } can take , r is the reward function , P is the transition kernel and H is the number of time steps . At each time step h ∈ [ H ] , player P1 and P2 take actions a ∈ A1 and b ∈ A2 respectively upon observing the state x ∈ S , and then both receive the reward rh ( x , a , b ) . The system then transitions to a new state x′ ∼ Ph ( ·|x , a , b ) according to the transition kernel P. Throughout this paper , we assume for simplicity that A1 = A2 = A and that the rewards rh ( x , a , b ) are deterministic functions of the tuple ( x , a , b ) taking values in [ −1 , 1 ] . Turn-based games are special cases of simultaneous games in the sense that at each state the reward and transition are independent of one player ’ s action ( Xie et al. , 2020 ) . Generalizations to A1 ̸= A2 and stochastic reward are also straightforward . Denote by ∆ : = ∆ ( A ) the probability simplex over the action space A . A stochastic policy of P1 is a length-H sequence of functions π : = { πh : S 7→ ∆ } h∈ [ H ] . At each step h ∈ [ H ] and state x ∈ S , P1 takes an action sampled from the distribution πh ( x ) over A . Similarly , a stochastic policy of P2 is given by the sequence ν : = { νh : S 7→ ∆ } h∈ [ H ] . The learning happens in K episodes . In each episode k , each player P1 and P2 proposes a policy πk and νk respectively based on history up to the end of episode k − 1 and then executes the policy to observe the trajectories { xkh , akh , bkh } Hh=1 . For a fixed policy pair ( π , ν ) , the value function and Q functions for zero-sum game is defined by V π , νh0 ( x ) : = E [ H∑ h=h0 rh ( xh , ah , bh ) |xh0 = x ] , Qπ , νh0 ( x , a , b ) : = E [ H∑ h=h0 rh ( xh , ah , bh ) |xh0 = x , ah0 = a , bh0 = b ] , where the expectation is over ah ∼ πh ( xh ) , bh ∼ νh ( xh ) and xh+1 ∼ Ph ( ·|xh , ah , bh ) . V π , ν1 and Qπ , ν1 are often abbreviated to V π , ν and Qπ , ν . In zero-sum games , PI aims to maximize V π , ν ( x1 ) and P2 aims to minimize V π , ν ( x1 ) . Given policy π of P1 , the best response policy of P2 is defined by ν∗π : = argminνV π , ν ( x1 ) . Similarly given policy ν of P2 , the best response policy of P1 is defined by π∗ν : = argmaxπV π , ν ( x1 ) . Then from definitions , V π , ν ∗ π ( x1 ) ≤ V π ∗ , ν∗ ( x1 ) ≤ V π ∗ ν , ν ( x1 ) and the equality holds for the Nash Equilibrium ( NE ) of the game ( π∗ , ν∗ ) . We abbreviate V π ∗ , ν∗ ( x1 ) as V ∗ and Qπ ∗ , ν∗ ( x1 ) as Q∗ . V ∗ is often referred to as the value of the game . As common in Markov games literature ( Pérolat et al. , 2015 ) , we will use the following Bellman operator ThQh+1 ( x , a , b ) : = r ( x , a , b ) + E x′∼Ph ( ·|x , a , b ) [ max π′ min ν′ V π ′ , ν′ h+1 ( x ′ ) ] , ( 1 ) and the following Bellman operator for fixed P1 ’ s policy T πh Qh+1 ( x , a , b ) : = r ( x , a , b ) + E x′∼Ph ( ·|x , a , b ) [ min ν′ V π , ν ′ h+1 ( x ′ ) ] . ( 2 ) Notations For a integer H , [ H ] denotes the set { 1 , 2 , . . . , H } . For a finite set S , |S| denotes its cardinality . For a matrix A ∈ Rd , Ai , ∗ and A∗ , j denote the i-th row and j-th column of A respectively . For a function f : S 7→ R , ∥f∥∞ denotes sups∈S |f ( s ) | . We use N ( F , ϵ ) to denote the ϵ-covering number of F under metric d ( f , g ) = maxh ∥fh − gh∥∞ .

Summary: This paper presents sample efficient algorithms for learning in two-player zero-sum markov games. The algorithms are for decoupled and coordinated settings, the latter of which is based on 'alternate optimism'. The authors also extend the Eluder dimension of MDPs to to zero-sum markov games using the minimax Bellman operator.