Towards General Function Approximation in Zero-Sum Markov Games

√ d factor in the regret when the reward function and transition kernel are parameterized with d-dimensional linear features . In the coordinated setting where both players are controlled by the agent , we propose a model-based algorithm and a model-free algorithm . In the model-based algorithm , we prove that sample complexity can be bounded by a generalization of Witness rank to Markov games . The model-free algorithm enjoys a √ K-regret upper bound where K is the number of episodes . 1 INTRODUCTION . In competitive reinforcement learning , there are two agents competing against each other by taking actions . Their actions together determine the state evolutions and rewards . Function approximation , especially deep neural networks ( LeCun et al. , 2015 ) , contributes to the success of RL in many real world applications , such as Atari ( Mnih et al. , 2013 ) , Go ( Silver et al. , 2015 ) , autonomous driving ( Shalev-Shwartz et al. , 2016 ) , Texas holdem poker ( Sandholm , 2017 ) , and Dota ( Berner et al. , 2019 ) . The goal of competitive reinforcement learning aims to learn the Nash Equilibrium ( NE ) policy in a trial-and-error fashion . In a NE , no agent can be better off by unilaterally deviating from her policy . Most of the existing sample efficient competitive RL algorithms focus on the tabular or linear function approximation cases ( Pérolat et al. , 2017 ; Bai & Jin. , 2020 ; Bai et al. , 2021 ; 2020 ; Xie et al. , 2020 ; Zhang et al. , 2020 ) . The huge empirical success of neural network based RL methods has remained largely unexplained . Thus , there exists a wide gap between theory and practice in competitive RL with general function approximation . One demanding question to ask is : Can we establish provably efficient competitive RL algorithms with general function approximation ? In this paper , we make a step towards answering this question by providing structural conditions and complexity measures of Markov Games and function classes that allow for efficient learning . We focus on episodic zero-sum Markov Game ( MG ) with simultaneous move , where each episode consists of H time steps and two players act simultaneously at each time step . The state space is arbitrarily large and can be infinite . We consider two function approximation settings : 1 ) use a general function class F to approximate the action-value function ( Q function ) ; 2 ) use a general function classM to approximate the environment . To this end , we introduce algorithms based on optimistic principles . Due to subtle game-theoretic issues , naive optimism is no longer working in Markov games where minimax optimization is performed . To deal with this issue , we use the algorithmic idea called ‘ alternate optimism ’ . Specifically , we develop algorithms for both coordinated and decoupled setting . In the decoupled setting , the agent controls one player and plays against an arbitrary and potentially adversarial opponent . The sample efficiency is measured by the gap between the learned value and value of the Nash Equilibrium . In the coordinated setting , the agent controls both players and the goal is to find the approximate Nash Equilibrium , i.e . with small duality gap . We identify key complexity measures of function classes to examine the effectiveness of elimination . By doing so , we prove upper bounds on sample complexity and regrets of the presented procedures that are independent of the number of states . Our contributions are summarized into the following two folds : • In the decoupled setting , we introduce Minimax Eluder dimension–a new complexity measure for competitive RL problems . We propose an algorithm that incurs at most Õ ( H √ dEK logNF ) 1 regret in K episodes where dE denotes the Minimax Eluder di- mension NF denotes the covering number of function class , with probability at least 1− p. As a special case , this result improves Xie et al . ( 2020 ) by a √ d multiplicative factor in their setting when the reward function and transition kernel are linearly parameterized and d is the dimension of feature mapping . • In coordinated settings , we propose both model-based and model free algorithms . In the model-based setting , we generalize the witness rank ( Sun et al. , 2019a ) to competitive RL . We prove that Õ ( H3W 2/ϵ2 ) samples are enough to learn a policy ϵ-close to the Nash Equilibrium , where W is witness rank . In the model-free setting , we develop algorithm for agnostic learning with a candidate policy class Π . The algorithm incurs at most Õ ( H √ dK log ( NFNΠ ) ) regret . Here d is a variant of Minimax Eluder dimension and NΠ denotes the covering number of policy class . 1.1 RELATED WORKS . There is a rich literature studying the learning and decision-making of Markov Games ( Littman & Szepesvari , 1996 ; Greenwald et al. , 2003 ; Grau-Moya et al. , 2018 ; Pérolat et al. , 2018 ; Srinivasan et al. , 2018 ; Sidford et al. , 2020 ; Wei et al. , 2017 ; Pérolat et al. , 2017 ; Bai & Jin. , 2020 ; Bai et al. , 2021 ; 2020 ; Zhang et al. , 2020 ; Zhao et al. , 2021 ) . The most related to us are perhaps ( Xie et al. , 2020 ; Chen et al. , 2021 ; Jin et al. , 2021b ) , where the authors address the challenge of explorationexploitation tradeoff in large state spaces . Due to space constraint , a detailed literature discussion is deferred to Appendix A . 1.2 TECHNICAL CHALLENGES . Previous work ( Xie et al. , 2020 ) imposes optimistic bonus on the action-value functions in every stateaction pairs and performs planning by the Coarse Correlated Equilibrium ( CCE ) on the optimistic value functions . To achieve improved rates , we leverage the idea of ‘ global optimism ’ ( Zanette et al. , 2020 ; Jin et al. , 2021a ; Du et al. , 2021 ) , which maintains a constraint set of candidate functions that do not deviate much from the empirical estimates and performs optimistic planning on the initial state . However , going beyond MDPs towards MGs , two problems arise . First , the concentration property of functions in constraint set is hard to characterize due to multi-agent interplay . For this , we use the concentration methods in Jin et al . ( 2021a ) and extend it from MDPs to MGs . The second and more prominent issue is the exploration and exploitation tradeoff . Since ‘ global optimism ’ only obtains optimism along the trajectories of behaviour policies , it may not guarantee optimism on the trajectories of target policies ( i.e . NE ) . As a result , directly using CCE to plan will cause the duality gaps to diverge . To deal with this problem , we apply ‘ alternate optimism ’ to guide explorations , which was previously used in Wei et al . ( 2017 ) for model-based methods . The ‘ alternate optimism ’ used in this work is slightly different for value-based methods . We prove two regret decomposition lemmata to support this optimism principle . 2 PRELIMINARIES . We consider two-player zero-sum simultaneous-moves episodic Markov game , defined by the tuple ( S , A1 , A2 , r , P , H ) 1We use Õ to hide logarithmic terms in H , 1/p and K. where S is the state space , Ai is a finite set of actions that player i ∈ { 1 , 2 } can take , r is the reward function , P is the transition kernel and H is the number of time steps . At each time step h ∈ [ H ] , player P1 and P2 take actions a ∈ A1 and b ∈ A2 respectively upon observing the state x ∈ S , and then both receive the reward rh ( x , a , b ) . The system then transitions to a new state x′ ∼ Ph ( ·|x , a , b ) according to the transition kernel P. Throughout this paper , we assume for simplicity that A1 = A2 = A and that the rewards rh ( x , a , b ) are deterministic functions of the tuple ( x , a , b ) taking values in [ −1 , 1 ] . Turn-based games are special cases of simultaneous games in the sense that at each state the reward and transition are independent of one player ’ s action ( Xie et al. , 2020 ) . Generalizations to A1 ̸= A2 and stochastic reward are also straightforward . Denote by ∆ : = ∆ ( A ) the probability simplex over the action space A . A stochastic policy of P1 is a length-H sequence of functions π : = { πh : S 7→ ∆ } h∈ [ H ] . At each step h ∈ [ H ] and state x ∈ S , P1 takes an action sampled from the distribution πh ( x ) over A . Similarly , a stochastic policy of P2 is given by the sequence ν : = { νh : S 7→ ∆ } h∈ [ H ] . The learning happens in K episodes . In each episode k , each player P1 and P2 proposes a policy πk and νk respectively based on history up to the end of episode k − 1 and then executes the policy to observe the trajectories { xkh , akh , bkh } Hh=1 . For a fixed policy pair ( π , ν ) , the value function and Q functions for zero-sum game is defined by V π , νh0 ( x ) : = E [ H∑ h=h0 rh ( xh , ah , bh ) |xh0 = x ] , Qπ , νh0 ( x , a , b ) : = E [ H∑ h=h0 rh ( xh , ah , bh ) |xh0 = x , ah0 = a , bh0 = b ] , where the expectation is over ah ∼ πh ( xh ) , bh ∼ νh ( xh ) and xh+1 ∼ Ph ( ·|xh , ah , bh ) . V π , ν1 and Qπ , ν1 are often abbreviated to V π , ν and Qπ , ν . In zero-sum games , PI aims to maximize V π , ν ( x1 ) and P2 aims to minimize V π , ν ( x1 ) . Given policy π of P1 , the best response policy of P2 is defined by ν∗π : = argminνV π , ν ( x1 ) . Similarly given policy ν of P2 , the best response policy of P1 is defined by π∗ν : = argmaxπV π , ν ( x1 ) . Then from definitions , V π , ν ∗ π ( x1 ) ≤ V π ∗ , ν∗ ( x1 ) ≤ V π ∗ ν , ν ( x1 ) and the equality holds for the Nash Equilibrium ( NE ) of the game ( π∗ , ν∗ ) . We abbreviate V π ∗ , ν∗ ( x1 ) as V ∗ and Qπ ∗ , ν∗ ( x1 ) as Q∗ . V ∗ is often referred to as the value of the game . As common in Markov games literature ( Pérolat et al. , 2015 ) , we will use the following Bellman operator ThQh+1 ( x , a , b ) : = r ( x , a , b ) + E x′∼Ph ( ·|x , a , b ) [ max π′ min ν′ V π ′ , ν′ h+1 ( x ′ ) ] , ( 1 ) and the following Bellman operator for fixed P1 ’ s policy T πh Qh+1 ( x , a , b ) : = r ( x , a , b ) + E x′∼Ph ( ·|x , a , b ) [ min ν′ V π , ν ′ h+1 ( x ′ ) ] . ( 2 ) Notations For a integer H , [ H ] denotes the set { 1 , 2 , . . . , H } . For a finite set S , |S| denotes its cardinality . For a matrix A ∈ Rd , Ai , ∗ and A∗ , j denote the i-th row and j-th column of A respectively . For a function f : S 7→ R , ∥f∥∞ denotes sups∈S |f ( s ) | . We use N ( F , ϵ ) to denote the ϵ-covering number of F under metric d ( f , g ) = maxh ∥fh − gh∥∞ .

Summary: This paper considers zero-sum Markov games in a general regime where the model is parameterized by general function classes. The goal of this research is to investigate reinforcement learning algorithms that learn a Nash policy in a trial-and-error fashion. The level of generality is achieved by introducing two complexity measures: The minimax Eluder dimension and the witness rank. The main result three algorithms with provable regret upper bounds involving covering numbers and the minimax Eluder dimension.