Title: Multi-Agent Meta-Reinforcement Learning: Sharper Convergence Rates with Task Similarity
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement learning

game theory

multi-agent systems

meta-learning
Abstract: Multi-agent reinforcement learning (MARL) has primarily focused on solving a single task in isolation, while in practice the environment is often evolving, leaving many related tasks to be solved. In this paper, we investigate the benefits of meta-learning in solving multiple MARL tasks collectively. We establish the first line of theoretical results for meta-learning in a wide range of fundamental MARL settings, including learning Nash equilibria in two-player zero-sum Markov games and Markov potential games, as well as learning coarse correlated equilibria in general-sum Markov games. Under natural notions of task similarity, we show that meta-learning achieves provable sharper convergence to various game-theoretical solution concepts than learning each task separately. As an important intermediate step, we develop multiple MARL algorithms with initialization-dependent convergence guarantees. Such algorithms integrate optimistic policy mirror descents with stage-based value updates, and their refined convergence guarantees (nearly) recover the best known results even when a good initialization is unknown. To our best knowledge, such results are also new and might be of independent interest. We further provide numerical simulations to corroborate our theoretical findings.

Multi-Agent Meta-Reinforcement Learning: Sharper Convergence Rates with Task Similarity

Abstract

Multi-agent reinforcement learning (MARL) has primarily focused on solving a single task in isolation, while in practice the environment is often evolving, leaving many related tasks to be solved. In this paper, we investigate the benefits of meta-learning in solving multiple MARL tasks collectively. We establish the first line of theoretical results for meta-learning in a wide range of fundamental MARL settings, including learning Nash equilibria in two-player zero-sum Markov games and Markov potential games, as well as learning coarse correlated equilibria in general-sum Markov games. Under natural notions of task similarity, we show that meta-learning achieves provable sharper convergence to various game-theoretical solution concepts than learning each task separately. As an important intermediate step, we develop multiple MARL algorithms with initialization-dependent convergence guarantees. Such algorithms integrate optimistic policy mirror descents with stage-based value updates, and their refined convergence guarantees (nearly) recover the best known results even when a good initialization is unknown. To our best knowledge, such results are also new and might be of independent interest. We further provide numerical simulations to corroborate our theoretical findings.

1 Introduction

Many real-world sequential decision-making problems involve multiple agents interacting in a shared environment, a scenario commonly captured by game theory and addressed using multi-agent reinforcement learning (MARL). Existing research in MARL has primarily focused on solving a single task (i.e., a game) independently. In practice, however, one often needs to collectively solve a set of similar tasks due to the dynamically evolving environment. For example, in sponsored search auctions [48], the advertising spaces and search results are dynamic, and each bidder with an active bid will participate in a sequence of related auctions. In multi-robot cooperation [31, 26], the learning agents are often first pre-trained in simplified environments and are then asked to quickly adapt to more complicated ones. In cloud computing [53, 70], a learning-based autoscaling policy needs to achieve fast model adaptation to deal with varied application workloads or constantly evolving cloud infrastructures. All of these intriguing applications call for the development of intelligent multi-agent systems that can continuously build on previous experiences to enhance the learning of new tasks.

Meta-learning, or learning-to-learn [64, 56, 65, 58], is a rapidly developing approach that is particularly suitable for learning in a set of related tasks. In essence, meta-learning studies the use of data from existing tasks to learn representations or model parameters that enable quick adaptation to new tasks. By exploiting the knowledge obtained from prior tasks, the meta-learner can ideally solve an unseen task using much fewer training samples than learning from scratch, especially when the tasks share some inherent similarities. Despite many empirical successes [71, 31, 26], the theoretical results of meta-learning in multi-agent scenarios are still relatively lacking. It remains elusive whether meta-learning can provably expedite the convergence of MARL, and if so, what the proper task similarity assumptions to impose are. In fact, it is even unclear whether a meta-learner converges at all in a highly non-stationary system with loosely-coupled learning agents and diverse task setups.

In this paper, we make an initial attempt toward characterizing some of the central theoretical properties of meta-learning in a wide range of fundamental MARL settings. We focus on the classic model-agnostic meta-learning (MAML) [20] type of algorithms that aim to learn a good initialization for quick adaptation to new tasks. To study the convergence rate of MAML, an important prerequisite is to understand how the convergence of MARL algorithms depends on the quality of policy initialization. However, the convergence guarantees of most existing MARL algorithms are initialization-independent: They fail to track how the sub-optimality of the initial policy propagates during the learning process, and only provide pessimistic guarantees with respect to worst-case initialization. As a crucial intermediate step to meta-MARL, we need to establish refined initialization-dependent convergence guarantees for MARL. Our main contributions are thus summarized as follows.

Contributions. 1) For learning Nash equilibria (NE) in two-player zero-sum Markov games, we first propose an MARL algorithm blessed with a refined convergence analysis that explicitly characterizes the dependence on policy initialization (Section 3.1). Our algorithm runs optimistic online mirror descent for policy optimization and performs stage-based value function updates. Even when initialized with random policies, our algorithm still matches the best-known convergence rates in the literature except for an extra logarithmic term. Our algorithm and analysis appear to be new and might be of independent interest. 2) Based on such refined analysis, we show that meta-learning provably achieves faster convergence to NE when learning a sequence of 'similar' zero-sum games collectively, where our similarity metric naturally depends on the closeness of the games' NE policies (Section 3.2). 3) For learning NE in Markov potential games (MPGs), we show that a simple refinement of an existing algorithm suffices to provide initialization-dependent guarantees. We establish sharper convergence rates of meta-learning when the MPGs have similar potential functions (Section 4.1). In addition, with a properly chosen policy update rule, we prove the non-asymptotic convergence of the exact MAML algorithm in MPGs (Section 4.2), despite the convoluted learning dynamics of multiple loosely-coupled agents. 4) For learning coarse correlated equilibria (CCE) in general-sum Markov games (Section 5), we analogously start by designing an initialization-dependent MARL algorithm, and then establish the sharper convergence rate of meta-learning under natural similarity metrics. 5) We provide numerical results to corroborate our theoretical findings (Section 6).

Related Work. Gradient-based meta-learning is a simple and effective approach that can be easily applied to any learning problem trained with gradient descent. The seminal MAML method [20] tries to learn a good model parameter initialization that leads to quick model adaptation. Theoretical properties of MAML have been investigated in a series of works [54, 17, 66, 18, 30]. In particular, [17, 30] have established the convergence of MAML to first-order stationarity for non-convex objectives. [18] has designed an unbiased gradient estimator for MAML in reinforcement learning tasks. Various first-order approximations [20, 49, 17] of MAML have been proposed to avoid the heavy computation of the Hessian. Meta-learning has also been studied in online convex optimization [21, 4, 14, 35], where regret bounds have been established under different metrics of task similarity. Another line of research [29, 51, 41] views meta-learning through the lens of task inference, where an RL policy is conditioned on a belief over tasks and perform Bayesian updates through interactions to adapt to different tasks.

MARL has been widely studied under the formulation of stochastic games (i.e., Markov games) [57]. Due to the fundamental difficulty of computing NE in generic games [10], most MARL research has focused on learning NE in games with special structures (such as zero-sum Markov games [67, 2, 69, 3, 12, 68, 7, 72, 73] and Markov potential games [43, 38, 76, 25, 22, 15, 78]) or learning weaker solution concepts such as (coarse) correlated equilibria [40, 44, 59, 33, 47, 16, 13]. The most relevant works are [77, 72], which have studied the convergence of optimistic no-regret

learning and smooth value updates in MARL with full-information feedback. For learning NE in MPGs, [38, 76, 15] have studied independent policy gradient methods and established their sample complexity results. These works have focused on learning a single game in isolation but have not considered exploiting the connections between multiple games to expedite the learning process.

Most related to ours, [27] has studied meta-learning in normal-form games. Under different notions of game similarities, [27] has shown faster convergences of meta-learning in zero-sum, general-sum, and Stackelberg games. [75] has investigated no-regret learning in time-varying zero-sum normalform games. Compared to [27, 75], we consider meta-learning in the more generic and challenging Markov game setup with state transitions. Other related works include meta-learning for regret minimization in a distribution of games [61] and meta-safe RL for quick adaptation in constrained Markov decision processes (CMDPs) under task similarity [34]. Finally, meta-learning has also been empirically applied to many important MARL scenarios, including multi-intersection traffic signal control [71], multi-agent communication with natural language [26], and multi-agent collaboration with first-person pixel observations in open-ended tasks [63].

2 Preliminaries

Markov game. An N -player episodic Markov game is defined by a tuple G = ( N , H, S , {A i } N i =1 , { r i } N i =1 , P ) , where (1) N = { 1 , 2 , . . . , N } is the set of agents; (2) H ∈ N + is the number of time steps in each episode; (3) S is the finite state space; (4) A i is the finite action space for agent i ∈ N ; (5) r i : [ H ] ×S × A all → [0 , 1] is the reward function for agent i , where A all = × N i =1 A i is the joint action space; and (6) P : [ H ] ×S×A all → ∆( S ) is the transition kernel. The agents interact in an unknown environment for T episodes. Without loss of generality, we make a standard assumption [33, 59] that each episode starts from a fixed initial state s 1 . Our results can be easily generalized to the setting where the initial state is sampled from a fixed distribution. At each time step h ∈ [ H ] , the agents observe the state s h ∈ S , and take actions a h,i ∈ A i , i ∈ N simultaneously. Agent i then receives its reward r h,i ( s h , a h ) , where a h = ( a h, 1 , . . . , a h,N ) , and the environment transitions to the next state s h +1 ∼ P h ( ·| s h , a h ) . Let S = |S| , A i = |A i | , ∀ i ∈ N , and A max = max i ∈N A i .

Policy and Nash equilibrium. A (Markov) policy π i ∈ Π i : [ H ] ×S → ∆( A i ) for agent i ∈ N is a mapping from the time index and state space to a distribution over its own action space. Each agent seeks to find a policy that maximizes its own cumulative reward. A joint, product policy π = ( π 1 , . . . , π N ) ∈ Π induces a probability measure over the sequence of states and joint actions. We use the subscript -i to denote the set of agents excluding agent i , i.e., N\{ i } . We can rewrite π = ( π i , π -i ) using this convention. For a joint policy π , and for any h ∈ [ H ] , s ∈ S , and a ∈ A all , we define the value function and Q-function for agent i as

For agent i , a policy π † i is a best response to π -i if V π † i ,π -i 1 ,i ( s 1 ) = sup π i V π i ,π -i 1 ,i ( s 1 ) . A joint (product) policy π = ( π i , π -i ) ∈ Π is a Nash equilibrium (NE) if π i is a best response to π -i for all i ∈ N . Similarly, for any ε > 0 , a joint policy π = ( π i , π -i ) is an ε -approximate NE if V π i ,π -i 1 ,i ( s 1 ) ≥ V π † i ,π -i 1 ,i ( s 1 ) -ε , ∀ i ∈ N .

Correlated policy and coarse correlated equilibrium. We define π = { π h : R × ( S × A ) h -1 × S → ∆( A ) } h ∈ [ H ] as a (non-Markov) correlated policy , where for each h ∈ [ H ] , π h maps from a coordination device z ∈ R and a history of length h -1 to a distribution over the joint action space. Let π i and π -i be the proper marginal distributions of π whose outputs are restricted to ∆( A i ) and ∆( A -i ) , respectively. The value functions for non-Markov correlated policies at step h = 1 are defined in a similar way as for product policies. Given the PPAD-hardness of calculating NE in general [11], people often study a relaxed solution concept named coarse correlated equilibrium (CCE), which allows possible correlations in the policies: In particular, for any ε > 0 , a correlated

Two-player zero-sum Markov game. An important special case of Markov games is (two-player) zero-sum Markov games, where there are two players ( N = 2 ) with exactly opposite rewards

( r 1 = -r 2 ). In a zero-sum game, we simply use r, V , and Q to denote the reward and (Q-)value functions for the max-player, i.e., agent 1. Correspondingly, the min-player has -r, -V , and -Q . For notational convenience, we denote the action space for the max-player (resp. min-player) by A (resp. B ), and let A = |A| , B = |B| . We also write their policies ( π 1 , π 2 ) as ( µ, ν ) for short. In zero-sum games, it is known that although the NE policy ( µ ⋆ , ν ⋆ ) may not be unique, all the NE have the same values. We use V ⋆ h and Q ⋆ h to denote the NE value function and the NE Q-function. For any fixed ( h, s ) ∈ [ H ] ×S and an arbitrary function Q : S × A × B → R , we may consider Q ( s, · , · ) as an A × B matrix. Then, for any policy pair ( µ h , ν h ) at step h ∈ [ H ] , we can write in shorthand:

Given the transition function P and an arbitrary function V : S → R , we define

The Bellman equations can hence be rewritten more succinctly as

Markov potential game. Another important class of games is Markov potential games [43, 39, 76]. MPGs cover Markov teams [36], a fully cooperative setting where all agents share the same rewards. A Markov game is an MPG if there exists a global potential function Φ : Π ×S → [0 , Φ max ] that can capture the variations of the agents' individual values: Specifically, ∀ i ∈ N and s ∈ S ,

Throughout the paper, we consider the classic full-information feedback setting [23, 8, 62, 68, 7], where the players are assumed to have exact information of the consequences of each of their candidate actions. In the case of zero-sum games, this implies that for any ( h, s ) , the max-player and min-player can query [ Q h ν h ]( s, · ) and [ µ ⊤ h Q h ]( s, · ) , respectively. Our meta-learning results can be easily extended to the stochastic bandit feedback setting using standard techniques as in [3, 44, 33, 15].

Meta-learning. Let G = { G k } be a set of different Markov games. Each game is defined by G k = ( N , H, S , {A i } N i =1 , { r k i } N i =1 , P k ) , where we assume without loss of generality that the games share the same agent set and state & action spaces, but can have different transition and reward functions. Most of our results are established in the online learning setting where we encounter a sequence of K games ( G 1 , . . . , G K ) one by one. To achieve faster convergence, the learning agents should use the knowledge obtained from previous games to expedite the learning process in future games.

The underlying principle of MAML [20] is to learn a good initialization such that running a few training steps from this initialization can lead to well-performing model parameters on any new task. An MAML-type algorithm in the context of RL typically involves two nested stages. The inner stage (or 'base algorithm') ψ performs T iterations of policy updates to optimize for an individual task G k :

When task G k is completed, the outer stage (or 'meta-algorithm') Ψ learns to form a good initialization π k +1 , 0 for a new task G k +1 using all the knowledge obtained from all previous tasks:

In this paper, we seek to properly instantiate both the base algorithm ψ and the meta-algorithm Ψ for a variety of MARL problems. We aim to show that a proper design of the meta-learning procedure ( ψ, Ψ) can largely reduce the number of iterations T required to find NE or CCE in a new game.

3 Meta-Learning for Two-Player Zero-Sum Markov Games

In this section, we study meta-learning for Nash equilibria in zero-sum Markov games, where players are fully competitive. Since MAML-type algorithms seek to learn a good initialization for quick adaptation, it is crucial to explicitly characterize how the convergence behavior of an MARL algorithm depends on the initial policy. To our best knowledge, such results are not directly achievable using existing algorithms. For this reason, in Section 3.1, we start by proposing a new base algorithm (1) for zero-sum Markov games that has a refined initialization-dependent convergence guarantee. Based on that, we present our meta-algorithm (2) in Section 3.2 and establish its sharper convergence rates.

3.1 Initialization-Dependent Convergence in an Individual Zero-Sum Markov Game

Algorithm 1 presents our optimistic online mirror descent algorithm with stage-based value updates for learning NE in a zero-sum Markov game. To establish initialization-dependent convergence, Algorithm 1 performs optimistic online mirror descent (OMD) [55, 62] for policy updates (Lines 5 and 6), in contrast to the popular optimistic follow the regularized leader (FTRL) method in recent MARL policy optimization [77, 72]. We choose the negative entropy as our regularizer R , in which case the Bregman divergence D R ( · , · ) reduces to the Kullback-Leibler divergence and optimistic OMDbecomes an optimistic variant of the classic multiplicative weights update (MWU) algorithm.

Algorithm 1: Optimistic Online Mirror Descent for Zero-Sum Markov Games

In order to establish convergence to (approximate) NE, we need to show that our optimistic OMD policy updates achieve 'no regret' with respect to the value estimate sequence at each state, i.e., to upper bound (3). If we were to use the celebrated α t = H +1 H + t learning rate [32] to update the value function estimates, we will inevitably need to show a no-weighted-regret guarantee for optimistic OMD, because such a time-varying learning rate assigns non-uniform weights to each history step. However, incorporating OMD with a dynamic learning rate is known to be challenging and can easily lead to linear regret [50]. While a stabilization technique [19] has been introduced to tackle this challenge, we take a different route by resorting to an alternative value update method, namely stage-based value updates [79]. Specifically, we divide the total T iterations into multiple stages and only update our value estimates at the end of a stage (Line 9). We let the lengths of the stages grow exponentially at a rate of (1 + 1 /H ) (Line 8) [79, 45]. The exponential growth ensures that the total T iterations can be covered by a small number of stages, while the (1 + 1 /H ) growth rate guarantees that the value estimation error does not blow up during the H steps of recursion (Lemma 9). Compared with the incremental α t = H +1 H + t update rule that modifies the value estimates at every step, stage-based updates are more stationary and allow us to assign uniform weights to each history step. This leads to a simpler no(-average)-regret problem [47] that can be easily addressed by (optimistic) OMD.

We introduce a few notations before presenting the convergence analysis of Algorithm 1. Let τ ( t ) denote the index of the stage that iteration t belongs to. We denote by ¯ τ the total number of stages, i.e., ¯ τ := τ ( T ) . For any ( τ, h, s ) ∈ [¯ τ ] × [ H ] ×S , define the per-state regrets for the max-player as

The per-state regret reg τ h, 2 ( s ) for the min-player can be defined symmetrically (see (14) in Appendix B). We define the maximal regret (over the states and the two players) as reg τ h := max s ∈S max i =1 , 2 { reg τ h,i ( s ) } . An upper bound for the per-state regrets is provided in Lemma 8 of Appendix B, which is useful in the analysis of Algorithm 1. We use the standard notion of

to measure the optimality of a policy pair ( µ, ν ) . The initialization-dependent convergence rate of Algorithm 1 is as follows.

Theorem 1. If Algorithm 1 is run on a two-player zero-sum Markov game for T iterations with a learning rate η ≤ 1 / (8 H 2 ) , the output policy pair (¯ µ, ¯ ν ) satisfies:

In addition, if the players' policies are initialized to be uniform policies, i.e., ˜ µ τ h ( ·| s ) = 1 /A and ˜ ν τ h ( ·| s ) = 1 /B, ∀ s ∈ S , τ ∈ [¯ τ ] , h ∈ [ H ] , we further have

Compared to existing results [77, 72], Theorem 1 directly associates the convergence rate with the quality of the initial policy (˜ µ, ˜ ν ) . Even when a good policy initialization is unknown and the algorithm is initialized with uniformly random policies, our convergence rate in (4) still matches the best-known result in the literature [72] except for an extra factor of O (log T ) . When suppressing the logarithmic terms, Theorem 1 immediately implies that for any ε > 0 , Algorithm 1 takes no more than T = ˜ O ( H 5 /ε ) steps to learn an ε -approximate NE in an individual zero-sum Markov game.

3.2 Sharper Convergence with Meta-Learning

Having settled the initialization-dependent convergence in a zero-sum game, we proceed to show how meta-learning can learn a set of related games collectively and more rapidly. We consider an online setting with a sequence of K games G = ( G 1 , . . . , G K ) . For the max-player, let ˜ µ k and ¯ µ k , respectively, denote the initial policy and output policy of Algorithm 1 on game G k . By putting together µ τ, † h ( ·| s ) over all ( τ, h, s ) ∈ [¯ τ ] × [ H ] ×S , we let µ k, † : [¯ τ ] × [ H ] ×S → ∆( A ) denote the best fixed policies in hindsight on G k . Define ˜ ν k , ¯ ν k and ν k, † analogously for the min-player. Let µ ⋆ = 1 K ∑ K k =1 µ k, † and ν ⋆ = 1 K ∑ K k =1 ν k, † be the empirical averages of the best response policies. To ensure that the knowledge gained from previous games is useful for learning future tasks, we need to impose some similarity assumptions on the games G . We consider the following similarity metric:

Intuitively, since { ν k,t } t ∈ [ T ] converges to an equilibrium policy for G k when T is large, the best fixed responses µ k, † can be considered as an approximation of the max-player's NE policy on G k . In this sense, ∆ µ,ν essentially measures the distances between the NE policies of different games. It considers a set of games G to be 'similar' if their NE policies lie in a close neighborhood of each other. We remark that there might be multiple NE policies (with the same value) in a zero-sum game, and ∆ µ,ν only takes into account the NE policy pairs that Algorithm 1 actually delivers.

Our meta-learning procedure proceeds as follows: Within each game G k , we run Algorithm 1 as our base algorithm (1) to find a NE of G k . In a new game G k +1 , the initial policy of Algorithm 1 is given

by the following meta-updates in the outer loop (2), which essentially averages the best response policies of the previous tasks under α -greedy parameterization:

In particular, for any vector x ∈ R d , we define its α -greedy parameterization [ x ] α := (1 -α ) x + α d 1 to be a weighted average with a uniform vector 1 /d ∈ R d of a proper dimension, where α ∈ (0 , 1 / 2) . Since µ k, † denotes a set of vectors, we apply the operator [ · ] α element-wise to each of the vectors. The reason for using α -greedy is mainly technical: KL( ·∥· ) is not Lipschitz continuous near the boundary of the probability simplex, and α -greedy parameterization helps to stay α -distance away from the boundary. We are now ready to present our sharper convergence rates for meta-learning.

Theorem 2. In a sequence of K two-player zero-sum Markov games, if Algorithm 1 is run for T iterations as the base algorithm and (5) with α = 1 / √ K as the meta-updates, we have

Consequently, for any ε > 0 , T = ˜ O ( H 3 ε ( ∆ µ,ν K + A + B + H 2 √ K )) steps on average suffice to find an ε -approximate Nash equilibrium in each game.

When the number of games K is large, the last two terms on the RHS of (6) become negligible. Hence, compared to the best-known results ˜ O ( H 5 /T ) of learning each game individually, Theorem 2 implies a significantly sharper convergence rate when the games are similar, i.e., when ∆ µ,ν ≪ KH 2 .

4 Meta-Learning for Markov Potential Games

In this section, we study meta-learning for NE in Markov potential games. We show that a straightforward refinement to the analysis of an existing algorithm [15] provides initialization-dependent bounds. Building on it, in Section 4.1, we first investigate the sharper convergence of meta-learning in a sequence of similar MPGs. Further, since there exists an optimization objective universally agreed on by all the players in an MPG (i.e., the potential function), we can formulate the meta-learning problem in the same way as MAML [20]. In Section 4.2, by choosing a proper base algorithm, we establish the non-asymptotic convergence of MAML in the highly non-stationary multi-agent scenario, without even imposing any smoothness assumptions as in existing works [17, 18, 30].

4.1 Sharper Rates in Similar Games

To be consistent with existing results in the literature, in this section, we consider an infinite-horizon γ -discounted reward setting for MPGs [43, 39, 76, 15]. A detailed description of the setup is provided in Appendix C for completeness. Equivalent results for the finite-horizon episodic setting (as we defined in Section 2) can be derived in a straightforward way. We choose an existing state-of-the-art algorithm, namely independent projected Q-descent [15], as our base algorithm (1). Specifically, in an MPG G k , each agent independently runs policy gradient ascents to update its own policy for T iterations:

where ¯ Q π i is the 'averaged' Q-function formally defined in Appendix C. Let Φ( · ; G k ) denote the potential function of G k . Through a simple refinement of the analysis in [15], we can establish the following initialization-dependence bound for our base algorithm (7).

Proposition 1. (Theorem 1 in [15]) Suppose that all players in a Markov potential game G k run independent projected Q-descent (7) for T iterations with α ≤ (1 -γ ) 4 8 κ 3 NA max . Then, we have

where κ ( G k ) is the standard distribution mismatch coefficient for G k formally defined in Appendix C.

Proposition 1 immediately implies that if we learn each MPG individually, it takes T = O ( NA max κ 4 Φ max (1 -γ ) 6 ε 2 ) steps to find an ε -approximate NE. To show the effectiveness of meta-learning, we consider the following similarity metric for a sequence of K games, which measures the maximal point-wise deviations of the potential functions:

As for the meta-updates, we simply instantiate (2) as π k, 0 i ← π k -1 ,T i , which lets each agent play the converged policy in the previous game. The intuition is that after running T steps on G k -1 , the agents will converge to an approximate NE policy of G k -1 . Since (8) requires the potential functions to be close, the converged policy π k -1 ,T should serve as a good starting point to search for NE in G k . We formally characterize such an intuition in the following theorem, which shows the sharper convergence of meta-learning in a large set of similar MPGs (i.e., when K is large and ∆ Φ is small):

Theorem 3. In a sequence of K Markov potential games, if (7) is run for T iterations as the base algorithm and π k, 0 i ← π k -1 ,T i as the meta-updates, then, for any ε > 0 , T = O ( NA max κ 4 (Φ max +∆ Φ ) K (1 -γ ) 6 ε 2 ) steps on average suffice to find an ε -approximate Nash equilibrium in each game.

4.2 Convergence to MAML Objective

In this subsection, we study meta-learning for MPGs under exactly the same formulation as in the seminal work of MAML [20]. Let G = { G j } be a set of different MPGs, where the games are now drawn from a fixed distribution p that we can sample from. We consider parametric policy classes where agent i 's policy is parameterized by θ i = { θ i ( a i | s ) ∈ R } s ∈S ,a i ∈A i . We focus on softmax parameterization where

Let ζ ( · ; G ) denote the operator of performing one step of policy gradient update on game G , i.e., ζ ( θ ; G ) := θ + α ∇ Φ( θ ; G ) , where α > 0 is the learning rate. The T -step MAML objective [20, 18, 30] can be formulated as

where θ = ( θ 1 , . . . , θ N ) ∈ Θ , and the operator ζ ( · ; G ) is applied T times. Intuitively, MAML tries to find a good parameter initialization from which running T steps of gradient ascents on any new task G leads to well-performing policy parameters.

Similar to Section 2, the MAML procedure consists of two nested stages. For the inner stage (1), we let each agent independently run T steps of policy gradient ascents to update its policy parameter θ ( t ) i on each encountered MPG. It is known (Theorem 5 of [78]) that T = O (1 /ε 2 ) steps will find an ε -approximate NE for each individual MPG. For the outer stage (2), MAML directly performs gradient ascents with respect to the meta-objective (9). The gradient of F T can be written in closed-form as

A detailed discussion of MAML and its instantiation in our problem are provided in Appendix D. Most importantly, Appendix D shows that both the policy gradient ∇ Φ( θ ) and the policy Hessian ∇ 2 Φ( θ ) can be written in closed-form, which allows us to construct unbiased estimators of (10) from samples. Despite the fact that the learning agents update their policies independently in an intertwined multi-agent system, our next result shows that the MAML updates converge to a stationary point of the meta-objective (9) in a non-asymptotic manner. A key step of the proof is to prove (rather than assume, as in existing works [17, 30]) that the meta-objective is Lipschitz smooth in the policy parameter θ . The smoothness constant can also be written in a closed form (Lemma 14).

Theorem 4. Suppose that the agents run independent policy gradient ascents with softmax parameterization on each encountered MPG as the inner stage, and perform gradient ascents w.r.t the MAML objective as the outer stage. For any ε > 0 , K = 4 NL F (1 -γ ) ε 2 iterations of MAML updates can find a policy θ ⋆ such that ∥∇ F T ( θ ⋆ ) ∥ ≤ ε , where L F is given in Lemma 14 of Appendix D.

5 Meta-Learning for General-Sum Markov Games

In this section, we consider learning coarse correlated equilibria in general-sum Markov games with no assumption on reward structures. Similar to Section 3, we start by developing an initializationdependent algorithm, followed by investigating the sharper convergence of meta-learning.

Our base algorithm for learning CCE also uses optimistic OMD with stage-based value updates. Detailed descriptions are deferred to Algorithm 2 in Appendix E due to space limitations. Algorithm 2 follows a similar structure as Algorithm 1, but the output policy ¯ π of Algorithm 2 is no longer a state-wise average policy and is instead a correlated policy [3, 44, 33]. For any correlated policy π , we use the notion

to measure its distance to a CCE. Let ¯ τ denote the total number of stages of Algorithm 2. Similar to zero-sum games (3), for any ( τ, h, s ) ∈ [¯ τ ] × [ H ] ×S , we define the per-state regret for each player i ∈ N as

where Q τ h,i is player i 's Q-function estimate at stage τ . We define the maximal regret (over all states and players) as reg τ h := max s ∈S max i ∈N { reg τ h,i ( s ) } . The initialization-dependent convergence rate of Algorithm 2 is established in the following theorem.

Theorem 5. If Algorithm 2 is run on a general-sum Markov game for T iterations with a learning rate η > 0 , the output policy ¯ π satisfies:

In addition, if the players' policies are initialized to be uniform policies ˜ π τ h,i ( ·| s ) = 1 /A i , ∀ i ∈ N and η is chosen as η = H -2 / 3 T -1 / 3 ( N -1) -2 / 3 , then we have

Compared to existing results, Theorem 5 directly associates the convergence rate with the quality of the initial policy ˜ π . With uniform initialization, the convergence rate in (11) has a slightly worse dependence on T than the best known result ˜ O ( √ NH 11 / 4 /T 3 / 4 ) [77]. Such deterioration is due to the potential lack of a smoothness condition for optimistic OMD that directly connects the stability of policies to the stability of utility functions (Lemma 18), unlike in optimistic FTRL. Although we believe that our rate in (11) can almost certainly be improved via a refined stability analysis, we leave the tightening of it to our future work as it would be a departure from the main focus of this work.

Let ˜ π k and ¯ π k , respectively, denote the initial policy and output policy of Algorithm 2 on game G k . For player i ∈ N , by putting together π τ, † h,i ( ·| s ) over all ( τ, h, s ) , we use π k, † i : [¯ τ ] × [ H ] ×S → ∆( A i ) to denote the best fixed policies in hindsight on G k . We consider a game similarity metric defined as

The following theorem presents the convergence rate of meta-learning, which again is sharper than learning each game individually when the games are similar, i.e., when ∆ π is sufficiently small.

Theorem 6. In a sequence of K general-sum Markov games, if Algorithm 2 is run for T iterations as the base algorithm and the meta-updates ˜ π k i = 1 k -1 ∑ k -1 k ′ =1 [ π k ′ , † i ] α , ∀ i ∈ N are used with α = 1 / √ K for policy initializations, then, for any ε > 0 , T = ˜ O ( HN ε 3 / 2 ( ∆ 3 / 2 π K 5 / 4 + A 3 / 2 max + H 3 K 1 / 2 )) steps on average suffice to find an ε -approximate CCE in each game.

6 Simulations

We numerically evaluate our meta-learning algorithms from Sections 3 and 4 on a sequence of K games. In this section, we evaluate on a sequence of K = 10 zero-sum Markov games and Markov potential games with two states, two players, and two candidate actions for each player. In Appendix F, we further demonstrate the scalability of our methods by providing numerical results on larger-scale tasks, including a simplified version of the Poker endgame considered in [27] and a 1D linear-quadratic tracking problem [37] with 4 cooperative players.

0

250

500

750

1000

Iterations

0

2

4

NE-gap

Individual NE-gap

Meta-learning NE-gap

(a) Zero-sum Markov game

0

250

500

750

1000

Iterations

0

.

0

0

.

5

1

.

0

1

.

5

NE-gap

Individual NE-gap

Meta-learning NE-gap

(b) Markov potential game

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

µ

/star

0

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ν

/star

0

State

s

0

State

s

1

(c) NE visualization

We generate the K = 10 games by first specifying a 'base game' and then adding random perturbations to its reward function to get K slightly different games. Each of the K games is run for T = 1000 iterations. To better visualize the similarity level of these games, in Figure 1(c), we plot the NE policies of the perturbed zero-sum matrix games at each of the two states for the K = 10 games. We remark that due to the existence of state transitions, the NE policies with respect to the stage Q-functions can be more diversified than Figure 1(c). Detailed descriptions of the simulation setup are deferred to Appendix F.

We evaluate the convergences of the algorithms in terms of NE-gap . Figures 1(a) and 1(b) compare the average NE-gap over the K games between individual learning and meta-learning for zero-sum Markov games and Markov potential games, respectively. We see that meta-learning can utilize knowledge from previous tasks to attain better policy initialization in a new task and converges to an approximate NE policy using much fewer iterations.

7 Concluding Remarks

In this paper, we have introduced meta-learning to solve multiple MARL tasks collectively. Under natural similarity metrics, we have shown that meta-learning achieves provably sharper convergence for learning NE in zero-sum and potential games and for learning CCE in general-sum games. Along the way, we have proposed new MARL algorithms with fine-grained initialization-dependent convergence guarantees. Our work appears to be the first to investigate the theoretical properties of meta-learning in MARL and provide reliable justifications for its usage. As for the limitations, our convergence rate for learning CCE (Theorem 5) is slightly less competitive than the best-known results when our policies are initialized conservatively, which might be improved via a refined policy stability analysis. Other future directions include further generalization of our results to alternative game similarity metrics and broader types of games (e.g., stochastic Stackelberg games).

Acknowledgments and Disclosure of Funding

This work was partially supported by the National Science Foundation (NSF) under grant CCF 20-29049; by the IBM-ILLINOIS Center for Cognitive Computing Systems Research (C3SR), a research collaboration that is part of the IBM AI Horizon Network; by the IBM-ILLINOIS Discovery Accelerator Institute (IIDAI); and by the Air Force Office of Scientific Research (AFOSR) under grant FA9550-19-1-0353. Any opinions, findings or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF or IBM or AFOSR.

References

A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22: 1-76, 2021.

Y. Bai and C. Jin. Provable self-play algorithms for competitive reinforcement learning. In International Conference on Machine Learning , pages 551-560, 2020.

Y. Bai, C. Jin, and T. Yu. Near-optimal reinforcement learning with self-play. Advances in Neural Information Processing Systems , 33, 2020.

M.-F. Balcan, M. Khodak, and A. Talwalkar. Provable guarantees for gradient-based metalearning. In International Conference on Machine Learning , pages 424-433. PMLR, 2019.

A. Banerjee, S. Merugu, I. S. Dhillon, J. Ghosh, and J. Lafferty. Clustering with Bregman divergences. Journal of Machine Learning Research , 6(10), 2005.

N. Brown and T. Sandholm. Superhuman AI for heads-up no-limit Poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018.

S. Cen, Y. Wei, and Y. Chi. Fast policy extragradient methods for competitive games with entropy regularization. Advances in Neural Information Processing Systems , 34:27952-27964, 2021.

N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

G. Chen and M. Teboulle. Convergence analysis of a proximal-like minimization algorithm using Bregman functions. SIAM Journal on Optimization , 3(3):538-543, 1993.

X. Chen, X. Deng, and S.-H. Teng. Settling the complexity of computing two-player Nash equilibria. Journal of the ACM , 56(3):1-57, 2009.

C. Daskalakis, P. W. Goldberg, and C. H. Papadimitriou. The complexity of computing a Nash equilibrium. Communications of the ACM , 52(2):89-97, 2009.

C. Daskalakis, D. J. Foster, and N. Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in Neural Information Processing Systems , 33: 5527-5540, 2020.

C. Daskalakis, N. Golowich, and K. Zhang. The complexity of Markov equilibrium in stochastic games. arXiv preprint arXiv:2204.03991 , 2022.

G. Denevi, C. Ciliberto, R. Grazzi, and M. Pontil. Learning-to-learn stochastic gradient descent with biased regularization. In International Conference on Machine Learning , pages 1566-1575. PMLR, 2019.

D. Ding, C.-Y. Wei, K. Zhang, and M. Jovanovic. Independent policy gradient for largescale Markov potential games: Sharper rates, function approximation, and game-agnostic convergence. In International Conference on Machine Learning , pages 5166-5220. PMLR, 2022.

L. Erez, T. Lancewicki, U. Sherman, T. Koren, and Y. Mansour. Regret minimization and convergence to equilibria in general-sum Markov games. arXiv preprint arXiv:2207.14211 , 2022.

A. Fallah, A. Mokhtari, and A. Ozdaglar. On the convergence theory of gradient-based modelagnostic meta-learning algorithms. In International Conference on Artificial Intelligence and Statistics , pages 1082-1092, 2020.

A. Fallah, K. Georgiev, A. Mokhtari, and A. Ozdaglar. On the convergence theory of debiased model-agnostic meta-reinforcement learning. Advances in Neural Information Processing Systems , 34:3096-3107, 2021.

H. Fang, N. J. Harvey, V. S. Portella, and M. P. Friedlander. Online mirror descent and dual averaging: Keeping pace in the dynamic case. Journal of Machine Learning Research , 23(1): 5271-5308, 2022.

C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning , pages 1126-1135, 2017.

C. Finn, A. Rajeswaran, S. Kakade, and S. Levine. Online meta-learning. In International Conference on Machine Learning , pages 1920-1930. PMLR, 2019.

R. Fox, S. M. Mcaleer, W. Overman, and I. Panageas. Independent natural policy gradient always converges in Markov potential games. In International Conference on Artificial Intelligence and Statistics , pages 4414-4425, 2022.

Y. Freund and R. E. Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior , 29(1-2):79-103, 1999.

T. Furmston, G. Lever, and D. Barber. Approximate Newton methods for policy search in Markov decision processes. Journal of Machine Learning Research , 17, 2016.

Z. Gao, Q. Ma, T. Ba¸ sar, and J. R. Birge. Finite-sample analysis of decentralized Q-learning for stochastic games. arXiv preprint arXiv:2112.07859 , 2021.

A. Gupta, M. Lanctot, and A. Lazaridou. Dynamic population-based meta-learning for multiagent communication with natural language. Advances in Neural Information Processing Systems , 34:16899-16912, 2021.

K. Harris, I. Anagnostides, G. Farina, M. Khodak, Z. S. Wu, and T. Sandholm. Meta-learning in games. arXiv preprint arXiv:2209.14110 , 2022.

E. Hazan. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

J. Humplik, A. Galashov, L. Hasenclever, P. A. Ortega, Y. W. Teh, and N. Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv:1905.06424 , 2019.

K. Ji, J. Yang, and Y. Liang. Theoretical convergence of multi-step model-agnostic metalearning. Journal of Machine Learning Research , 2022.

H. Jia, B. Ding, H. Wang, X. Gong, and X. Zhou. Fast adaptation via meta learning in multiagent cooperative tasks. In Symposia and Workshops on Ubiquitous, Autonomic and Trusted Computing , pages 707-714, 2019.

C. Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan. Is Q-learning provably efficient? Advances in Neural Information Processing Systems , 31, 2018.

C. Jin, Q. Liu, Y . Wang, and T. Yu. V-learning-A simple, efficient, decentralized algorithm for multiagent RL. In ICLR Workshop on Gamification and Multiagent Solutions , 2022.

V. Khattar, Y. Ding, B. Sel, J. Lavaei, and M. Jin. A CMDP-within-online framework for meta-safe reinforcement learning. In International Conference on Learning Representations , 2022.

M. Khodak, M.-F. F. Balcan, and A. S. Talwalkar. Adaptive gradient-based meta-learning methods. Advances in Neural Information Processing Systems , 32, 2019.

M. Lauer and M. Riedmiller. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In International Conference on Machine Learning , 2000.

M. Lauriere, S. Perrin, S. Girgin, P. Muller, A. Jain, T. Cabannes, G. Piliouras, J. Pérolat, R. Elie, O. Pietquin, et al. Scalable deep reinforcement learning algorithms for mean field games. In International Conference on Machine Learning , pages 12078-12095. PMLR, 2022.

S. Leonardos, W. Overman, I. Panageas, and G. Piliouras. Global convergence of multi-agent policy gradient in Markov potential games. arXiv preprint arXiv:2106.01969 , 2021.

S. Leonardos, W. Overman, I. Panageas, and G. Piliouras. Global convergence of multiagent policy gradient in Markov potential games. In International Conference on Learning Representations , 2021.

Q. Liu, T. Yu, Y. Bai, and C. Jin. A sharp analysis of model-based reinforcement learning with self-play. In International Conference on Machine Learning , 2021.

S. Liu, M. Lanctot, L. Marris, and N. Heess. Simplex neural population learning: Any-mixture bayes-optimality in symmetric zero-sum games. In International Conference on Machine Learning , pages 13793-13806. PMLR, 2022.

R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems , 30, 2017.

S. V. Macua, J. Zazo, and S. Zazo. Learning parametric closed-loop policies for Markov potential games. In International Conference on Learning Representations , 2018.

W. Mao and T. Ba¸ sar. Provably efficient reinforcement learning in decentralized general-sum Markov games. Dynamic Games and Applications , 13:165-186, 2023.

W. Mao, K. Zhang, R. Zhu, D. Simchi-Levi, and T. Ba¸ sar. Near-optimal model-free reinforcement learning in non-stationary episodic MDPs. In International Conference on Machine Learning , pages 7447-7458. PMLR, 2021.

W. Mao, H. Qiu, C. Wang, H. Franke, Z. Kalbarczyk, R. Iyer, and T. Basar. A mean-field game approach to cloud resource management with function approximation. Advances in Neural Information Processing Systems , 35:36243-36258, 2022.

W. Mao, L. Yang, K. Zhang, and T. Ba¸ sar. On improving model-free algorithms for decentralized multi-agent reinforcement learning. In International Conference on Machine Learning , pages 15007-15049, 2022.

D. Nekipelov, V. Syrgkanis, and E. Tardos. Econometrics for learning agents. In Proceedings of the 16th ACM Conference on Economics and Computation , pages 1-18, 2015.

A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 , 2018.

F. Orabona and D. Pál. Scale-free online learning. Theoretical Computer Science , 716:50-69, 2018.

P. A. Ortega, J. X. Wang, M. Rowland, T. Genewein, Z. Kurth-Nelson, R. Pascanu, N. Heess, J. Veness, A. Pritzel, P. Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arXiv:1905.03030 , 2019.

S. Perrin, J. Pérolat, M. Laurière, M. Geist, R. Elie, and O. Pietquin. Fictitious play for mean field games: Continuous time analysis and applications. Advances in Neural Information Processing Systems , 33:13199-13213, 2020.

H. Qiu, W. Mao, C. Wang, H. Franke, A. Youssef, Z. T. Kalbarczyk, T. Ba¸ sar, and R. K. Iyer. AWARE: Automate workload autoscaling with reinforcement learning in production cloud systems. In USENIX Annual Technical Conference , 2023.

A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-learning with implicit gradients. Advances in Neural Information Processing Systems , 32, 2019.

S. Rakhlin and K. Sridharan. Optimization, learning, and games with predictable sequences. Advances in Neural Information Processing Systems , 26, 2013.

A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented neural networks. In International Conference on Machine Learning , pages 1842-1850. PMLR, 2016.

L. S. Shapley. Stochastic games. Proceedings of the National Academy of Sciences , 39(10): 1095-1100, 1953.

J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems , 30, 2017.

Z. Song, S. Mei, and Y. Bai. When can we learn general-sum Markov games with a large number of players sample-efficiently? In International Conference on Learning Representations , 2022.

R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems , 12, 1999.

D. Sychrovsky, M. Sustr, E. Davoodi, M. Lanctot, and M. Schmid. Learning not to regret. arXiv preprint arXiv:2303.01074 , 2023.

V. Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire. Fast convergence of regularized learning in games. Advances in Neural Information Processing Systems , 28, 2015.

A. A. Team, J. Bauer, K. Baumli, S. Baveja, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg, M. Chang, N. Clay, A. Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608 , 2023.

S. Thrun and L. Pratt. Learning to Learn . Springer Science & Business Media, 1998.

O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems , 29, 2016.

L. Wang, Q. Cai, Z. Yang, and Z. Wang. On the global optimality of model-agnostic metalearning. In International Conference on Machine Learning , pages 9837-9846, 2020.

C.-Y. Wei, Y.-T. Hong, and C.-J. Lu. Online reinforcement learning in stochastic games. In International Conference on Neural Information Processing Systems , pages 4994-5004, 2017.

C.-Y. Wei, C.-W. Lee, M. Zhang, and H. Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive Markov games. In Conference on Learning Theory , pages 4259-4299, 2021.

Q. Xie, Y. Chen, Z. Wang, and Z. Yang. Learning zero-sum simultaneous-move Markov games using function approximation and correlated equilibrium. In Conference on Learning Theory , pages 3674-3682, 2020.

S. Xue, C. Qu, X. Shi, C. Liao, S. Zhu, X. Tan, L. Ma, S. Wang, S. Wang, Y. Hu, et al. A meta reinforcement learning approach for predictive autoscaling in the cloud. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4290-4299, 2022.

S. Yang and B. Yang. A meta multi-agent reinforcement learning algorithm for multi-intersection traffic signal control. In IEEE International Symposium on Dependable, Autonomic and Secure Computing , pages 18-25, 2021.

Y. Yang and C. Ma. O ( T -1 ) convergence of optimistic-follow-the-regularized-leader in twoplayer zero-sum Markov games. arXiv preprint arXiv:2209.12430 , 2022.

S. Zeng, T. Doan, and J. Romberg. Regularized gradient descent ascent for two-player zero-sum Markov games. Advances in Neural Information Processing Systems , 35:34546-34558, 2022.

K. Zhang, A. Koppel, H. Zhu, and T. Ba¸ sar. Global convergence of policy gradient methods to (almost) locally optimal policies. arXiv preprint arXiv:1906.08383 , 2019.

M. Zhang, P. Zhao, H. Luo, and Z.-H. Zhou. No-regret learning in time-varying zero-sum games. In International Conference on Machine Learning , pages 26772-26808, 2022.

R. Zhang, Z. Ren, and N. Li. Gradient play in stochastic games: Stationary points, convergence, and sample complexity. arXiv preprint arXiv:2106.00198 , 2021.

R. Zhang, Q. Liu, H. Wang, C. Xiong, N. Li, and Y. Bai. Policy optimization for Markov games: Unified framework and faster convergence. In Advances in Neural Information Processing Systems , 2022.

R. Zhang, J. Mei, B. Dai, D. Schuurmans, and N. Li. On the global convergence rates of decentralized softmax gradient play in Markov potential games. In Advances in Neural Information Processing Systems , 2022.

Z. Zhang, Y. Zhou, and X. Ji. Almost optimal model-free reinforcement learningvia referenceadvantage decomposition. Advances in Neural Information Processing Systems , 33:1519815207, 2020.

Peer reviews:
comments: SUMMARY:
This paper studies the interdependence between the convergence of MARL and the quality of policy initialization.

STRENGTHS:
1. It proposes a new algorithm that has an initialization-dependent convergence guarantee.
2. It establishes several theoretical results that connect policy initialization and convergence of MARL in various types of games.

WEAKNESSES:
No

QUESTIONS:
No.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a meta-learning method to initialize the OOMD algorithm. Combined with the introduced initialization-dependent convergence guarantees, authors then can show faster convergence when the meta-learning initialization is close.\n\nSTRENGTHS:\nThe paper is well written, and the prior work is well referenced. \n\n\nThe authors also provide extensive theoretical analysis of their algorithms. I think the most interesting result is Theorem 1 (other theorems often seem like an application of Theorem 1). I believe that the result is novel and interesting.\n\n\nI also appreciate that authors analyze two-player zero-sum markov games as well as markov potential games\n\nWEAKNESSES:\nFirst, I agree that “closest to this work” is Harris, Keegan, et al. \"Meta-learning in games.\" arXiv preprint arXiv:2209.14110 (2022). More recently, a closely related (also meta learning in games and regret minimization) paper presents an algorithm combining meta-learning and regret guarantees: Sychrovsky, David, et al. \"Learning not to Regret.\" arXiv preprint arXiv:2303.01074 (2023). I believe that paper should be included in the related work.\n\n\nMy biggest issue though is the empirical analysis (Section 6 - Simulations).\nThere are no details in the main text and one can only find it in the appendix. It is fine to move details to the appendix, but the authors do not even mention the size of the games in the main text.\nLooking into the appendix, I think the games are trivially small (2x2 matrix games). Furthermore, the games in the meta-learning sequence seem very similar (the epsilon noisy seems very small). I think this is rather confirmed by Figure 1c. It is then not a big surprise that the convergence is so fast, when the games are so small and the Nash Equilibria very similar/close. I think the authors should to evaluate substantially more interesting/diverse/larger games (e.g. some epsilon/noisy parametrization of small poker games).\n\n\nIf the authors add experiments in larger games than the trivially small currently included, I am happy to move my score to \"Accept\" as otherwise I think the paper is good.\n\n\n----------------------------------------------------------------------------------------------------\nAuthors addressed my comments and included more experiments, increasing the score.\n\nQUESTIONS:\nDid you have a chance to run the algorithm on larger games than the small ones included in the current version of the submission?\n\nLIMITATIONS:\nSufficient", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a meta-learning approach based on MAML for multi-agent domains where tasks with similar NE policies, when learned sequentially, converges faster to desired equilibria solutions.\n\nSTRENGTHS:\n* Originality \nThe authors investigated theoretical convergence properties of multi-agent learning through the lens of meta-learning of a sequence of similar tasks. This is novel and the problem setting is relevant as non-stationarity is inherent to multi-agent learning that may benefit from the MAML family of methodologies.  \n\n* Quality\nI find the theoretical results thorough but the work could benefit significantly with more substantial results on a few more standard benchmark domains. \n\n* Clarity\nThe paper is generally well written and easy to follow.\n\nWEAKNESSES:\n* Task similarity metric: my main reservation with the proposed approach is related to the proposed task similarity metric (Sec 3.2) which translates to measuring similarity in different tasks' NE policy. Wouldn't selecting tasks in such way naturally promote faster convergence to NE policies when learning on \"similar\" tasks sequentially? I would appreciate further clarification on why this metric is used and if alternatives have been considered. \n\n* Multi-agent learning inherently deals with non-stationarity during learning as the environment dynamics (from the perspective of any one player) is non-stationary. Quite often such non-stationarity does not translate to similar NE policy though (e.g. rock-paper-scissors, where NE polices would be quite different). Could you clarify if the proposed method should be applicable to such \"similar tasks\" of this nature?\n\nQUESTIONS:\nL82-83: \" ... weaker solution concepts such as (C)CE ...\", perhaps clarify that in general-sum games NE would not allow for coordination and is therefore restrictive in such games? By \"weaker\" perhaps you meant computationally tractable beyond two-player zero-sum games?\n\nL49: \"... convergence guarantees for MARL.\", do you mean convergence to equilibria specifically? Perhaps clarify in writing that convergence bounds are wrt equilibria upfront? \n\nRelated Works: \"meta-learning\" is overloaded in the RL literature and another line of work in meta-learning follows from prior works such as  [1-3] where a policy is conditioned on prior belief over tasks and can infer Bayes-optimally through interaction at test time to adapt to different tasks. This idea is then extended to multi-agent setting [3-4], with a focus on faster convergence / transfer learning via shared representation learning. Would it make sense to include a brief discussion of this line of meta-learning works in related works? \n\n[1] Meta-learning of Sequential Strategies: https://arxiv.org/abs/1905.03030\n[2] Meta reinforcement learning as task inference: https://arxiv.org/abs/1905.06424\n[3] Meta-Learning with Memory-Augmented Neural Networks: https://proceedings.mlr.press/v48/santoro16.pdf\n[3] NeuPL: Neural Population Learning: https://arxiv.org/abs/2202.07415\n[4] Simplex NeuPL: https://arxiv.org/abs/2205.15879\n\nLIMITATIONS:\nPlease see my comments on the choice of task similarity metric which may be a limitation on when and where the proposed method could be effectively applied.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper establish theoretical results for meta-learning in a wide range of fundamental MARL settings, including learning Nash equilibria in two-player zero-sum Markov games and Markov potential games. Numerical results are shown to demonstrate the advantages of meta-learning.

STRENGTHS:
This paper establish theoretical results for meta-learning in a wide range of fundamental MARL settings, including learning Nash equilibria in two-player zero-sum Markov games and Markov potential games. Numerical results are shown to demonstrate the advantages of meta-learning.

WEAKNESSES:
The simulation part is relatively simple since only toy examples are given. The advantages of meta-learning are not fully demonstrated.

QUESTIONS:
The simulation part is relatively simple since only toy examples are given. Is it possible to have much more sophisticated examples (at least more than two player) to demonstrate the theoretical results?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce theoretical results on Model-Agnostic Meta-Learning in a multi-agent reinforcement learning setting. In particular they show that meta-learning can achieve stronger convergence guarantees than an RL baseline when tasks are similar. The results hold for zero-sum, potential and general-sum games. In order to establish this, they provide theoretical results on convergence for online mirror descent which are dependent on initial conditions. Finally they conduct some small-scale experiments to validate the faster convergence empirically.\n\nSTRENGTHS:\n- Well motivated and timely article: meta-learning is becoming increasingly prevalent in the multi-agent setting, and theoretical results are lagging behind empirical ones, to the best of my knowledge.\n- The notation is consistent and clear throughout. \n- The theoretical results seem convincing, although I haven't checked the proofs in detail. \n- There is an interesting and intuitive definition of game similarity, which may well find use in other algorithms in the future.\n\nWEAKNESSES:\n- The empirical results are quite small scale, and it would be useful to have a little more explanation about the environment in the main text. \n- It would be nice to have at least one sketch proof in the main text. Perhaps there is a reorganisation of some other material to the Appendix that might permit this.\n- There are a few places where more discussion or justification would be beneficial, for example: why is this \"without loss of generality\" on line 102; to what extent is it standard to neglect the logarithmic terms in line 201, and what is the reason that these terms show up.\n- The limitations / future work section could have a little more detail. Would the authors be happy to comment on scaleability, and to discuss how similar results might be obtained for other meta-learning approaches outside MAML?\n\nQUESTIONS:\nOn lines 94-95, the authors may want to cite another recent paper using meta-learning in the context of a distribution over multi-agent cooperative tasks: https://arxiv.org/abs/2301.07608.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}