Title: Transformers as Game Players: Provable In-context Game-playing Capabilities of Pre-trained Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: In-context Learning; Multi-agent Competitive Games; Transformers; Decision-making
Abstract: The in-context learning (ICL) capability of pre-trained models based on the transformer architecture has received growing interest in recent years. While theoretical understanding has been obtained for ICL in reinforcement learning (RL), the previous results are largely confined to the single-agent setting. This work proposes to further explore the in-context learning capabilities of pre-trained transformer models in competitive multi-agent games, i.e., in-context game-playing (ICGP). Focusing on the classical two-player zero-sum games, theoretical guarantees are provided to demonstrate that pre-trained transformers can provably learn to approximate Nash equilibrium in an in-context manner for both decentralized and centralized learning settings. As a key part of the proof, constructional results are established to demonstrate that the transformer architecture is sufficiently rich to realize celebrated multi-agent game-playing algorithms, in particular, decentralized V-learning and centralized VI-ULCB.

Transformers as Game Players: Provable In-context Game-playing Capabilities of Pre-trained Models

Abstract

The in-context learning (ICL) capability of pre-trained models based on the transformer architecture has received growing interest in recent years. While theoretical understanding has been obtained for ICL in reinforcement learning (RL), the previous results are largely confined to the single-agent setting. This work proposes to further explore the in-context learning capabilities of pre-trained transformer models in competitive multi-agent games, i.e., in-context game-playing (ICGP). Focusing on the classical two-player zero-sum games, theoretical guarantees are provided to demonstrate that pre-trained transformers can provably learn to approximate Nash equilibrium in an in-context manner for both decentralized and centralized learning settings. As a key part of the proof, constructional results are established to demonstrate that the transformer architecture is sufficiently rich to realize celebrated multi-agent game-playing algorithms, in particular, decentralized V-learning and centralized VI-ULCB.

1 Introduction

Since proposed in Vaswani et al. [57], the transformer architecture has received significant interest. It has powered many recent breakthroughs in artificial intelligence [11, 12, 21, 23], including the extremely powerful large language models such as GPT [42] and Llama [55, 56]. One of the most striking observations from the research of these transformer-powered models is that they demonstrate remarkable in-context learning (ICL) capabilities. In particular, after appropriate pre-training, the models can handle new tasks when prompted by a few descriptions or demonstrations without any parameter updates, e.g., Brown et al. [11], Chowdhery et al. [15], Liu et al. [39].

ICL is practically attractive as it provides strong generalization capabilities across different downstream tasks without requiring further training or a large amount of task-specific data. These appealing properties have motivated many empirical studies to better understand ICL [18, 26, 59]; see the survey by Dong et al. [22] for key findings and results. In addition to the empirical investigations, recent years have witnessed growing efforts in gaining deeper theoretical insights into ICL, e.g., Ahn et al. [2], Akyürek et al. [3], Bai et al. [7], Cheng et al. [14], Li et al. [37], Raventós et al. [44], Wu et al. [64], Xie et al. [66], Zhang et al. [71].

Among these empirical and theoretical studies, one emerging direction focuses on the capability of pretrained transformer models to perform in-context reinforcement learning (ICRL) [28, 34, 35, 50, 73]. In particular, the transformer is pre-trained with interaction data from diverse environments, modeling

the interaction as a sequential prediction task. During inference, the pre-trained transformer is prompted via the interaction trajectory in the current environment for it to select actions. The work by Lin et al. [38] provides some theoretical understanding of ICRL, including both a general pretraining guarantee and specific constructions of transformers to realize some well-known designs in multi-armed bandits and RL (especially, LinUCB [1], Thompson sampling [54], and UCB-VI [6]). Wang et al. [62] further provides understandings on the capability of transformers learning temporal difference (TD) methods [52] via an in-context fashion. A detailed literature review can be found in Sec. 6.

The insights from Lin et al. [38], Wang et al. [62] are largely confined to the single-agent scenario, i.e., a single-agent multi-armed bandit or Markov decision process (MDP). The power of RL, however, extends to the much broader multi-agent scenario, especially the multi-player competitive games such as GO [49], Starcraft [58], and Dota 2 [10]. To provide a more comprehensive understanding of ICRL, this work targets further studying the in-context game-playing (ICGP) capabilities of transformers in multi-agent competitive settings. To the best of our knowledge, this is the first work providing theoretical analyses and empirical pieces of evidence on the ICGP capabilities of transformers. The contributions of this work are further summarized as follows.

· A general framework is proposed to model in-context game-playing via transformers, where we focus on the representative two-player zero-sum Markov games and target learning Nash equilibrium (NE). Compared with the single-agent scenario [38], the multi-agent setting considered in this work broadens the ICRL research scope while it is also more complicated due to its game-theoretic nature.

· The challenging decentralized learning setting is first studied, where two distinct transformers are trained to learn NE, one for each player, without observing the opponent's actions. A general realizability-conditioned guarantee is first derived that characterizes the generalization error of the pre-trained transformers. Then, the capability of the transformer architecture is demonstrated by providing a concrete construction so that the famous V-learning algorithm [8] can be exactly realized. Lastly, a finite-sample upper bound on the approximation error of NE is proved to establish the ICGP capability of transformers. As a further implication, the result of realizing V-learning demonstrates the capability of pre-trained transformers to perform model-free RL designs, in addition to the model-based ones (e.g., UCB-VI [6] as studied in Lin et al. [38]).

· To obtain a complete understanding, the centralized learning setting is also investigated, where one transformer is pre-trained to control both players' actions. A similar set of results is provided: a general pre-training guarantee, a constructional result to demonstrate realizability, and a finite-sample upper bound on the approximation error of NE. Distinctly, the transformer construction is presented as a specific parameterization to implement the renowned centralized VI-ULCB algorithm [8].

· Furthermore, experiments are also performed to practically test the ICGP capabilities of the pretrained transformers. The obtained results not only corroborate the derived theoretical claims, but also empirically motivate this and further studies on the interesting direction of pre-trained models in game-theoretic settings.

2 A Theoretical Framework for In-Context Game Playing

2.1 The Basic Setup of Environments

To demonstrate the ICGP capability of transformers, we focus on one of the most basic gametheoretic settings: two-player zero-sum Markov games [47], while the discussions provided later conceivably extend to more general games. An illustration of the different settings (i.e., decentralized and centralized) considered in this work (with details explained later) is given in Fig. 1. The overall framework is introduced in the following, which extends Lin et al. [38] from the single-agent decision-making setting to the competitive multi-agent domain.

Considering a set of two-player zero-sum Markov games denoted as M . Each environment M ∈ M shares the number of episodes G , the number of steps H in each episode, the state space S (with |S| = S ) , the action spaces {A , B} (with |A| = A and |B| = B ), and the reward space R . Here A and B denote the action spaces of two players, respectively, which are referred to as the max-player and the min-player for convenience.

In-context

CentralizedLearning

Game-playing

Pre-trained

001

Inference

Env.PriorDist.A

Inference

Pre-trainedTransformer+

Transformer

Env.M

Env.M

Pre-trainedTransformer.

FiniteInteraction

Joint

FiniteEnv.Samples

Individual ContextAlg+

Dataset

{Di,+:i e [N]}

Dataset

Context

(D:ie [N]}

Alg

Individual Context Alg-

Dataset

{(Di,:ie [N]}

Each environment M = { T h -1 M , R h M : h ∈ [ H ] } has its transition model T h M : S × A × B → ∆( S ) and reward functions R h M : A × B → ∆( R ) , where T 0 M ( · ) denotes the initial state distribution. Particularly, overall G episodes of H steps happen in each environment M , with each episode starting at s g, 1 ∼ T 0 M ( · ) . If the action pair ( a g,h , b g,h ) is taken upon state s g,h at step h in episode g , the state is transited to s g,h +1 ∼ T h M ( ·| s g,h , a g,h , b g,h ) , and reward r g,h ∼ R h M ( s g,h , a g,h , b g,h ) (respectively, -r g,h ) is collected by the max-player (respectively, the min-player). For simplicity, we assume that the max-player rewards are bounded in [0 , 1] and deterministic, i.e., for each ( s, a, b, h ) , there exists r ∈ [0 , 1] such that R g,h M ( r | s, a, b ) = 1 . Also, the initial state s g, 1 is assumed to be a fixed one s 1 , i.e., T 0 M ( s 1 ) = 1 .

We further leverage the notation T := GH , while using time t and episode-step pair ( g, h ) in an interleaving manner with t := ( g -1) H + h . The partial interaction trajectory up to time t is then denoted as D t := { ( s τ , a τ , b τ , r τ ) : τ ∈ [ t ]) and we use the abbreviated notation D := D T . Individually, for the max-player, we denote her observed interaction trajectory up to time t by D t + := { ( s τ , a τ , r τ ) : τ ∈ [ t ]) and write D + := D T + for short. Similarly, for the min-player, we denote D t -:= { ( s τ , b τ , r τ ) : τ ∈ [ t ]) and D -:= D T -.

2.2 Game-playing Algorithms and Nash Equilibrium

A game-playing algorithm Alg can map a partial trajectory D t -1 and state s t to a distribution over the actions, i.e., Alg ( · , ·| D t -1 , s t ) ∈ ∆( A×B ) . If one algorithm Alg is decoupled for the two players (as in the later decentralized setting), we denote it as Alg = ( Alg + , Alg -) , where Alg + ( ·| D t -1 + , s t ) ∈ ∆( A ) and Alg -( ·| D t -1 -, s t ) ∈ ∆( B ) . Given an environment M and an algorithm Alg , the distribution over a full trajectory D can be expressed as

If further considering an environment prior distribution Λ ∈ ∆( M ) such that M ∼ Λ , the joint distribution of ( M,D ) is denoted as P Alg Λ ( D ) , where M ∼ Λ( · ) and D ∼ P Alg M ( · ) .

For environment M and a game-playing algorithm π , we define its value function over one episode as V π M ( s 1 ) = E D H ∼ P π M [ ∑ t ∈ [ H ] r t ] . With the marginalized policies of π denoted as ( µ, ν ) , we define their best responses as

whose corresponding values are

With the notion of best responses, the following classical definition of approximate Nash equilibrium (NE) can be introduced [8, 32, 40, 47].

Definition 2.1 (Approximate Nash equilibrium) . A decoupled policy pair (ˆ µ, ˆ ν ) is an ε -approximate Nash equilibrium for environment M if V ˆ µ, † M ( s 1 ) + ε ≥ V ˆ µ, ˆ ν M ( s 1 ) ≥ V † , ˆ ν M ( s 1 ) -ε , i.e., the Nash equilibrium gap V † , ˆ ν M ( s 1 ) -V ˆ µ, † M ( s 1 ) ≤ 2 ε .

For each environment, our learning goal is to approximate its NE policy pair. In other words, we target outputting a policy pair (ˆ µ, ˆ ν ) that is ε -approximate NE with an error ε that is as small as possible, after interacting with the environment for an overall T rounds.

2.3 The Transformer Architecture

With the basics of the game-playing environment and the learning target established, we now introduce the transformer architecture [57], which has demonstrated great potential in processing sequential inputs. First, for an input vector x ∈ R d , we denote σ r ( x ) := ReLU ( x ) := max { x , 0 } ∈ R d as the entry-wise ReLU activation function and σ s ( x ) := softmax( x ) ∈ R d as the softmax activation function, while using σ ( · ) to refer to a non-specified activation function (i.e., both ReLU and softmax may be used). Then, the masked attention layer and the MLP layer can be defined as follows.

Definition 2.2 (Masked Attention Layer) . A masked attention layer with M heads is denoted as Attn θ ( · ) with parameters θ = { ( V m , Q m , K m ) } m ∈ [ M ] ⊂ R d × d . On any input sequence H = [ h 1 , · · · , h N ] ∈ R d × N , we have H = Attn θ ( H ) = [ h 1 , · · · , h N ] ∈ R d × N , where

Definition 2.3 (MLP Layer) . An MLP layer with hidden dimension d ′ is denoted as MLP θ with parameters θ = ( W 1 , W 2 ) ∈ R d ′ × d × R d × d ′ . On any input sequence H = [ h 1 , · · · , h N ] ∈ R d × N , we have H = MLP θ ( H ) = [ h 1 , · · · , h N ] ∈ R d × N , where

The combination of masked attention layers and MLP layers leads to the overall decoder-based transformer architecture studied in this work, as defined in the following.

Definition 2.4 (Decoder-based Transformer) . An L -layer decoder-based transformer, denoted as TF θ ( · ) , is a composition of L masked attention layers, each followed by an MLP layer and a clip operation: TF θ ( H ) = H ( L ) ∈ R d × N , where H ( L ) is defined iteratively by taking H (0) = H ∈ R d × N and for l ∈ [ L ] ,

where parameter θ = { ( θ ( l ) mattn , θ ( l ) mlp ) : l ∈ [ L ] } consists of θ ( l ) mattn = { ( V ( l ) m , Q ( l ) m , K ( l ) m ) : m ∈ [ M ] } ⊂ R d × d and θ ( l ) mlp = ( W ( l ) 1 , W ( l ) 2 ) ∈ R d ′ × d × R d × d ′ .

We further define the parameter class of transformers as Θ d,L,M,d ′ ,F := { θ = ( θ (1: L ) mattn , θ (1: L ) mlp ) : ∥ θ ∥ ≤ F } , where the norm of a transformer TF θ is denoted as

Other Notations. The total variation distance between two algorithms { π, π ′ } upon D t -1 ∪ { s } is denoted as TV ( π, π ′ | D t -1 , s ) := TV ( π ( ·| D t -1 , s ) , π ′ ( ·| D t -1 , s )) . Also, the notation x ≲ y indicates that x is a lower or equivalent order term compared with y , i.e., x = O ( y ) , ˜ O ( · ) hides poly-logarithmic terms in H,G,S,A,B , and poly ( · ) compactly denotes a polynomial term with respect to the input.

3 Decentralized Learning

First, we study the decentralized learning setting, i.e., each player takes actions following her own model independently without observing the opponent's actions, as it better captures the unique game-playing scenario considering in this work. This setting is aligned with the canonical study of normal-form games [20, 53], and has been extended to Markov games in recent years [32, 41, 51]. In the following, we start by introducing the basic setup of supervised pre-training and provide a general performance guarantee relying on a realizability assumption. Then, we provide a constructional result

to demonstrate that the algorithms induced by transformers are rich enough to realize the celebrated V-learning algorithm [32]. With these results, we finally establish that with V-learning providing training data, the pre-trained transformer can effectively approximate NE when interacting with different environments in an in-context fashion.

3.1 Supervised Pre-training Results

3.1.1 Basic Setups

Training Dataset. In the supervised pre-training, we use a context algorithm Alg 0 to collect the offline trajectories. For the decentralized setting, the context algorithm Alg 0 used for data collection is assumed to be consisted of two decoupled algorithms ( Alg + , 0 , Alg -, 0 ) for the max- and min-players, respectively. With the context algorithm, we consider N i.i.d. offline trajectories { D i : i ∈ [ N ] } are collected, where D i := D i ∪ D ′ i with

It can be observed that D i is the commonly considered offline interaction trajectory of Alg 0 , while D ′ i is the sampled actions of each state s at each step t with Alg 0 . Compared with Lin et al. [38], D ′ i is an augmented component. We first note that collecting D ′ i is relatively easy in practical applications, as it only needs to additionally sample from the distribution Alg 0 ( · , ·| D t -1 i , s ) for each s ∈ S (i.e., no additional interactions with the environment). Moreover, the reason to incorporate such an augmentation is to provide additional diverse pre-training data due to the unique game-theoretic environment, with further discussions provided after the later Lemma 3.6. It has also been recognized previously [16, 72] that the data requirement for learning Markov games is typically much stronger than that for single-agent RL.

To facilitate the decentralized training, the overall dataset is further split into two parts: { D + ,i : i ∈ [ N ] } and { D -,i : i ∈ [ N ] } , where

In other words, D i, + denotes the observations of the max-player, while D i, -those of the min-layer. Note that neither player can observe the opponent's actions.

Algorithm Induced by Transformers. Due to the decentralized nature, two embedding mappings of d + and d -dimensions are considered as h + : S ∪ ( A×R ) → R d + and h -: S ∪ ( B × R ) → R d -, together with two transformers TF θ + and TF θ -. Taking the max-player's transformer as representative, for trajectory ( D t -1 + , s t ) , let H + = h + ( D t -1 + , s t ) = [ h + ( s 1 ) , h + ( a 1 , r 1 ) , · · · , h + ( s t )] be the input to TF θ + , and the obtained output is H + = TF θ + ( H + ) = [ h + , 1 , h + , 2 , · · · , h + , -2 , h + , -1 ] , which has the same shape as H + . Similarly, the mapping h -is used for the min-player's transformer TF θ -to embed trajectory ( D t -1 -, s t ) .

We further assume that two fixed linear extraction mappings, A ∈ R A × d + and B ∈ R B × d -, are used to induce algorithms Alg θ + and Alg θ -over the action spaces A and B of the max- and min-players, respectively, as

where proj ∆ denotes the projection to a probability simplex.

Training Scheme. We consider the standard supervised pre-training to maximize the log-likelihood of observing training datasets D + (resp., D -) over algorithms { Alg θ + : θ + ∈ Θ + } (resp., { Alg θ -: θ -∈ Θ -} ) with Θ + := Θ d + ,L + ,M + ,d ′ + ,F + (resp., Θ -:= Θ d -,L -,M -,d ′ -,F -). In particular, the pre-training outputs ̂ θ + and ̂ θ -are determined as

3.1.2 Theoretical Guarantees

In this section, we provide a generalization guarantee of the algorithms Alg ̂ θ + and Alg ̂ θ -pre-trained following the scheme introduced above. First, the standard definition regarding the covering number and an assumption of approximate realizability are introduced to facilitate the analysis, which are also leveraged in Lin et al. [38].

Definition 3.1 (Decentralized Covering Number) . For a class of algorithms { Alg θ + : θ + ∈ Θ + } , we say ˜ Θ + ⊆ Θ + is a ρ + -cover of Θ + , if ˜ Θ + is a finite set such that for any θ + ∈ Θ + , there exists ˜ θ + ∈ ˜ Θ + such that for all D t -1 + , s ∈ S , t ∈ [ T ] , it holds that

The covering number N Θ + ( ρ + ) is the minimal cardinality of ˜ Θ + such that ˜ Θ + is a ρ + -cover of Θ + . Similarly, we can define the ρ --cover of Θ -and the covering number N Θ -( ρ -) .

Assumption 3.2 (Decentralized Approximate Realizability) . There exist θ ∗ + ∈ Θ + and ε + , real > 0 such that for all t ∈ [ T ] , s ∈ S , a ∈ A , it holds that

We also similarly assume ε -, real -approximate realizability of Alg -, 0 via Alg θ ∗ -with θ ∗ -∈ Θ -.

Then, we can establish the following generalization guarantee on the TV distance between ( Alg ̂ θ + , Alg ̂ θ -) and Alg 0 = ( Alg 0 , + , Alg 0 , -) , capturing their similarities.

Theorem 3.3 (Decentralized Pre-training Guarantee) . Let ̂ θ + be the max-player's pre-training output defined in Sec. 3.1.1. Take N Θ + = N Θ + (1 /N ) as in Def. 3.1. Then, under Assumption 3.2, with probability at least 1 -δ , it holds that 1

A similar result holds for the min-players' pre-training output ̂ θ -.

Theorem 3.3 demonstrates that in expectation of the pre-training data distribution, i.e., P Alg 0 Λ ( D ) , the TV distance between the pre-trained algorithm Alg ̂ θ + (resp, Alg ̂ θ -) and the context algorithm Alg + , 0 (resp, Alg -, 0 ) can be bounded via two terms: one from the approximate realizability, i.e., ε + , real (resp, ε -, real), and the other from the limited amount of pre-training trajectories, i.e., finite N . While we can diminish the second term via a large pre-training dataset (i.e., sufficient pre-training games), the key question is whether the transformer structure is sufficiently expressive to realize the context algorithm, i.e., having a small ε + , real, which we affirmatively answer via an example of realizing V-learning [32] in the next subsection.

3.2 Realizing V-learning

To demonstrate the capability of transformers in the decentralized game-playing setting, we choose to prove that they can realize the renowned V-learning algorithm [32], the first design that breaks the curse of multiple agents in learning Markov games. Particularly, V-learning leverages techniques from adversarial bandits [5] to perform policy updates without observing the opponent's actions. The details of V-learning are provided in Appendix G.1, where its unique output rule is also elaborated.

In the following theorem, we demonstrate that a transformer can be constructed to exactly perform V-learning with a suitable parameterization. One additional Assumption G.2 on the existence of

a transformer parameterized by the class of Θ d,L D ,M D ,d D ,F D to perform exact division is adopted for the convenience of the proof, while in Appendix G.2, we further demonstrate that the required division operation can be approximated to any arbitrary precision.

Theorem 3.4. With embedding mapping h + and extraction mapping A defined in Appendix G.3, under Assumption G.2, there exists a transformer TF θ + with

which satisfies that for all D t -1 + , s ∈ S , t ∈ [ T ] , Alg θ + ( ·| D t -1 + , s ) = Alg V-learning ( ·| D t -1 + , s ) . A similar construction TF θ -exists for the min-player's transformer such that for all D t -1 -, s ∈ S , t ∈ [ T ] , Alg θ -( ·| D t -1 -, s ) = Alg V-learning ( ·| D t -1 -, s ) .

The proof of Theorem 3.4 (presented in Appendix G.3) is challenging because V-learning is a modelfree design, while UCB-VI [6] studied in Lin et al. [38] and VI-ULCB [8] later presented in Sec. 4.2 are both model-based ones. We believe this result deepens our understanding of the capability of pre-trained transformers in decision-making, i.e., they can realize both model-based and model-free designs, showcasing their further potentials.

More specifically, with the embedded trajectory as the input, the model-based philosophy is natural for the masked attention mechanism, i.e., the value computation at each step is directly over all raw inputs in previous steps. Thus, the construction procedure is straightforward as (input 1 , input 2 , ..., input t ) → (value 1 , value 2 , ..., value t ). However, the model-free designs are different, where value computation at one step requires previous values (instead of raw inputs). In other words, the construction procedure is a recursive one as (input 1 , input 2 , ..., input t ) → (value 1 , input 2 , ..., input t ) → (value 1 , value 2 , ..., input t ) → ... → (value 1 , value 2 , ..., value t ), whose realization requires carefully crafted constructions.

3.3 The Overall ICGP Capablity

Finally, built upon the obtained results, the following theorem demonstrates the ICGP capability of pre-trained transformers in the decentralized setting.

Theorem 3.5. Let Θ + and Θ -be the classes of transformers satisfying the requirements in Theorem 3.4 and ( Alg + , 0 , Alg -, 0 ) both be V-learning. Let (ˆ µ, ˆ ν ) be the output policies via the output rule of V-learning. Denoting Alg ̂ θ = ( Alg ̂ θ + , Alg ̂ θ -) and N Θ = N Θ + N Θ -. Then, with probability at least 1 -δ , it holds that

With the obtained upper bound on the approximation error of NE, Theorem 3.5 demonstrates the ICGP capability of pre-trained transformers as the algorithms Alg ˆ θ + and Alg ˆ θ -are fixed during interactions with varying inference games (i.e., no parameter updates). When prompted by the interaction trajectory in the current game, they are capable of deciding the future interaction strategy and finally provide policy pairs that are approximate NE. We further note that during both pre-training and inference, each player's transformer takes inputs of its own observed trajectories, but not the opponent's actions, which reflects the decentralized requirement. Moreover, the approximation error in Theorem 3.5 depends on A ∨ B instead of AB as in the later Theorem 4.2, evidencing the benefits of decentralized learning.

Proof Sketch. The proof of Theorem 3.5 (presented in Appendix H) rely on the following, decomposition, where E 0 [ · ] and E ̂ θ [ · ] are with respect to P Alg 0 Λ and P Alg ̂ θ Λ , respectively:

It can be observed that the first decomposed term is the performance of the considered V-learning, which can be obtained following Jin et al. [32] as in Theorem G.1.

Then, the remaining terms concern the performance of the policy pair (ˆ µ, ˆ ν ) learned from Alg 0 and Alg ̂ θ against their own best responses , respectively. This is drastically different from the consideration in Lin et al. [38], which only bounds the performance of the learned policies, i.e.,

The involvement of best responses complicates the analysis. After careful treatments in Appendix H, we obtain the following lemma to characterize these terms.

Lemma 3.6. For any two decentralized algorithms Alg α and Alg β , we denote their performed policies for episode g are ( µ g α , ν g α ) and ( µ g β , ν g β ) , and their final output policies via the output rule of V-learning (see Appendix G.1) are (ˆ µ α , ˆ ν α ) and (ˆ µ β , ˆ ν β ) . For { ˆ µ α , ˆ µ β } , it holds that

where E α [ · ] and E β [ · ] are with respect to P Alg α Λ and P Alg β Λ . A similar result holds for { ˆ ν α , ˆ ν β } .

With Lemma 3.6, we can incorporate Theorem 3.3 to upper bound the TV distance between Alg 0 and Alg ̂ θ , which together with Theorem 3.4 establish ε real , + = ε real , -= 0 in this case, leading to the desired performance guarantee in Theorem 3.5. We here further note that the effectiveness of Theorem 3.3 in capturing the bound in Lemma 3.6 over all s ∈ S credits to the augmented dataset D ′ , which provides diverse data of all s ∈ S .

4 Centralized Learning

In this section, we discuss the scenario of centralized learning, i.e., training one joint model to control both players' interactions. This is also known as the self-play setting [8, 9, 33, 40, 67]. Following a similar procedure as the decentralized discussions, we first provide supervised pre-training guarantees and then demonstrate that transformers are capable of realizing the renowned VI-ULCB algorithm [8]. It is thus established that in a centralized learning setting, the pre-trained transformer can still effectively perform ICGP and approximate NE.

4.1 Supervised Pre-training Results

The same training dataset { D i : i ∈ [ N ] } as in Section 3.1.1 is considered. As the centralized setting is studied here, no further split of the dataset is needed. Moreover, one d -dimensional mapping h : S ∪ ( A×B×R ) → R d can be designed to embed the trajectories, and the induced algorithm Alg θ ( · , ·| D t , s t ) from the transformer TF θ can be obtained via a fixed linear extraction mapping E similarly as Eqn. (1). Finally, the MLE training is performed with Θ := Θ d,L,M,d ′ ,F as ̂ θ = arg max θ ∈ Θ 1 N ∑ i ∈ [ N ] ∑ t ∈ [ T ] ∑ s ∈S log ( Alg θ ( a t i,s , b t i,s | D t -1 i , s ) ) .

Then, a generalization guarantee of Alg ̂ θ can be provided similarly as Theorem 3.3, which is deferred to Theorem C.3. This centralized result also implies that the pre-trained centralized algorithm performs similarly as the context algorithm, with errors caused by the approximate realizability and the finite pre-training data.

4.2 Realizing VI-ULCB

The VI-ULCB algorithm [8] is one of the first provably efficient centralized learning designs for Markov games. It extends the key idea of using confidence bounds to incorporate uncertainties from stochastic bandits and MDPs [4, 6] to handle competitive environments, and has further inspired many extensions in Markov games [9, 30, 33, 40, 65]. As VI-ULCB is highly representative, we choose it as the example for realization in the centralized setting to demonstrate the capability of transformers.

To make VI-ULCB practically implementable, we adopt an approximate CCE solver powered by multiplicative weight update (MWU) in the place of its originally required general-sum NE solver

(which is computationally demanding). This modification is demonstrated as provably efficient in later works [9, 40, 65]. Then, the following result illustrates that a transformer can be constructed to exactly perform the MWU-version of VI-ULCB.

Theorem 4.1. With embedding mapping h and extraction mapping E defined in Appendix D.2, there exists a transformer TF θ with

which satisfies that for all D t -1 , s ∈ S , t ∈ [ T ] , Alg θ ( · , ·| D t -1 , s ) = Alg VI-ULCB ( · , ·| D t -1 , s ) .

One observation from the proof of Theorem 4.1 (presented in Appendix D.2) is that transformer layers can perform MWU so that an approximate CCE can be found, which is not reported in Lin et al. [38] and further demonstrates the in-context learning capability of transformers in playing normal-form games (since MWU is one of the most basic designs).

4.3 The Overall ICGP Capability

With Theorem 4.1 showing VI-ULCB can be exactly realized (i.e., ε real = 0 in Assumption C.2), we can further prove an overall upper bound of the approximation error of NE by Alg ̂ θ via the following theorem, demonstrating the ICGP capability of transformers.

Theorem 4.2. Let Θ be the class of transformers satisfying the requirements in Theorem 4.1 and Alg 0 be VI-ULCB. For all ( g, h, s ) ∈ [ G ] × [ H ] ×S , let ( µ g,h ( ·| s ) , ν g,h ( ·| s )) be the marginalized policies of Alg ̂ θ ( · , ·| D t -1 , s ) . Then, with probability at least 1 -δ , it holds that

where ˆ µ and ˆ ν are uniformly sampled as ˆ µ ∼ Unif { µ 1 , · · · , µ G } and ˆ ν ∼ Unif { ν 1 , · · · , ν G } , with µ g := { µ g,h ( ·| s ) : ( h, s ) ∈ [ H ] ×S} and ν g := { ν g,h ( ·| s ) : ( h, s ) ∈ [ H ] ×S} , and E ˆ µ, ˆ ν is with respect to the process of policy sampling.

This result demonstrates the ICGP capability of pre-trained transformers in the centralized setting, complementing the discussions in the decentralized results.

5 Empirical Experiments

Experiments are performed on two-player zero-sum normal-form games ( H = 1 ) and Markov games ( H = 2 ), with the decentralized EXP3 [5] (which can be viewed as a one-step V-learning) and the centralized VI-ULCB being the context algorithms as demonstrations, respectively. Additional experimental setups and details can be found in Appendix J. It can be first observed from Fig. 2 that, the transformers pre-trained with N = 20 games performs better on the inference tasks than the ones pre-trained with N = 10 games. This observation empirically validates the theoretical result that more pre-training games benefit the final game-playing performance during inference (i.e., the √ 1 /N -dependencies established in Theorems 3.5 and 4.2). Moreover, when the number of pre-training games is sufficient (i.e., N = 20 in Fig. 2), the obtained transformers can indeed learn to approximate NE in an in-context manner (i.e., having a gradually decaying NE gap), and also the obtained performance is similar to the context algorithm, i.e., EXP3 or VI-ULCB. These observations provide empirical pieces of evidence to support the ICGP capabilities of pre-trained transformers, motivating and validating the theoretical analyses performed in this work.

6 Related Works

In-context Learning. Since GPT-3 [11] demonstrates the ICL capability of pre-trained transformers, growing attention has been paid to this direction. In particular, an emerging line of work targets providing a deeper understanding of the fundamental mechanism behind the ICL capability [3, 24, 29, 31, 37, 44, 59, 60, 64, 66, 68], where many interesting results have been obtained. In particular, transformers have been shown to be capable of performing in-context gradient descent so that

0

500

1000

1500

2000

2500

3000

Episodes

0.225

0.250

0.275

0.300

0.325

0.350

0.375

0.400

NE Gap

Transformer (20 envs)

Transformer (10 envs)

Exp3

(a) Decentralized Comparisons With EXP3.

0

50

100

150

200

250

300

Episodes

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

NE Gap

Transformer (20 envs)

Transformer (10 envs)

VI-ULCB

(b) Centralized Comparisons With VI-ULCB.

varying optimization-based algorithms can be realized [2, 3, 7, 26, 59]. Also, Giannou et al. [27] demonstrates that looped transformers can emulate basic computing blocks, whose combinations can lead to complex operations.

This work is more focused on the in-context reinforcement learning (ICRL) capability of pre-trained transformers, as demonstrated in Grigsby et al. [28], Laskin et al. [34], Lee et al. [35], Wang et al. [62]. The recent work by Lin et al. [38] initiates the theoretical investigation of this topic. In particular, Lin et al. [38] provides generalization guarantees after pre-training in the single-agent RL scenario, and further constructs transformers to realize provably efficient single-agent bandits and RL algorithms (in particular, LinUCB [1], Thompson sampling [54], UCB-VI [6]). This work extends Lin et al. [38] to the domain of competitive multi-agent RL by studying the in-context game-playing setting. A recent concurrent work [36] also touches upon the in-context game-playing capability of pre-trained transformers, while focusing on practical aspects and exploiting different opponents.

Competitive Multi-agent RL. The study of RL in the competitive multi-agent domain has a long and fruitful history [10, 47, 49, 58, 70]. In recent years, researchers have gained a deeper theoretical understanding of this topic. The centralized setting (also known as self-play) has been investigated in Bai and Jin [8], Bai et al. [9], Cui et al. [17], Huang et al. [30], Jin et al. [33], Liu et al. [40], Wang et al. [63], Xiong et al. [67], Zhang et al. [69], and this work focuses on the representative VI-ULCB design [9]. On the other hand, decentralized learning is more challenging, and the major breakthrough is made by V-learning [9, 32, 41, 51], which is thus adopted as the target algorithm in this work.

7 Conclusions

This work investigated the in-context game-playing (ICGP) capabilities of pre-trained transformers, broadening the research scope of in-context RL from the single-agent scenario to the more challenging multi-agent competitive games. Focusing on the classical two-player zero-sum Markov games, a general learning framework was first introduced, laying down a solid ground for this and later studies. Through concrete theoretical results, this work further demonstrated that in both decentralized and centralized learning settings, properly pre-trained transformers are capable of approximating Nash equilibrium in an in-context manner. As a key part of the proof, concrete sets of parameterization were provided to demonstrate that the transformer architecture can realize two famous designs, decentralized V-learning and centralized VI-ULCB. Empirical experiments further validate the theoretical results (especially that pre-trained transformers can indeed approximate NE in an incontext manner) and motivate future studies on this under-explored research direction.

Acknowledgments and Disclosure of Funding

The work of CSs and KY was supported in part by the US National Science Foundation (NSF) under awards CNS-2002902, ECCS- 2029978, ECCS-2143559, and CNS-2313110, and the Bloomberg Data Science Ph.D. Fellowship. The work of JY was supported in part by the US NSF under awards CNS-1956276 and CNS-2114542.

References

Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24.

Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. (2023). Transformers learn to implement preconditioned gradient descent for in-context learning. arXiv preprint arXiv:2306.00297 .

Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. (2022). What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661 .

Auer, P., Cesa-Bianchi, N., and Fischer, P. (2002a). Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256.

Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. (2002b). The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77.

Azar, M. G., Osband, I., and Munos, R. (2017). Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR.

Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. (2023). Transformers as statisticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04637 .

Bai, Y. and Jin, C. (2020). Provable self-play algorithms for competitive reinforcement learning. In International conference on machine learning , pages 551-560. PMLR.

Bai, Y., Jin, C., and Yu, T. (2020). Near-optimal reinforcement learning with self-play. Advances in neural information processing systems , 33:2159-2170.

Berner, C., Brockman, G., Chan, B., Cheung, V., D˛ ebiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. (2019). Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 .

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901.

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020). End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer.

Cesa-Bianchi, N. and Lugosi, G. (2006). Prediction, learning, and games . Cambridge university press.

Cheng, X., Chen, Y., and Sra, S. (2023). Transformers implement functional gradient descent to learn non-linear functions in context. arXiv preprint arXiv:2312.06528 .

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2023). Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1-113.

Cui, Q. and Du, S. S. (2022). When are offline two-player zero-sum markov games solvable? Advances in Neural Information Processing Systems , 35:25779-25791.

Cui, Q., Zhang, K., and Du, S. (2023). Breaking the curse of multiagents in a large state space: Rl in markov games with independent linear function approximation. In The Thirty Sixth Annual Conference on Learning Theory , pages 2651-2652. PMLR.

Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., and Wei, F. (2023). Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 4005-4019.

Daskalakis, C. (2013). On the complexity of approximating a nash equilibrium. ACM Transactions on Algorithms (TALG) , 9(3):1-35.

Daskalakis, C., Fishelson, M., and Golowich, N. (2021). Near-optimal no-regret learning in general games. Advances in Neural Information Processing Systems , 34:27604-27616.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 .

Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. (2022). A survey for in-context learning. arXiv preprint arXiv:2301.00234 .

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 .

Fu, D., Chen, T.-Q., Jia, R., and Sharan, V . (2023). Transformers learn higher-order optimization methods for in-context learning: A study with linear models. arXiv preprint arXiv:2310.17086 .

Gao, B. and Pavel, L. (2017). On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805 .

Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. (2022). What can transformers learn incontext? a case study of simple function classes. Advances in Neural Information Processing Systems , 35:30583-30598.

Giannou, A., Rajput, S., Sohn, J.-y., Lee, K., Lee, J. D., and Papailiopoulos, D. (2023). Looped transformers as programmable computers. In International Conference on Machine Learning , pages 11398-11442. PMLR.

Grigsby, J., Fan, L., and Zhu, Y . (2023). Amago: Scalable in-context reinforcement learning for adaptive agents. arXiv preprint arXiv:2310.09971 .

Guo, T., Hu, W., Mei, S., Wang, H., Xiong, C., Savarese, S., and Bai, Y. (2023). How do transformers learn in-context beyond simple functions? a case study on learning with representations. arXiv preprint arXiv:2310.10616 .

Huang, B., Lee, J. D., Wang, Z., and Yang, Z. (2021). Towards general function approximation in zero-sum markov games. arXiv preprint arXiv:2107.14702 .

Huang, Y., Cheng, Y., and Liang, Y. (2023). In-context convergence of transformers. arXiv preprint arXiv:2310.05249 .

Jin, C., Liu, Q., Wang, Y., and Yu, T. (2023). V-learning-a simple, efficient, decentralized algorithm for multiagent reinforcement learning. Mathematics of Operations Research .

Jin, C., Liu, Q., and Yu, T. (2022). The power of exploiter: Provable multi-agent rl in large state spaces. In International Conference on Machine Learning , pages 10251-10279. PMLR.

Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al. (2022). In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215 .

Lee, J. N., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. (2023). Supervised pretraining can learn in-context reinforcement learning. arXiv preprint arXiv:2306.14892 .

Li, S., Yang, C., Zhang, Y ., Li, P., Wang, X., Huang, X., Chan, H., and An, B. (2024). In-context exploiter for extensive-form games. arXiv preprint arXiv:2408.05575 .

Li, Y., Ildiz, M. E., Papailiopoulos, D., and Oymak, S. (2023). Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning , pages 19565-19594. PMLR.

Lin, L., Bai, Y., and Mei, S. (2023). Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. arXiv preprint arXiv:2310.08566 .

Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. (2021a). What makes good in-context examples for gpt3 ? arXiv preprint arXiv:2101.06804 .

Liu, Q., Yu, T., Bai, Y., and Jin, C. (2021b). A sharp analysis of model-based reinforcement learning with self-play. In International Conference on Machine Learning , pages 7001-7010. PMLR.

Mao, W. and Ba¸ sar, T. (2023). Provably efficient reinforcement learning in decentralized general-sum markov games. Dynamic Games and Applications , 13(1):165-186.

OpenAI (2023). GPT-4 technical report. arXiv preprint , arXiv:2303.08774.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners.

Raventós, A., Paul, M., Chen, F., and Ganguli, S. (2023). Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. arXiv preprint arXiv:2306.15063 .

Reddy, G. (2023). The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Representations .

Roughgarden, T. (2010). Algorithmic game theory. Communications of the ACM , 53(7):78-86.

Shapley, L. S. (1953). Stochastic games. Proceedings of the national academy of sciences , 39(10):1095-1100.

Shoham, Y. and Leyton-Brown, K. (2008). Multiagent systems: Algorithmic, game-theoretic, and logical foundations . Cambridge University Press.

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017). Mastering the game of go without human knowledge. nature , 550(7676):354-359.

Sinii, V ., Nikulin, A., Kurenkov, V ., Zisman, I., and Kolesnikov, S. (2023). In-context reinforcement learning for variable action spaces. arXiv preprint arXiv:2312.13327 .

Song, Z., Mei, S., and Bai, Y. (2021). When can we learn general-sum markov games with a large number of players sample-efficiently? arXiv preprint arXiv:2110.04184 .

Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine learning , 3:9-44.

Syrgkanis, V., Agarwal, A., Luo, H., and Schapire, R. E. (2015). Fast convergence of regularized learning in games. Advances in Neural Information Processing Systems , 28.

Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3/4):285-294.

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023a). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 .

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023b). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 .

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30.

Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019). Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354.

Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. (2023a). Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151-35174. PMLR.

Von Oswald, J., Niklasson, E., Schlegel, M., Kobayashi, S., Zucchet, N., Scherrer, N., Miller, N., Sandler, M., Vladymyrov, M., Pascanu, R., et al. (2023b). Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858 .

Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press.

Wang, J., Blaser, E., Daneshmand, H., and Zhang, S. (2024). Transformers learn temporal difference methods for in-context reinforcement learning. arXiv preprint arXiv:2405.13861 .

Wang, Y., Liu, Q., Bai, Y., and Jin, C. (2023). Breaking the curse of multiagency: Provably efficient decentralized multi-agent rl with function approximation. arXiv preprint arXiv:2302.06606 .

Wu, J., Zou, D., Chen, Z., Braverman, V., Gu, Q., and Bartlett, P. L. (2023). How many pretraining tasks are needed for in-context learning of linear regression? arXiv preprint arXiv:2310.08391 .

Xie, Q., Chen, Y., Wang, Z., and Yang, Z. (2020). Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. In Conference on learning theory , pages 3674-3682. PMLR.

Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. (2021). An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080 .

Xiong, W., Zhong, H., Shi, C., Shen, C., and Zhang, T. (2022). A self-play posterior sampling algorithm for zero-sum markov games. In International Conference on Machine Learning , pages 24496-24523. PMLR.

Yun, C., Bhojanapalli, S., Rawat, A. S., Reddi, S. J., and Kumar, S. (2019). Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077 .

Zhang, K., Kakade, S., Basar, T., and Yang, L. (2020). Model-based multi-agent rl in zero-sum markov games with near-optimal sample complexity. Advances in Neural Information Processing Systems , 33:1166-1178.

Zhang, K., Yang, Z., and Ba¸ sar, T. (2021). Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control , pages 321-384.

Zhang, R., Frei, S., and Bartlett, P. L. (2023). Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.09927 .

Zhong, H., Xiong, W., Tan, J., Wang, L., Zhang, T., Wang, Z., and Yang, Z. (2022). Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. In International Conference on Machine Learning , pages 27117-27142. PMLR.

Zisman, I., Kurenkov, V., Nikulin, A., Sinii, V., and Kolesnikov, S. (2023). Emergence of in-context reinforcement learning from noise distillation. arXiv preprint arXiv:2312.12275 .

A An Overview of the Appendix

In this section, an overview of the appendix is provided. First, additional discussions are presented in Appendix B, which cover broader impacts of this work and our thoughts on the future directions.

Then, the proof details omitted in this main paper are provided. While the decentralized learning setting is the major focus in the main paper, the discussions and proofs for the centralized learning setting are first provided to facilitate the presentation and understanding as the decentralized learning setting is more challenging.

The supervised pre-training guarantee (i.e., Theorem C.3) for the centralized learning setting is proved in Appendix C. The details and realization of VI-ULCB (i.e., Theorem 4.1) are presented in Appendix D. The proofs for the overall performance guarantee (i.e., Theorem 4.2) can be found in Appendix E.

Subsequently, the proofs for the supervised pre-training guarantee (i.e., Theorem 3.3) in the decentralized learning setting are provided in Appendix F. Appendix G contains the details and realization of V-learning (i.e., Theorem 3.4). The overall performance guarantee (i.e., Theorem 3.5) is proved in Appendix H.

A detailed discussion of the covering number is provided in Appendix I.

Finally, the setups and details of the experiments presented in Sec. 5 are reported in Appendix J.

B Additional Discussions

B.1 Broader Impacts

This work mainly provides a theoretical understanding of the in-context game-playing capabilities of pre-trained transformers, broadening the research scope of in-context reinforcement learning from single-agent settings to multi-agent competitive games. Due to its theoretical nature, we do not foresee major negative societal impacts; however, we still would like to acknowledge the need for responsible usage of the practical implementation of the proposed game-playing transformers due to their high capability in various environments.

B.2 Limitations and Future Works

The research direction of in-context game-playing is currently under-explored, and we believe that there are many interesting topics to be further investigated.

· Different game forms and game-solving algorithms. This work mainly studies the classical twoplayer zero-sum Markov games, which can be viewed as the most basic form of competitive games, and has particular focuses on constructing transformers to realize V-learning [32] and VI-ULCB [8]. The cooperative games, on the other hand, are conceptually more similar to the single-agent setting [38]. There are more complicated game forms [46, 48], e.g., the mixed cooperative-competitive games, which requires different game-solving algorithms. We believe the framework built in this work is beneficial to further explore the capabilities of pre-trained models in game-theoretical scenarios.

· Pre-training dataset construction. This work considering the pre-training dataset is collected from the context algorithm with an additional augmentation. First, while the current proofs rely on the augmentation, it will be an interesting topic to understand whether it is necessary. As mentioned Sec. 3.1.1, learning in Markov games typically require more diverse data than learning in singleagent settings; however, the minimum requirement to perform effective pre-training is worth further exploring. Moreover, in the study of single-agent RL [35], it is shown that pre-training with the data from the optimal policy is more efficient, which is further theoretically investigated in Lin et al. [38]. In multi-agent competitive games, it is currently unclear whether similar strategies can be incorporated, e.g., pre-training with data collected by Nash equilibrium policies or best responses for certain other policies.

· Large-scale empirical evaluations. Due to the limited computation resources, the experiments reported in Sec. 5 are relatively small-scale compared with the current size of practically adopted transformers. It would be an important and interesting direction to further evaluate the ICGP capabilities of pre-trained transformers in large-scale experiments and practical game-theoretic

applications. Also, the training dynamics are also worth further investigation, e.g., the sudden shifts in learning effectiveness reported by Reddy [45].

Besides these directions, from the theoretical perspective, we believe it would be valuable to investigate how to extend the current study on the tabular setting to incorporate function approximation, where we conjecture it is sufficient for the pre-training dataset to cover information of certain representative states and actions (e.g., a well-coverage of the feature space) [72]. Another attractive theoretical question is how to learn from a dataset collected by multiple context algorithms. From the practical perspective, a future study on the impact of the practical training recipe (e.g., model structure, training hyperparameters, etc.) would be desirable to bring additional insights.

C Proofs for the Centralized Supervised Pre-training Guarantees

First, the definition of the centralized covering number and an assumption of centralized approximate realizability are introduced to facilitate the analysis, which are also leveraged in Lin et al. [38].

Definition C.1 (Centralized Covering Number) . For a class of algorithms { Alg θ : θ ∈ Θ } , we say ˜ Θ ⊆ Θ is a ρ -cover of Θ , if ˜ Θ is a finite set such that for any θ ∈ Θ , there exists ˜ θ ∈ ˜ Θ such that for all D t -1 , s ∈ S , t ∈ [ T ] , it holds that

The covering number N Θ ( ρ ) is the minimal cardinality of ˜ Θ such that ˜ Θ is a ρ -cover of Θ .

Assumption C.2 (Centralized Approximate Realizability) . There exist θ ∗ ∈ Θ and ε real > 0 such that for all s ∈ S , t ∈ [ T ] , ( a, b ) ∈ A × B , it holds that

Then, the following pre-training guarantee can be established.

Theorem C.3 (Centralized Pre-training Guarantee) . Let ̂ θ be the maximum likelihood pre-training output. Take N Θ = N Θ (1 /N ) as in Definition C.1. Then, under Assumption C.2, with probability at least 1 -δ , it holds that

Proof of Theorem C.3. This proof extends that of Theorem 6 in Lin et al. [38] to the multi-agent scenario. Let ˜ Θ be a ρ -covering set of Θ with covering number N Θ = N Θ ( ρ ) as defined in Definition C.1. With Lemma 15 in Lin et al. [38], we can obtain that for any θ ∈ Θ , there exists ˜ θ ∈ ˜ Θ such that for all D t -1 , t ∈ [ T ] and s ∈ S ,

For m ∈ [ N Θ ] , t ∈ [ T ] , i ∈ [ N ] , s ∈ S , we define that

According to Lemma 14 in Lin et al. [38], with probability at least 1 -δ , for all m ∈ [ N Θ ] , t ∈ [ T ] , s ∈ S , it holds that

Furthermore, it can be established that

which implies that

where the inequality is from the fact that the Hellinger distance is smaller than the TV distance.

Then, we can obtain that for any θ covered by θ m , it holds that

which further implies that

Thus, for the obtained ̂ θ , with probability at least 1 -δ , it holds that

≤ 2 NSTε real + ST log( TS/δ ) + 2 NSTρ 2 +2 NSTρ +4 ST log( N Θ TS/δ ) ,

Further by Cauchy-Schwarz inequality, we can obtain that

Taking ρ = 1 /N concludes the proof.

D Proofs for Realizing VI-ULCB

D.1 Details of MWU VI-ULCB

We here note one distinction from the VI-ULCB design considered in this work from its vanilla version proposed in Bai and Jin [8], which makes VI-ULCB practically implementable. Especially, Bai and Jin [8] requires an oracle solver that can provide the exact NE policy pair ( µ ∗ , ν ∗ ) from any two general input payoff matrices ( Q,Q ) ∈ R A × B × R A × B . However, it is known that approximating such a general-sum NE is computationally hard (specifically, PPAD-complete) [19], which makes this vanilla version impractical. Luckily, later studies [9, 40, 65] have demonstrated that a solver finding one weaker notation of equilibrium, i.e., coarse correlated equilibrium (CCE), is already sufficient. Following these recent results, we replace the NE solver with an approximate CCE solver in VI-ULCB. Moreover, we consider finding such CCEs via no-regret learning. 2 In particular, both players virtually run multiplicative weight update (MWU) (which is also known as Hedge ), a classical no-regret algorithm, with payoff matrices ( Q,Q ) for several rounds; then, an aggregated policy can be generated as an approximate CCE. The details of the VI-ULCB algorithm are provided in Alg. 1.

More specifically, we consider that an approximate CCE solver is adopted such that with each pair of inputs ( Q h ( s, · , · ) , Q h ( s, · , · )) , we can obtain an ε CCE-approximate CCE policy π h ( · , ·| s ) which satisfies that

We also specifically choose to obtain such approximate CCEs by having both players (virtually) perform MWU against each other. The details of MWU are included in Alg. 2, where we use the following notations to denote normalized losses:

Standard online learning results [13] guarantee that using learning rates η A = √ log( A ) /N MWU and η B = √ log( B ) /N MWU, after N MWU rounds of MWU, the policy

is an ε CCE-approximate CCE policy, with

Algorithm 1 VI-ULCB

1: Initialize : for any ( s, a, b, h ) , Q h ( s, a, b ) ← H , Q h ( s, a, b ) ← 0 , N h ( s, a, b ) ← 0 , N h ( s, a, b, s ′ ) ← 0

2: for episode g = 1 , · · · , G do

3: for ( s, a, b ) ∈ S × A × B do

6: end for

7: for s ∈ S do

9: Compute V h ( s ) ← ∑ a,b π h ( a, b | s ) Q h ( s, a, b )

8: Update π h ( · , ·| s ) ← ε N -approximate CCE ( Q h ( s, · , · ) , Q h ( s, · , · ) ) solved by N -round MWU

11: end for

12: for step h = 1 , · · · , H do

13: Take action ( a h , b h ) ∼ π h ( · , ·| s h )

14: Observe reward r h and next state s h +1

15: Update N h ( s h , a h , b h ) and N h ( s h , a h , b h , s h +1 )

16: Update ˆ P h ( ·| s h , a h , b h ) and ˆ r h ( s h , a h , b h )

17: end for

18: end for

Algorithm 2 MWU

1: Input : learning rates η A = √ log( A ) /N and η B = √ log( B ) /N , action sets A and B with size A and B , loss matrices L h ( s, · , · ) and L h ( s, · , · )

2: Initialize : cumulative loss O + ← 0 A and O -← 0 A

3: for n = 1 , · · · , N do

4: Compute µ h n ( ·| s ) ← σ s ( -η A O + ) ∈ ∆( A )

6: Observe vectors o + ,n ∈ R A with o + ,n ( a ) = ν h n ( ·| s ) · L h ( s, a, · )

5: Compute ν h n ( ·| s ) ← σ s ( -η B O -) ∈ ∆( B )

7: Observe vectors o -,n ∈ R B with o -,n ( b ) = µ h n ( ·| s ) · L h ( s, · , b )

8: Update O + = O + + o + ,n and O -= O -+ o -,n

9: end for

10: Output: policy ∑ n ∈ [ N ] µ h n ( ·| s ) ν h n ( ·| s ) /N

Furthermore, for a certain bounded ε CCE, Xie et al. [65] demonstrated that the performance degradation can still be controlled. Following the results therein, the following theorem can be easily established.

Theorem D.1 (Modified from Theorem 2 from Bai and Jin [8]) . With probability at least 1 -δ , in any environment M , the output policies { ( µ g , ν g ) : g ∈ [ G ] } from the MWU-version of VI-ULCB satisfy that

With δ = 1 /T , to have a non-dominant loss caused by the approximate CCE solver, we can choose N MWU = G . Then, for any environment M , it holds that

D.2 Proof of Theorem 4.1: The Realization Construction

D.2.1 Embeddings and Extraction Mapping

We consider each episode of observations to be embedded in 2 H tokens. In particular, for each t ∈ [ T ] , we construct that

where s g,h , a g,h , b g,h are represented via one-hot embedding. The positional embedding pos i is defined as

where e h is a one-hot vector with the h -th element being 1 and v i := 1 { h a i = 0 } denote the tokens that do not embed actions and rewards.

In summary, for observations D t -1 ∪ { s t } , we obtain the following tokens of length 2 t -1 :

With the above input H , the transformer outputs H = TF θ + ( H ) of the same size as H . The extraction mapping E is directly set to satisfy the following

i.e., the part c of the output tokens is used to store the learned policy.

D.2.2 An Overview of the Proof

In the following, for the convenience of notations, we will consider step t +1 , i.e., with observations D t ∪ { s t +1 } . Given an input token matrix

we construct a transformer to perform the following steps

where we use N h ( s, a, b ) , N h ( s, a, b, s ′ ) , N h ( s, a, b ) r h ( s, a, b ) , ˆ P h ( s ′ | s, a, b ) , ˆ r h ( s, a, b ) , Q h ( s, a, b ) , Q h ( s, a, b ) and π h ( a, b | s ) to denote their entire vectors over h ∈ [ H ] , s ∈ S , a ∈ A , b ∈ B , s ′ ∈ S . The notation ⋆ denotes other quantities in h d 2( t +1) .

The following provides a sketch of the proof.

Step 1 There exists an attention-only transformer TF θ to complete Step 1 with

Step 2 There exists a transformer TF θ to complete Step 2 with

Step 3 There exists a transformer TF θ to complete Step 3 with

Step 4 There exists a transformer TF θ to complete Step 4 with

Step 5 There exists an attention-only transformer TF θ to complete Step 5 with

Step 6 There exists an attention-only transformer TF θ to complete Step 6 with

Thus, the overall transformer TF θ can be summarized as

Also, from the later construction, we can observe that log( R ) = ˜ O (1) .

This can be similarly completed by an attention-only transformer constructed in Step 1 of realizing UCB-VI in Lin et al. [38].

D.2.4 Proof of Step 2: Update ˆ P ( s ′ | s, a, b ) and ˆ r ( s, a, b )

This can be similarly completed by a transformer constructed in Step 2 of realizing UCB-VI in Lin et al. [38].

The computation of Q h ( s ′ | s, a, b ) can be similarly completed by a transformer constructed in Step 3 of realizing UCB-VI in Lin et al. [38]. The Q part can also be obtained by modifying a few plus signs to minuses.

D.2.6 Proof of Step 4: Compute CCE

This is the most challenging part of realizing the VI-ULCB design, which distinguishes it from the single-agent algorithms, e.g., UCB-VI [6]. As mentioned in Appendix D.1, we obtain an approximate CCE via virtually playing MWU. In the following, for one tuple ( s, h ) , we prove that one transformer can be constructed to perform a one-step MWU update with that

To obtain this result, we construct a transformer to perform the following computation from inputs to output for all t ′ ≤ t :

Note that here we again use the notations L h ( s, · , · ) = H -Q h ( s, · , · ) H and L h ( s, · , · ) = H -Q h ( s, · , · ) H to denote the normalized losses. It can be seen that this computation can be performed via one ReLU MLP layer.

Step 4.1: Get o + ,n and o -,n .

First, we can construct that for all t ′ ≤ t

With ReLU activation, this constructed transformer leads to updates that h d 2 t ′ = 0 and h d 2 t ′ +1 = o + ,n ( a ) . With another A -1 paralleling heads, the whole vector o + ,n can be computed. Similarly, with B more paralleling heads, the whole vector o -,n can be computed.

Then, with one ReLU MLP layer, we can obtain that

The required transformer can be summarized as

We can construct a softmax MLP layer such that

where η A = √ log( A ) /G . Thus, µ n +1 ( ·| s ) can be provided. Similarly, another MLP layer { W (2) 1 , W (2) 2 } with softmax activation can provide ν n +1 ( ·| s ) . The current output can be expressed as

We can further construct one more attention layer as

With ReLU activation, this construction would result in that

At last, one ReLU MLP layer { W (3) 1 , W (3) 2 } can be constructed to replace µ n ( ·| s ) and ν n ( ·| s ) with µ n +1 ( ·| s ) and ν n +1 ( ·| s ) :

The required transformer can be summarized as

We can construct

With ReLU activation, this construction can update that

Using an overall AB paralleling heads, we can then obtain µ n +1 ( ·| s ) ν n +1 ( ·| s ) . Then, with a ReLU MLP layer { W (5) 1 , W (5) 2 } , we can obtain ∑ τ ≤ n +1 µ τ ( ·| s ) ν τ ( ·| s ) /N .

The required transformer can be summarized as

Combining all the sub-steps provides proof of a one-step MWU update. The same transformer can be stacked for G times, which completes the G -step MWU.

D.2.7 Proof of Step 5: Compute V h ( s ) and V h ( s )

We can construct that

With ReLU activation, this construction leads to that

Thus, with overall 2 HS paralleling heads, the values of { V h ( s ) , V h ( s ) : h ∈ [ H ] , s ∈ S} can be computed.

The required transformer can be summarized as

D.2.8 Proof of Step 6: Obtain π h +1 ( · , ·| s h +1 )

We can construct one HS -head transformer that for all ( s, h ) ∈ S × [ H ]

With ReLU activation, this construction leads to the update that

Then, we can construct that

With ReLU activation, this construction leads to the update that

The required transformer can be summarized as

E Proofs for the Centralized Overall Performance

Proof of Theorem 4.2. First, we can obtain the decomposition that

Then, via Lemma E.1 and Theorem C.3, we can obtain that

and similarly,

With Theorem 4.1 providing that ε real = 0 , combining the above terms completes the proof of the regret bound. The bound on the covering number, i.e., log( N Θ ) can be obtained via Lemma I.4.

Lemma E.1. For any two centralized algorithms Alg α and Alg β , we denote their performed policies for episode g are ( π g α , π g β ) , whose marginal policies are ( µ g α , ν g α ) and ( µ g β , ν g β ) . For { µ g α , ν g β } , it holds that

where E α [ · ] and E β [ · ] are with respect to P Alg α Λ and P Alg β Λ . A similar result holds for { ν g α , ν g β } .

Proof of Lemma E.1. It holds that

Denoting D H := D ( g -1) H +1: gH and f ( D H ) = ∑ r g,h , we can further obtain that

where (a) is from the definition of best responses, (b) is from the variational representation of the TV distance, (c) is from the fact that TV ( µ × ν, µ ′ × ν ) = TV ( µ, µ ′ ) , and (d) is from the fact that TV ( ∑ b π ( · , b ) , ∑ b π ′ ( · , b )) ≤ TV ( π ( · , · ) , π ′ ( · , · )) . The above relationship further leads to that

Also, denoting g ( D ) = ∑ g ∈ [ G ] V µ β , † ( s 1 ) , it holds that

Combining (term I) and (term II) finishes the proof.

F Proofs for the Decentralized Supervised Pre-training Guarantees

Definition F.1 (The Complete Version of Definition 3.1) . For a class of algorithms { Alg θ + : θ + ∈ Θ + } , we say ˜ Θ + ⊆ Θ + is a ρ + -cover of Θ + , if ˜ Θ + is a finite set such that for any θ + ∈ Θ + , there exists ˜ θ + ∈ ˜ Θ + such that for all D t -1 + , s ∈ S , t ∈ [ T ] , it holds that

The covering number N Θ + ( ρ + ) is the minimal cardinality of ˜ Θ + such that ˜ Θ + is a ρ + -cover of Θ + .

Similarly, for a class of algorithms { Alg θ -: θ -∈ Θ -} , we say ˜ Θ -⊆ Θ -is a ρ --cover of Θ -, if ˜ Θ -is a finite set such that for any θ -∈ Θ -, there exists ˜ θ -∈ ˜ Θ -such that for all D t -1 -, s ∈ S , t ∈ [ T ] , it holds that

The covering number N Θ -( ρ -) is the minimal cardinality of ˜ Θ -such that ˜ Θ -is a ρ --cover of Θ -. Assumption F.2 (The Complete Version of Assumption 3.2) . There exists θ ∗ + ∈ Θ + such that there exists ε + , real > 0 , for all t ∈ [ T ] , s ∈ S , a ∈ A , it holds that

Similarly, there exists θ ∗ -∈ Θ -such that there exists ε -, real > 0 , for all t ∈ [ T ] , s ∈ S , b ∈ B , it holds that

Theorem F.3 (The Complete Version of Theorem 3.3) . Let ̂ θ + be the max-player's pre-training output defined in Section 3.1.1. Take N Θ + = N Θ + (1 /N ) as in Definition F .1. Then, under Assumption F .2, with probability at least 1 -δ , it holds that

Let ̂ θ -be the min-player's pre-training output defined in Section 3.1.1. Take N Θ -= N Θ -(1 /N ) as in Definition F .1. Then, under Assumption F .2, with probability at least 1 -δ , it holds that

Proof of Theorem F.3. This theorem can be similarly proved as Theorem 3.3.

G Proofs for Realizing V-learning

In the following proof, we focus on the max-player's perspective, which can be easily extended for the min-player.

G.1 Details of V-learning

The details of V-learning [32], discussed in Sec. 3.2, are presented in Alg. 3, where the following notations are adopted

After the learning process, V-learning requires an additional procedure to provide the output policy. We include this procedure in Alg. 4, where the following notations are adopted

N g,h ( s ) = the number of times s is visited at step h before episode g, g h i ( s ) = the index of the episode s is visited at step h for the i -th time ,

Following the results in Jin et al. [32], we can obtain the following performance guarantee of V-learning.

Theorem G.1 (Theorem 4 in Jin et al. [32]) . With probability at least 1 -δ , in any environment M , the output policies (ˆ µ, ˆ ν ) from V-learning satisfy that

Thus, with δ = 1 /T , we can obtain that

Algorithm 3 V-learning [32]

1: Initialize: for any ( s, a, h ) ∈ S ×A× [ H ] , V h ( s ) ← H +1 -h , N h ( s ) ← 0 , µ h ( a | s ) ← 1 /A

2: for episode g = 1 , · · · , G do

3: receive s g, 1

4: for step h = 1 , · · · , H do

5: Take action a g,h ∼ π h ( ·| s g,h ) , observe reward r g,h and next state s g,h +1

6: Update n = N h ( s g,h ) ← N h ( s g,h ) + 1

11: end for

12: end for

G.2 An Additional Assumption About Transformers Performing Division

Before digging into the proof of Theorem 3.4, we first state the following assumption that there exists one transformer that can perform the division operation.

Assumption G.2. There exists one transformer TF θ with

Algorithm 4 V-learning Executing Output Policy ˆ µ [32]

1: Sample g ∼ Unif (1 , 2 , · · · , G )

2: for step h = 1 , · · · , H do

3: Observe s h and set n ← N g,h ( s h )

4: Set g ← g h i ( s h ) where i ∈ [ n ] is sampled with probability α n,i

5: Take action a h ∼ µ g,h ( ·| s h )

6: end for

such that for any x ∈ [0 , 1] , y ∈ [ √ log( A ) / ( AG ) , √ log( A ) /A + 1] , φ > 0 , with input H = [ h 1 , · · · , h e , · · · ] , where

̸

the transformer can provide output H = [ h 1 , · · · , h e , · · · ]

̸

We note that this assumption on performing exact division is only for the convenience of the proof as one can approximate the division to any arbitrary precision only via one MLP layer with ReLU activation.

In particular, let Ball k ∞ ( R ) = [ -R,R ] ∞ denote the standard ℓ ∞ ball in R k with radius R > 0 , we introduce the following definitions and result.

Definition G.3 (Approximability by Sum of ReLUs, Definition 12 in Bai et al. [7]) . A function g : R k → R is ( ε approx , R, M, C ) -approximable by sum of ReLUs, if there exists a ' ( M,C ) -sum of ReLUs' function

with such that

Definition G.4 (Sufficiently Smooth k -variable Function, Definition A.1 in Bai et al. [7]) . We say a function g : R k → R is ( R,C ℓ ) -smooth if for s = ⌈ ( k -1) / 2 ⌉ +2 , g is a C s function on Ball k ∞ ( R ) , and

for all i ∈ { 0 , 1 , · · · , s } , with

Proposition G.5 (Approximating Smooth k -variable Functions, Proposition A.1 in Bai et al. [7]) . For any ε approx > 0 , R ≥ 1 , C ℓ > 0 , we have the following: Any ( R,C ℓ ) -smooth function g : R k → R is ( ε approx , R, M, C ) approximable by sum of ReLUs with M ≤ C ( k ) C 2 ℓ log(1 + C ℓ /ε approx ) /ε 2 approx and C ≤ C ( k ) C ℓ , where C ( k ) > 0 is a constant that depends only on k .

Then, we can consider g ( x, y ) = ( c 1 + x ) / ( c 2 + y ) with c 1 ≥ 1 , c 2 ≥ 1 + c ′ 2 > 1 , x ∈ [ -1 , 1] and y ∈ [ -1 , 1] . It can be verified that

Then, from Definition G.4 there exists one C l such that g ( x, y ) is (1 , C l ) -smooth. Thus, by Proposition G.5, this function can be approximated by a sum of ReLUs (defined in Definition G.3). With this observation, the division operation required in Assumption G.2 can be approximated.

G.3 Proof of Theorem 3.4: The Realization Construction

G.3.1 Embeddings and Extraction Mappings

We consider each episode of the max-player's observations to be embedded in 2 H tokens. In particular, for each t ∈ [ T ] , we construct that

where s g,h , a g,h are represented via one-hot embedding. The positional embedding pos i is defined as

where v i := 1 { h a i = 0 } denote the tokens that do not embed actions and rewards.

In summary, for observations D t -1 + ∪ { s t } , we obtain the tokens of length 2 t -1 which can be expressed as the following form:

With the above input H , the transformer outputs H = TF θ + ( H ) of the same size as H . The extraction mapping A is directly set to satisfy the following

i.e., the part c of the output tokens is used to store the learned policy.

G.3.2 An Overview of the Proof

In the following, for the convenience of notations, we will consider step t +1 , i.e., with observations D t + ∪ { s t +1 } . Given an input token matrix

we construct a transformer to perform the following steps

where n := N h ( s g,h ) .

To ease the notations, in the proof, we will slightly abuse TF θ as TF θ + . The following provides a sketch of the proof.

Step 1 There exists an attention-only transformer TF θ to complete Step 1 with

Step 2 There exists a transformer TF θ to complete Step 2 with

Step 3 There exists a transformer TF θ to complete Step 3 with

Step 4 There exists a transformer TF θ to complete Step 4 with

Thus, the overall transformer TF θ can be summarized as

Also, from the later construction, we can observe that log( R ) = ˜ O (1) . The bound on the covering number, i.e., log( N Θ + ) can be obtained via Lemma I.4.

G.3.3 Proof of Step 1: Update N h ( s g,h ) .

From the proof of Step 1 in realizing UCB-VI in Lin et al. [38], we know that there exists a transformer with 3 heads that for all t ′ ≤ t , can move s t ′ , ( a t ′ , r t ′ ) from h a 2 t ′ -1 and h b 2 t ′ to h d 2 t ′ +1 while maintaining h d 2 t ′ not updated. This constructed transformer is shown in Lin et al. [38] to be an attention-only one with ReLU activation and

Then, still following Lin et al. [38], another attention-only transformer can be constructed so that N t ( s t ) can be computed in h d 2 t +1 . The constructed transformer has ReLU activation and

In summary, at the end of Step 1, we can obtain that for all t ′ ≤ t ,

with an attention-only transformer that uses ReLU activation and

G.3.4 Proof of Step 2: Compute ˜ V h ( s g,h ) and V h ( s g,h )

In Step 2, let n t = N t ( s t ) , the following computations will be performed:

First, similar to Step 2 in realizing UCB-VI in Lin et al. [38], we can obtain α n t ′ and β n t ′ in each h d 2 t ′ +1 via a transformer with ReLU activation and

Then, we can assume that the values of { ˜ V h old ( s ) , V h old ( s ) : h ∈ [ H ] , s ∈ S} is already computed in token h d 2 τ -1 , and prove via induction to show that the set of new values can be computed in token h d 2 τ +1 , i.e.,

In the following, we will show that this one-step update can be completed via a transformer that requires

and the overall updates can be completed via stacking T similar transformers.

First, we obtain an auxiliary value τ - | t ′ -1 -τ | in each h d 2 t ′ -1 and h d 2 t ′ for all t ′ ≤ t . In particular, we can construct three MLP layers with ReLU activation, i.e., { W (1) 1 , W (1) 2 , W (2) 1 , W (2) 2 , W (3) 1 , W (3) 2 } to sequentially compute that

It can be observed that τ -| t ′ -1 -τ | reaches its maximum τ at t ′ = τ +1 .

The required transformer can be summarized as

Step 2.2: Move ˜ V 1: H old ( · ) and V 1: H old ( · ) .

Then, we move ˜ V 1: H old ( · ) and V 1: H old ( · ) from h 2 τ -1 to h 2 τ +1 . In particular, we can construct that

With ReLU activation, this construction leads to that

The transformer required in Step 2.3 can be summarized as

Step 2.3: Compute ˜ V 1: H new ( · ) and V 1: H new ( · ) .

Finally, we get to compute ˜ V 1: H new ( · ) and V 1: H new ( · ) , where

We can have a HS -head transformer that

It holds that for all t ′ ≤ t ,

which completes the computation (1 -α n τ ) ˜ V τ old ( s τ ) .

Two similar HS -head transformers can further perform (1 -α n τ ) ˜ V τ old ( s τ ) + α n τ r τ + α n τ β n τ . and a similar HS 2 -head transformer can finalize ˜ V τ new ( s τ ) as (1 -α n τ ) ˜ V τ old ( s τ ) + α n τ r τ + α n τ β n τ + α n τ V τ +1 old ( s τ +1 ) .

Finally, from the proof of Step 3 in realizing UCB-VI in Lin et al. [38], one MLP layer can perform

The required transformer in step 2.4 can be summarized as

Thus, we can see that the update of h 2 τ +1 can be done. Repeating the similar step for each τ ∈ [ T ] would complete the overall updates.

G.3.5 Proof of Step 3: Compute ˜ ℓ n t ( s g,h , a ) and µ h ( a | s g,h )

In step 3, let n t = N t ( s t ) , for step t , we update

First, similar to Step 2, we can obtain ω n t ′ , η n t ′ /ω n t ′ and γ n t ′ in each h d 2 t ′ +1 via a transformer with ReLU activation and

Denoting vectors L ∈ R HSA and µ ∈ R HSA containing the cumulative losses and policies, i.e., ∑ i ∈ [ n ] ω i ˜ ℓ h i ( s, a ) and µ h ( a | s ) . Similar to step 2, we will assume that the values of {L old , µ old } ,

which are computed via information before time τ , are already contained in token h d 2 τ -1 , and prove via induction to show that the set of new values can be computed in token h d 2 τ +1 , i.e.,

In the following, we will show that this one-step update can be completed via a transformer that requires

and the overall updates can be completed via stacking T similar transformers.

First, similar to step 2.1, we can have an auxiliary value τ -| t ′ -1 -τ | in each h d 2 t ′ -1 and h d 2 t ′ for all t ′ < t via three MLP layers. The required transformer can be summarized as

Step 3.2: Compute ˜ ℓ τ n τ ( s τ , a ) .

First, similar to Step 2.2, we can move L old and µ old from h d 2 τ -1 to h d 2 τ +1 while keeping other tokens unchanged. In particular, we can construct that

With ReLU activation, this construction leads to that

Then, using similar constructions as Step 2.3, we can specifically extract V τ +1 ( s τ +1 ) and µ τ old ( a τ | s τ ) to h d 2 τ +1 while keeping other tokens unchanged. Following the extraction, one MLP layer can compute the value H -r τ -V τ +1 ( s τ +1 ) H and add it to h d 2 τ +1 . The required transformer can be summarized as

With Assumption G.2, for token h 2 τ +1 , we can further have one transformer to compute

The overall required transformer can be summarized as

Step 3.3: Update µ τ ( a | s τ ) .

First, one transformer can be constructed to update L old to L new in h d 2 τ +1 . In particular, we can have

With ReLU activation, this construction leads to that

With a similar A -head transformer, we can add ω n τ · ˜ ℓ τ n τ ( s τ , a τ ) to L old ( h ( τ ) , s τ , a τ ) and thus obtain the new values L new ( h ( τ ) , s τ , · ) . Furthermore, a one-head transformer can obtain

Finally, with another softmax MLP layer, we can obtain µ τ new ( ·| s τ ) .

The required transformer can be summarized as

G.3.6 Proof of Step 4: Obtain µ h +1 ( ·| s h +1 )

This step is essentially the same as Step 6 in realizing VI-ULCB, which can be completed with a transformer that

H Proofs for the Decentralized Overall Performance

Proof of Theorem 3.5. We use the decomposition that

First, via Theorem G.1, it holds that

Then, via Lemma 3.6 and Theorem 3.3, we can obtain that

and similarly,

With Theorem 3.4 providing that ε + , real = ε -, real = 0 , combining the above terms completes the proof of the regret bound. The bound on the covering number, i.e., log( N Θ + ) and log( N Θ -) can be obtained via Lemma I.4.

In the following, we provide the proof for the Lemma 3.6, which plays a key role in the above proof of the overall performance.

Proof of Lemma 3.6. It holds that

where (a) is from the definition of best responses, (b) uses the notation ˆ µ g to denote the output policy from Alg. 4 with the initial random sampling result to be g , (c) uses the variational representation of the TV distance, (d) uses the abbreviations n = N g,h ( s h ) and g i = g h i ( s h ) , (e) leverages the property of TV distance, and (f) uses the fact that α n,i < 1 .

With the above result, we can further obtain that

Also, for term (II), denoting g ( D ) = V ˆ µ β , † M ( s 1 ) , it holds that

Combining (term I) and (term II) concludes the proof.

I Discussions on the Covering Number

In the following, we characterize the covering number of algorithms induced by transformers parameterized by θ ∈ Θ d,L,M,d ′ ,F , i.e., { Alg θ : θ ∈ Θ d,L,M,d ′ ,F } . Here we will mainly take the perspective of the centralized setting, while the extension to the decentralized setting is straightforward.

To facilitate the discussion, we introduce the following clipped transformer, where an additional clip operator is adopted to bound the output of the transformer.

Definition I.1 (Clipped Decoder-based Transformer) . An L -layer clipped decoder-based transformer, denoted as TF R θ ( · ) , is a composition of L masked attention layers, each followed by an MLP layer and a clip operation: TF R θ ( H ) = H ( L ) ∈ R d × N , where H ( L ) is defined iteratively by taking H (0) = clip R ( H ) ∈ R d × N and for l ∈ [ L ] ,

Furthermore, for ζ ∈ (0 , 1] , we define the following ζ -biased algorithm induced by transformer TF R θ ( H ) as

with 1 AB denoting an all-one vector of dimension AB , which introduces a lower bound ζ/AB for the probably each pair ( a, b ) ∈ A × B to be sampled.

Finally, for any H = [ h 1 , · · · , h N ] ∈ R d × N , we denote

Proposition I.2 (Modified from Proposition J.1 in Bai et al. [7]) . For any θ 1 , θ 2 ∈ Θ d,L,M,d ′ ,F , we have

where F Θ := FR (2 + FR 2 + F 3 R 2 ) and F H := (1 + F 2 )(1 + F 2 R 3 ) .

Proof. This proposition can be obtained similarly as Proposition J.1 in Bai et al. [7] with the following Lemma I.3 in the place of Lemma J.1 in Bai et al. [7].

Lemma I.3 (Modified from Lemma J.1 in Bai et al. [7]) . For a single MLP layer θ mlp = ( W 1 , W 2 ) , we introduce its norm

For any fixed hidden dimension D ′ , we consider

Then, for any H ∈ H R , θ mlp ∈ Θ mlp ,F , it holds that MLP θ mlp ( H ) is 2 FR -Lipschitz in θ mlp and (1 + F 2 ) -Lipschitz in H .

Proof. It holds that

Inequality (a) is from that

and

where the last result on the Lipschitzness of softmax is adopted from Gao and Pavel [25]. Similarly, we can obtain that

which concludes the proof.

Lemma I.4 (Modified from Lemma 16 in Lin et al. [38]) . For the space of transformers { TF R θ : θ ∈ Θ d,L,M,d ′ ,F } , the covering number of the induced algorithms { Alg ζ θ : θ ∈ Θ d,L,M,d ′ ,F } satisfies that

Proof. First, similar to Lemma 16 in Lin et al. [38], we can use Example 5.8 in Wainwright [61] to obtain that the δ -covering number of the ball B ∥·∥ ( F ) with radius F under norm ∥ · ∥ , i.e., B ∥·∥ ( F ) = { θ : ∥ θ ∥ ≤ F } , can be bounded as

Recall that the projection operation to a probability simplex is Lipschitz continuous, i.e.,

Then, we can see that there exists a subset ˜ Θ ⊂ Θ d,L,M,d ′ ,F with size

such that for any θ ∈ Θ d,L,M,d ′ ,F , there exists ˜ θ ∈ ˜ Θ with

Let δ = ζρ ABLF L -1 H F Θ , we can obtain ∥ log Alg ζ ˜ θ ( · , ·| D t -1 , s ) -log Alg ζ θ ( · , ·| D t -1 , s ) ∥ ∞ ≤ ρ , which proves that

which concludes the proof.

From the transformer construction in the proofs for Theorems 3.4 and 4.1, we can observe that it is sufficient to specify one R with log( R ) = ˜ O (1) without impacting the transformers' operations. Also, in Theorems 3.3 and C.3, ρ is taken to be 1 /N .

Finally, for the introduced ζ parameter, it can be recognized that the induced algorithms discussed in the main paper, i.e. Alg θ , can be interpreted as ζ = 0 , which does not lead to a meaningful log( N Θ d,L,M,d ′ ,F ( ρ )) provided in Lemma I.4. However, a non-zero ζ can tackle this situation by only introducing an additional realization error. Especially, assuming Assumption C.2 can be achieved with ε real = 0 , i.e., exactly realizing the context algorithm (as in Theorem 4.1), we can obtain that

With ζ = O (1 /N ) , an additional realization error ε real = O (1 /N ) occurs, whose impact on the overall performance bound (i.e., Theorems 3.5 and 4.2) is non-dominating. As a result, for the parameterization provided in Theorem 4.1, a covering number satisfies log( N Θ ) = ˜ O ( poly ( GHSAB ) log( N )) can be obtained. Similarly, the results can be extended to the decentralized setting, where for the parameterization provided in Theorem 3.4, it holds that log( N Θ + ) = ˜ O ( poly ( GHSAL D M D d D ) log( NF D )) .

J Details of Experiments

J.1 Detail of Games

The normal-form games for the decentralized setting. The normal-form games (i.e., matrix games) used in decentralized experiments of Sec. 5 have A = B = 5 actions for both players and G = 3000 episodes (i.e., a horizon of T = 3000 with H = 1 ), which can be interpreted as having S = 1 state.

The Markov games for the centralized setting. The Markov games used in centralized experiments of Sec. 5 have A = B = 5 actions for both players, S = 4 states, H = 2 steps in each episode and G = 300 episodes (i.e., a horizon of T = GH = 600 ). The transitions are fixed for different games: when both players take the same actions, the state transits to the next one (i.e., 1 → 2 , · · · , 4 → 1 ); otherwise, the state stays the same.

At the start of each game (during both training and inference), a A × B reward matrix R h ( s, · , · ) is generated for each step h ∈ [ H ] and state s ∈ S with its elements independently sampled from a standard Gaussian distribution truncated on [0 , 1] . Then, the interactions proceed as follows: at each time step h ∈ [ H ] , the players select action a and b on state s based on their computed policy distributions. After selecting their actions, the players receive rewards R h ( s, a, b ) and -R h ( s, a, b ) , respectively, while the state transits to the new one.

J.2 Collection of Pre-training Data

The EXP3 algorithm for the decentralized setting. In the decentralized setting, both players are equipped with the EXP3 algorithm [5] to collect pre-training data. Up to time step t , the trajectory of the max-player is recorded as ' a 1 , r 1 , · · · , a t , r t ', and that of the min-player as ' b 1 , 1 -r 1 , · · · , b t , 1 -r t ',

The VI-ULCB algorithm for the centralized setting. In the centralized setting, both players jointly follow the VI-ULCB algorithm [8] to collect pre-training data. Up to time step t , the trajectory is recorded as ' s 1 , a 1 , b 1 , r 1 , 1 -r 1 , · · · , s t , a t , b t , r t , 1 -r t '.

We note that the decimal digits of the rewards are limited to two to facilitate tokenization, while 1 -r i instead of -r i is adopted for the min-player to avoid the additional complexity of negative numbers.

J.3 Transformer Structure and Training

The transformer architecture employed in our experiments is primarily based on the well-known GPT-2 model [43], and our implementation follows the miniGPT realization 3 for simplicity. The

numbers of transformer layers and attention heads have been modified to make the entire transformer much smaller. In particular, we utilize a transformer with 2 layers and 4 heads. Given that the transformer is used to compute the policy, we modify the output layer of the transformer to make it aligned with the action dimension. We focus solely on the last output from this layer to determine the action according to the computed transformer policy.

For the training procedure, we use one Nvidia 6000 Ada to train the transformer with a batch size of 32, trained for 100 epochs, and we set the learning rate as 5 × 10 -4 . The experimental codes are available at https://github.com/ShenGroup/ICGP .

J.4 Performance Measurement

To test the performance of the pre-trained transformer (in particular, how well it approximates EXP3 and VI-ULCB), we adopt the measurement of Nash equilibrium gap. In particular, for either the transformer induced policy or EXP3, we denote the max-player's policy at time step t as µ t , and the min-player's policy at time step t as ν t . Furthermore, the average policy is computed as

The NE gap at step t is computed as

For VI-ULCB, the process is similar except that µ t and ν t are taken as the marginalized policies of the joint policy learned at step t while the NE gap is cumulated over one episode. The NE gaps averaged over 10 randomly realized games at each step are plotted in Fig. 2.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: This work investigates the in-context game-playing (ICGP) capabilities of pretrained transformers, where both the decentralized and centralized settings are considered. The abstract and introduction (i.e., Sec. 1) accurately reflect the considered problem and the obtained main results, highlighting our contributions in providing a comprehensive understanding of ICGP.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The major limitations of this work in our mind are illustrated in Appendix B.2. In particular, it is emphasized that this work mainly focuses on the two-player zero-sum Markov games and two specific game-solving algorithms. Also, the construction of the pre-trained dataset and the need of further large-scale experiments are discussed.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The main paper (particularly, Sec. 3.1.1 and the beginning of Sec. 4.1) provides the main setups and assumptions considered in this work. Additional Assumptions C.2, F.2, and G.2 are deferred to the appendix due to the page limits while being noted in the main paper with corresponding pointers. The complete proofs are provided in the appendix (see Appendix A for an overview), which have been carefully checked multiple times and are correct to the best of our knowledge. A proof sketch is included in Sec. 4.3 to facilitate the readers' understanding.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: A detailed description of the experimental setups and details is provided in Appendix J. The complete set of codes for the experiments are available at https: //github.com/ShenGroup/ICGP .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The complete set of codes can be found at https://github.com/ ShenGroup/ICGP .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: A detailed description of the experimental setups and details is provided in Appendix J. The experimental codes are available at https://github.com/ShenGroup/ ICGP .

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars have been provided in Fig. 2, which is the main experimental results in this work.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computer resources used to perform the experiments reported in this work are described in Appendix J.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All the authors have reviewed the NeurIPS Code of Ethics and practiced it during this submission.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Discussions on the broader impacts of this work are provided in Appendix B.1. Due to the theoretical focus of this work, we do not foresee major negative social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The transformers trained in this work are only for the purpose of validating the theoretical claims, which we do not foresee risks of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have included the URL of the adopted transformer model in Appendix J, which is under a standard MIT license.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets have been released with this work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing and research with human subjects has been performed during this work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing and research with human subjects has been performed during this work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the in-context learning capabilities of pre-trained transformer models in competitive multi-agent games, i.e., in-context game-playing (ICGP). The authors provide extensive theoretical guarantees to validate that pre-trained transformers can approximate NE in an in-context manner for both decentralized and centralized learning settings. They also demonstrate that the transformer architecture can realize well-known multi-agent game-playing algorithms. Finally, experimental results support their theoretical findings, showing that transformers can effectively approximate NE in an in-context manner.\n\nSTRENGTHS:\nThe main strengths of this paper are:\n\n1. The paper focuses on a novel and important aspect of in-context learning in competitive multi-agent settings. Specifically, they introduce a framework for modeling ICGP via transformers. \n\n2. This paper provides strong theoretical guarantees and backs them with empirical validation.\n\n3. The methodology is sound, and the paper is well-structured, with clear explanations and thorough analyses.\n\nWEAKNESSES:\n1. The paper seems to be an extended work that generalizes the analysis of in-context learning in RL to the analysis of in-context learning in game theory. Certainly, this setting has not been explored before. However, this paper did not explain why this research problem is important. \n\n2. Although the paper provides strong theoretical results, the empirical experiments are conducted on relatively simple two-player zero-sum normal-form games. This limits the generalizability of the findings.\n\n3. The paper lacks a more detailed discussion of the practical implications.  Including such a discussion would provide a more comprehensive understanding of the applicability of the results.\n\nQUESTIONS:\n1. The empirical experiments are currently limited to simple two-player zero-sum normal-form games. Have you considered testing your methods in more complex environments, such as multi-player games or games with larger state-action spaces? If so, what were the results, and if not, why?\n\n2. This paper focuses on two-player zero-sum Markov games. Could your method be applied to other types of games, such as cooperative or general-sum games? Do you have any insights or challenges you anticipate in these games?\n\n3. Could you discuss more about the broader impacts and potential limitations of your work?\n\nLIMITATIONS:\nYes, the authors provide limitations in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the in-context learning capabilities of pre-trained transformer models in two-player zero-sum games. The authors provide the theoretical guarantees that pre-trained transformers can learn the approximate Nash equilibrium in an in-context manner, both in decentralized and centralized settings.\n\nSTRENGTHS:\n1. Considering multi-agent settings (two-player zero-sum games in this work) is well-motivated compared to previous works that focused on single-agent settings.\n\n2. The theoretical results in this work provide a deeper understanding of how the pre-trained transformers can approximate NE in an in-context manner.\n\n3. The paper is well-written, easy to follow, and makes non-trivial contributions to the relevant communities. However, because I am not an expert in theory, so it is quite difficult for me to check all the details of the proofs of all the theoretical results. Therefore, I am positive for acceptance but with relatively low confidence.\n\nWEAKNESSES:\nThere are some questions:\n\n1. In this work, the game instances are assumed to have the same length of time horizon H, which may not be always the case. For example, when playing SMAC, each game-play could terminate at any time. In this case, will the results of this work still be applicable?\n\n2. In the current setup, the results seem only suitable for games with small state spaces because the augmented component (Line 163) needs to enumerate all the states. This could be impractical for more realistic and complex games.\n\n3. The experiments are only conducted using normal-form games, which is simple. The performance of the proposed framework for more complex Markov games is unclear. I would have guessed that it is not very practical for the current framework as one needs to sample actions over the state space which could be extremely large.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe authors have discussed the limitations and future directions of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors built on the recent work of Lin et. al 2023, extending their ICLR framework so that instead of being for one agent, it is for multi-agent systems; at the same time, they also analyze and provide evidence of \"ICGP\" (in-context game-playing) capabilities in transformers.\n\nThe paper analyzes zero-sum Markov games for two players and explores learning the Nash Equilibrium. In these games, two setups are studied: one of decentralized learning (where the Nash Equilibrium must be reached without each player observing the opponent's actions) and another of centralized learning (where a transformer controls the actions of both players).\n\nThe authors find important theoretical results related to transformers in decision-making, e.g., that pre-trained transformers can realize both model-based and model-free designs, and the in-context learning capability of transformers in playing normal-form games\n\nSTRENGTHS:\n1. **Soundness of the Claims**\n\n   a) **Theoretical Grounding**\n\n   The paper does a great job of formalizing and mathematically demonstrating decentralized and centralized training of transformers to achieve ICGP, and I think it is a high-impact contribution. One could argue that it is also necessary to provide more empirical evidence (which is scarce in this paper) of the dynamics being modeled, but results like Theorem 4.1 are, in my opinion, a sufficient contribution. Among other things, this work shows the capability of transformers to not only implement existing algorithms (not new in the literature) but also to adapt them to the needs of model-free designs (as far as I know, new), demonstrated by the transformer’s ability to perform exact and approximate implementations, which is important as we move to real-world applications.\n\n   b) **Empirical Evaluation**\n\n   The methodology followed by the authors (\"J. Details of the Experiments\") seems to be solid. They use the EXP3 algorithm to collect pre-training data to ensure that the data is reflective of competitive strategies and behaviors. They also customize GPT-2, with only 2 layers and 4 attention heads, tailored to match the action dimensionality in its output layer, which directly supports computing policies from the model outputs (Lines 1002-1008). The Nash Equilibrium (NE) gap is calculated by comparing the expected rewards of the max player's policy against the min player's policy over time. The gap measures how close the transformer’s induced policy is to an ideal Nash Equilibrium.\n\n   Since the area of applied game theory for LLMs/multi-agent LLMs is still largely unexplored, providing such detailed information about the experiments and the code is likely going to be useful and impactful for the community.\n\n2. **Significance**\n\n   In general, I think that ICL work has a lot of impact; at this point, it is folk knowledge that ICL > fine-tuning, so papers like this one have a lot of significance for the advancement of the state of the art. In particular, Theorems 3.4 and 4.1 can prove to be of great impact.\n\n3. **Novelty**\n\n   This paper is timely and tackles a timely and interesting topic. To the best of my knowledge, it is the first paper providing a theoretical analysis of this level on game theory applied to LLMs.\n\nWEAKNESSES:\n1. **Soudness of the claims**\n\n   My concerns about the paper relate to whether this theoretical framework might struggle with generalization and performance in more complex games. In this section, I provide some examples of limitations that suggest this possibility. I'm concerned that it may excel in controlled environments but not in more complex scenarios, which are typical in the multi-agent literature.\n\n   Also, please note that the improved performance of transformers trained with 20 games compared to those trained with 10 (Lines 336-341) suggests a heavy dependence on the volume of training data. Thus, I'm also highlighting some potential problems related to this aspect. \n\n   a) **Theoretical grounding**\n\n      a1) **Decentralized learning scenario**\n\n   In the decentralized setting, my primary concerns revolve around the diversity and representativeness of the training data, overly optimistic assumptions about model approximation capabilities, and potential issues with the suitability of transformer outputs for generating valid action probabilities.\n\n* (Lines 157-170) If the data collected through $Alg+,0$ and $Alg−,0$ only captures a limited range of strategic interactions (e.g., typical or frequent scenarios but not edge cases or unusual strategies) the training data might lack the necessary diversity, leading to overfitting/lack of robustness. If the data does not sufficiently cover the state-action space, the expected log probability of correct actions given the state, as modeled by the transformer, may not reflect true gameplay dynamics and $\\mathbb{E}\\left[\\log P(\\text{actual action} \\mid \\text{state})\\right]$ may overestimate actual performance.\n* (Lines 200-202) Take into account that with Assumption \"3.2. Decentralized Approximate Realizability\" you assume that the true data-generating distribution ($Alg_0^+$) can be closely approximated by the learned model ($Alg_\\theta^+$). This assumption may not hold if the real complexity of strategies in the environment cannot be captured by the model's architecture or training data -- causing the models have generalization issues.\n* (Lines 195-190) Potential convergence/generalization problems with the learning algorithm if the covering number $\\Theta$ is underestimated. The clipping operation (Lines 936-938), which is designed to maintain the norm of activations within bounds, can significantly impact the parameter sensitivity and the gradient flow during backpropagation. So, if $\\Theta$ is underestimated, it might not sufficiently account for the reduced effective parameter space available for optimization, possibly leading to convergence on local minima that do not generalize well\n* (Lines 182-185) The use of linear extraction mappings followed by a projection to the probability simplex to determine action probabilities assumes that the transformer’s outputs can be linearly mapped to form valid probability distributions. Buuut, if the transformer outputs are not suitable for this kind of linear transformation due to scale, bias, or other distributional issues, the resulting action probabilities may not be valid or optimal --> leading to suboptimal decisions and erratic behavior, especially in complex games.\n\n   a2) **Centralized learning scenario**\n\n   For the centralized learning scenario, I want to provide feedback on some specific concerns related to computational demands, dependency on precise model and algorithm alignment, and assumptions about uniform sampling that may not adequately capture the complexity of strategic behaviors across varied game states\n\n* (Lines 307-309) The required dimensions and mappings such as $d⪅HS^2AB,L⪅GHSd⪅HS^2AB,L⪅GHS$ etc., indicate a high computational complexity and dependency on specific game parameters (like $S,A,B,G,HS,A,B,G,H$). As a result, this could make the model computationally expensive and potentially overfit to specific types of game environments.\n* (Lines 307-309) This performance requirement: $Algθ(⋅,⋅∣Dt−1,s)=AlgVI−ULCB(⋅,⋅∣Dt−1,s)Algθ​(⋅,⋅∣Dt−1​,s)=AlgVI−ULCB​(⋅,⋅∣Dt−1​,s)$ assumes an ideal alignment between the transformer’s output and the algorithm’s requirements; note that any deviation in this alignment due to model approximation errors or misestimations in the transformer's training could lead to significant performance drops...\n* (Lines 321-322) Note that the assumption that $\\hat{\\mu}\\$ and $\\hat{\\nu}\\$ are uniformly sampled and provide an effective representation of strategic options in every game state might not hold if some strategies are inherently more complex or contextually sensitive than others.\n\nQUESTIONS:\n* Given that the decentralized setup involves collecting offline trajectories using different algorithms for max- and min-players (Section \"3.1 Supervised Pre-training Results\") -- how do you ensure that the diversity and distributional properties of the training data do not adversely affect the learning outcomes?\n* The clipping operation $clip_R$ is defined to constrain the norm of each vector (Lines 936 - 938). How does this norm constraint affect the learning dynamics, particularly in terms of gradient propagation and stability during backpropagation? This seems to be important as it directly affects the model's ability to learn and fine-tune strategies based on gameplay experience.\n* In Appendix G.3, you detail the use of embedding and extraction mappings within the transformer architecture for implementing V-learning (Lines 810-819). These mappings essentially allow the transformer to understand and manipulate game states and decisions effectively. Are there specific state spaces where these mappings prove particularly beneficial/limited?\n\nJust out of curiosity:\n* Have you observed any sudden shifts in learning effectiveness or strategy adaptation in your decentralized learning models, based on the results from \"The Mechanistic Basis of Data Dependence and Abrupt Learning in an In-Context Classification Task\" by Gautam Reddy (2023)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework for pre-trained transformers to approximate Nash equilibrium in two-player zero-sum normal-form games and provides theoretical guarantees to show that these pre-trained transformers can learn to approximate Nash equilibrium in an in-context manner. This is shown for both the decentralized and centralized learning settings. The decentralized case follows the idea that each agent will have it's own model, and uses V-learning. The centralized case uses a centralized joint model to control both players, and uses VI-ULCB. The paper shows that there is a provable upper bound to the approximation error of the Nash equilibrium which demonstrates the in-context game player capability of transformers. Further more, empirical results show that given a sufficiently well pre-trained transformer that it can approximate Nash equilibrium in an in context manner, similar to that of decentralized EXP3 in two-player zero-sum normal-form games, that is it has a similar profile of the gradually decaying Nash equilibrium gap.\n\nSTRENGTHS:\nOriginality:\nThis work offers new insights into the the theoretical analyses and empirical evidence for the in context game player capabilities of transformers.\n\nClarity and Quality:\nThe paper is well cited and explains the concepts it is conveying quite well, with the appendix showing the details of the proofs and supporting code demonstrating the reproducibility of the empirical work.\n\nSignificance:\nThe paper shows the theoretical capabilities of transformers adapted to V-learning (and thus aiding in the scalability of MARL) as well as approximating Nash equilibrium are important to MARL settings.\n\nWEAKNESSES:\nI have concerns that since the pre-training dataset is collected from a context algorithm that the transformers are simply learning to mimic this algorithm.\n\nThe fact that empirical results are only collected from bandit settings, I have concerns that this may not hold in a setting where there are state transitions. I would have hoped that there would have been a more complex domain than just matrix games.\n\nQUESTIONS:\n1) Do you believe that this implementation might have different results given a different pre-training dataset (produced from a different algorithm)?\n\n2) Do you believe that in more complex domains, with state transitions, we will still observe similar results? Or is this a factor that might break the current state of this work?\n\nLIMITATIONS:\nThe authors have identified the limitations and have stated that there should be no negative societal impact due to the theoretical nature of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}