Title: On the Role of Information Structure in Reinforcement Learning for Partially-Observable Sequential Teams and Games
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning

partially-observable

information structure

sequential decision-making

causal

causal structure

graphical model
Abstract: In sequential decision-making problems, the *information structure* describes the causal dependencies between system variables, encompassing the dynamics of the environment and the agents' actions. Classical models of reinforcement learning (e.g., MDPs, POMDPs) assume a restricted and highly regular information structure, while more general models like predictive state representations do not explicitly model the information structure. By contrast, real-world sequential decision-making problems typically involve a complex and time-varying interdependence of system variables, requiring a rich and flexible representation of information structure. In this paper, we formalize a novel reinforcement learning model which explicitly represents the information structure.
We then use this model to carry out an information-structural analysis of the statistical complexity of general sequential decision-making problems, obtaining a characterization via a graph-theoretic quantity of the DAG representation of the information structure. We prove an upper bound on the sample complexity of learning a general sequential decision-making problem in terms of its information structure by exhibiting an algorithm achieving the upper bound. This recovers known tractability results and gives a novel perspective on reinforcement learning in general sequential decision-making problems, providing a systematic way of identifying new tractable classes of problems.

On the Role of Information Structure in Reinforcement Learning for Partially-Observable Sequential Teams and Games

Abstract

In sequential decision-making problems, the information structure describes the causal dependencies between system variables, encompassing the dynamics of the environment and the agents' actions. Classical models of reinforcement learning (e.g., MDPs, POMDPs) assume a restricted and highly regular information structure, while more general models like predictive state representations do not explicitly model the information structure. By contrast, real-world sequential decisionmaking problems typically involve a complex and time-varying interdependence of system variables, requiring a rich and flexible representation of information structure. In this paper, we formalize a novel reinforcement learning model which explicitly represents the information structure. We then use this model to carry out an information-structural analysis of the statistical complexity of general sequential decision-making problems, obtaining a characterization via a graph-theoretic quantity of the DAG representation of the information structure. We prove an upper bound on the sample complexity of learning a general sequential decision-making problem in terms of its information structure by exhibiting an algorithm achieving the upper bound. This recovers known tractability results and gives a novel perspective on reinforcement learning in general sequential decision-making problems, providing a systematic way of identifying new tractable classes of problems.

1 Introduction

The information structure of a sequential decision-making problem is a description of the causal dependencies between system variables. In particular, the information structure specifies the subset of past events that causally influence the present state. This includes information affecting system dynamics and information available to each agent when choosing an action. The control community has long recognized the importance of information structure, leading to the development of the celebrated Witsenhausen intrinsic model [1], and extensive study since the 1970s [e.g., 2-14].

In general, sequential decision-making problems with arbitrary causal dependencies between system variables can be computationally and statistically intractable in both control and learning settings [3, 15]. In response, early research has identified classes of tractable information structures. For example, Markov Decision Processes (MDP) and Markov teams/games assume an observable Markovian state, which serves as a sufficient statistic for the system's evolution. These structural assumptions enable practical planning and learning algorithms, for example based on dynamic programming [16, 17]. Although such restricted model classes can enable more fruitful analysis, they lack the expressiveness needed to naturally capture the causal structures of complex real-world sequential decision-making problems, where each event in the system may depend arbitrarily on past events.

The concept of information structure is also fundamental to studying the phenomenon of partial observability . In general, partial observability refers to situations where a system's evolution depends on a potentially large number of sequential events, but only a subset of these events is observable by the learning agent. However, commonly studied models capture only a restrictive form of this phenomenon. For example, in a Partially-Observable Markov Decision Process (POMDP)-the standard model of partial observability -it is assumed that a Markovian state exists and that each observation is a noisy measurement of the current state. This assumption is often unrealistic, as general systems may not have clearly defined 'states', and observations may be generated by more complex dependencies. Information structure provides a more powerful framework for understanding partial observability, capturing a general notion in which system variables evolve with an arbitrary causal structure (not necessarily Markovian), and the set of observables is any subset of all system variables.

The highly-regular information structures of classical models make analysis more tractable, enabling favorable theoretical results [e.g., 18-22] and driving notable empirical success across a wide range of domains [e.g., 23-28]. Despite this empirical success, a general theory addressing the role of information structure in the statistical aspects of reinforcement learning is missing. We argue for the perspective that information structure is an important component of analyzing and solving reinforcement learning problems. A rich and flexible representation of information structure is essential to faithfully capture real-world sequential decision-making problems, where the system evolves according to a complex and time-varying dependence on the past, and different agents have different information available to them at different points in time.

In this work, we formulate a general model of sequential decision-making with an explicit representation of information structure, and study the role of information structure in the statistical complexity of reinforcement learning. Explicitly modeling information structure allows us to identify a broader class of tractable decision-making problems and ultimately develop more tailored reinforcement learning approaches that leverage this structure.

Summary of Contributions

A model for representing information structure. We propose partially-observable sequential teams (POST) and partiallyobservable sequential games (POSG) as general models that contain an explicit representation of information structure as part of the problem specification. This forms a unifying framework that enables an analysis of the role of information structure in RL and captures commonly studied RL models as special cases, including MDPs, Markov teams/games, POMDPs, and Dec-POMDPs/POMGs (Figure 1). The models introduced in this work draw inspiration from Witsenhausen's intrinsic model [1] and its variants from the control literature [29, 30], with added elements to model partial observability in the context of reinforcement learning.

Theoretical analysis of sequential decision-making through information structure. We characterize the complexity of the observable dynamics of any sequential decision-making problem through a graph-theoretic analysis of the information struc-

POSG

POMG

Dec-POMDP

POMDP

MDP

POST

ture. Moreover, we propose a generalization of predictive state representations-which may be of independent interest-and construct such a representation for POSTs/POSGs by exploiting information structure. This provides a robust and efficient parameterization amenable to learning.

Learning theory & information-structural analysis of statistical complexity. We prove an upper bound on the sample complexity of learning general sequential decision-making problems, expressed as a function of information structure. In particular, the dependence is on an interpretable quantity derived from a graphical representation that can be thought of as an effective 'information-structural state', generalizing the typical notion of a Markovian state. We prove this result by exhibiting an algorithm achieving this upper bound, making use of the generalized predictive state representation constructed earlier which provides a robust representation amenable to learning. In doing so, we identify a larger class of statistically tractable reinforcement learning problems.

Related Work. We provide a detailed discussion of related work in Appendix B.

2 Background & Preliminaries

What follows is an abridged description of the relevant background and preliminaries. We refer the reader to Appendix C for a more detailed treatment, and to Appendix A for a summary of notations.

2.1 Generic Sequential Decision Making Problems

Consider a controlled stochastic process ( X 1 , . . . , X H ) , where X h is a random variable corresponding to the variable at time h . At each time h ∈ [ H ] , the variable X h may be either an 'observation' (i.e., observable system variable) or an 'action'. The dynamics of this stochastic process are described by a tuple ( H, { X h } h , O , A , P ) , where H is the time horizon, X h is the variable space at time h , O ⊂ [ H ] is the index set of observations (i.e., X h is an observation if h ∈ O ), A ⊂ [ H ] is the index set of actions, and P = { P h } h ∈O is a set of probability kernels which describes the probability of any trajectory x 1 , . . . , x H given the actions, P [ { x s : s ∈ O} | { x s : s ∈ A} ] = ∏ h ∈O P h [ x h | x 1 , . . . , x h -1 ] . A choice of policy π induces a probability distribution P π on X 1 × · · · × X H . The objective of the agent(s) is to choose a policy which maximizes their expected reward V R ( π ) = E π [ R ( X 1 , . . . , X H )] .

Let H h = ∏ s ∈ 1: h X s denote the space of histories at time h and F h = ∏ s ∈ h +1: H X s denote the space futures at time h . Similarly, let H o h = obs ( H h ) = ∏ s ∈O 1: h X s denote the observation component of histories and let H a h = act ( H h ) = ∏ s ∈A 1: h X s denote the action component. The observation and action components of the futures, F o h and F a h respectively, are defined similarly. Here, obs ( · ) and act ( · ) extract the observation and action components, respectively, of any trajectory.

We define the system dynamics matrix D h ∈ R | H h |×| F h | as the matrix giving the probability of each possible pair of history and future at time h given the execution of the actions, [ D h ] τ h ,ω h := P [ τ h , ω h ] ≡ P [ τ o h , ω o h | do( τ a h , ω a h )] , where τ h is the history, ω h is the future, and τ o h , τ a h , ω o h , ω a h separate the history and future into observation components and action components.

The rank of the dynamics of a sequential decision-making problem is a measure of its complexity. It is defined as the maximal rank of its dynamics matrices.

Definition 1 (Rank of dynamics) . The rank of the dynamics { D h } h ∈ [ H ] is r = max h ∈ [ H ] rank( D h ) .

2.2 Generalized Predictive State Representations

Predictive state representations (PSR) [31, 32] are a representation of dynamical systems and sequential decision-making problems based on predicting future observations given the past, without explicitly modeling a latent state. In this section, we propose and formalize a generalization of standard PSRs which allows for an arbitrary order of observations and actions. This generalization is necessary to capture POSTs/POSGs (introduced in Section 3), but may also be of independent interest.

The 'PSR rank' of a sequential decision-making problem coincides with the rank of its dynamics (Definition 1). Denote r h := rank( D h ) . At the heart of predictive state representations is the concept of 'core test sets.' A core test set at time h is a set of futures such that the vector of probabilities of those futures conditioned on the past encodes all the information that the past contains about the future. Definition 2 (Core test sets) . A core test set at time h is a subset of d h ≥ r h futures, Q h := { q 1 , . . . , q d h } ⊂ F h , such that the submatrix D h [ Q h ] ∈ R | H h |× d h is full-rank.

h h

The d h -dimensional vector ψ h ( τ h ) := ( P [ τ h , q 1 h ] , . . . , P [ τ h , q d h h ]) serves as a sufficient statistic for computing the probability of any future conditioned on τ h . Intuitively, core test sets relate the rank of the dynamics to a representation based on predicting future outcomes. A PSR parameterization of the system dynamics represents the probability of any trajectory using the sufficient statistics ψ h via a set of operators that iteratively update the predictive representations at each step as new observations come in. We proceed to formally define our generalized predictive state representation.

Definition 3 (Generalized Predictive State Representations) . Consider a sequential decision-making problem ( X h ) h ∈ [ H ] where A , O partition [ H ] into actions and observations, respectively. Then, a generalized predictive state representation for this sequential decision-making problem is a tuple ( { Q h } 0 ≤ h ≤ H -1 , ϕ H , { M h } 1 ≤ h ≤ H -1 , ψ 0 ) satisfying

An important condition for the learnability of PSR models, which was used in prior work [including 33-35], is the so-called 'well-conditioning assumption'. We state the analogous assumption for our generalized PSR model below. For a core test set Q h , let Q A h = act ( Q h ) be the set of core action sequences, let Q A = max h | Q A h | be the number of core action sequences, and let d = max h d h .

Assumption 1 ( γ -well-conditioned generalized PSR) . A generalized PSR model is said to be γ -well conditioned for γ > 0 if, for any h ∈ [ H ] and any π , it satisfies

To understand this condition, recall that m h ( ω h ) ⊤ ψ h ( τ h ) = P [ τ h , ω h ] . Thus, we can interpret z as the error in estimating the prediction features ψ h . This condition ensures that the error in estimating the probability of system trajectories remains controlled when the estimation error of ψ h is small.

Although we focus on finite spaces X h in this work, we briefly discuss possible extensions to continuous spaces. In finite settings, the core tests q ∈ Q h are represented by future trajectories q = ( x h +1 , ..., x H ) ∈ F h . In continuous spaces, the core tests become trajectories over subsets of the underlying continuous space: q = ( X h +1 , . . . , X H ) ⊂ F h , where each X k ⊂ X k is a measureable subset. We point to [33] for more discussion on how to extend (standard) PSRs to continuous spaces.

3 Information Structure

The 'information structure' of a sequential decision-making problem defines the causal dependencies between events in the system occurring at different points time, whether those events are observable by the learning agent or not. In this section, we will introduce a novel reinforcement learning model that explicitly represents information structure. We demonstrate that this enables a rich analysis of the system's dynamics and is crucial for characterizing the statistical complexity of general RL problems.

3.1 Partially-Observable Sequential Teams

We propose a model of sequential decision-making problems that includes an explicit representation of information structure, which we call partially-observable sequential teams (POST). A POST is a controlled stochastic process consisting of a sequence of variables, where each variable is either a 'system variable' or an 'action variable'. POSTs also model the observability of each system variable with respect to the learning algorithm (i.e., which system variables are available to the learning algorithm). The information structure of a POST describes the causal dependence between these variables. The information set of each system variable describes the subset of past variables that are coupled to it in the dynamics. The information set of an action variable describes the information available to the agent when choosing an action, hence defining the policy class they optimize over.

Definition 4 (POST) . A partially-observable sequential team is a controlled stochastic process that specifies the joint distribution of T variables ( X t ) t ∈ [ T ] , together with a specification of the observability of each variable. Here each X t is either a system variable or an action variable, and is either observable by the learning agent or not. A POST is specified by the following components.

Variable Structures. The variables { X t } t ∈ [ T ] are partitioned into two disjoint subsets - system variables and action variables. S ⊂ [ T ] indexes system variables and A ⊂ [ T ] indexes action variables, with S ∩ A = ∅ , S ∪ A = [ T ] .

Information Structure. For t ∈ [ T ] , the 'information set' I t ⊂ [ t -1] of the variable X t is the set of past variables that are coupled to X t in the dynamics. That is, the transition to X t depends on the value of I t := ( X s : s ∈ I t ) . We call I t the 'information variable' at time t , and call I t = ∏ s ∈I t X s the 'information space'.

System Kernels. For any t ∈ S , T t ∈ P ( X t | I t ) is kernel from I t to X t that specifies the conditional distribution of a system variable X t given the information variable I t .

Decision Kernels. Each agent chooses a decision kernel π t : I t → P ( X t ) , specifying a (potentially randomized) policy for choosing an action at time t ∈ A .

Observability. We denote the observable system variables by O ⊂ S . We require that the information sets of the action variables are observable to the learning algorithm: O ⊃ ∪ t ∈A ( I t ∩ S ) . We define U := O∪A , and let H := |U| be the time-horizon of the observable variables.

Reward Function. At the end of an episode, the team receives a reward determined by the function R : ∏ s ∈U X s → [0 , 1] .

With the above components, any set of decision kernels (i.e., joint policy) π = ( π t : t ∈ A ) induces a unique probability measure over X 1 ×··· × X T , which is given by

We will be interested in modeling the observable dynamics of the POST. We index the observable variables by their position among the observables h ∈ [ H ] rather than their position among all variables, as follows: ( X t ) t ∈U = ( X t (1) , . . . , X t ( H ) ) , where t : [ H ] →U ⊂ [ T ] maps the index over observables to the index over all variables. The distribution of the observables is obtained by marginalizing over the unobservable variables, P π [ x t (1) , . . . , x t ( H ) ] = ∑ s ∈O ∁ ∑ x s ∈ X s P π [ X 1 = x 1 , . . . X T = x T ] .

The value of a policy is given by its expected reward V ( π ) := E π [ R ( X t (1) , . . . , X t ( H ) ) ] , where E π is the expectation associated with the probability measure P π . The objective of a POST is to learn a policy π = ( π t ) t ∈A which maximizes the expected reward, sup π V ( π ) . When the variable spaces X t are finite, this supremum is attained by a deterministic policy, π t : I t → X t , t ∈ A .

Representation of the information structure as a directed acyclic graph. The information structure of a POST can be naturally represented as a labeled directed acyclic graph (DAG). Given the variable structure and information structure of a POST, ( S , A , {I t } t ) , its DAG representation is given by G ( V , E , L ) . The nodes of the graph are the set of variables, V = [ T ] = S ∪ A . The edges E ⊂ V × V of the DAG are given by E = { ( i, t ) : t ∈ [ T ] , i ∈ I t } . That is, there exists an edge from i to t if i is in the information set of t . Finally, L contains labels for each node as being a system variable (in S ) or an action variable (in A ). Further, the observability of system variables (in O ) is also labeled.

This DAG represents a directed graphical model for the POST. In particular, the probability distribution on X 1 ×··· × X T factors according to G ,

where pa( X t ) is the set of parents of X t in G (which are I t ). This representation of the information structure as a DAG will be crucial for our analysis of the dynamics of sequential teams in Section 3.2.

Partially observable sequential games . We define partially-observable sequential games (POSGs) in a similar manner. The main difference is that, in the game setting, there exists an expanded reward structure with N different reward objectives R 1 , ..., R N , with different agents pursuing different objectives. The action index set is partitioned into N subsets A = A 1 ∪ · · · A N , where A i ⊂ A denotes the action index set associated with the agent(s) optimizing for objective R i . The extension to the game setting is treated in detail in Appendix F.

3.2 Information Structure Determines the Rank of POSTs/POSGs

POSTs and POSGs form a highly general framework that captures any sequential decision-making problem that can be described by a controlled stochastic process, subsuming classical models such as MDPs, POMDPs, Dec-POMDPs, and POMGs. By introducing a model with an explicit representation of information structure, we gain the ability to perform a richer analysis of sequential decision-making problems. In particular, we will show that the rank of the dynamics, and ultimately the statistical complexity of reinforcement learning, can be characterized as a function of the information structure.

We begin by defining the central quantity in our analysis, which we call the 'information-structural state', hinting at the role it will play. The information structural state is defined for each point in time as a subset of the past (whether observed or latent) which forms a sufficient statistic for the future.

Definition 5 (Information-structural state) . Let G † be the DAG obtained from G by removing all edges directed towards actions. That is, it consists of the edges E † := E \ { ( x, a ) : x ∈ N , a ∈ A} . For each h ∈ [ H ] , let I † h ⊂ [ t ( h )] be the minimal set of past variables (observed or unobserved) which d -separates the past observations ( X t (1) , . . . , X t ( h ) ) from the future observations ( X t ( h +1) , . . . , X t ( H ) ) in the DAG G † . Define I † h := ∏ s ∈I † h X s as the joint space of those variables.

The following theorem, whose proof is given in Appendix H, states that the rank of the observable system dynamics of POSTs and POSGs is bounded by the cardinality of I † h .

Theorem 1. The rank of the observable system dynamics of a POST or POSG is bounded in terms of its information structure by r ≤ max h ∈ [ H ] | I † h | .

This result shows that the complexity of the observable system dynamics is characterized by the information structure through I † h . In particular, i † h ∈ I † h describes a set of system variables, either observable or latent, which provide a sufficient statistic of the past at time h for predicting future observationsI † h 'separates' the past from the future. Hence, the quantity | I † h | admits an interpretation as the size an effective state space at time h . This is a generalization of the standard notion of a latent state. For example, in the case of POMDPs or Dec-POMDPs, the information-structural state indeed coincides with the latent Markovian state (Figure 2a). We emphasize that I † h may contain observable variables as well as unobservable system variables. In fact, unobservable system variables can introduce crucial structure that simplifies the observable system dynamics.

3.3 Examples of Information Structures and their Rank

In this section, to provide some intuition, we present examples of information structures and apply Theorem 1 to obtain a bound the rank of their observable system dynamics. We see that classical models such as MDPs, POMDPs, and POMGs are special cases of the POST/POSG framework, and known results about their rank [31] are recovered by the generalized graph-theoretic analysis of their information structure. For notational convenience, we adopt a modified notation in this section where information sets I are indexed by the symbol of the variable rather than its time-index. For example, in an MDP, we write I ( s t ) = { s t -1 , a t -1 } for the information set of the state variable s t .

Decentralized POMDPs and POMGs. At each time t , the system variables of a decentralized POMDP (or POMG) consists of a latent state s t , observations for each agent o 1 t , . . . , o N t , and actions of each agent a 1 t , . . . , a N t . The latent state transitions are Markovian and depend on the agents' joint action. The observations are sampled via a kernel conditioned on the latent state. Each agent can use their own history of observations to choose an action. Thus, the information structure is given by

Here, the observable variables are U = { o i 1: T , a i 1: T , i ∈ [ N ] } . By Theorem 1, we have I † ( o i t ) = I † ( a i t ) = { s t } , ∀ t, i , as shown in Figure 2a. Thus, the rank of a Dec-POMDP is bounded by | S | , where S is the state space. Note that in the case of models with a true latent state (e.g., POMDPs, DecPOMDPs, and POMGs), the information-structural state coincides with the Markovian latent state.

Point-to-Point Real-Time Communication with Feedback. Consider the following model of realtime communication with feedback. Let x t be a Markov source. At time t , the encoder receives the source x t ∈ X and sends an encoded symbol z t ∈ Z . The symbol is sent through a memoryless noisy channel which outputs y t to the receiver. The decoder produces the estimate ̂ x t . The output of the noisy channel is also fed back to the encoder. The encoder and decoder have full memory of their observations and previous 'actions'. The observation variables are O = { x 1: T , y 1: T } and the 'actions' are A = { z 1: T , ̂ x 1: T } . Hence, the information structure is given by the following,

By Theorem 1, we have that,

Hence, the rank is bounded by | X || Z | . This is depicted in Figure 2c.

Limited-memory information structures. Consider a sequential decision making problem with variables o t , a t , t ∈ [ T ] and an information structure with lengthm 'memory'. That is, observations do not directly depend on variables more than m steps in the past. That is, the information structure is

The observables consist of all observations and actions, U = { o 1: T , a 1: T } . By Theorem 1 we have that I † ( o t ) = { o t -m : t -1 , a t -m : t -1 } , as shown in Figure 2d. Hence, the rank of this sequential decision-making problem is bounded by | O | m | A | m .

s

1

o

1

1

o

2

1

s

2

o

1

2

o

2

2

s

3

o

1

3

o

2

3

s

4

o

1

4

o

2

4

s

5

o

1

5

o

2

5

a

1

1

a

2

1

a

1

2

a

2

2

a

1

3

a

2

3

a

1

4

a

2

4

a

1

5

a

2

5

(a) Decentralized POMDP/POMG information-structure.

s

1

1

a

1

1

s

2

1

a

2

1

s

3

1

a

3

1

s mf

1

a mf

1

s

1

2

s

2

2

s

3

2

a

1

2

a

2

2

a

3

2

s mf

2

a mf

2

s

1

3

s

2

3

s

3

3

a

1

3

a

2

3

a

3

3

s mf

3

a mf

3

s

1

4

s

2

4

s

3

4

a

1

4

a

2

4

a

3

4

s mf

4

a mf

4

(b) 'Mean-field' information structure.

x

1

z

1

y

1

x

1

x

2

z

2

y

2

x

2

x

3

z

3

y

3

x

3

x

4

z

4

y

4

x

4

x

5

z

5

y

5

x

5

(c) Point-to-point real-time communica-

tion with feedback information structure.

o

1

o

2

a

1

o

3

a

2

o

4

a

3

o

5

a

4

o

6

a

5

o

7

a

6

a

7

o

1

o

2

a

1

o

3

a

2

o

4

a

3

o

5

a

4

o

6

a

5

o

7

a

6

a

7

The examples above show that the complexity of the dynamics of a sequential decision-making problem depends directly on its information structure. An expanded version of this discussion is provided in Appendix D. Next, we use an information-structural analysis to construct a generalized PSR representation for a class of POSTs/POSGs, which we will ultimately use to prove an upper bound on the statistical complexity of reinforcement learning as a function of information structure.

4 Constructing a PSR Parameterization for POSTs and POSGs

A key challenge in reinforcement learning is constructing representations which enable robustly and efficiently modeling probabilities of system trajectories (i.e., probabilities of the form P [ future | history ] ). In this section, we will construct a generalized predictive state representation for a class of POSTs/POSGs, ultimately enabling sample-efficient reinforcement learning.

4.1 Core test sets for POSTs/POSGs

Recall that a core test set is a set of futures such that the probabilities of those futures given the past encode all the information that the past contains about the future. For systems with a simple and

Observables

Unobservables

ℎ

𝐺ℎ

∈ ℝ ℚ ℎ

𝑚 ×𝕀 ℎ

†

𝑞ℎ

∈ ℚℎ

𝑚

𝑚

-step future

𝑖

ℎ

†

∈ 𝕀

ℎ

†

ഥ

ℙ 𝑞ℎ  𝑖

ℎ

†

)

Information-structural state

𝑥𝑡 ℎ

: =

𝐺 ℎ

†

⊤

ഥ

ℙ 𝑥𝑡(ℎ) , 𝑞 ℎ+1

1

𝑖

ℎ

†

𝑖 ℎ

†

∈𝕀 ℎ

†

⋮

𝐺 ℎ

†

⊤

ഥ

ℙ 𝑥𝑡(ℎ), 𝑞

ℎ+1

𝑑ℎ+1

𝑖

ℎ

†

𝑖 ℎ

†

∈𝕀 ℎ

†

regular information structure, a core test set may be simple to obtain. For example, undercomplete POMDPs with a full-rank emission matrix admit the 1-step observation futures as core test sets.

For POSTs/POSGs with arbitrary information structures, obtaining a core test set is much more challenging without knowing the system dynamics. In this section, we identify a condition in terms of the information structure under which m -step futures are a core test set for POSTs/POSGs. Let us denote the candidate core test set of m -step futures at time h by Q m h := ∏ s ∈U h +1:min( h + m,H ) X s . We define the matrix G h as encoding the probability of observing each m -step future conditioned on the information-structural state i † h ∈ I † h :

The operational meaning of G h is depicted in Figure 3. Next, we formulate a condition in terms of information structure that we will show implies that the m -step futures are core test sets.

Definition 6 ( m -step I † -weakly revealing) . We say that a sequential team is m -step I † -weakly revealing if for all h ∈ [ H ] , rank( G h ) = | I † h | . Furthermore, we say that the sequential team is α -robustly m -step I † -weakly revealing if for all h ∈ [ H -m +1] , σ | I † h | ( G h ) ≥ α .

̸

The I † -weakly revealing condition is essentially a statistical identifiability condition. If a POST/POSG is I † -weakly revealing, then, for any two mixtures of the information-structural state, the distributions of the m -step futures are distinct. Formally, for any ν 1 , ν 2 ∈ P ( I † h ) with supp( ν 1 ) ∩ supp( ν 2 ) = ∅ , we have G h ν 1 = G h ν 2 . That is, the future observations contain information that can distinguish between mixtures of the latent information-structural state. The α -robust version of the I † -weakly revealing condition requires that G h is not only full rank, but that its | I † h | -th singular value is bounded away from zero, so that ∥ G h ν 1 -G h ν 2 ∥ ≥ α ∥ ν 1 -ν 2 ∥ .

The condition holds whenever there exists a sequence of actions within the m -step futures such that executing these actions results in a sequence of observations that is informative about the informationstructural state i † h ∈ I † h . In general, this condition will be harder to satisfy when I † h is large since it would require the m -step future observations to encode more information. In particular, G h cannot be full rank when | Q m h | < | I † h | . As a heuristic, when we don't have prior knowledge about the dynamics (e.g., in the learning setting), we can choose m such that | Q m h | ≥ | I † h | . In general, it will be possible to find a smaller core test set when the d -separating set I † h is small. This happens when the system dynamics contain state-like variables that are low-dimensional.

The I † -weakly-revealing condition is a generalization of the 'weakly-revealing' condition for POMDPs introduced by Liu et al. [36]. Liu et al. [37] proposed an algorithm for learning weaklyrevealing POMGs. Our analysis here recovers weakly-revealing POMGs as a special case and enables learning a much more general class of problems. We note that such an identifiability condition is necessary, and is reflective of the fundamental difficulty of the partially-observable setting. For example, in the case of POMDPs, there exist hardness results that state that if an analogous condition does not hold, the statistical complexity can scale exponentially with the relevant quantities [36].

We now proceed to show that under the I † -weakly-revealing condition, m -step futures are core test sets for POSTs/POSGs which can be used to construct a generalized PSR representation amenable

to learning. Recall that the vector of core test set probabilities for the history τ h is given by the mappings ψ h , ψ h : H h → R | Q m h | ,

Define the mapping m h : F h → R | Q m h | as,

The following lemma, whose proof is given in Appendix I, shows that the m -step futures Q m h are core test sets for any m -step I † -weakly revealing POST/POSG. In particular, given any future ω h ∈ F h and history τ h ∈ H h , the conditional probability P [ ω h | τ h ] can be written as a linear combination of the core test probabilities ψ h ( τ h ) , with weights given by m h ( ω h ) , depending only on the future ω h .

Lemma 1 (Core test set for POSTs) . Suppose that the POST is m -step I † -weakly revealing. Then, Q m h is a core test set for all h ∈ [ H ] . Furthermore, we have P [ τ h , ω h ] = ⟨ m h ( ω h ) , ψ h ( τ h ) ⟩ and P [ ω h | τ h ] = 〈 m h ( ω h ) , ψ h ( τ h ) 〉 .

4.2 Generalized PSR parameterization of POST/POSG

Consider a POST/POSG which is m -step I † -weakly revealing. Lemma 1 shows that the m -step futures Q m h are core test sets. In this section, we will explicitly construct a generalized PSR parameterization for this class of sequential decision-making problems. Moreover, we will show that this generalized PSR representation is well-conditioned when the weakly revealing condition is robust.

Let d h := | Q m h | . The key observation that allows us to construct the generalized PSR representation is that the vector mappings m h : F h → R d h and ψ h : H h → R d h can be used to derive a recursive form of the dynamics of the POST/POSG. We define the operator mapping M h : X t ( h ) → R d h × d h -1 by

That is, M h ( x t ( h ) ) is the matrix whose rows are indexed by the core tests at the h -th observable step, where the row corresponding to each q ∈ Q h is the weights returned by the mapping m h -1 when applied to the future consisting of x t ( h ) followed by core test q . The operator map M h allows us to update the probabilities of the core test sets after receiving an additional observation x t ( h ) : ψ h ( x t (1) , ..., x t ( h ) ) = M h ( x t ( h ) ) ψ h -1 ( x t (1) , ..., x t ( h -1) ) . The following result, whose proof is given in Appendix I, states that the set of operators { M h } h forms a generalized PSR.

Theorem 2. Consider an m -step I † -weakly revealing POST/POSG. Let { M h } h ∈ [ H -1] be defined as above, and let ψ 0 = [ P [ q ] ] q ∈ Q m 0 , ϕ H ( x t ( H ) ) = e x t ( H ) . Then, ( { Q m h } h , ϕ H , { M h } h ∈ [ H -1] , ψ 0 ) forms a generalized predictive state representation. Moreover, if the weakly-revealing property is α -robust, then this PSR is γ -well-conditioned with γ = α/ max h | I † h | 1 / 2 .

Thus, we have constructed a robust parameterization of the sequential decision-making problem, making use of its information structure, that will enable us to design an efficient learning algorithm.

5 Characterizing the Statistical Complexity of General Reinforcement Problems via Information Structure

In this section, we establish an upper bound on the achievable sample complexity of general reinforcement learning problems in terms of their information structure. In plain language, this is a result that roughly says 'any sequential decision-making problem with an information structure I can be learned with a sample complexity at most f ( I ) '. This identifies a class of sequential decision-making problems that are statistically tractable via conditions on the information structure, expanding the set of known-tractable problems while recovering existing tractability results as a special case.

We will prove this result by exhibiting an algorithm that achieves this upper bound. Our approach will be to use the generalized predictive state representation constructed for POSTs/POSGs in Section 4, which provides a robust representation amenable to learning. We will introduce an algorithm for

learning generalized PSRs and prove a corresponding sample complexity result, which will in turn imply a bound on the sample complexity of learning POST/POSG models via the informationstructural characterization of the rank of observable dynamics established in Section 3.2.

There exist several works in the literature which study learning in PSRs [e.g. 33-35, 38, 39]. Using the technical tools developed in this paper, most of these algorithms can be directly extended to our generalization of PSRs. With such an algorithm, Theorems 1 and 2 then imply a bound on the achievable sample complexity for learning general sequential decision-making problems in terms of their information structure. In this work, we will adapt the model-based UCB-type algorithm of Huang et al. [35], extending it to generalized PSRs to obtain a bound on the achievable sample complexity for POSTs/POSGs. We will defer the details of the algorithm to Appendix E and formally verify the proof in Appendix J. Here, we will focus on discussing the role of information structure in determining the statistical complexity of reinforcement learning.

The following result states that the size of the information-structural state, | I † h | , characterizes an upper bound on the statistical complexity of learning a sequential decision-making problem.

Theorem 3. Suppose a sequential decision-making problem described by a POST is α -robustly m -step I † -weakly revealing. Let Q m := max h | Q m h | be the size of the m -step observable trajectories, and let A = max s ∈A | X s | be the size of largest action space. Then, there exists an algorithm that can learn an ϵ -optimal policy with a sample complexity (omitting log factors) bounded by

Under the game setting, the same assumption implies the existence of a self-play algorithm that learns an ϵ (Nash or coarse-correlated) equilibrium with the same sample complexity.

This result identifies I † -weakly revealing POSTs/POSGs as a class of statistically tractable sequential decision-making problems. The sample complexity is polynomial in the size of the informationstructural state space max h | I † h | , the size of the action space A , and the time horizon H . Further, it depends on Q m , the size of m -step observable trajectories, and the robustness parameter α -1 , which corresponds to the m -step I † -weakly revealing identifiability condition. We note that the algorithm constructed to prove Theorem 3 only needs to know the parameters of the I † -weakly revealing condition (i.e., m and α ), and does not need to know the full information structure.

This result shows that the size of the information-structural state is a fundamental measure of the statistical complexity of reinforcement learning. As a result, learning is tractable when max h | I † h | is of modest size, and the information structural state is strongly coupled to the observable system variables.

One notable special case of POSTs/POSGs is POMDPs. Learning in POMDPs has been studied extensively in the literature. Theorem 3 implies a poly( S, O, A, H, α -1 ) · ϵ -2 bound on the sample complexity of learning in α -weakly POMDPs and poly( S, ( OA ) m , H, α -1 ) · ϵ -2 for learning m -step weakly revealing POMDPs. This recovers a similar sample complexity as was shown by more specialized analysis tailored to POMDPs [e.g., 36, 40].

6 Conclusion

Summary. This paper examines the role of information structure in general reinforcement learning problems. We introduced novel models that explicitly represent information structure, and proved an upper bound on the sample complexity of general reinforcement learning problems in terms of information structure. The central quantity in this upper bound is derived from a graphical representation of the information structure, and admits an interpretation as an effective informationstructural state, generalizing the typical notion of a Markovian state.

Limitations. The results of this paper are theoretical in nature, and the algorithm proposed to prove our main result is not computationally practical. Current SOTA algorithms for partially-observable algorithms are based on recurrent neural networks or other deep learning models, which create internal representations of latent states based on the history, akin to belief states. Our theory may offer insights into the design of improved architectures in such deep learning-based algorithms. For example, the information structure can be incorporated as an inductive bias of the neural network.

References

Hans S Witsenhausen. 'The Intrinsic Model for Discrete Stochastic Control: Some Open Problems'. In: Control Theory, Numerical Methods and Computer Systems Modelling . Ed. by M. Beckmann, H. P. Künzi, A. Bensoussan, and J. L. Lions. Berlin, Heidelberg: Springer Berlin Heidelberg, 1975 (cited on pages 1, 2, 17).

Hans S Witsenhausen. 'On information structures, feedback and causality'. In: SIAM Journal on Control (1971) (cited on page 1).

Y. Ho and K. Chu. 'Team decision theory and information structures in optimal control problems-Part I'. In: IEEE Transactions on Automatic Control (1972) (cited on pages 1, 17).

Y. Ho and K. Chu. 'On the Equivalence of Information Structures in Static and Dynamic Teams'. In: IEEE Transactions on Automatic Control (1973) (cited on page 1).

Tsuneo Yoshikawa. 'Decomposition of dynamic team decision problems'. In: IEEE Transactions on Automatic Control (1978) (cited on page 1).

Hans S Witsenhausen. 'Equivalent Stochastic Control Problems'. In: Mathematics of Control, Signals, and Systems (1988) (cited on page 1).

Mark S Andersland and Demosthenis Teneketzis. 'Information structures, causality, and nonsequential stochastic control I: Design-independent properties'. In: SIAM journal on control and optimization (1992) (cited on page 1).

Demosthenis Teneketzis. 'On Information Structures and Nonsequential Stochastic Control'. In: CWI Quarterly (1996) (cited on page 1).

Aditya Mahajan and Sekhar Tatikonda. 'A Graphical Modeling Approach to Simplifying Sequential Teams'. In: 2009 7th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks . 2009 (cited on page 1).

Ashutosh Nayyar, Aditya Mahajan, and Demosthenis Teneketzis. 'Optimal Control Strategies in Delayed Sharing Information Structures'. In: IEEE Transactions on Automatic Control (2011) (cited on page 1).

Aditya Mahajan, Nuno C. Martins, Michael C. Rotkowitz, and Serdar Yuksel. 'Information Structures in Optimal Decentralized Control'. In: 2012 IEEE 51st IEEE Conference on Decision and Control (CDC) . Maui, HI, USA: IEEE, 2012 (cited on pages 1, 17).

Ashutosh Nayyar, Aditya Mahajan, and Demosthenis Teneketzis. 'The Common-Information Approach to Decentralized Stochastic Control'. In: Information and Control in Networks . Ed. by Giacomo Como, Bo Bernhardsson, and Anders Rantzer. Cham: Springer International Publishing, 2014 (cited on pages 1, 18).

Andreas A. Malikopoulos. 'On Team Decision Problems with Nonclassical Information Structures'. 2022. arXiv: 2101.10992 [math] (cited on page 1).

Naci Saldi and Serdar Yüksel. 'Geometry of Information Structures, Strategic Measures and Associated Stochastic Control Topologies'. In: Probability Surveys (2022) (cited on page 1).

Hans S Witsenhausen. 'Separation of estimation and control for discrete time systems'. In: Proceedings of the IEEE (1971) (cited on pages 1, 17).

Richard Bellman. 'Dynamic programming and stochastic control processes'. In: Information and control (1958) (cited on page 1).

Christopher John Cornish Hellaby Watkins. 'Learning from delayed rewards'. In: (1989) (cited on page 1).

Satinder Singh, Tommi Jaakkola, Michael L Littman, and Csaba Szepesvári. 'Convergence results for single-step on-policy reinforcement-learning algorithms'. In: Machine learning (2000) (cited on page 2).

Richard S Sutton, Hamid Maei, and Csaba Szepesvári. 'A convergent o ( n ) temporal-difference algorithm for off-policy learning with linear function approximation'. In: Advances in neural information processing systems (2008) (cited on page 2).

Rémi Munos and Csaba Szepesvári. 'Finite-Time Bounds for Fitted Value Iteration.' In: Journal of Machine Learning Research (2008) (cited on page 2).

Yasin Abbasi-Yadkori and Csaba Szepesvári. 'Regret bounds for the adaptive control of linear quadratic systems'. In: Proceedings of the 24th Annual Conference on Learning Theory . JMLR Workshop and Conference Proceedings. 2011 (cited on page 2).

Tor Lattimore and Marcus Hutter. 'PAC bounds for discounted MDPs'. In: Algorithmic Learning Theory: 23rd International Conference, ALT 2012, Lyon, France, October 29-31, 2012. Proceedings 23 . Springer. 2012 (cited on page 2).

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 'Playing Atari with Deep Reinforcement Learning'. 2013. arXiv: 1312.5602 [cs] (cited on page 2).

Jens Kober, J. Andrew Bagnell, and Jan Peters. 'Reinforcement Learning in Robotics: A Survey'. In: The International Journal of Robotics Research (2013) (cited on pages 2, 18).

Volodymyr Mnih et al. 'Human-Level Control through Deep Reinforcement Learning'. In: Nature (2015) (cited on page 2).

David Silver et al. 'Mastering the Game of Go with Deep Neural Networks and Tree Search'. In: Nature (2016) (cited on pages 2, 18).

Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. 'Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving'. 2016. arXiv: 1610.03295 [cs, stat] (cited on pages 2, 18).

Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojciech M. Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, and Richard Powell. 'Alphastar: Mastering the Real-Time Strategy Game Starcraft Ii'. In: DeepMind blog (2019) (cited on pages 2, 18).

Aditya Mahajan and Sekhar Tatikonda. 'An Axiomatic Approach for Simplification of Sequential Teams'. In: (2011) (cited on page 2).

Aditya Mahajan and Sekhar Tatikonda. 'An Algorithmic Approach to Identify Irrelevant Information in Sequential Teams'. In: Automatica (2015) (cited on page 2).

Michael Littman and Richard S Sutton. 'Predictive representations of state'. In: Advances in neural information processing systems (2001) (cited on pages 3, 6, 17, 19, 20).

Herbert Jaeger. 'Observable Operator Models for Discrete Stochastic Time Series'. In: Neural Computation (2000) (cited on pages 3, 17, 19).

Qinghua Liu, Praneeth Netrapalli, Csaba Szepesvári, and Chi Jin. 'Optimistic MLE - A Generic Model-based Algorithm for Partially Observable Sequential Decision Making'. 2022. arXiv: 2209.14997 [cs, stat] (cited on pages 4, 10, 17, 18, 21, 25, 30, 39).

Masatoshi Uehara, Ayush Sekhari, Jason D. Lee, Nathan Kallus, and Wen Sun. 'Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systems'. 2022. arXiv: 2206.12020 [cs, math, stat] (cited on pages 4, 10, 17).

Ruiquan Huang, Yingbin Liang, and Jing Yang. 'Provably Efficient UCB-type Algorithms For Learning Predictive State Representations'. 2023. arXiv: 2307.00405 [cs, stat] (cited on pages 4, 10, 17, 18, 21, 24, 26, 27, 38, 40, 41, 47, 53, 54).

Qinghua Liu, Alan Chung, Csaba Szepesvári, and Chi Jin. 'When Is Partially Observable Reinforcement Learning Not Scary?' 2022. arXiv: 2204.08967 [cs, eess, stat] (cited on pages 8, 10, 17, 18, 25, 27).

Qinghua Liu, Csaba Szepesvári, and Chi Jin. 'Sample-Efficient Reinforcement Learning of Partially Observable Markov Games'. 2022. arXiv: 2206.01315 [cs, stat] (cited on pages 8, 18, 27).

Fan Chen, Yu Bai, and Song Mei. 'Partially Observable RL with B-Stability: Unified Structural Condition and Sharp Sample-Efficient Algorithms'. 2022. arXiv: 2209.14990 [cs, math, stat] (cited on pages 10, 17, 18, 25, 48).

Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D. Lee. 'PAC Reinforcement Learning for Predictive State Representations'. 2022. arXiv: 2207.05738 [cs] (cited on pages 10, 17, 18).

Chi Jin, Sham M. Kakade, Akshay Krishnamurthy, and Qinghua Liu. 'Sample-Efficient Reinforcement Learning of Undercomplete POMDPs'. 2020. arXiv: 2006.12484 [cs, math, stat] (cited on pages 10, 17).

Christos H. Papadimitriou and John N. Tsitsiklis. 'The Complexity of Markov Decision Processes'. In: Mathematics of operations research (1987) (cited on page 17).

Ashutosh Nayyar, Aditya Mahajan, and Demosthenis Teneketzis. 'Optimal control strategies in delayed sharing information structures'. In: IEEE Transactions on Automatic Control (2010) (cited on page 17).

Ashutosh Nayyar, Abhishek Gupta, Cedric Langbort, and Tamer Ba¸ sar. 'Common information based Markov perfect equilibria for stochastic games with asymmetric information: Finite games'. In: IEEE Transactions on Automatic Control (2013) (cited on page 17).

Ashutosh Nayyar, Aditya Mahajan, and Demosthenis Teneketzis. 'Decentralized stochastic control with partial history sharing: A common information approach'. In: IEEE Transactions on Automatic Control (2013) (cited on page 17).

Yi Ouyang, Hamidreza Tavafoghi, and Demosthenis Teneketzis. 'Dynamic games with asymmetric information: Common information based perfect bayesian equilibria and sequential decomposition'. In: IEEE Transactions on Automatic Control (2016) (cited on page 17).

Aditya Dave, Nishanth Venkatesh, and Andreas A Malikopoulos. 'Decentralized Control of Two Agents with Nested Accessible Information'. In: 2022 American Control Conference (ACC) . IEEE. 2022 (cited on page 17).

Yue Guan, Mohammad Afshari, and Panagiotis Tsiotras. 'Zero-Sum Games between MeanField Teams: A Common Information and Reachability based Analysis'. 2023. arXiv: 2303. 12243 [eess.SY] (cited on page 17).

Serdar Yuksel and Tamer Basar. 'Stochastic Teams, Games and Control under Information Constraints'. Springer, 2023 (cited on page 17).

Joseph A Tatman and Ross D Shachter. 'Dynamic programming and influence diagrams'. In: IEEE transactions on systems, man, and cybernetics (1990) (cited on page 17).

Daphne Koller and Brian Milch. 'Multi-agent influence diagrams for representing and solving games'. In: Games and economic behavior (2003) (cited on page 17).

Ian Osband and Benjamin Van Roy. 'Near-optimal reinforcement learning in factored mdps'. In: Advances in Neural Information Processing Systems (2014) (cited on page 17).

Peter Auer, Thomas Jaksch, and Ronald Ortner. 'Near-Optimal Regret Bounds for Reinforcement Learning'. In: Advances in neural information processing systems (2008) (cited on page 17).

Shipra Agrawal and Randy Jia. 'Optimistic Posterior Sampling for Reinforcement Learning: Worst-Case Regret Bounds'. In: Advances in Neural Information Processing Systems (2017) (cited on page 17).

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. 'Minimax Regret Bounds for Reinforcement Learning'. In: International Conference on Machine Learning . PMLR, 2017 (cited on page 17).

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. 'Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism'. In: Advances in Neural Information Processing Systems (2021) (cited on page 17).

Martin Mundhenk, Judy Goldsmith, Christopher Lusena, and Eric Allender. 'Complexity of Finite-Horizon Markov Decision Process Problems'. In: Journal of the ACM (JACM) (2000) (cited on page 17).

Nikos Vlassis, Michael L. Littman, and David Barber. 'On the Computational Complexity of Stochastic Controller Optimization in POMDPs'. In: ACM Transactions on Computation Theory (TOCT) (2012) (cited on page 17).

Elchanan Mossel and Sébastien Roch. 'Learning Nonsingular Phylogenies and Hidden Markov Models'. In: Proceedings of the Thirty-Seventh Annual ACM Symposium on Theory of Computing . 2005 (cited on page 17).

Akshay Krishnamurthy, Alekh Agarwal, and John Langford. 'PAC Reinforcement Learning with Rich Observations'. In: Advances in Neural Information Processing Systems (2016) (cited on page 17).

Yonathan Efroni, Chi Jin, Akshay Krishnamurthy, and Sobhan Miryoosefi. 'Provable Reinforcement Learning with a Short-Term Memory'. In: International Conference on Machine Learning . PMLR, 2022 (cited on page 17).

Jiacheng Guo, Zihao Li, Huazheng Wang, Mengdi Wang, Zhuoran Yang, and Xuezhou Zhang. 'Provably Efficient Representation Learning with Tractable Planning in Low-Rank POMDP'. 2023. arXiv: 2306.12356 [cs.LG] (cited on page 17).

Hongming Zhang, Tongzheng Ren, Chenjun Xiao, Dale Schuurmans, and Bo Dai. 'Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning'. 2023. arXiv: 2311.12244 [cs.LG] (cited on page 17).

Noah Golowich, Ankur Moitra, and Dhruv Rohatgi. 'Learning in observable pomdps, without computationally intractable oracles'. In: Advances in Neural Information Processing Systems (2022) (cited on page 17).

Noah Golowich, Ankur Moitra, and Dhruv Rohatgi. 'Planning in Observable POMDPs in Quasipolynomial Time'. 2022. arXiv: 2201.04735 [cs.LG] (cited on page 17).

Noah Golowich, Ankur Moitra, and Dhruv Rohatgi. 'Planning and Learning in Partially Observable Systems via Filter Stability'. In: Proceedings of the 55th Annual ACM Symposium on Theory of Computing . 2023 (cited on page 17).

Qi Cai, Zhuoran Yang, and Zhaoran Wang. 'Reinforcement learning from partial observation: Linear function approximation with provable sample efficiency'. In: International Conference on Machine Learning . PMLR. 2022 (cited on page 17).

Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. 'Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiency'. 2022. arXiv: 2205.13476 [cs, eess, stat] (cited on page 17).

Satinder P. Singh, Michael L. Littman, Nicholas K. Jong, David Pardoe, and Peter Stone. 'Learning Predictive State Representations'. In: Proceedings of the 20th International Conference on Machine Learning (ICML-03) . 2003 (cited on page 17).

Satinder Singh, Michael R James, and Matthew R Rudary. 'Predictive State Representations: A New Theory for Modeling Dynamical Systems'. In: Proceedings of the Twentieth Conference on Uncertainty in Artificial Intelligence (UAI2004) (2004) (cited on page 17).

Michael R. James, Satinder Singh, and Michael L. Littman. 'Planning with Predictive State Representations'. In: 2004 International Conference on Machine Learning and Applications, 2004. Proceedings. IEEE, 2004 (cited on page 17).

Peter McCracken and Michael Bowling. 'Online Discovery and Learning of Predictive State Representations'. In: Advances in neural information processing systems (2005) (cited on page 17).

Byron Boots, Sajid M. Siddiqi, and Geoffrey J. Gordon. 'Closing the Learning-Planning Loop with Predictive State Representations'. In: The International Journal of Robotics Research (2011) (cited on page 17).

Nan Jiang, Alex Kulesza, and Satinder Singh. 'Completing State Representations Using Spectral Learning'. In: Advances in Neural Information Processing Systems (2018) (cited on page 17).

Zhi Zhang, Zhuoran Yang, Han Liu, Pratap Tokekar, and Furong Huang. 'Reinforcement Learning under a Multi-agent Predictive State Representation Model: Method and Theory'. In: International Conference on Learning Representations . 2021 (cited on page 17).

Ahmed Hefny, Carlton Downey, and Geoffrey J. Gordon. 'Supervised Learning for Dynamical System Learning'. In: Advances in neural information processing systems (2015) (cited on page 17).

Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. 'GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond'. 2023. arXiv: 2211.01962 [cs, math, stat] (cited on pages 17, 18, 48).

Shuang Qiu, Ziyu Dai, Han Zhong, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. 'Posterior Sampling for Competitive RL: Function Approximation and Partial Observation'. 2023. arXiv: 2310.19861 [cs.LG] (cited on pages 17, 18, 48).

Zhihan Liu, Miao Lu, Wei Xiong, Han Zhong, Hao Hu, Shenao Zhang, Sirui Zheng, Zhuoran Yang, and Zhaoran Wang. 'Maximize to explore: One objective function fusing estimation, planning, and exploration'. In: Advances in Neural Information Processing Systems (2024) (cited on pages 17, 18).

Richard S Sutton and Andrew G Barto. 'Reinforcement learning: An introduction'. MIT press, 2018 (cited on page 18).

Noam Brown and Tuomas Sandholm. 'Superhuman AI for Multiplayer Poker'. In: Science (2019) (cited on page 18).

Ronen I. Brafman and Moshe Tennenholtz. 'R-Max-a General Polynomial Time Algorithm for near-Optimal Reinforcement Learning'. In: Journal of Machine Learning Research (2002) (cited on page 18).

Yu Bai, Chi Jin, and Tiancheng Yu. 'Near-Optimal Reinforcement Learning with Self-Play'. In: Advances in Neural Information Processing Systems . Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin. Curran Associates, Inc., 2020 (cited on page 18).

Ziang Song, Song Mei, and Yu Bai. 'When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?' In: (2021). arXiv: 2110.04184 (cited on page 18).

Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. 'Regret Minimization in Games with Incomplete Information'. In: Advances in neural information processing systems (2007) (cited on page 18).

Tadashi Kozuno, Pierre Ménard, Rémi Munos, and Michal Valko. 'Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recall'. In: (2021). arXiv: 2106.06279 (cited on page 18).

Gabriele Farina and Tuomas Sandholm. 'Model-Free Online Learning in Unknown Sequential Decision Making Problems and Games'. In: Proceedings of the AAAI Conference on Artificial Intelligence . 2021 (cited on page 18).

Xiangyu Liu and Kaiqing Zhang. 'Partially Observable Multi-agent RL with (Quasi)Efficiency: The Blessing of Information Sharing'. 2024. arXiv: 2308.08705 [cs.LG] (cited on page 18).

Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan. 'Approximate information state for approximate planning and reinforcement learning in partially observed systems'. In: The Journal of Machine Learning Research (2022) (cited on page 18).

Weichao Mao, Kaiqing Zhang, Erik Miehling, and Tamer Ba¸ sar. 'Information state embedding in partially observable cooperative multi-agent reinforcement learning'. In: 2020 59th IEEE Conference on Decision and Control (CDC) . IEEE. 2020 (cited on page 18).

Ali Kara and Serdar Yuksel. 'Near optimality of finite memory feedback policies in partially observed markov decision processes'. In: Journal of Machine Learning Research (2022) (cited on page 18).

Hsu Kao and Vijay Subramanian. 'Common information based approximate state representations in multi-agent reinforcement learning'. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2022 (cited on page 18).

Dengwang Tang, Ashutosh Nayyar, and Rahul Jain. 'A novel point-based algorithm for multiagent control using the common information approach'. In: 2023 62nd IEEE Conference on Decision and Control (CDC) . IEEE. 2023 (cited on page 18).

John Nash. 'Non-cooperative games'. In: Annals of mathematics (1951) (cited on page 28).

Sara van de Geer. 'Rates of Convergence for Maximum Likelihood Estimators'. In: Applications of Empirical Process Theory . Reprint. Cambridge Series on Statistical and Probabilistic Mathematics. Cambridge: Cambridge Univ. Pr, 2006 (cited on pages 33, 40).

Thomas Verma and Judea Pearl. 'Causal networks: Semantics and expressiveness'. In: Machine intelligence and pattern recognition . Elsevier, 1990 (cited on page 35).

Varsha Dani, Thomas P Hayes, and Sham M Kakade. 'Stochastic linear optimization under bandit feedback'. In: (2008) (cited on pages 53, 54).

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. 'Improved algorithms for linear stochastic bandits'. In: Advances in neural information processing systems (2011) (cited on pages 53, 54).

Alexandra Carpentier, Claire Vernade, and Yasin Abbasi-Yadkori. 'The elliptical potential lemma revisited'. In: (2020). arXiv: 2010.10182 [stat.ML] (cited on pages 53, 54).

A Summary of Notation

Generic Sequential Decision-Making Problems

B Related Work

The study of information structure in the control literature. In stochastic control, the construct of the 'information structure' is used to model the structural properties of a system which may restrict the flow, storage, and processing of information. The role of information structure in decentralized control has been extensively studied since Witsenhausen [15] and Ho and Chu [3] began investigating information structures in the context of team decision theory. For example, early work showed that the information structure can determine the tractability of optimal decentralized control problems [15, 41]. More generally, information structure plays an important role in the analysis of multi-agent team decision problems and games, as well as in the design of efficient algorithms, especially in the decentralized setting. See, e.g., Mahajan et al. [11], Nayyar et al. [42-44], Ouyang et al. [45], Dave et al. [46], and Guan et al. [47] and the references therein. We also refer the reader to Yuksel and Basar [48] for a comprehensive overview of the interaction between information and control, including recent progress in the field. The models we propose in this paper are closely related to Witsenhausen's intrinsic model [1], but with some added elements to model partial-observability in the context of reinforcement learning. Related to this is the framework of multi-agent influence diagrams [e.g., 49, 50] which allow for an explicit representation of dependence relations among variables in games. Whereas the above-mentioned work studies the role of information structure in planning and control, we study the the role of information structure in reinforcement learning (i.e., statistical estimation and sample complexity). We note that specific types of structural assumptions related to information structure have been studied in the learning setting as well. For example, previous work has studied factored MDPs [51], which assume an information structure where the state transitions and the reward function are factored, enabling improved learning results. In the present work, we study the role of information structure in reinforcement learning in greater generality, focusing particularly on the partially-observable setting.

Learning under partial observations. In an MDP, where the system dynamics obey a Markovian property and are fully observable, reinforcement learning has been shown to be both computationally and statistically efficient [e.g., 52-55]. However, under partial observability, reinforcement learning can be computationally and statistically intractable in the worst case, even when assuming a Markovian latent state. Such worst-case hardness results are well-known. For example, [41, 56, 57] show that planning is computational intractable and [58, 59] show that learning is statistically intractable, in the worst-case. In these worst cases, the hardness comes from instances where the observations reveal little information about the latent state, which causes errors in learned representations to be uncontrollable. Accordingly, sub-classes of POMDPs have been identified in recent work where added structural conditions make efficient learning possible. One such condition is decodability [see e.g., 59-62], which assumes that the latent state can be decoded from the current observation (i.e., Block MDP), or an m -step history of observations. Another set of conditions is the 'observability' condition [63-65] and its cousin the 'weakly revealing' condition [36, 40] which require different belief states to induce distinguishable distributions over observations. These conditions are further extended to POMDP models in the function approximation setting, where the state or observation spaces are large and function approximators (e.g., linear functions) are used to represent the model. See, e.g., Uehara et al. [34], Guo et al. [61], Zhang et al. [62], Cai et al. [66], and Wang et al. [67]. In our work, we build on the above by identifying a class of POSTs/POSGs which can be learned efficiently. This is significant since POSTs/POSGs are much more general than POMDPs and do not assume the existence of a latent state.

Predictive state representations. Predictive state representations were introduced by Littman and Sutton [31] building on prior work on observable operator models by Jaeger [32] which proposed the idea of predictive representations as an alternative to belief states for modeling HMMs and POMDPs [see also 68-71]. PSRs are a way to represent the dynamics of a sequential decision-making problem by modeling the (conditional) probabilities of a small set of future trajectories, typically called 'core tests'. In a PSR, the probability of any future trajectory is a deterministic function of the conditional probabilities of the core tests. That is, the probabilities of the core tests encode all the information that the past contains about the future. Littman and Sutton [31] showed that any POMDP can be represented as a PSR. Various reinforcement learning methods for PSRs have been proposed under the assumption that data distribution is explorative, including spectral algorithms [72-74] and supervised learning approaches [75]. In addition, when it comes to the online setting where the algorithm needs to explore, there is a line of work that extends the theory and algorithms for online POMDP learning to PSRs. Moreover, some of these works propose generic theory and algorithms that can be applied to a large class of models including MDPs, POMDPs, and two-player zero-sum dynamic games with partial observability. See, e.g., Liu et al. [33], Huang et al. [35], Liu et al. [36],

Chen et al. [38], Zhan et al. [39], Zhong et al. [76], Qiu et al. [77], and Liu et al. [78]. Our work is particularly related to the works that combine the idea of optimism in the face of uncertainty [79] and maximum likelihood model estimation [33, 35, 36, 38, 39]. Specifically, our algorithm extends the UCB-type algorithm proposed by Huang et al. [35] for standard PSRs to learn a generalization of PSRs which captures POSTs/POSGs.

Learning in multi-agent systems. Most applications of interest in reinforcement learning involve the participation of multiple agents in the same environment. Empirical research has achieved striking success in several domains, including for example in the games of Go [26], Starcraft [28], and Poker [80], as well as in robotic control [24] and autonomous driving [27]. There also exists a growing literature of theoretical work. For example, Brafman and Tennenholtz [81], Bai et al. [82], and Song et al. [83] tackle learning in Markov games (MGs)-a generalization of single-agent MDPs that assumes the existence of a Markovian state which is observable by all agents. Another model which has been explored in the literature is imperfect-information extensive-form games (IIEFG), which assumes tree-structured transitions and deterministic emission, and can be viewed as a subclass of partially-observable Markov games (POMGs). Learning under this model has been studied in Zinkevich et al. [84], Kozuno et al. [85], and Farina and Sandholm [86]. More recently, Liu et al. [37] studied reinforcement learning in POMGs using an MLE-based algorithm building on their previous work in the single-agent setting [36]. To address the computational intractability of the planning step for such model-based algorithms, Liu and Zhang [87] proposed a quasi-efficient algorithm for multi-agent POMGs that runs in quasi-polynomial time with quasi-polynomial sample complexity. Their proposed algorithm leverages the common information approach [see 12] to construct an approximate Markov game where the state space of this new game corresponds to the space of approximate common information among agents. The idea of leveraging an informationsharing structure in multi-agent reinforcement learning has also appeared in Subramanian et al. [88], Mao et al. [89], Kara and Yuksel [90], Kao and Subramanian [91], and Tang et al. [92].

In each of the above-mentioned models (e.g., MDP, POMDP, MG, IIEFG, POMG, etc.), a particular fixed information structure is assumed. We emphasize that the POST/POSG model proposed in our work allows the information structure to be specified arbitrarily and hence captures these models as special cases within a unifying framework. Moreover, our analysis and proposed algorithm significantly expand the class of multi-agent sequential decision-making problems that can be efficiently learned.

C Preliminaries

C.1 Generic Sequential Decision-Making Problems

Consider a controlled stochastic process ( X 1 , . . . , X H ) , where X h is a random variable corresponding to the variable at time h . At each time h ∈ [ H ] , the variable X h may be either an 'observation' (i.e., observable system variable) or an 'action'. The dynamics of this stochastic process are described by a tuple ( H, { X h } h , O , A , P ) , where H is the time horizon, X h is the variable space at time h (i.e., X h ∈ X h ), O ⊂ [ H ] is the index set of observations (i.e., X h is an observation if h ∈ O ), A ⊂ [ H ] is the index set of actions, and P = { P h } h ∈O is a set of probability kernels which describes the the probability of any trajectory x 1 , . . . , x H given that the actions are executed,

A choice of policy π = { π h } h ∈A induces a probability distribution on X 1 ×··· × X H as follows

We now define some notation. Let H h = ∏ s ∈ 1: h X s denote the space of histories at time h and F h = ∏ s ∈ h +1: H X s denote the space futures at time h . Similarly, let H o h = obs ( H h ) = ∏ s ∈O 1: h X s denote the observation component of histories and let H a h = act ( H h ) = ∏ s ∈A 1: h X s denote the action component. Here, O i : j denotes O∩{ i, . . . , j } , and similarly for A i : j . The observation and action components of the futures, F o h and F a h respectively, are defined similarly.

We define the system dynamics matrix D h ∈ R | H h |×| F h | as the matrix giving the probability of each possible pair of history and future at time h given the execution of the actions,

where ω o h = obs ( ω h ) are is the observation component of the future ω h , ω a h = act ( ω h ) is the action component, and similarly for τ o h , τ a h . Note that the actions are actively executed via the do -operation. Hence, the system dynamics matrices are independent of any action-selection criteria. Note that D H ∈ R | H H |× 1 is defined as [ D H ] τ H = P [ τ H ] , and D 0 = D ⊤ H .

We introduce the notion of the rank of the dynamics. The rank of such a controlled stochastic process is the maximal rank of its dynamics matrices. This is a measure of the complexity of the dynamics.

Definition (Rank of dynamics; Definition 1) . The rank of the dynamics { D h } h ∈ [ H ] is r = max h ∈ [ H ] rank( D h ) .

This defines the dynamics of the system. A sequential decision-making problem is such a controlled stochastic process together with an objective . The objective is defined by a reward function R : X 1 × · · · × X H → [0 , 1] mapping a trajectory to a reward in [0 , 1] . The agent(s) can affect the dynamics of the system through their choice of actions or policies. Each action X h , h ∈ A may be chosen by either a single agent or one of several agents (e.g., a team). The policy at time h ∈ A is a mapping π h : H h -1 →P ( X h ) from previous observations to an action (or a distribution over actions, if randomized). The collection of policies at all time steps is denoted π = ( π h : h ∈ A ) , and induces a probability distribution over trajectories, denoted P π . Then, the value of a policy π is the expected value of the reward under the measure P π , V R ( π ) := E π [ R ( X 1 , . . . , X H )] , where E π is the expectation associated with P π .

The formalism of sequential decision-making problems introduced in this section is highly generic, but does not explicitly model the information structure . In the next section, we introduce the models of partially observable sequential teams/games , which explicitly represent information structures. We then show that the information structure characterizes the rank of a sequential decision-making problem as per Definition 1.

C.2 (Generalized) Predictive State Representations

Predictive state representations (PSR) [31, 32] are a model of dynamical systems and sequential decision-making problems based on predicting future observations given the past, without explicitly modeling a latent state. In this section, we propose and formalize a generalization of standard PSRs.

In the standard formulation of sequential decision-making and predictive state representations, the sequence of variables is such that observations and actions always occur in an alternating manner (i.e., o h , a h , o h +1 , a h +1 , . . . ). The POST/POSG models we will propose are more general, and hence require a more flexible formalization of PSRs which allows for arbitrary order of observations and actions as well as arbitrary variable spaces at each time point. This generalization of PSRs will be used in our reinforcement learning algorithms.

The 'PSR rank' of a sequential decision-making problem coincides with the rank of its dynamics, as defined in Definition 1. Recall that the system dynamics matrix D h ∈ R | H h |×| F h | is indexed by all possible observable histories τ h and futures ω h . Denote the rank of the system dynamics at time h by r h := rank( D h ) .

Consider a sequential decision-making problem as defined in Section 2.1 (i.e., with an arbitrary order of observations and actions, and arbitrary variable spaces). At the heart of predictive state representations is the concept of 'core test sets.' A core test set at time h is a set of futures such that the set of probabilities of those futures conditioned on the past encodes all the information that the past contains about the future. This is formalized in the definition below as a set of futures such that the submatrix of the full dynamics matrix restricted to those futures is full rank.

Definition (Core test sets) . A core test set at time h is a subset of d h ≥ r h futures, Q h := { q 1 h , . . . , q d h h } ⊂ F h , such that the submatrix D h [ Q h ] ∈ R | H h |× d h is full-rank, rank( D h [ Q h ]) = rank( D h ) = r h .

A core test set implies the existence of a matrix W h ∈ R | F h |× d h such that D h = D h [ Q h ] · W ⊤ h .

Denote the τ h -th row of D h [ Q h ] by,

The vector ψ h ( τ h ) is a sufficient statistic for the history τ h in predicting the probabilities of all futures conditioned on τ h . This is sometimes called the prediction features of a history τ h .

For any integer d h ≥ r h , there exists a core test set of size d h . In particular, for any low-rank sequential decision-making problem, there exists a minimal core test set of size r h at each h . However, the minimal core test set depends on the system dynamics matrix D h , which is unknown in the learning setting. In the literature on reinforcement learning in PSRs, it is typically assumed that a core test set is known. We address the problem of constructing a PSR representation for POSTs/POSGs in Section 4.

For a core test set Q h , let Q A h = { act( q ) : q ∈ Q h } , where act( q ) denotes the action components of the test q ∈ Q h . Let Q A = max h ∣ ∣ Q A h ∣ ∣ and d = max h d h .

With core test sets defined, we are now ready to present the definition of a generalized predictive state representation. The essential element in a PSR is a set of operators M h : X h → R d h × d h -1 for each time point h ∈ [ H ] . Given the prediction features at time h -1 , ψ h -1 ( x 1 , . . . , x h -1 ) ∈ R d h -1 , the linear map M h ( x h ) computes the prediction features at time h , incorporating the additional observation x h . That is, ψ h ( x 1 , . . . , x h ) = M h ( x h ) ψ h -1 ( x 1 , . . . , x h -1 ) . The full definition is given below.

Definition (Generalized Predictive State Representations; Definition 3) . Consider a sequential decision-making problem ( X h ∈ X h ) where A , O partition [ H ] into actions and observations, respectively. Then, a predictive state representation of this sequential decision-making problem is a tuple θ = ( { Q h } 0 ≤ h ≤ H -1 , ϕ H , M , ψ 0 ) given by

{ Q h } 0 ≤ h ≤ H -1 are core test sets, including for h = 0 , where Q 0 = { q 1 0 , . . . , q d 0 0 } ⊂ F 0 are core tests before the system begins.

ψ 0 ∈ R d 0 is the vector ψ ( ∅ ) = ( P [ q 1 0 ] , . . . , P [ q d 0 0 ]) .

M = { M h } 1 ≤ h ≤ H -1 is a set of mappings M h : X h → R d h × d h -1 , from an observation/action to a matrix of size d h × d h -1 .

ϕ H : X H → R d H -1 is a mapping from the final observation to a d H -1 -dimensional vector.

This tuple satisfies,

To obtain a probability for a trajectory τ h = ( x 1 , . . . , x h ) , with h < H , note that ∑ ω h ∈ F h P [ τ h , ω h ] = | F a h | P [ τ h ] . Hence

Thus, if we recursively define ϕ h , h < H via

with ϕ H as the terminating condition, then, we can obtain P [ τ h ] for any h < H , via an inner product between ϕ h and ψ h ( τ h ) ,

Finally, if we define ψ h ( τ h ) = ψ h ( τ h ) / P [ τ h ] , then we obtain the conditional probability of the core tests given the history, ψ h ( τ h ) = ( P [ q 1 h | τ h ] , . . . , P [ q d h h | τ h ]) ∈ R d h . ψ h ( τ h ) is known as the (normalized) prediction feature of the history τ h [31].

Remark 1 (Generality and difference from standard PSRs) . In standard PSRs, observations and actions are assumed to occur in an alternating manner, and hence observable operators are defined on pairs of observations and actions (i.e., M h ( o h , a h ) ). This structure leads to a somewhat simpler description compared to the above. However, our formulation is more general, as it allows each variable to be treated independently, and allows for an arbitrary sequence of variables with arbitrary spaces. This generality will be needed when modeling problems with an explicit representation of information structure.

An important condition for the learnability of PSR models, which was used in prior work [including 33, 35], is the so-called 'well-conditioning assumption'. We state the analogous assumption for our generalized PSR model below.

Assumption ( γ -well-conditioned generalized PSR; Assumption 1) . A PSR model θ = ( { Q h } 0 ≤ h ≤ H -1 , ϕ H , M , ψ 0 ) , as defined in Definition 3, is said to be γ -well conditioned for γ > 0 if it satisfies

1. For any h ∈ [ H ] ,

where m h ( ω h ) ⊤ = ϕ H ( x H ) ⊤ M H -1 ( x H -1 ) · · · M h +1 ( x h +1 ) with ω h = ( x h +1 , . . . , x H ) ∈ F h . The maximization is over policies π such that for any fixed future observations ω o h , ∑ ω a h π ( ω o h , ω a h ) = 1 .

For any h ∈ [ H -1] ,

where π ( x h ) = 1 when h / ∈ A and ∑ x h π ( x h ) = 1 when h ∈ A .

To understand this condition, recall that m h ( ω h ) ⊤ ψ h ( τ h ) = P [ τ h , ω h ] . We may think of z in Assumption 1 as representing the error in estimating ψ h ( τ h ) , the probabilities of core tests at time h given the history τ h . The γ -well-conditioned assumption ensures that the error in estimating the overall PSR (i.e., the probability of a particular trajectory) does not blow up when the estimation error of ψ h ( τ h ) is small.

The following result states that any sequential decision-making problem of the form described in Section 2.1 admits a generalized PSR representation. The proof and explicit construction are given in Appendix G.

Proposition 1. Let ( X 1 , . . . , X H ) be any sequential decision-making problem with observation index set O , action index set A , and variable spaces { X h } h ∈ [ H ] . Let r h = rank( D h ) , where D h , h ∈ [ H ] are the system dynamics matrices. Then, there exists a PSR representation ψ 0 , ϕ H : X H → R r H -1 , M h : X h → R r h +1 × r h , h ∈ [ H -1] , satisfying Definition 3.

Proof. The proof is given in Appendix G.

D Examples of Information Structures and their Rank

The analysis in Section 3 and Theorem 1 characterizes the rank of any sequential decision-making problem as a function of its information structure. In this section, we illustrate this on several sequential decision-making problems, characterizing the information-structural complexity of their dynamics. The procedure is as follows: 1) formulate the sequential decision-making problem as a POST/POSG; 2) represent the information structure as a labeled directed acyclic graph G ; 3) remove incoming edges into the action variables to produce G † ; 4) apply Theorem 1 to find the information structural state at each point in time through a d -separation analysis.

Illustration: translating to the POST/POSG framework. We begin by illustrating how an arbitrary sequential decision-making problem can be formulated in the POST/POSG framework. Consider

s

1

s

2

a

1

s

3

a

2

s

4

a

3

s

5

a

4

s

6

a

5

o

1

o

2

o

3

o

4

o

5

o

6

a

6

s

1

s

2

a

1

s

3

a

2

s

4

a

3

s

5

a

4

s

6

a

5

o

1

o

2

o

3

o

4

o

5

o

6

a

6

a POMDP with variables ( s 1 , o 1 , a 1 , s 2 , o 2 , a 2 , ... ) . This can be formulated as a POST/POSG by a simple relabelling of variables as follows.

Here, the system variables S are the s -type and o -type variables, with system index set S = { 1 , 2 , 4 , 5 , 7 , 8 , ... } , and the action variables are the a -type variables with action index set A = { 3 , 6 , 9 , ... } . The observable system variables are the o -type variables only, with index set O = { 2 , 5 , 8 , ... } ⊂ S . This can be done for any sequential decision-making problem.

To ease notation, let us not explicitly write the indices in this section, but rather use the original notation for the variables in the problem formulation. For example, we'll write S = { s t , o t , t ∈ [ T ] } . Similarly, we use the notation I ( x ) to mean the information set corresponding to the variable x . Similarly, I † ( x ) denotes the information-structural state at the time when x occurs. For example, in a POMDP I ( s t ) = { s t -1 , a t -1 } , I ( o t ) = { s t } , and I ( a t ) = { o 1: t , a 1: t } .

Below, we will consider several examples of sequential decision-making problems, and apply the information-structural analysis of Theorem 1 to obtain a bound on the rank of the observable dynamics (which in turn implies a bound on the sample complexity, by Theorem 3).

Decentralized POMDPs and POMGs. At each time t , the system variables of a decentralized POMDP (or POMG) consist of a latent state s t , observations for each agent o 1 t , . . . , o N t , and actions of each agent a 1 t , . . . , a N t . The latent state transitions are Markovian and depend on the agents' joint action. The observations are sampled via a kernel conditional on the latent state. Each agent can use their own history of observations to choose an action. Thus, the information structure is given by,

Here, the observable variables are U = { o i 1: T , a i 1: T , i ∈ [ N ] } . By Theorem 1, we have I † ( o i t ) = { s t } , ∀ t, i , as shown in Figure 5a. Thus, the rank of a Dec-POMDP is bounded by | S | , where S is the state space. Note that in the case of models with a true latent state (e.g., POMDPs, Dec-POMDPs, and POMGs), the information-structural state coincides with the true latent state.

Limited-memory information structures. Consider a sequential decision making problem with variables o t , a t , t ∈ [ T ] and an information structure with m -length memory. That is, observations can only depend directly on at most m of the most recent observations and actions. That is, the information structure is

The observables are all observations and actions, U = { o 1: T , a 1: T } . By Theorem 1 we have that I † ( o t ) = { o t -m : t -1 , a t -m : t -1 } , as shown in Figure 5d. Hence, the rank of this sequential decisionmaking process is bounded by | O | m | A | m .

s

1

o

1

1

o

2

1

s

2

o

1

2

o

2

2

s

3

o

1

3

o

2

3

s

4

o

1

4

o

2

4

s

5

o

1

5

o

2

5

a

1

1

a

2

1

a

1

2

a

2

2

a

1

3

a

2

3

a

1

4

a

2

4

a

1

5

a

2

5

s

1

1

a

1

1

s

2

1

a

2

1

s

3

1

a

3

1

s mf

1

a mf

1

s

1

2

s

2

2

s

3

2

a

1

2

a

2

2

a

3

2

s mf

2

a mf

2

s

1

3

s

2

3

s

3

3

a

1

3

a

2

3

a

3

3

s mf

3

a mf

3

s

1

4

s

2

4

s

3

4

a

1

4

a

2

4

a

3

4

s mf

4

a mf

4

(b) 'Mean-field' information structure.

x

1

z

1

y

1

x

1

x

2

z

2

y

2

x

2

x

3

z

3

y

3

x

3

x

4

z

4

y

4

x

4

x

5

z

5

y

5

x

5

(c) Point-to-point real-time communica- tion with feedback information structure.

o

1

o

2

a

1

o

3

a

2

o

4

a

3

o

5

a

4

o

6

a

5

o

7

a

6

a

7

o

1

o

2

a

1

o

3

a

2

o

4

a

3

o

5

a

4

o

6

a

5

o

7

a

6

a

7

Symmetric / 'Mean-field' Information Structures. Consider a sequential decision-making problem with N agents. Each agent has their own local state, s i t ∈ S local . Similarly, at each time point, each agent takes an action a t i ∈ A loc . The global state s t = ( s 1 t , . . . , s N t ) ∈ S N loc =: S is composed by of all agents' local states. Similarly, the joint action space is A := A N loc . Consider a symmetric information structure where the evolution of each agent's local state depends only on a symmetric aggregation of all agents' states and actions, rather than on the local state/action of any particular agent. That is, the identity of who is in what state or takes which action does not matter-only the distribution of states and actions. This is often referred to as a 'mean-field' setting (in the limit). Here, the transition depends only on the distribution of local states and actions, defined as s mf t = dist( s t ) := 1 N δ s i t , a mf t = dist( a t ) := 1 N δ a i t , for s t ∈ S , a t ∈ A . Different agents can have different transition kernels for their local state. Hence, by introducing dist( s t ) , dist( a t ) as auxiliary unobserved variables at each time t , we obtain the following information structure,

and an application of Theorem 1 bounds the rank by

This is compared to | S loc | N · | A loc | N (e.g., if we modeled this as an MDP with the state s t ), which is much larger when the number of agents is large. The information structure and d -separation decomposition are depicted in Figure 5b.

Point-to-Point Real-Time Communication with Feedback. Consider the following model of realtime communication with feedback. Let x t be the Markov source. At time t , the encoder receives the source x t ∈ X and encodes sending a symbol z t ∈ Z . The symbol is sent through a memoryless noisy channel which outputs y t to the receiver. The decoder produces the estimate ̂ x t . The output of the noisy channel is also fed back to the encoder. The encoder and decoder have full memory of their observations and previous 'actions'. The observation variables are O = { x 1: T , y 1: T } and the 'actions' are A = { z 1: T , ̂ x 1: T } . Hence, the information structure is given by the following,

By Proposition 1, we have that,

Hence, the rank is bounded by | X || Z | . This is depicted in Figure 5c.

Fully-Connected Information Structures. Consider a sequential decision making problem with variables o t , a t , t ∈ [ T ] and a fully-connected information structure. That is, each observation directly depends on the entire history of observations and actions. Thus, the information structure is

The observables are all observations and actions, U = { o 1: T , a 1: T } . By Theorem 1 we have that I † ( o t ) = { o 1: t -1 , a 1: t -1 } , as shown in Figure 5e. Hence, the rank of this sequential decision-making process can be exponential in the time horizon.

The examples above show that the tractability of a sequential decision-making problem in terms of the complexity of its dynamics depends directly on its information structure. This gives an interpretation of why certain models, like POMDPs, are more tractable than those with arbitrary information structures. Previous work primarily considers particular problem classes with fixed and highly regular information structures. In this work we argue for the importance of explicitly modeling the information structure of a sequential decision-making problem.

Remark 2 (Necessity of generalized PSRs) . The formalization of generalized PSRs in Section 2.2 was necessary to enable the study of information structure through POSTs/POSGs. An alternative (naive) solution to construct PSR representations for models with non-alternating observations and actions is to aggregate consecutive observations and actions to force them to obey the standard formulation of PSRs. This approach results in a loss of 'resolution' in the information structure. That is, when you aggregate consecutive system variables, you also aggregate the DAG which represents the information structure, losing potentially important structure. In particular, in the worst case, such aggregation could result in an exponential increase in the rank of the dynamics. The examples given above elucidate this. Consider for example the 'mean-field' information structure. If we aggregated local states and actions into a combined global state and joint action, the PSR rank would indeed be | S loc | N | A loc | N . By comparison, by considering each local state separately without aggregation, we are able to obtain a decomposition with a much smaller PSR rank.

E UCB-Type Reinforcement Learning Algorithm for Generalized PSRs

We will adapt the model-based UCB-type algorithm of Huang et al. [35], extending it to generalized PSRs, including those representing POSTs. The algorithm involves the estimation of an upper confidence bound which captures the uncertainty in the estimated model and drives exploration so as to minimize this uncertainty. The UCB-based approach has the advantage of providing a last-iterate guarantee and requiring a weaker notion of planning oracle (a standard planning oracle instead of an optimistic planning oracle as required by similar algorithms). The technical contribution of this section is to extend the algorithm and its theoretical guarantees to generalized PSRs. The tools developed in doing so can be used to directly extend any other PSR-based algorithm to generalized PSRs.

When learning generalized PSRs, we suppose that the core test sets { Q h } 0 ≤ h ≤ H -1 are known by the algorithm. For example, if the sequential decision-making problem is a POST, Section 4 provides conditions under which m -step futures form core test sets. Let Θ be the set of γ -well-conditioned generalized PSR representations with { Q h } 0 ≤ h ≤ H -1 as core test sets. Denote by Θ ϵ an optimistic ϵ -cover of Θ (defined formally in Appendix G).

Recall that d h := | Q h | and d = max h d h . Moreover, Q A h := act ( Q h ) are the action components of the core test sets and Q A := max h ∣ ∣ Q A h ∣ ∣ is the maximal size of those action components. We define the exploration action sequences at time h to be Q exp h -1 = act ( X h × Q h ∪ Q h -1 ) . Moreover, we define u exp h -1 as the policy, defined from time h -1 onwards, in which each selection of action sequences in Q exp h -1 are chosen uniformly at random. For a model θ and reward function R , we define the value of a policy under this model and reward as V R θ ( π ) := ∑ τ H R ( τ H ) P π θ ( τ H ) .

The algorithmic description is given in Algorithm 1. At each iteration k , the learner collects a trajectory τ k,h H for each time index h ∈ [ H ] by using a particular policy that drives exploration so as to better estimate the parameters associated with the h -th time step. To collect the trajectory τ k,h H , the learner executes the policy at the previous iteration, π k -1 , until time h -1 collecting the trajectory τ k,h h -1 then executes u exp h -1 which samples action sequences from Q exp h -1 uniformly. The particular choice of the exploratory action sequences Q exp h -1 comes out of the proof (see proof of Lemma 5 in the appendix). Intuitively, act ( Q h -1 ) allows us to estimate the prediction features ψ ∗ ( τ k,h h -1 ) = [ P ( q | τ k,h h -1 )] q ∈ Q h -1 , and act ( X h × Q h ) allows us to estimate M ∗ h ( x h ) ψ ∗ ( τ k,h h -1 ) .

The collected trajectories are added to the dataset, together with the policies used to collect them. The next step is model estimation via (constrained) maximum likelihood estimation. The algorithm estimates a model ̂ θ k by selecting any model in a constrained set B k defined as

The introduction of Θ k min ensures that P π k -1 θ ∗ ( τ k,h h -1 ) is not too small so that the estimates of the prediction features ψ ∗ ( τ k,h h -1 ) = [ P ( q | τ k,h h -1 )] q ∈ Q h -1 are accurate. This design differs from other MLE-based estimators [e.g., 33, 36, 38] due to the estimation of parameters capturing conditional probabilities.

Next, the algorithm chooses a policy which drives the algorithm to trajectories τ h whose prediction features have so far been unexplored. To do this, Algorithm 1 constructs an upper confidence bound on the total variation distance between the estimated model and the true model. This is done via a bonus function ̂ b k ( τ H ) ,

where λ and α are pre-specified parameters to the algorithm. Thus, the bonus function captures the degree of uncertainty in the estimated prediction features ̂ ψ ( τ h ) . In particular, the bonus ̂ b ( τ H ) will be large for trajectories whose prediction feature ̂ ψ ( τ h ) lie far away from the empirical distribution of prediction features sampled in the dataset D k h . This is captured by computing the norm with respect to the covariance ̂ U k h .

The algorithm then chooses an exploration policy for the next iteration which maximizes this upper confidence bound, hence collecting trajectories that have high uncertainty in their prediction features. When the estimated model is sufficiently accurate on all trajectories, the algorithm terminates and returns the optimal policy with respect to the reward function R under the estimated model.

We extend Huang et al.'s theoretical guarantees to show that Algorithm 1 enjoys polynomial sample complexity for generalized PSRs (Definition 3).

Theorem 4. Suppose Assumption 1 holds. Suppose the parameters p min , λ, α, β are chosen appropriately. In particular, let

Then, with probability at least 1 -δ , Algorithm 1 returns a model θ ϵ and a policy π that satisfy

In addition, the algorithm terminates with a sample complexity of,

Proof. The proof is given in Appendix J.

This result shows that the sample complexity of learning a generalized PSR depends on the problem size through a few key quantities. In particular, the sample complexity scales polynomially in the underlying rank r , the dimension of the PSR parameterization d , the size of the action component of the core tests Q A , the time horizon H , the conditioning number γ -1 , the size of the action spaces max s ∈A | X s | , the log covering number log | Θ ϵ | , and the desired suboptimality error ϵ . Note that ˜ O omits logarithmic dependence.

To apply this algorithm to a POST, we can use the generalized PSR parameterization constructed in Section 4. By Theorem 1 the PSR rank is bounded by r ≤ max h | I † h | . If this POST is α -robustly I † -weakly revealing, then by Theorem 2 it admits a γ -well-conditioned generalized PSR parameterization with γ = α/ max h | I † h | 1 / 2 and the m -step futures as core test sets. Moreover, we have d = max h d h = max h | Q m h | . The following corollary states that Algorithm 1 can learn a partially-observable sequential team with a sample complexity which is polynomial in the size of the information-structural state space max h | I † h | .

Corollary 1. Suppose a partially-observable sequential team is m -step α -robustly I † -weakly revealing as per Definition 6. Applying Algorithm 1 to this PSR representation, with parameters p min , λ, α, β chosen as in Theorem 4, returns a ε -optimal policy with a sample complexity of,

We can interpret this result as saying that the information structure of a sequential decision-making problem, through the quantity max h | I † h | , is fundamentally a measure of the complexity of the dynamics which need to be modeled. As a result, learning is tractable when max h | I † h | is of modest size, and intractable otherwise. Recall that max h | I † h | is small when there exists 'state-like' variables, whether they are observable or unobservable. In this sense, max h | I † h | is a fundamental quantity which generalizes the notion of a 'state'. For example, in the case of an m -step α -weakly revealing POMDP, our algorithm has a sample complexity of poly( S, ( OA ) m , H, α -1 ) · ϵ -2 , where S is the size of the state space, O is the size of the observation space, and A is the size of the action space. This is similar to the sample complexity of [36, 37], which designed an algorithm tailored specifically for weaklyrevealing POMDPs. Our algorithms, together with the POST/POSG models, enable sample-efficient reinforcement learning for a much broader class of models all within a unified framework.

In this section, we extended the algorithm in Huang et al. [35] to generalized PSRs, enabling sampleefficient learning of POSTs. We emphasize that other PSR-based algorithms can be extended in a similar manner. In the next section, we tackle the problem of learning in the game setting where different agents have different objectives.

F Extension to the game setting

F.1 Partially Observable Sequential Games

In a POST, all agents share the same objective. In the game setting, different agents may have different objectives which compete with each other in interesting ways. Information structures play a crucial role in the study of games. The information available to one agent when making its decisions, compared to the information available to competing agents, determines how well it can achieve its objective. In particular, the information structure of a problem determines the set of equilibria it admits. There has been a plethora of work in the game theory community studying such problems.

Analogously to partially-observable sequential teams, we define partially-observable sequential games (POSGs). The dynamics of a POSG are identical to a POST, with the same formalization of variable structure, variable spaces, information structure, system kernels, and decision kernels. In contrast to a POST, agents in a POSG may have different objectives. In a POSG, there exists N agents, with agent i ∈ [ N ] deciding the actions at times t ∈ A i , where A i ⊂ A . Each agent has its own objective defined by a reward function R i . This is defined formally below.

Definition 7 (Partially-Observable Sequential Game Model) . A partially-observable sequential game (POSG) is a controlled stochastic process consisting of the following components: variable structure, variable spaces, information structure, system kernels, decision kernels, and observability. These are defined in an identical manner to Definition 4. Additionally, POSGs define a reward structure as follows. Let N be the number of agents. Each agent may act several times. Denote by A i ⊂ A the index of action variables associated to agent i ∈ [ N ] . Each agent has a reward function R i : ∏ t ∈U X t → [0 , 1] which they aim to maximize.

Denote by π i = ( π t : t ∈ A i ) the collection of decision kernels belonging to agent i , one for each action they take. Denote by π = ( π 1 , . . . , π N ) the collection of all agents' policies. Fixing π induces

a probability distribution over X 1 ×··· X T in the same way as in the team setting,

The value of a policy π for agent i ∈ [ N ] is defined as the expected value of their reward R i under P π ,

̸

where π -i = ( π j : j = i ) .

The nature of randomization in agents' policies is crucial to the analysis of solution concepts in the game setting. To model randomized policies, which are potentially correlated, we introduce a random seed ω ∈ Ω which is sampled at the beginning of an episode. Then, the policy at time t ∈ A can be modeled as a deterministic function mapping the seed ω and information variable i t ∈ I t to an action X t . That is, π t : Ω × I t → X t . To model independently randomized policies with each agent having private randomness, we consider the special case where the seed has the product structure ω = ( ω 1 , . . . , ω N ) ∈ Ω 1 × · · · × Ω N , and ω i is the seed belonging to agent i ∈ [ N ] . Then, for t ∈ A i , π t : Ω i × I t → X t . For each agent i ∈ [ N ] , define the three policy spaces,

Deterministic policies, Γ i det = { π i : π i = ( π t : I t → X t , t ∈ A i )} ,

Independently-randomized policies, Γ i ind = { π i : π i = ( π t : Ω i × I t → X t , t ∈ A i )} ,

Correlated randomized policies, Γ i cor = { π i : π i = ( π t : Ω × I t → X t , t ∈ A i )} .

Define the joint deterministic policy space, as Γ det = Γ 1 det × · · · × Γ N det , and similarly for the independently-randomized policy space Γ ind , and the correlated randomized policy space Γ cor .

When studying games, a common question is to find an equilibrium within a particular policy space. At a high-level, an equilibrium is a joint policy where no agent can do better by deviating from their policy when the other agents keep their policies fixed. We will consider several notions of equilibrium. We begin by defining the notion of a best-response . Suppose that agent i 's policy space is Γ i (e.g., Γ i det , Γ i ind , or Γ i cor ). Then, we say that agent i 's policy π i is a best response to π -i if there is no policy in Γ i which achieves a higher value. This is formalized in the definition below.

Definition 8 (Best response) . For a joint policy π , π i is said to be a best-response to π -i in the policy space Γ i (e.g., Γ i det , Γ i ind , or Γ i cor ), if V i ( π i , π -i ) = max ˜ π i ∈ Γ i V i (˜ π i , π -i ) =: V i, † ( π -i ) .

This leads to the definition of two notions of equilibria. A Nash Equilibrium (NE) is a joint policy where all agents are best-responding in the space of independently-randomized policies. A Coarse Correlated Equilibrium (CCE) is a joint policy where all agents are best-responding in the space of correlated randomized policies. The difference between NE and CCE is that the randomness in the joint policy must be independent in an NE but can be correlated in a CCE. Since Γ ind ⊂ Γ cor , coarse correlated equilibria are a generalization of Nash equilibria. We define them formally below.

Definition 9 (Nash Equilibrium) . A joint policy π ∈ Γ ind is said to be a Nash equilibrium if for all agents i ∈ [ N ] , V i ( π ) = max ˜ π i ∈ Γ i ind V i (˜ π i , π -i ) =: V i, † ( π -i ) . A joint policy π ∈ Γ ind is said to an ε -approximate Nash equilibrium if V i ( π ) ≥ V i, † ( π -i ) -ε for all i ∈ [ N ] .

Definition 10 (Coarse Correlated Equilibrium) . A joint policy π ∈ Γ cor is said to be a coarse correlated equilibrium if for all agents i ∈ [ N ] , V i ( π ) = max ˜ π i ∈ Γ i cor V i (˜ π i , π -i ) =: V i, † ( π -i ) . A joint policy π ∈ Γ cor is said to an ε -approximate Nash equilibrium if V i ( π ) ≥ V i, † ( π -i ) -ε for all i ∈ [ N ] .

Since we consider finite-space sequential games, an equilibrium is guaranteed to exist [93].

Remark 3 (Notion of equilibrium can be represented through information structure) . The policy classes defined above (i.e., deterministic, independently-randomized, correlated randomized) can be directly modeled by the information structure. For example, to represent correlated randomized policies, the random seed ω ∈ Ω can be modeled as an observable variable at time t = 0 which is in all agents' information sets. Similarly, independently randomized policies can be represented through a different random seed for each agent at time t = 0 , and including the appropriate random seed in each action's information set. Hence, the information structure itself can decide which equilibrium notion we are interested in. Moreover, this allows us to consider additional notions of equilibrium

where, for example, only subsets of agents can be correlated with each other (e.g., this may be useful in modeling multi-team problems). Note that adding random seeds in order to model randomized policies does not affect the information-structural state I † h since the seeds don't appear in G † . For concreteness, we focus on NE and CCE in our presentation.

F.2 Characterizing the Sample Complexity of Learning an Equilibrium via a Self-Play Algorithm as a function of the Information Structure

We now introduce a sample-efficient reinforcement learning algorithm for learning well-conditioned generalized predictive state representations in the game setting with each agent having their own objective. In particular, since partially-observable sequential games with a I † -weakly revealing information structure admit a well-conditioned generalized PSR representation, they can also be learned sample-efficiently by this algorithm.

The algorithm we propose is a self-play algorithm for learning an equilibrium of the dynamic game problem. That is, the algorithm specifies the policies of all agents during the learning phase, collecting the trajectory of observables at each episode to improve its estimate of the system dynamics. This can be thought of as a centralized agent playing against itself. We will propose an algorithm which can find a Nash equilibrium or coarse correlated equilibrium in a sample-efficient manner. We begin with some preliminaries.

Game setting. Recall that a sequential decision-making problem falls within the game setting if each agent has their own objective. Following Section 2.1, we consider a sequential decisionmaking problem ( X 1 , . . . , X H ) where O denotes the index set of (observable) system variables and A denotes the set of action variables. We suppose the game involves N agents, and denote the action index set of each agent by A i ⊂ A , where {A i } i ∈ [ N ] partitions A . Each agent has their own reward function R i ( X 1 , . . . , X H ) . Note that POSGs as defined in Definition 7 are structured models which fall within this framework.

Equilibria and policy classes. Recall that in the game setting, the type of randomization in each agent's policy affects the set of equilibria in the game. In Appendix F.1, we formalized this randomization by introducing a random seed ω ∈ Ω and allowing each agent's policy to be a function of their information set and this seed. If the seed has a product structure ω = ( ω 1 , . . . , ω N ) with each agent observing their own seed, this results in independently-randomized policies, denoted by Γ i ind . If all agents use the same seed, this results in correlated randomized policies, which we denote by Γ i cor . An equilibrium among independently randomized policies is called a Nash equilibrium and an equilibrium among correlated randomized policies is called a coarse correlated equilibrium.

Estimating probabilities in the planner. The probability of any trajectory under a joint policy π is given by P π ( τ H ) = ∑ ω P [ τ H ] π ( τ H | ω ) P [ ω ] , where P [ τ H ] = P [ τ o H | τ a H ] as before, and π ( τ H | ω ) = ∏ h ∈A 1 { x h = π h ( τ h -1 , ω ) } . Recall that the probabilities P [ τ H ] are estimated by the generalized PSR model ̂ θ . We assume that the planner has knowledge of the randomization, P [ ω ] . Hence, the planner in the self-play algorithm is able to compute the probability of any trajectory for each choice of policy.

Algorithm. The algorithmic description is presented in Algorithm 2. In the first stage of the algorithm, the centralized learning agent has a unified goal: to explore the environment. This is done by executing policies which maximize the bonus function ̂ b k ( τ H ) by visiting trajectories with imprecise estimates of their probability, as measured by the upper confidence bound on the total variation distance. This part is identical to Algorithm 1. Once the algorithm is sufficiently confident about the estimated probabilities of all trajectories, it computes the equilibrium using the estimated model directly. That is, ComputeEquilibrium computes either NE or CCE. The only difference in the exploration stage of the algorithm compared to Algorithm 1 is that the termination condition involves ε/ 4 rather than ε/ 2 in order to guarantee an ε -approximate equilibrium under the added complications of the game setting.

Theorem 5. Suppose Assumption 1 holds. Suppose the parameters p min , λ, α, β are chosen as in Theorem 4. Then, with probability at least 1 -δ , Algorithm 2 returns a model θ ϵ and a policy π which is an ε -approximate equilibrium (either NE or CCE). That is,

In addition, the algorithm terminates with a sample complexity of,

Proof.

To apply this algorithm to a partially-observable sequential game, we can use the generalized PSR parameterization constructed in Section 4.

Corollary 2. Suppose a partially-observable sequential game is m -step α -robustly I † -weakly revealing as per Definition 6. Applying Algorithm 2 to this PSR representation, with parameters p min , λ, α, β chosen as in Theorem 5, returns a ε -approximate equilibrium π with a sample complexity of,

G Existence of Generalized PSR representations and their covering number

In this section we show that any rankr sequential decision-making problem (as per Section 2.1) can be represented via a rankr generalized PSR (Definition 3). Next, we bound the covering number of the class of rank r PSRs, which will be important for our MLE analysis. Similar results have been established in previous work for sequential decision-making problems with alternating observations and actions [e.g., 33]. Recall that our formulation of the generic sequential decision-making problem and generalized PSRs is more general than the standard formulation since it allows for an arbitrary sequence of variables. Here, we follow a similar procedure to prove a slightly generalized result.

Proposition 2 (Existence of Generalized PSR representation) . Consider a sequential decision-making problem with rank( D h ) = r h , h ∈ 0 : H -1 . There exists a generalized PSR representation (i.e, observable operator model) b 0 , { B h ( x h ) } h ∈ [ H ] ,x h ∈ X h , { v h } h ∈ 0: H such that,

B h ( x h ) ∈ R r h × r h -1 and ∥ B h ( x h ) ∥ 2 ≤ 1 for any x h .

| b 0 | ≤ √ | H a H | .

∥ v h ∥ 2 ≤ √ | F o h | / | F a h | .

For any h , 1 | X h | 1 { h ∈A} v ⊤ h ∑ x h ∈ X h B h ( x h ) = v ⊤ h -1 .

For any τ h ∈ H h , P [ τ h ] = v ⊤ h B h ( x h ) · · · B 1 ( x 1 ) b 0 .

Proof. We construct the representation via the singular value decomposition of the matrix D ⊤ h . Let U h ∈ R | F h |× r h , Σ h ∈ R r h × r h , V ⊤ h ∈ R r h ×| H h | be the SVD such that D ⊤ h = U h Σ h V ⊤ h . Define b 0 , B h , v ⊤ h as follows,

Here, [ U h -1 ] ( x h , Ω h ) , : denotes an | F h | by r h -1 submatrix of U h -1 consisting of the rows ( x h , ω h ) , ω h ∈ F h (i.e., the set of futures where the variable at time h is x h ). Note that | F a H | = 1 by convention, a product over an empty set. We verify each property in turn.

First, ∥ B h ( x h ) ∥ 2 = ∥ U ⊤ h [ U h -1 ] ( x h , F h ) , : ∥ 2 ≤ 1 since U h , U h -1 are unitary matrices. Second,

where the inequality is since P [ τ H ] ∈ [0 , 1] . For property 3, we have

where the inequality is since U h is unitary, and the final equality is since | F h | = | F o h | | F a h | . Next, to prove properties 4 and 5, we first show the following claim.

Claim. For any history τ h = ( x 1 , . . . , x h ) ∈ H h , h ∈ 0 : H , we have B h ( x h ) · · · B 1 ( x 1 ) b 0 = U ⊤ h [ D ⊤ h ] : ,τ h .

Proof of claim. We prove the claim by induction. In the base case, h = 0 , D ⊤ 0 is a vector in R F 0 (note that F 0 = H H ). Hence, U 0 is simply the normalized vector U 0 = D ⊤ 0 / ∥ D ⊤ 0 ∥ 2 , and hence U ⊤ 0 D ⊤ 0 = D 0 D ⊤ 0 / ∥ D 0 ∥ 2 = ∥ D 0 ∥ 2 = b 0 . Proceeding by induction, suppose the claim holds for h -1 . Then, we have,

Using this fact, we can now show property 5 as follows,

Finally, it remains to show property 4. Consider the linear equation x ⊤ U ⊤ h D ⊤ h = | F a h | -1 1 ⊤ D ⊤ h . Note that U ⊤ h D ⊤ h ∈ R r h ×| H h | is rank r h . Thus, this equation has a unique solution. Our strategy is to show that v ⊤ h and v ⊤ h +1 ∑ x h +1 B h +1 ( x h ) are both solutions to this linear equation, and hence v ⊤ h = v ⊤ h +1 ∑ x h +1 B h +1 ( x h ) . That v ⊤ h is a solution is clear by definition of v h , v ⊤ h U ⊤ h D ⊤ h = | F a h | -1 1 ⊤ U h U ⊤ h D ⊤ h = | F a h | -1 1 ⊤ D ⊤ h . First, recall by the calculation above that | F a h | -1 1 ⊤ D ⊤ h is a vector in R H h where the τ h -th entry is P [ τ o h | τ a h ] . We will calculate the τ h -th entry of the vector x ⊤ U ⊤ h D h when x ⊤ = v ⊤ h +1 ∑ x h +1 B h +1 ( x h +1 ) ,

where the final inequality is since | F a h | = ∏ s ∈ h +1: H ( | X s | 1 { s ∈A} ) .

Corollary 3. Consider a sequential decision-making problem with rank( D h ) ≤ r . Then, there exists a generalized PSR b 0 ∈ R r , { B h ( x h ) } h ∈ [ H ] ,x h ∈ X h ⊂ R r × r , v H ∈ R r such that,

For any τ H ∈ H H , P [ τ H ] = v ⊤ H B H ( x H ) · · · B 1 ( x 1 ) b 0 .

Proof. In Proposition 2 we constructed such a representation with dimensions in terms of r h instead of r . Since r h ≤ r , we can pad this representation with dummy columns and/or rows filled with zeros to obtain a representation with dimensions in terms of r .

An important part of maximum likelihood analysis is the notion of a 'bracketing number' which controls the complexity of the model class Θ [e.g., 94]. In our analysis, the model class is the set of generalized PSRs of a given rank. As shown in the results above, rankr generalized PSRs can represent any rankr sequential decision-making problem, with operators whose norm is bounded. In the next result, we will consider a closely related notion to the bracketing number which crucially incorporates optimism. Θ ε is said to be an 'optimistic ε -cover' for Θ if for each θ ∈ Θ , there exists ̂ θ ∈ Θ ε with an associated probability measure P ε ̂ θ such that,

The first condition ensures optimism and the second condition ensures that Θ ε ε -covers Θ , in the sense that the probability of any trajectory is approximated within an error ε . Recall that the parameter β in Algorithms 1 and 2, which appears in the sample complexity results in Theorems 4 and 5, is defined in terms of | Θ ε | . The next proposition bounds the size of ∣ ∣ Θ ε ∣ ∣ .

Proposition 3 (Optimistic cover of sequential decision making problems) . Let M be the set of all rankr sequential decision-making problems with a horizon of length H , observation index set O ⊂ [ H ] , action index set A ⊂ [ H ] , and variable spaces X 1 , . . . , X H . Then, there exists an optimistic ε -cover Θ ε of Θ with cardinality bounded by,

Proof. Define the set of generalized PSR representations constructed in Corollary 3,

Let C δ be a δ -cover of the above set with respect to the ℓ ∞ -norm. For ̂ θ = ( b 0 , { B h ( x h ) } , v H ) ∈ C δ , define the ε -optimistic probabilities as,

We will show that for an appropriate choice of δ , C δ is an optimistic ε -cover. In particular, for each θ ∈ Θ , there exists ̂ θ ∈ C δ such that,

To choose the value of δ for which the above holds, observe that

where the second inequality uses ∥ ̂ v H ∥ 2 = ∥ v H ∥ 2 = ∥ B h ( x h ) ∥ 2 = 1 , ∥ ̂ B h ( x h ) -B h ( x h ) ∥ 2 ≤ r ∥ ̂ B h ( x h ) -B h ( x h ) ∥ max ≤ rδ , ∥ b 0 ∥ 2 ≤ √ | H a H | , and ∥ ̂ b 0 -b 0 ∥ 2 ≤ √ r ∥ ̂ b 0 -b 0 ∥ ∞ ≤ √ rδ . Hence, choosing δ := ε · max h | X h | -cH for c an absolute constant large enough achieves a ε -optimistic covering of Θ . Hence, we let Θ ε = C δ , with δ = ε · max h · | X h | -cH . It remains to bound the size of | Θ ε | .

Recall that ∥·∥ ∞ ≤ ∥·∥ 2 and that an interval [ -x, x ] in R admits a δ -cover of size bounded by 2 x/δ . Now, observe that max ij | [ B h ( x h )] ij | ≤ ∥ B h ( x h ) ∥ 2 ≤ 1 . Hence, for a fixed h , { B h ( x h ) } x h admits a cover of size bounded by (2 /δ ) r 2 | X h | . Considering all h , the cover is bounded by (2 /δ ) r 2 ∑ h | X h | ≤ (2 /δ ) r 2 max h | X h | H . For, b 0 , we have ∥ b 0 ∥ ∞ ≤ ∥ b 0 ∥ 2 ≤ √ | H a H | , hence the covering number is bounded by (2 √ | H a H | /δ ) r . Finally for v H , we have ∥ v H ∥ ∞ ≤ ∥ v H ∥ 2 ≤ 1 , hence the covering number is bounded by (2 /δ ) r . Thus, we have,

Recalling that δ = ε max s | X s | -cH , we obtain that,

H Proofs of Section 3.2

Theorem (Restatement of Theorem 1) . The rank of the observable system dynamics of a POST or POSG is bounded by

Proof. We have

where step (a) is simply the law of total probability, step (b) is that { x k , k ∈ I † h } is conditionally independent of do( ω a h ) (future actions) given ( τ o h ; do( τ a h )) (the past), and step (c) is that ω o h is conditionally independent of ( τ o h ; do( τ a h )) given { x k , k ∈ I † h } . This is due to a result by Verma and Pearl [95] which states: for three sets of variables A,B,C in a directed graphical model, if A and B are d -separated by C , then A ⊥ B | C . Recall that I † h is defined as the minimal set which d -separates ( X t (1) , . . . , X t ( h ) ) from ( X t ( h +1) , . . . , X t ( H ) ) .

As a technical remark, note that i † h = ( x k , k ∈ I † h ) may include actions and hence,

since the action components of i † h are contained in the history τ h .

Now define two matrices

We have that D h = D h, 1 D h, 2 , where both D h, 1 and D h, 2 have rank upper bounded by | I † h | = ∏ s ∈I † h | X s | . Hence, rank( D h ) ≤ | I † h | , and the result follows.

I Proofs of Section 4

Lemma (Restatement of Lemma 1) . Suppose that the POST/POSG is m -step I † -weakly revealing. Then, Q m h is a core test set for all h ∈ [ H ] . Furthermore, we have

Proof. Let τ h ∈ H h , ω h ∈ F h be any history and future, respectively. By Theorem 1, recall that we have

Recall that i † h may overlap with τ h . In particular, the action component of i † h is contained in τ h . Thus, P [ i † h | τ h ] = P [ { x k , k ∈ I † h \ U 1: h } | τ h ] · 1 { ( x k , k ∈ I † h ∩ U 1: h ) matches τ h } . Note that I † h \ U 1: h ⊂ S does not contain any actions. Hence, the summation over I † h is equivalent to summing over its unobservable components with the restriction that its observable components match τ h .

Define the mappings ˜ m h : F h → R | I † h | and p h : H h → R | I † h | by

Then, we have that the conditional probability of the future ω h given the past τ h is given by the inner product of the above mappings, P [ ω h | τ h ] = ⟨ ˜ m h ( ω h ) , p h ( τ h ) ⟩ . Recall that the vector of (conditional) core test set probabilities for the history τ h is given by ψ h ( τ h ) = [ P [ q o | τ o h ; do( τ a h ) , do( q a )]] q ∈ Q m h ∈ R | Q m h | . By the definition of G h and Equation (22), we have G h p h ( τ h ) = ψ h ( τ h ) , since, for q ∈ Q m h ,

Since by assumption rank( G h ) = ∣ ∣ ∣ I † h ∣ ∣ ∣ , its pseudo-inverse G † h is a left inverse of G h (i.e., G † h G h = I ). Hence, multiplying on the left by G † h , we obtain p h ( τ h ) = G † h ψ h ( τ h ) . Hence,

That P [ τ h , ω h ] = ⟨ m h ( ω h ) , ψ h ( τ h ) ⟩ follows directly by noting the definition of ψ h ( τ h ) := ψ h ( τ h ) / P [ τ h ] .

Hence, we have shown that for the test set Q m h , the probability of each future ω h given a history τ h is a linear combination of the probabilities of each test in the core test set with weights m h ( ω h ) := ( G † h ) ⊤ ˜ m h ( ω h ) ∈ R | Q m h | depending only on the future and not the history.

Theorem (Restatement of Theorem 2) . Suppose a POST/POSG is α -robustly m -step I † -weakly revealing. Then, the corresponding generalized PSR as constructed in Section 4 is γ -well-conditioned with γ = α/ max h | I † h | 1 / 2 .

We will first show that ( { Q m h } h , ϕ H , { M h } h , ψ 0 ) indeed forms a PSR through a series of simple calculations.

A direct corollary of Lemma 1 is the following.

Lemma 2. For any h ∈ [ H ] , τ h -1 ∈ H h -1 , x t ( h ) ∈ X t ( h ) , ω h ∈ F h , we have

Hence, given a history τ h -1 = ( x t (1) , . . . , x t ( h -1) ) , having observed another variable x t ( h ) , we can update our predictions of the future and obtain the probability of trajectories of the form ( τ h -1 , x t ( h ) , ω h ) for any future ω h ∈ F h . Note that x t ( h ) may be either an observation or an action. Hence, we can update our prediction of the future after deciding an action, and before receiving the next observation. This is in contrast to the standard PSR formulation where predictions of the future can only be updated with a pair of observation and action. Our formulation provides additional flexibility, which is crucial for the general information structures modeled by POSTs and POSGs.

This means that, after observing x t ( h ) , we can use the m h : F h → R d h mapping constructed in Lemma 1 to update the probability of any candidate future ω h . We are particularly interested in updating the probabilities of the futures corresponding to the core test set at the next time point, since this provides a sufficient statistic of the past. Thus, we define the matrix mapping M h : X t ( h ) → R d h × d h -1 by,

That is, M h ( x t ( h ) ) is the matrix whose rows are indexed by the core tests at the h -th observable step, where the q ∈ Q h row is the weights given by the m h -1 mapping for the future consisting of x t ( h ) followed by q . This mapping enables us to update the probabilities of the core test sets.

Lemma 3. For any h ∈ [ H -1] , τ h ∈ H h , x t ( h +1) ∈ X t ( h +1) , we have

Hence, for a history τ h = ( x t (1) , . . . , x t ( h ) ) ∈ H h , we have

where ψ 0 = ψ 0 ( ∅ ) .

Finally, observe that Q m H -1 = X t ( H ) . Hence,

Thus, letting ϕ H : X t ( H ) → R | X t ( H ) | be ϕ H ( x t ( H ) ) = e x t ( H ) (the canonical basis vector), yields

Hence, Equation (27) together with Equation (26) imply that ( M , ϕ H , ψ 0 ) is a valid generalized PSR representation for the POST/POSG (as per Definition 3).

What remains is to show that if the POST/POSG is α -robustly I † -weakly revealing, then the PSR constructed above is well-conditioned.

Theorem (Restatement of Theorem 2) . Suppose a POST/POSG is α -robustly m -step I † -weakly revealing. Then, the corresponding generalized PSR as constructed in Section 4 is γ -well-conditioned with γ = α/ max h | I † h | 1 / 2 .

Proof. We first show condition (1) in Assumption 1. Suppose h > H -m and hence the core tests are the full futures, which have length smaller than m . Then for any x ∈ R d h , d h = ∏ H s = h | X s | , we have

where x [ ω h ] indexes the component of the vector x corresponding to the future ω h .

Now suppose h ≤ H -m (and hence the core tests consist of m -step futures). Then, we have,

Now observe that for any policy π and any i † ∈ I † h , we have

where we used the definition of m h ( ω h ) := ˜ m h ( ω h ) ⊤ G † h , and [ ˜ m h ( ω h )] i † := P [ ω h | i † ] . Recall that π ( ω h ) is such that for any fixed sequence of observations ω o h , ∑ ω a h π ( ω o h , ω a h ) = 1 .

Putting this observation together with the preceding inequality yields

where the final inequality is from the relation between the one-norm and two-norm ∥ ∥ ∥ G † h ∥ ∥ ∥ 1 ≤ √∣ ∣ ∣ I † h ∣ ∣ ∣ ∥ ∥ ∥ G † h ∥ ∥ ∥ 2 , and ∥ ∥ ∥ G † h ∥ ∥ ∥ 2 ≤ 1 α , by the assumption on its singular values.

Now we show condition (2) in Assumption 1. For ease of notation, we denote x t ( h ) by x h . When h > H -m , note that [ M h ( x h )] q h +1 ,q h = 1 { q h = ( x h , q h +1 ) } , for all q h ∈ Q h , q h +1 ∈ Q h +1 . Hence,we have

Now, when h ≤ H -m , by a similar line of reasoning to the proof for condition (1), we have,

where step (a) uses the definition of M h and step (b) uses the definition of m h ( ω h ) ⊤ := ˜ m h ( ω h ) ⊤ G † h and [ ˜ m h ( ω h )] i † := P [ ω h | i † ] . Now note that,

where the second line is since for any fixed action sequence, the sum over the probabilities of all observation sequences is 1.

Thus, putting this together, we obtain the following,

where the last line again follows by the assumption on the singular values of G h .

J Proof of Theorem 4: UCB Algorithm for Generalized PSRs (Team Setting)

In this section, we prove Theorem 4 which states that Algorithm 1 returns a near-optimal policy in a polynomial number of iterations. The proof is adapted from [35] and generalized to our setting with generalized PSRs (Definition 3). The proof is organized into several subsections. In Appendix J.1, we show that the total variation distance between trajectories under the true model and the estimated model can be bounded in terms of the estimation error of the observable operators { M h } h . In Appendix J.2 we state some general results on maximum likelihood estimation which show that the MLE model has small error on the collected dataset. In Appendix J.3 we prove that the bonus term is an upper confidence bound for the total variation distance. In Appendix J.4 we show that the estimation error is sublinear in the number of iterations (i.e., O ( √ K )) . Finally, in Appendix J.5 we put this all together to prove the theorem.

J.1 Properties of Generalized PSRs

Recall that a PSR model θ = ( M , ψ 0 , ϕ H ) consists of operators M = { M h } H -1 h =1 , M h : X h → R d h × d h -1 , ϕ H : X H → R d H -1 (assumed to be the identity mapping), and ψ 0 (assumed to be known

for the purposes of presentation). Recall that, for any trajectory τ h -1 = ( x 1 , . . . , x h -1 ) , under model θ , we have

Here, the notation P θ ( x h | τ h -1 ) means the probability of x h conditioned on the history τ h -1 , with all actions executed. In particular, if x h is an action, then P θ ( x h | τ h -1 ) = 1 and M h ( x h ) ψ h -1 ( τ h -1 ) = ψ h ( τ h ) .

The following proposition shows that the total variation distance between the distribution of trajectories of two PSR models can be bounded in terms of the difference in their observable operators.

Proposition 4. For any policy π and θ, ̂ θ ∈ Θ , we have,

Proof. The probability of any trajectory τ H = ( x 1 , . . . , x H ) can be written in terms of products of the observable operators M h ( x h ) of a PSR model (Equation (1)). Hence, we have,

where the second line follows by the triangle inequality after noting that for any trajectory τ H = x 1: H ∈ H H , the following holds for any h = 1 , . . . , H ,

By the same argument, we obtain the second inequality,

In this result, recall that we assume ψ 0 is known to the agent, to simplify the presentation. If ψ 0 was not known, there would be another term due to the estimation as ̂ ψ 0 [see 33, Lemma C.3]. Note that the sample complexity of estimating ψ 0 is small compared to learning the other parameters.

J.2 General Results on MLE

In this section, we state some general results on maximum likelihood estimation which ultimately guarantee that the estimated model produced by the procedure in Algorithm 1 has a small estimation error. The results are stated without proof. The proofs are given in [35] and use standard techniques on MLE analysis [94]. This ultimately leads us to a lemma which states that the estimation error of the MLE model is small on the collected data.

The first proposition states that the log-likelihood of the true model θ ∗ is large compared to any other model.

Proposition 5 (Proposition 4 of [35]) . Fix ε < 1 KH . With probability at least 1 -δ , for any θ ∈ Θ ε and any k ∈ [ K ] , the following holds:

The second proposition provides an upper bound on the total variation distance between the distributions of futures given histories on the empirical history of trajectories. This result ensures that the model estimated by Algorithm 1 is accurate on the sampled trajectories.

Proposition 6 (Proposition 5 in [35]) . Fix p min and ε ≤ p min KH . Let

Θ k min = { θ : ∀ h, ( τ h , π ) ∈ D k h , P π θ ( τ h ) ≥ p min } . Then, with probability at least 1 -δ , for any k ∈ [ K ] , θ ∈ Θ k min , we have,

The next proposition is standard in the analysis of maximum likelihood estimation. D H denotes the Hellinger distance.

Proposition 7 (Proposition 6 of [35]) . Let ε < 1 K 2 H 2 . Then, with probability at least 1 -δ , the following holds for all θ ∈ Θ and k ∈ [ K ] ,

The final proposition of this section states that when p min is chosen as in Theorem 4, the true model θ ∗ lies in the constraint Θ k min with high probability.

Proposition 8. Fix p min ≤ δ KH ∏ H h =1 | X h | . Then, with probability at least 1 -δ , we have θ ∗ ∈ Θ k min ∀ k .

Proof. For each k ∈ [ K ] , we have θ ∗ ∈ Θ k min if P π k θ ∗ ( τ k h ) ≥ p min for all h ∈ [ H ] , ( τ k h , π k ) ∈ D k h . Consider the probability of θ ∗ violating this constraint for some trajectory in the dataset. For each k, h, ( τ k h , π k ) , we have

In the above, the first line is by the law of total probability, where the expectation is over the policy π k used while collecting the ( h, k ) -th trajectory, and the inner probability is over trajectories τ k h . The second line calculates the probability of the event { P π k θ ∗ ( τ k h ) < p min } . Taking a union bound over k ∈ [ K ] , h ∈ [ H ] , and ( τ h , π ) ∈ D h implies that P [ θ ∗ ∈ Θ k min ] ≥ 1 -δ .

In what follows, let E ω , E π , E min be the events in Propositions 6 to 8, respectively. Let E = E ω ∩ E π ∩E min be the intersection of all events. Propositions 6 to 8 guarantee the event E occurs with high probability, P [ E ] ≥ 1 -3 δ , by a union bound.

The following result states that the estimated model is accurate on the past exploration policies and dataset of collected trajectories. This holds for both the conditional probabilities of futures given past trajectories in the dataset as well as over full trajectories. The result follows from the MLE analysis in Propositions 6 to 8.

Lemma 4. Let β = 31log K | Θ ε | δ , and suppose ε ≤ δ K 2 H 2 ∏ h | X h | , where Θ ε is the optimistic ε -net in Proposition 3. Then, under event E , the following holds,

Proof. The proof follows by Propositions 6 to 8. The argument is direct and is identical to Lemma 1 of [35].

J.3 UCB for Total Variation Distance

Notation. Let m ∗ , { M ∗ h } h be the observable operators of the true PSR θ ∗ , and let { ̂ M k h } h be the algorithm's estimates of the observable operators corresponding to ̂ θ k .

Recall that Proposition 4 shows that the total variation distance between the distribution over trajectories of two PSRs is bounded by the estimation error of the observable operators M h . The following result constructs a bound on the estimation error of the observable operators M h ( x h ) . The proof is adapted from [35, Lemma 2] to our setting with generalized PSRs.

Lemma 5. Under event E , for any policy π and k ∈ [ K ] , we have,

where,

Proof. To ease notation, we index the future trajectories ω h -1 = ( x h , . . . , x H ) ∈ F h -1 by i and history trajectories τ h -1 = ( x 1 , . . . , x h -1 ) ∈ H h -1 by j . We denote m ⋆ ( ω h ) ⊤ ( ̂ M k h ( x h ) -M ⋆ h ( x h ) )

The following bound follows from the Cauchy-Schwarz inequality,

Step (a) follows from the fact that ̂ ψ k h -1 ( τ h -1 ) = ̂ ψ k h ( τ h -1 ) · ( ̂ ϕ k h -1 ) ⊤ ̂ ψ k h -1 ( τ h -1 ) = ̂ ψ k h ( τ h -1 ) · P ̂ θ k [ τ h -1 ] and P ̂ θ k [ τ h -1 ] · π ( τ H ) = π ( ω h -1 | τ h -1 ) · P π ̂ θ k ( τ h -1 ) . Step (b) is the Cauchy-Schwarz inequality.

Fix τ h -1 = j 0 . Let I 1 := ∥ ∥ ∑ i π i | j 0 · sign( w ⊤ i x j 0 ) · w i ∥ ∥ 2 ̂ U k h -1 , which we bound next. By the definition of ̂ U k h -1 , we partition this term into two parts,

We bound I 2 and I 3 separately. By the triangle inequality, √ I 2 is bound by a sum of two terms,

where step (a) is by the definition of w ⊤ i , π i | j 0 and the triangle inequality, and step (b) is by the triangle inequality.

Consider the first term. It can be bound via the definition of γ -well-conditioning as follows,

where step (a) is by the first condition in Assumption 1, step (b) is by the second condition of Assumption 1, and step (c) is by the fact that max z ∈ R d h -1 : ∥ z ∥ 2 =1 ∥ z ∥ 1 = √ d h -1 ≤ √ d and ∣ ∣ Q A h +1 ∣ ∣ ≤ Q A . In the above, note that we used the γ -well-conditioning of PSR ̂ θ k in step (a) and the γ -wellconditioning of PSR θ ∗ in step (b). The second term in √ I 2 admits an identical bound, simply by using the well-conditioning of the PSR θ ∗ in both steps. Hence, we have that

Now we upper bound I 3 ,

where the second equality follows from the triangle inequality by adding and subtracting m ∗ ( ω h ) ⊤ M ∗ h ( x h ) ψ ∗ ( τ h -1 ) inside the absolute value. We now bound each of I 4 and I 5 .

where step (a) is by the fact that M h ( x h ) ψ h -1 ( τ h -1 ) = P [ x h | τ h -1 ] ψ ( τ h ) , as shown in Equation (28), step (b) uses ω h -1 = ( x h , ω h ) and π ( ω h -1 | j 0 ) = π ( x h | j 0 ) π ( ω h | j 0 , x h ) , step (c) is by Assumption 1, and step (d) follows by the definition ψ h , [ ψ h ( τ h ) ] l = P θ [ q l h ∣ ∣ τ h ] .

Now, we turn to bound the I 5 term. We have

where step (a) is since m ∗ h ( ω h ) ⊤ M ∗ h ( x h ) = m ∗ h -1 ( ω h -1 ) ⊤ , step (b) is by the first condition of Assumption 1, and the final equality is again by the definition of ψ .

Combining the above, we have that,

Now, we decompose the summations above over X h × Q h and Q h -1 into separate summations over observation futures and action futures. That is, ( x h , q h ) is decomposed into ( ω a h -1 , ω o h -1 ) , where ω a h -1 = act ( x h , q h ) and ω o h -1 = obs ( x h , q h ) , and the summations are over ω a h -1 ∈ act ( X h × Q h )

and ω o h -1 ∈ obs ( X h × Q h ) . Similarly, q h -1 can be decomposed into ( q o h -1 , q a h -1 ) ∈ obs ( Q h -1 ) × act ( Q h -1 ) . Hence, the bound on I 3 can be written as,

Where the second inequality is by the definition of Q exp h -1 = act ( X h × Q h ∪ Q h -1 ) . Here, the second summation is over ω o h -1 ∈ obs ( X h × Q h ∪ Q h -1 ) . The final equality uses the fact the under the policy u exp h -1 the probability of each action sequence ω o h -1 is 1 / ∣ ∣ Q exp h -1 ∣ ∣ . Note that ∣ ∣ Q exp h -1 ∣ ∣ ≤ | act ( X h × Q h ) | + | act ( Q h -1 ) | , and hence we have ∣ ∣ Q exp h -1 ∣ ∣ ≤ 2 max s ∈A | X s | Q A for all h . Hence, we have,

Putting this together with the bounds on I 2 and I 3 , we get that,

Using the above bound on the difference between the observable operators of the true model and the estimated model, we now bound the total variation distance between the distributions of trajectories through Proposition 4.

Lemma 6. Under even E , the total variation distance between the estimated model at iteration k , ̂ θ k , and the true model θ ∗ , is bounded by,

for any policy π , where

Proof. Consider α k h -1 in the previous lemma. We have that,

where the inequality is by the bound on the total variation distance established in Lemma 4. Now, by Proposition 4, the total variation distance is bounded by the estimation error:

where step (a) is by Proposition 4, step (b) is by Lemma 5, and step (c) is by the Cauchy-Schwarz inequality and the calculation above bounding ∑ h ( α k h -1 ) 2 .

Adirect corollary is the following bound on the error in the estimated value function, which establishes that the bonus term ̂ b k gives an upper confidence bound.

Corollary 4 (Upper confidence bound) . Under the event E , for any k ∈ [ K ] , any reward function R : ∏ h ∈ [ H ] X h → [0 , 1] , and any policy π , we have,

Proof. By a direct calculation,

where step (a) is by the triangle inequality and the fact that R ( τ H ) ∈ [0 , 1] , step (b) is by Lemma 6, and step (c) is by the definition of ̂ b k .

J.4 ∑ K k =1 V ̂ b k ̂ θ k is sublinear

The next step is to prove that ∑ K k =1 V ̂ b k ̂ θ k = O ( √ K ) . To do that, we first prove that the estimated prediction features and the ground-truth prediction features can be related through the total-variation distance between the estimated model and the true model.

Lemma 7. Under event E , for any k ∈ [ K ] , we have:

Proof. First, we recall the definition of ̂ U k h , and we define its ground-truth counterpart replacing estimated features with true features,

For any trajectory τ H ∈ H H , we have,

where step (a) is simply using ∥ x ∥ 2 ≤ ∥ x ∥ 1 and step (b) is by the identity [35, Lemma 13]. Note that r is the rank of the PSR and r ≥ rank( { ̂ ψ k ( τ h ) : τ h ∈ H h } ) , rank( { ψ ∗ ( τ h ) : τ h ∈ H h } ) . Moreover, we have,

where we used the definition of ψ in (a) and the definition of the u exp h -1 in (b). Now, by Lemma 4, we have,

where the first line is combining the calculations above and the second line is by the estimation guarantee of Lemma 4.

The first term can be bounded in expectation under P π θ ∗ for any π as follows,

where step (a) is the triangle inequality, step (b) is the definition of ψ ( τ h ) , step (c) is since ∥ ∥ ∥ ∥ ̂ ψ k ( τ h ) ∥ ∥ ∥ ∥ 1 ≤ ∣ ∣ Q A h ∣ ∣ ≤ Q A for any τ h and the definition of ψ ( τ h ) , and step (d) is simply D TV ( P π θ ∗ ( τ h ) , P π ̂ θ k ( τ h ) ) ≥ D TV ( P π θ ∗ ( τ h ) , P π ̂ θ k ( τ h ) ) .

Putting this together concludes the proof,

The following lemma bounds the cumulative estimation error of the probability of trajectories. It can be proved via an ℓ 2 Eluder argument [38, 76, 77]. A significant portion of the proof is very similar to that of Proposition 4, involving an exchange of ̂ ( · ) and ( · ) ∗ . We include the proof for completeness.

Lemma 8. Under event E , for any h ∈ { 0 , . . . , H -1 } , we have

Here, a ≲ b indicates that there is an absolute positive constant c s.t. a ≤ c · b .

Proof. Recall that, by the first inequality in Proposition 4, we have:

This is very similar to the inequality in Lemma 5, with the difference being that the quantities associated with the estimated model and the true model are exchanged. Since both correspond to a PSR, the analysis follows a similar series of steps. We will use analogous notation to Lemma 5. We index the future trajectory ω h -1 = ( x h , . . . , x H ) by i and history trajectory τ h -1 = ( x 1 , . . . , x h -1 ) by j . We denote ̂ m k ( ω h ) ⊤ ( ̂ M k h ( x h ) -M ⋆ h ( x h ) ) as w i , ψ ⋆ ( τ h -1 ) as x j , and π ( ω h -1 | τ h -1 ) as π i | j .

Define the matrix,

where λ 0 is a constant to be determined later.

For any policy π , using a similar calculation as in Lemma 5, we have,

where the last line is the Cauchy-Schwarz inequality.

Fix j = j 0 and consider the term: ∥ ∥ ∑ i π i | j 0 sign( w ⊤ i x j 0 ) w i ∥ ∥ Λ h in the above. This term can be partitioned in the same manner as in Lemma 5 by simply using the definition of Λ h and expanding,

We bound each term separately. The process is nearly identical to the proof of Lemma 5, but we show it for completeness.

√ I 1 is bounded by the sum of two terms,

where step (a) is the definition of w i , π i | j 0 , and the triangle inequality, and step (b) is the triangle inequality.

Both terms can be bounded by the γ -well-conditioning assumption on ̂ θ k and θ ∗ . Consider the first term,

where step (a) is by the first condition in Assumption 1, step (b) is by the second condition of Assumption 1, and step (c) is by the fact that max z ∈ R d h -1 : ∥ z ∥ 2 =1 ∥ z ∥ 1 = √ d h -1 ≤ √ d and ∣ ∣ Q A h +1 ∣ ∣ ≤ Q A . In the above, note that we used the γ -well-conditioning of PSR ̂ θ k in both step (a) and step (b). The second term in √ I 1 admits an identical bound, simply by using the well-conditioning of the PSR ̂ θ k in the first step and θ ∗ in the second step. Hence, we have that

Now, we consider the term I 2

where the line follows by the fact that x ≤ | x | and the line follows from the triangle inequality by adding and subtracting ̂ m h ( ω h ) ̂ M h ( x h ) ̂ ψ k ( τ h -1 ) inside the absolute value. We now bound each of I 3 and I 4 .

First, we bound I 3 as follows,

where step (a) is since ̂ m ( ω h ) ⊤ ̂ M h ( x h ) = ̂ m ( ω h -1 ) ⊤ , step (b) is by Assumption 1, and the final equality is by the definition of ψ .

where step (a) is by the fact that M h ( x h ) ψ h -1 ( τ h -1 ) = P [ x h | τ h -1 ] ψ ( τ h ) , as shown in Equation (28), step (b) is by Assumption 1, and step (c) is since [ ψ h ( τ h ) ] l = P θ [ q l h ∣ ∣ τ h ] . Combining the above, we have that,

where step (a) follows from the definition of Q exp h -1 (same as Lemma 5), and step (b) is because the Hellinger distance bounds the total variation distance and since ∣ ∣ Q exp h -1 ∣ ∣ ≤ 2 max s ∈A | X s | Q A . Hence, we have,

Now, combining the bound on I 1 and I 2 allows us to finally bound ∥ ∥ ∑ i π i | j sign( w ⊤ i x j ) w i ∥ ∥ 2 Λ h as follows,

We choose λ 0 = γ 4 4 Q 2 A d , and bound ˜ α 2 := ∑ h ( ˜ α k h -1 ) 2 as follows,

where the second line is by the estimation guarantee of Lemma 4. Thus, we have,

where step (a) is by the Cauchy-Schwarz inequality and step (b) is by the bound established above. Since the total variation distance is bounded above by 2, we have

Finally, the proof is completed by summing over k using the elliptical potential lemma as follows,

Here, step (a) is uses the relationship between the ℓ 1 and ℓ 2 norms ∥·∥ 1 ≤ √ d ∥·∥ 2 . Step (b) is by the elliptical potential lemma ([35, Lemma 14]; see also [96-98]). Step (c) uses the bound on ˜ α established above and the fact that √ 1 + 4 / ˜ α 2 is bounded by an absolute constant. Step (d) uses the definition of λ 0 and the fact that √ 28(1 + 4 / ˜ α 2 ) is bounded by an absolute constant.

Using the two lemmas above, we are now ready to show that ∑ K k =1 V ̂ b k ̂ θ k ( π k ) = O ( √ K ) . The argument is identical to [35, Lemma 6] and does not require modification for generalized PSRs. We recount the argument for completeness.

Lemma 9. Under the event E , with probability at least 1 -δ , we have:

where β 0 = max { log(1 + K/λ ) , log(1 + dQ A K/γ ) } , and λ = γ max s ∈A | X s | Q A β max { √ r,Q A √ H/γ } √ dH

Proof. First, we note that,

where we recall that ̂ b k ( · ) ∈ [0 , 1] . Hence, we may focus on bounding the value of ̂ b k under the true model θ ∗ and use the bound on the cumulative total variation estimation error established in Lemma 8. Recall the definition of the bonus term,

which is defined in terms of the estimated prediction features ̂ ψ . Recall also that in Lemma 7 we established a bound on the expectation of the prediction features under the true model, which

corresponds to V ̂ b k θ ∗ . Hence, we proceed to bound ∑ k V ̂ b k θ ∗ ( π k ) as follows,

where step (a) is by Lemma 7 and step (b) is since min( a + b, c ) ≤ min( a, c ) + b when a, b, c are non-negative.

Next, we bound the term I 1 . Recall the definition of U k h := λI + ∑ τ h ∈D k h ψ ∗ ( τ h ) ψ ∗ ( τ h ) ⊤ . Also, note that the process

is a martingale. Hence, by the Azuma-Hoeffding inequality, we have that with probability at least 1 -δ ,

where the second line is by the Elliptical potential lemma ([35, Lemma 14]; see also [96-98]).

We now return to bounding ∑ K k =1 V ̂ b k ̂ θ k ( π k ) . For convenience we define β 0 := max { log(1 + K/λ ) , log(1 + dQ A K/γ ) } and we choose λ as follows,

We have,

where step (a) is by Lemma 8 and the bound on I 1 established above, step (b) uses the definition of α and the fact that α ≲ Q A √ Hdλ γ 2 + max s ∈A | X s | Q A √ β γ , and step (c) is by plugging in the choice of λ .

J.5 Proof of Theorem 4

Theorem (Restatement of Theorem 4) . Suppose Assumption 1 holds. Let p min = O ( δ KH ∏ H h =1 | X h | ) , λ = γ max s ∈A | X s | 2 Q A β max { √ r,Q A √ H/γ } √ dH , α = O ( Q A √ Hd γ 2 √ λ + max s ∈A | X s | Q A √ β γ ) , and let β = O (log ∣ ∣ Θ ε ∣ ∣ ) , where ε = O ( p min KH ) . Then, with probability at least 1 -δ , Algorithm 1 returns a model θ ϵ and a policy π that satisfy

In addition, the algorithm terminates with a sample complexity of,

Proof. By Propositions 6 to 8, the event E occurs with high probability, P [ E ] ≥ 1 -3 δ . Suppose E holds. Then, by the upper confidence bound established in Corollary 4, if Algorithm 1 terminates, then the following must hold,

where the maximization is over reward functions R : H H → [0 , 1] . The last inequality is simply the termination condition of Algorithm 1.

Now, the difference between the optimal value and the value of π (the policy returned by the algorithm) can be bounded as follows,

where the inequality follows from the fact that π = arg max π V R θ ϵ ( π ) and by Corollary 4. Recall that by Lemma 9, we have,

By the pigeon-hole principle and the termination condition of Algorithm 1, the algorithm must terminate within

episodes. Since each episode contains H iterations, this implies a sample complexity of

This concludes the proof of Theorem 4.

K Proof of Theorem 5: UCB Algorithm for Generalized PSRs (Game Setting)

Theorem (Restatement of Theorem 5) . Suppose Assumption 1 holds. Let p min = O ( δ KH ∏ H h =1 | X h | ) , λ = γ max s ∈A | X s | 2 Q A β max { √ r,Q A √ H/γ } √ dH , α = O ( Q A √ Hd γ 2 √ λ + max s ∈A | X s | Q A √ β γ ) , and let β = O (log ∣ ∣ Θ ε ∣ ∣ ) , where ε = O ( p min KH ) . Then, with probability at least 1 -δ , Algorithm 2 returns a model θ ϵ and a policy π which is an ε -approximate equilibrium (either NE or CCE). That is,

In addition, the algorithm terminates with a sample complexity of,

Proof. Recall that the model-estimation portion of Algorithm 2 is identical to Algorithm 1. Hence, by Theorem 4, the returned estimated model θ ε satisfies,

for any collection of policies π = ( π i : i ∈ [ N ]) . This implies that V i θ ∗ ( π ) ≥ V i θ ε ( π ) -ε/ 2 for all i ∈ [ N ] .

Let Γ i = Γ i ind in the case of running the algorithm to find a Nash equilibrium and Γ i = Γ i cor in the case of a coarse correlated equilibrium. Recall that the collection of policies π = ( π 1 , . . . , π N ) returned by the algorithm are an equilibrium under θ ε . That is, for all i ∈ [ N ] ,

Moreover, note that,

where the final inequality is since D TV ( P π θ ε ( τ H ) , P π θ ∗ ( τ H )) ≤ ε/ 2 for any π . Thus, V i, † θ ε ( π -i ) ≥ V i, † θ ∗ ( π -i ) -ε/ 2 .

Putting this together, we have,

Hence, π is an ε -approximate equilibrium (either NE or CCE).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction summarize the main message and technical results of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This is discussed in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides a complete, self-contained description of all technical assumptions, and a full proof (in the appendix) for all theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: There is no data or code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is theoretical in nature and does not have immediately-foreseeable negative or positive societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not release data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use any existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an abstract model of RL with information structures. Under this mode, the paper investigates the necessary conditions for sample-efficient learning.\n\nSTRENGTHS:\nThe results of this paper are very general, in the sense that they encompass the results of several previous works.\n\nWEAKNESSES:\n(1) It is hard to identify the technical novelty & originality of this paper. The POSTs and POSGs are special cases of \"generalized PSRs\", but the so-called \"generalized PSR\" is largely a direct abstraction of the analysis framework of Liu et al. 2022b. The algorithm and the proofs are also largely patching the analysis in the previous papers (in particular, Liu et al. 2022b).\n\n(2) It is not clear what new concrete tractable class does this paper identify. Tractable POMDP classes and POMG classes have already been studied in previous works. Several examples of information structure are provided, but there isn't a definition of a new, meaningful, and tractable problem class.\n\nOverall, this paper indeed provides a general framework, but it seems that generality is the only advantage of this framework. However, \"general framework\" cannot be an excuse. After all, there have already been several general frameworks for sequential decision making. For example, for model-based learning, the so-called Decision-Estimation Coefficient characterizes the statistical complexity of *any* model class [1,2,3,etc]. The SAIL framework of Liu et al. (2022b) is more concrete and it also applies to a broad class of sequential decision making problems. The authors have to justify what is the significance of this framework other than its (vacuous) generality.\n\n[1] Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making.\n\n[2] Dean Foster, Dylan J Foster, Noah Golowich, and Alexander Rakhlin. On the complexity of multi-agent decision making: From learning in games to partial monitoring.\n\n[3] Dylan J Foster, Noah Golowich, and Yanjun Han. Tight guarantees for interactive decision making with the decision-estimation coefficient.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new representation of sequential decision making problems with an information structure. They show that the information structure can be used to estimate the complexity of the decision-making problems and can guide learning algorithm development.\n\nSTRENGTHS:\n1. The authors present a good study on complexity of decision-making problems in finite state and action spaces\n1. They identify scalable decision-making problems for learning applications.\n1. The work appears to be technically sound, but I haven't checked all the proofs\n\nWEAKNESSES:\n1. If I understood correctly, the authors are dealing with finite state and action spaces and the complexity measure is effectively a rank of a certain matrix that they construct. If the state or the action spaces are infinite or continuous then the measure becomes infinite for all systems and we cannot use the measure for judging complexity. If this is so, I see it as a major weakness\n1. Presentation can be improved. It’s hard to understand what different concepts mean, say core tests. Can a “layman” summary be given after these kinds of definitions?\n1. While I appreciate that the paper is theoretic in nature, it is hard to judge it's practical implications in present or even in future. \n1. Some claims need for further clarification or justification. For example, I am not sure that Figure 1 is fully substantiated. Hopefully, the authors can clarify\n\nQUESTIONS:\n1. The dynamics matrix seems to bear resemblance to the Hankel operator for linear systems theory in control literature. The Hankel operator is mapping past inputs to feature outputs, since the system is linear its information structure is rather simple.  The rank of the Hankel operator is actually a well-established complexity measure for linear systems. It would be good to understand if there’s more than just some general similarity. \n1. Is there an example of a system that is POST but not POMG or not dec-POMDP as claimed in Figure 1?\n1. Are the authors deal with finite state and action spaces? It seems that the dynamics matrix with probabilities of potential futures and histories is finite-dimensional, but it’s not clearly stated\n1. “This   assumption is often unrealistic, since general systems may not have “states” per se, and observations may be generated with more complex dependencies.” - I am bit confused what does it mean that the system may not have states. Can you add an example here? I think that POMDPs are generally quite realistic when it comes to single agent systems hence this comment is confusing for me. \n1. Please define dec-POMDP, POMG before using the acronyms. Or at least define them in the Appendix.\n1. Why in def 5 we need to remove all the arrows pointing to actions? Can actions be dependent on history?\n1. What’s $\\sigma$ in Def 6?\n1. Line 268. If I understand correctly $G_h$ is possibly a non-square matrix, what’s an eigenvalue in this context? Do the authors mean singular value as they speak about rank?\n\nLIMITATIONS:\nlimitations are discussed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the tractability of modeling and learning general sequential decision-making problems with an explicit representation of information structure. This gives a unifying framework that captures a range of commonly studied RL models. Through a graph-theoretic analysis it characterizes the complexity of observable dynamics in sequential decision-making problems. It then gives an upper bound on the sample complexity of learning these problems characterized by their information structure.\n\nSTRENGTHS:\n- The paper is well-written and well-structured. The proposed framework of partially-observable sequential teams (POST) and partially-observable sequential games (POSG) incorporates more general system dynamics than classical MDP/POMDP models through much more general information structures. The model is also practically relevant for application with complex and time-varying dependence on the past or time-varying observations dynamics. \n- The paper provides a thorough theoretical foundation to understanding the complexity and tractability of POST and POSG problems. This includes characterizing the complexity of the observable system dynamics, as well as upper bound of on the statistical hardness of learning both through the lens of the information structure. Moreover the results recovers known sample complexity results in POMDPs. \n- This paper gives a key insight that the complexity of the dynamics and the statistical hardness of sequential decision-making problems can be characterized in terms of its information structure. The proposed method of observation generalized predictive state representation is also of independent interest for other adjacent problems.\n\nWEAKNESSES:\n- The m-step $\\mathcal{I}^{\\dagger}$ -weakly revealing condition can be stringent. As the paper points out, the condition will be harder to satisfy when $\\mathbb{I}_h^{\\dagger}$ is large. It would be helpful to address the sensitivity of claimed results when this condition is not satisfied. \n- The results of the paper are theoretical in nature, and it could benefit from empirical results to demonstrate the effectiveness of the proposed algorithm and framework on experiments / simulations.\n\nQUESTIONS:\n- How realistic are the assumptions required for the theoretical results, e.g. the m-step $\\mathcal{I}^{\\dagger}$ -weakly revealing condition, the $\\alpha$-robust weakly-revealing property etc.? E.g., what are the some of the implications of these assumptions in a practical context?\n- The paper shows that the sample complexity upper bound in Theorem 3 for POST implies a reasonable bound for the case of $\\alpha$-weakly POMDPs. Does this sample complexity upper bound also give reasonable upper bounds for other well-studies cases incorporated by the POST framework such as MDPs, Markov games, Dec-POMDPs?\n\nLIMITATIONS:\nThe author(s) have adequately addressed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}