Title: Provable Partially Observable Reinforcement Learning with Privileged Information
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning

pomdp

partial observability

computational

privileged information

expert distillation

teacher-student learning
Abstract: Partial observability of the underlying states generally presents significant challenges for reinforcement learning (RL). In practice, certain *privileged information* , e.g., the access to states from simulators, has been exploited in training and achieved prominent empirical successes. To better understand the benefits of privileged information, we revisit and examine several simple and practically used paradigms in this setting, with both computation and sample efficiency analyses. Specifically, we first formalize the empirical paradigm of *expert distillation* (also known as  *teacher-student* learning), demonstrating its pitfall in finding near-optimal policies. We then identify a condition of the partially observable environment, the deterministic filter condition, under which expert distillation achieves sample and computational complexities that are *both* polynomial. Furthermore, we investigate another successful empirical paradigm of *asymmetric actor-critic*, and focus on the more challenging setting of observable partially observable Markov decision processes. We develop a belief-weighted optimistic asymmetric actor-critic algorithm with polynomial sample and quasi-polynomial computational complexities, where one key component is a new provable oracle for learning belief states that preserve *filter stability* under a misspecified model, which may be of independent interest. Finally, we also investigate the provable efficiency of partially observable multi-agent RL (MARL) with privileged information. We develop algorithms with the feature of centralized-training-with-decentralized-execution, a popular framework in empirical MARL, with polynomial sample and (quasi-)polynomial computational complexity in both paradigms above. Compared with a few recent related theoretical studies, our focus is on understanding practically inspired algorithmic paradigms, without computationally intractable oracles.

Provable Partially Observable Reinforcement Learning with Privileged Information

Abstract

Partial observability of the underlying states generally presents significant challenges for reinforcement learning (RL). In practice, certain privileged information , e.g., the access to states from simulators, has been exploited in training and has achieved prominent empirical successes. To better understand the benefits of privileged information, we revisit and examine several simple and practically used paradigms in this setting. Specifically, we first formalize the empirical paradigm of expert distillation (also known as teacher-student learning), demonstrating its pitfall in finding near-optimal policies. We then identify a condition of the partially observable environment, the deterministic filter condition , under which expert distillation achieves sample and computational complexities that are both polynomial. Furthermore, we investigate another successful empirical paradigm of asymmetric actor-critic , and focus on the more challenging setting of observable partially observable Markov decision processes. We develop a belief-weighted asymmetric actor-critic algorithm with polynomial sample and quasi-polynomial computational complexities, in which one key component is a new provable oracle for learning belief states that preserves filter stability under a misspecified model, which may be of independent interest. Finally, we also investigate the provable efficiency of partially observable multi-agent RL (MARL) with privileged information. We develop algorithms featuring centralized-training-with-decentralized-execution , a popular framework in empirical MARL, with polynomial sample and (quasi)polynomial computational complexities in both paradigms above. Compared with a few recent related theoretical studies, our focus is on understanding practically inspired algorithmic paradigms, without computationally intractable oracles.

1 Introduction

In most real-world applications of reinforcement learning (RL), e.g., perception-based robot learning [46, 3], autonomous driving [73, 41], dialogue systems [88], and clinical trials [77], only partial observations of the environment state are available for sequential decision-making. Such partial observability presents significant challenges for efficient decision-making and learning, with known computational [66] and statistical [43, 36] barriers under the general model of partially observable Markov decision processes (POMDPs). The curse of partial observability becomes severer when multiple RL agents interact, where not only the environment state, but also other agents' information, are not fully-observable in decision-making [85, 80].

On the other hand, a flurry of empirical paradigms has made partially observable (multi-agent) RL promising in practice. One notable example is to exploit the privileged information that may be available (only) during training. The privileged information usually includes direct access to the underlying states, as well as access to other agents' observations/actions in multi-agent RL (MARL), due to the use of simulators and/or high-precision sensors for training. The latter is also known as

2

the centralized-training-with-decentralized-execution (CTDE) framework in deep MARL, and has become prevalent in practice [53, 70, 22, 82]. These approaches can be mainly categorized into two types: i) privileged policy learning, where an expert/teacher policy is trained with privileged information, and then distilled into a student partially observable policy. This expert distillation , also known as teacher-student learning , approach has been the key to some empirical successes in robotic locomotion [45, 59] and autonomous driving [14]; ii) privileged value learning, where a value function is trained conditioned on privileged information, and used to improve a partially observable policy. It is typically instantiated as the asymmetric actor-critic algorithm [68], and serves as the backbone of some high-profiled successes in robotic manipulation [46, 3] and MARL [53, 82].

Despite the remarkable empirical successes, theoretical understandings of partially observable RL with privileged information have been rather limited, except for a few recent prominent advances in RL with hindsight observability [44, 30] (see Appendix B for a detailed discussion). However, most of these theoretically sound algorithms are different from those used in practice, and require computationally intractable oracles to achieve provable sample efficiency. The soundness and efficiency of the aforementioned paradigms used in practice remain elusive. In this work, we examine both paradigms of expert distillation and asymmetric actor-critic, with foresight privileged information as in these empirical works. In contrast to [44, 30], which purely focused on sample efficiency, we aim to understand the benefits of privileged information by examining these practically inspired paradigms under several POMDP models, without computationally intractable oracles. We defer a detailed literature review to Appendix B, and summarize our contribution as follows.

Contributions. We first formalize the empirical paradigm of expert distillation , and demonstrate its pitfall in distilling near-optimal policies even in observable POMDPs, a model class that was recently shown to allow provable partially observable RL without computationally intractable oracles [25]. We then identify a new condition for POMDPs, the deterministic filter condition, and establish sample and computational complexities that are both polynomial for expert distillation. The new condition is weaker and thus encompasses several known (statistically) tractable POMDP models (see Figure 1 for a summary). Further, we revisit the asymmetric actor-critic paradigm and analyze its efficiency under the more challenging setting of observable POMDPs above (where expert distillation fails). Identifying the inefficiency of vanilla asymmetric actor-critic, and inspired by the empirical success in belief-state-learning , we develop a new belief-weighted version of asymmetric actorcritic, with polynomial-sample and quasi-polynomial-time complexities. Key to the results is a new belief-state learning oracle that preserves filter stability under a misspecified model, which may be of independent interest. Finally, we also investigate the provable efficiency of partially observable multi-agent RL with privileged information, by studying algorithms under the CTDE framework, with polynomial-sample and (quasi-)polynomial-time complexities in both paradigms above.

2 Preliminaries

2.1 Partially Observable RL (with Privileged Information)

Model. Consider a POMDP characterized by a tuple P = ( H, S , A , O , T , O , µ 1 , r ) , where H denotes the length of each episode, S is the state space with |S| = S , A denotes the action space with |A| = A . We use T = { T h } h ∈ [ H ] to denote the collection of transition matrices, so that T h ( · | s, a ) ∈ ∆( S ) gives the probability of the next state if action a is taken at state s and step h . In the following discussions, for any given a , we treat T h ( a ) ∈ R |S|×|S| as a matrix, where each row gives the probability for reaching each next state from different current states. We use µ 1 to denote the distribution of the initial state s 1 , and O to denote the observation space with |O| = O . We use O = { O h } h ∈ [ H ] to denote the collection of emission matrices, so that O h ( · | s ) ∈ ∆( O ) gives the emission distribution over the observation space O at state s and step h . For notational convenience, we will at times adopt the matrix convention, where O h is a matrix with rows O h ( · | s ) for each s ∈ S . Finally, r = { r h } h ∈ [ H ] is a collection of reward functions, so that r h ( s, a ) ∈ [0 , 1] is the reward given the state s and action a at step h . When privileged information is available, the agent can observe the underlying state s ∈ S directly during training (only). We thus denote the trajectory until step h with states as τ h = ( s 1: h , o 1: h , a 1: h -1 ) , the one without states as τ h = ( o 1: h , a 1: h -1 ) , and its space as T h . Finally, we use b h ( τ h ) to denote the posterior distribution over the underlying state at step h given history τ h , which is known as the belief state (c.f. Appendix C.1 for more details).

Policy and value function. We define a stochastic policy at step h as:

where the agent bases on the entire (partially observable) history for decision-making. The corresponding policy class is denoted as Π h . We further denote Π = × h ∈ [ H ] Π h . We also define Π gen := { π 1: H | π h : S h ×O h ×A h -1 → ∆( A ) for h ∈ [ H ] } to be the most general policy space in partially observable RL with privileged state information, which can potentially depend on all historical states, observations, and actions. It can be seen that Π ⊆ Π gen . We may also use policies that only receive a finite memory instead of the whole history as inputs: fix an integer L > 0 , we define the policy space Π L to be the space of all possible policies π = π 1: H := ( π h ) h ∈ [ H ] such that π h : Z h → ∆( A ) with Z h := O min { L,h } × A min { L,h } for each h ∈ [ H ] . Finally, we define the space of state-based policies as Π S , i.e., for any π = π 1: H ∈ Π S , π h : S → ∆( A ) for all h ∈ [ H ] .

Given the POMDP model P , we write P P s 1: H +1 ,a 1: H ,o 1: H ∼ π ( E ) to denote the event E when ( s 1: H +1 , a 1: H , o 1: H ) is drawn as a trajectory following the policy π in the model P . We will also use the shorthand notation P π, P ( · ) if ( s 1: H +1 , a 1: H , o 1: H ) is evident. We write E P π [ · ] to denote the expectation similarly. We define the value function at step h as V π, P h ( y h ) := E P π [ ∑ H t = h r t ( s t , a t ) | y h ] , denoting the expected accumulated rewards from step h , where y h ⊆ ( s 1: h , o 1: h , a 1: h -1 ) , and we slightly abuse the notation by treating as a set the sequence of states s 1: h , the sequence of observations o 1: h , and the sequence of actions a 1: h -1 up to time h , which are the available information to the agent at step h . We say y h is reachable if there exists some policy π ∈ Π gen such that P π, P ( y h ) > 0 . For h = 1 , we adopt the simplified notation v P ( π ) = E P π [ ∑ H h =1 r h ( s h , a h )] . Meanwhile, we also define Q π, P h ( y h , a h ) := E P π [ ∑ H t = h r t ( s t , a t ) | y h , a h ] . We denote the occupancy measure on the state space as d π, P h ( s h ) = P π, P ( s h ) . The goal of learning in POMDPs is to find the optimal policy that maximizes the expected accumulated reward over the policies that take τ h as input at each step h ∈ [ H ] , i.e., those π ∈ Π . Formally, we define:

Definition 2.1 ( ϵ -optimal policy) . Given ϵ > 0 , a policy π ⋆ ∈ Π is ϵ -optimal, if v P ( π ⋆ ) ≥ max π ∈ Π v P ( π ) -ϵ .

Learning with privileged information. Common RL algorithms for POMDPs deal with the scenario where during both the training and test time, the agent can only observe its historical observations and actions τ h at step h , while the states are not accessible. In other words, the agent can only utilize policies from Π to interact with the environment. In contrast, in settings with privileged information , e.g., training in simulators and/or using sensors with higher precision, the underlying state can be used in training. Thus, the agent is allowed to utilize policies from the class Π gen during training. Meanwhile, the objective is still to find the optimal history-dependent policy in the space of Π , since at test time, the agent cannot access the state information anymore, and it is the performance for such policies that matters eventually. For simplicity, we assume the reward function is known since under our privileged information setting, learning the reward function is much easier than learning the transition and emission, and the sample/computational complexity for the former is dominated by that for the latter. This assumption has also been made for learning in POMDPs without privileged information [36, 47, 48].

2.2 Partially Observable Multi-agent RL with Information Sharing

Partially observable stochastic games (POSGs) are a natural generalization of POMDPs with multiple agents of potentially independent interests. We define a POSG with n agents by a tuple G = ( H, S , {A i } n i =1 , {O i } n i =1 , T , O , µ 1 , { r i } n i =1 ) , where each agent i has its individual action space A i , observation space O i , and reward function r i = { r i,h } h ∈ [ H ] with r i,h ( s, a ) ∈ [0 , 1] denoting the reward given state s and joint action a for agent i at step h . An episode of POSG proceeds as follows: at each step h and state s h , a joint observation is drawn from ( o i,h ) i ∈ [ n ] ∼ O h ( · | s h ) , and each agent receives its own observation o i,h , takes the corresponding action a i,h , obtains the reward r i,h ( s h , a h ) , where a h := ( a i,h ) i ∈ [ n ] , and then the system transitions to the next state as s h +1 ∼ T h ( · | s h , a h ) . Notably, each agent i may not only know its local information ( o i, 1: h , a i, 1: h -1 ) , but also information from some other agents. Therefore, we denote the information available to each agent i at step h also as τ i,h ⊆ ( o 1: h , a 1: h -1 ) and define the common information as c h = ∩ i ∈ [ n ] τ i,h and private information as p i,h = τ i,h \ c h . We denote the space for common information and private information as C h and P i,h for each agent i and step h . The joint private information at step h is denoted as p h = ( p i,h ) i ∈ [ n ] , where the collection of the joint private information is given by P h = P 1 ,h × · · · × P n,h . We refer more examples of this setting of POSG with information-sharing to Appendix C.2 (and also [62, 63, 51]). Correspondingly, the policy each agent i deploys at test time takes the form of π i,h : Ω h ×C h ×P i,h → ∆( A i ) , where Ω h is the space of random seeds. We denote the policy

Without PI

With PI (Ours)

Block MDP

With STD:

Oracle-efficient

[17, 18, 60]

Without additional assump.:

Computationally harder than SL [28]

Tabular:

Poly sample

+ time

FA:

Poly sample +

Classification

(SL) oracle

k

-decodable

POMDP

Exponential-in-

k

sample + time [19]

Det. POMDP

Without WSE:

Statistically hard [47]

With WSE:

Poly sample + time [36]

POSG with

det. filter

N/A

Poly sample + time

Observable

POMDP

Quasi-poly

sample + time

[25] [51]

Poly sample +

Quasi-poly time

Observable

POSG

Transition

"'Determinism"

Deterministic

POMDP

Deterministicfilter:

poly sample

（polysample,polytime)

polytime

poly sample

quasi-polytime

k-decodablePOMDP

poly sample

O(k is a small constant)

unknowntime

Only with PI:

(poly sample,

knownsetting

unknowntime)

Observable

BlockMDP MDP

POMDP

Weaklyrevealing

Observation

POMDP

"Informativeness"

Observability:

(poly sample,quasi-polytime)

space for agent i as Π i . If π i,h takes the state s h instead of ( c h , p i,h ) as input, we denote its policy space as Π S ,i , e.g., for each agent i , and policy π 1: H ∈ Π S ,i , we have π i,h : S → ∆( A i ) for each step h ∈ [ H ] . Similar to the POMDP setting, we define Π gen to be the most general policy space, i.e., Π gen := { π 1: H | π h : S h ×O h ×A h -1 → ∆( A ) for h ∈ [ H ] } . Note that this model covers several recent POSG models studied for partially observable MARL, e.g., [49, 27]. For example, at each step h , if there is no shared information, then c h = ∅ , and if all history information is shared, then p i,h = ∅ for all i ∈ [ n ] . In privileged-information -based learning, the training algorithm may exploit not only the underlying state information, but also the observations and actions of other agents.

Solution concepts. The solution concepts for POSGs are usually the equilibria , particularly Nash equilibrium (NE) for two-player zero-sum games (i.e., when n = 2 and r 1 ,h + r 2 ,h = 1 ), 1 and correlated equilibrium (CE) or coarse correlated equilibrium (CCE) for general-sum games. We defer the formal definitions of these standard solution concepts to Appendix C.2.

2.3 Technical Assumptions for Computational Tractability

A key technical assumption is that the POMDPs/POSGs we consider satisfy an observability assumption, as outlined below. This observability assumption allows us to use short memory policies to approximate the optimal policy, and yields quasi-polynomial-time complexity for both planning and learning in POMDPs/POSGs [26, 25, 51]. Meanwhile, we defer an additional assumption to ensure the traceability for solving POSGs to Appendix C.3.

Assumption 2.2 ( γ -observability [20, 26, 25]) . Let γ > 0 . For h ∈ [ H ] , we say that the matrix O h satisfies the γ -observability assumption if for each h ∈ [ H ] , for any b, b ′ ∈ ∆( S ) , ∥ ∥ O ⊤ h b -O ⊤ h b ′ ∥ ∥ 1 ≥ γ ∥ b -b ′ ∥ 1 . A POMDP/POSG satisfies γ -observability if all its O h for h ∈ [ H ] do so.

3 Revisiting Empirical Paradigms of RL with Privileged Information

Most empirical paradigms of RL with privileged information can be categorized into two types: i) privileged policy learning, where the policy in training is conditioned on the privileged information, and the trained policy is then distilled to a policy that does not take the privileged information as input. This is usually referred to as either expert distillation [14, 64, 58] or teacher-student learning [45, 59, 75] in the literature; ii) privileged value learning, where the value function is conditioned on the privileged information, and is then used to directly output a policy that takes partial observation (history) as input. One prominent example of ii) is asymmetric-actor-critic [68, 3]. It is worth noting that asymmetric-actor-critic is also closely related to one of the most successful paradigms for multi-agent RL, centralized-training-with-decentralized-execution [53, 86, 21], which is usually instantiated under the actor-critic framework, with the critic taking privileged information as input

in training. Here we formalize and revisit the potential pitfalls of these two paradigms, and further develop theoretical guarantees under certain additional conditions and/or algorithm variants.

3.1 Privileged Policy Learning: Expert Policy Distillation

The motivation behind expert policy distillation is that learning an optimal fully observable policy in MDPs is a much easier and better-studied problem with many known efficient algorithms. The (expected) distillation objective can be formalized as follows:

where π ′ ∈ Π gen is some given behavior policy to collect exploratory trajectories, π ⋆ ∈ arg max π ∈ Π S v P ( π ) denotes the optimal fully observable policy, and D f denotes the general f -divergence to measure the discrepancy between π ⋆ and π .

Such a formulation looks promising since it essentially circumvents the challenging issue of exploration in partially observable environments , by directly mimicking an expert policy that can be obtained from any off-the-shelf MDP learning algorithm. However, we point out in the following proposition that even if the POMDP satisfies Assumption 2.2, the distilled policy can still be strictly suboptimal even with infinite data, i.e., by solving the expected objective Equation (3.1) completely. We postpone the proof of Proposition 3.1 to Appendix E.

Proposition 3.1 (Pitfall of expert policy distillation) . For any ϵ, γ ∈ (0 , 1) , there exists a γ -observable POMDP P ϵ with H = 1 , S = O = A = 2 such that for any behavior policy π ′ ∈ Π gen and choice of D f in Equation (3.1), it holds that v P ϵ ( ̂ π ⋆ ) ≤ max π ∈ Π v P ϵ ( π ) -(1 -ϵ )(1 -γ ) 4 .

The key reason Equation (3.1) fails is that in general, the underlying state can remain highly uncertain even given the full history. Thus, the distilled policy may not be able to mimic the state-based expert policy well at different states s h if the associated π ⋆ h ( · | s h ) differs significantly across s h . To see how we may rule out such an issue, notice that if γ = 1 (note that according to Assumption 2.2, we have γ is at most 1 since γ ≤ ∥ O h ∥ ∞ ≤ 1 ), implying that the observation can decode the underlying state, the bound in Proposition 3.1 becomes vacuous. Inspired by this, we propose the following condition that incorporates this case of γ = 1 , and will be shown to suffice to make expert distillation effective.

Definition 3.2 (Deterministic filter condition) . We say a POMDP P satisfies the deterministic filter condition if for each h ≥ 2 , the belief update operator under P satisfies that there exists an unknown function ψ h : S × A × O → S such that for any reachable s h -1 ∈ S , o h ∈ O , a h -1 ∈ A , U h ( b s h -1 ; a h -1 , o h ) = b ψ h ( s h -1 ,a h -1 ,o h ) , where we define for any s ∈ S , b s ∈ ∆( S ) and b s ( s ) = 1 is a one-hot vector. In addition, for h = 1 , there exists a function ψ 1 : O → S such that for any reachable o 1 , B 1 ( µ 1 ; o 1 ) = b ψ 1 ( o 1 ) , where B h ( b ; o h ) := P P s h ∼ b ( · | o h ) ∈ ∆( S ) , U h ( b ; a h -1 , o h ) := P P s h -1 ∼ b ( · | a h -1 , o h -1 ) ∈ ∆( S ) are the belief update operators under the Bayes rule for any b ∈ ∆( S ) , for which we defer the formal introduction to Appendix C.1.

Notably, this condition is weaker than and thus covers several known tractable classes of POMDPs with sample and computation efficiency guarantees including Block MDP, deterministic POMDP, k -decodable POMDP as well as a new setting we have identified and existing literature cannot handle. We refer the formal introduction to Appendix E and Figure 1 for an illustration.

In light of the pitfall in Proposition 3.1, we will analyze both the computational and statistical efficiencies of expert distillation in Section 4, under the condition in Definition 3.2.

3.2 Privileged Value Learning: Asymmetric Actor-Critic

Asymmetric actor-critic [68] iterates between two procedures as in standard actor-critic algorithms [42]: policy improvement and policy evaluation . As the name suggests, its key difference from the standard actor-critic is that the algorithm maintains Q -value functions (the critic) based on the state/privileged information , while the policy receives only the (partially observable) history as input.

Policy evaluation. At iteration t -1 , given the policy π t -1 , the algorithm estimates Q -functions in the form of { Q t -1 h ( τ h , s h , a h ) } h ∈ [ H ] , where we adopt the 'unbiased' version [6] such that Q -

functions are conditioned on both the history and the states . 2 One key to achieving sample efficiency is by adding some bonus terms in policy evaluation to encourage exploration, i.e., obtaining some optimistic Q -function estimates, similarly as in the fully-observable MDP setting, see e.g., [11, 74], for which we defer the detailed introduction to Section 4.

Policy improvement. At each iteration t , given the critic { Q t -1 h ( τ h , s h , a h ) } h ∈ [ H ] for π t -1 , the vanilla asymmetric actor-critic algorithm updates the policy according to the sample-based gradient estimation via K trajectories { s k 1: H +1 , o k 1: H , a k 1: H } k ∈ [ K ] sampled from π t -1

where λ t is the step-size and PROJ Π is the projection operator onto the space of Π , which corresponds to projecting onto the simplex of ∆( A ) for each h ∈ [ H ] . Here we point out the potential drawback of the vanilla algorithm as in [68, 6], where the key insight is that for each iteration of policy evaluation and improvement, one roughly only performs the computation of order O ( KH ) , while needing to collect K new episodes of samples. Thus, the sample complexity will scale in the same order as the computational complexity when the algorithm converges after some iterations to an ϵ -optimal solution, which will be super-polynomial even for γ -observable POMDPs [27]. Proof of the result is deferred to Appendix E.

Proposition 3.3 (Inefficiency of vanilla asymmetric actor-critic) . Under the tabular parameterization for both the policy and the value function, the vanilla asymmetric actor-critic algorithm (Equation (3.2)) suffers from super-polynomial sample complexity for γ -observable POMDPs under standard hardness assumptions.

To address such an issue, one may need to perform more computation per iteration , so that although the total computational complexity (iteration number × per-iteration computational complexity) is super-polynomial, the total iteration number can be lower such that the total sample complexity may be lower as well. This desideratum is hard to achieve if one computes policy update only on the sampled trajectories τ h per iteration, i.e., update asynchronously , since this will couple the scales of computational and sample complexities similarly as Equation (3.2). In contrast, we first propose to update all trajectories per iteration in a synchronous way, with the following proximal-policy optimization-type [72] policy improvement update with the state-history-dependent Q -functions { Q t -1 h ( τ h , s h , a h ) } h ∈ [ H ] :

where we recall b h ( τ h ) ∈ ∆( S ) denotes the belief state and η > 0 is the learning rate. This update rule also reduces to the natural policy gradient (NPG) [40] update under the softmax policy parameterization in the fully-observable case [1], when updated for each state s h separately [74]. We defer the detailed derivation of Equation (3.3) to Appendix E.

However, such an update presents two challenges: (1) It requires enumerating all possible τ h , whose number scales exponentially with the horizon, making it still computationally intractable; (2) An explicit belief function b h is needed. Motivated by these two challenges, we propose to consider finitememory -based policy and assume access to an approximate belief function { b apx h : Z h → ∆( S ) } h ∈ [ H ] (the learning for which will be made clear later). Correspondingly, the policy update is modified as:

Then we develop and analyze one possible approach to learning such an approximate belief efficiently (c.f. Section 5). It is worth noting that the policy optimization algorithm we aim to develop and analyze does not depend on the specific algorithm approximate belief learning. Such a decoupling enables a more modular algorithm design framework, and can potentially incorporate the rich literature on learning approximate beliefs in practice, see e.g., [24, 65, 16, 87, 83], which has mostly not been theoretically analyzed before. We will thus analyze such an oracle in Section 5.

4 Provably Efficient Expert Policy Distillation

We now focus on the provable correctness and efficiency of expert policy distillation, under the deterministic filter condition in Definition 3.2. We will defer all the proofs in this section to Appendix F. Definition 3.2 motivates us to consider only succinct policies that incorporate an auxiliary parameter representing the most recent state, as well as the most recent observations and actions. We consider policies that are the composition of two functions: at step h a function g h : S × A × O → S that decodes the state based on the previous state, the most recent action, and the most recent observation, and a policy π E ∈ Π S that takes as input the current (decoded) underlying state and outputs a distribution over actions.

Definition 4.1. We define a policy class Π D as:

where π E stands for an arbitrary expert policy, and Π D stands for the distilled policy class, and for h = 1 , a 0 , s 0 are some fixed dummy action and state. Intuitively, the distilled policy π ∈ Π D executes as follows: it firstly decodes the underlying states by applying { g h } h ∈ [ H ] recursively along the history, and then takes actions using π E based on the decoded states.

Our goal is to learn the two functions independently, that is, we want to learn an approximately optimal policy π E ∈ Π S with respect to the MDP M derived from POMDP P by omitting the observations and observing the underlying state (see Definition 4.2 for a formal definition), and for each step h ∈ [ H ] , a decoding function g h ( s h -1 , a h -1 , o h ) such that the probability of incorrectly decoding a state-action-observation triplet over the trajectories induced by the policy π E is low.

Definition 4.2 (POMDP-induced MDP) . Given a POMDP P = ( H, S , A , O , T , O , µ 1 , r ) , we define its associated Markov Decision Process (MDP) M as M = ( H, S , A , T , µ 1 , r ) without observations.

Definition 4.3. Consider a POMDP P that satisfies Definition 3.2, and let ψ = { ψ h } h ∈ [ H ] be the promised set of functions that always correctly decode a state-action-observation triplet into an underlying state. Consider policy ˜ π E = { π E h ( ψ ( · )) : S ×A×O → ∆( A ) } h ∈ [ H ] ∈ Π D . We slightly abuse the notation and simply denote by v P ( π E ) = v P ( ˜ π E ) .

̸

Lemma 4.4. Let P = ( H, S , A , O , T , O , µ 1 , r ) be a POMDP that satisfies Definition 3.2, and consider a policy π E ∈ Π S . Consider a set of decoding functions { g h } h ∈ [ H ] such that, P π E , P [ ∃ h ∈ [ H ] : g h ( s h -1 , a h -1 , o h ) = s h ] ≤ ϵ . Consider the policy π = { π E h ( g h ( · )) : S × A × O → ∆( A ) } h ∈ [ H ] on the POMDP P , then: v P ( π ) ≥ v P ( ˜ π E ) -Hϵ.

We can use any off-the-shelf algorithm to learn an approximate optimal policy π E for the associated MDP M (see Definition 4.2). Thus, in the rest of the section, we focus on learning the decoding function { g h } h ∈ [ H ] . To efficiently learn the decoding function, we model the access to the underlying state by keeping track of the most recent pair of the action and the observation, as well as the two most recent states. We summarize the algorithm of decoding-function learning in Algorithm 1.

̸

Theorem 4.5. Consider a POMDP P that satisfies Definition 3.2, a policy π E ∈ Π S , and let { g h } h ∈ [ H ] be the output of Algorithm 1 with M = AOS +log( H/δ ) ϵ 2 . Then, with probability at least 1 -δ , for each step h ∈ [ H ] : P π E , P [ ∃ h ∈ [ H ] : g h ( s h -1 , a h -1 , o h ) = s h ] ≤ ϵ, using POLY ( H,A,O,S, 1 ϵ , log ( 1 δ ) ) episodes in time POLY ( H,A,O,S, 1 ϵ , log ( 1 δ ) ) .

The following is an immediate consequence of Lemma 4.4 and Theorem 4.5. Note that both the sample and computation complexities are polynomial , which is in stark contrast to the k -decodable POMDP case [19] (a special one covered by our Definition 3.2), for which the sample complexity is necessarily exponential in k when there is no privileged information [19]. In fact, thanks to privileged information, the complexities are only polynomial in horizon H even when the decodable length is unknown . For the benefits of using privileged information in several other subclasses of problems, we refer to Table 1 for more details.

Theorem 4.6. Let P satisfy Definition 3.2 and consider any policy π E ∈ Π S . Using POLY ( H,A,O,S, 1 ϵ , log ( 1 δ ) ) episodes and in time POLY ( H,A,O,S, 1 ϵ , log ( 1 δ ) ) , we can compute a policy π ∈ Π D (see Definition 4.1) such that with probability at least 1 -δ , v P ( π ) ≥ v P ( π E ) -ϵ.

Extension to the case with general function approximation. Due to the modularity of our algorithmic framework and its compatibility with supervised learning oracles, it can be readily generalized to the function approximation setting to handle large observation spaces. We defer the corresponding results to Appendix G.

5 Provable Asymmetric Actor-Critic with Approximate Belief Learning

Unlike most existing theoretical studies on provably sample-efficient partially observable RL [36, 25, 47], which directly learn an approximate POMDP model for planning near-optimal policies, we consider a general framework with two steps: firstly learning an approximate belief function , followed by adopting a fully observable RL subroutine on the belief state space.

5.1 Belief-Weighted Optimistic Asymmetric Actor-Critic

We now introduce our main algorithmic contribution to the privileged policy learning setting. Our algorithm is conceptually similar to the natural policy gradient methods [40, 1, 74] in the fullyobservable setting, with additional weighting over the states s h using some learned belief states, to handle the additional state -dependence in the asymmetric critic. The overall algorithm is presented in Algorithm 2. The algorithm requires a belief-learning subroutine that takes the stored memory as input and outputs a belief about the underlying state (c.f. { b apx h } h ∈ [ H ] ). Additionally, similar to the fully observable setting, we include a subroutine to estimate the Q -function, which introduces additional challenges due to partial observability (see Appendix H). We establish the performance guarantee of Algorithm 2 in the following theorem. We defer the proof to Appendix H.

Theorem 5.1 (Near-optimal policy) . Fix ϵ, δ ∈ (0 , 1) . Given a POMDP P and an approximate belief { b apx h : Z h → ∆( S ) } h ∈ [ H ] , with probability at least 1 -δ , Algorithm 2 can learn an approximate optimal policy π ⋆ of P in the space of Π L such that v P ( π ⋆ ) ≥ max π ∈ Π L v P ( π ) -O ( ϵ + H 2 ϵ belief ) , with sample complexity POLY ( S, A, O, H, 1 ϵ , log 1 δ ) and time complexity POLY ( S, A, O, H, Z, 1 ϵ , log 1 δ ) , where ϵ belief is the belief-learning error defined as ϵ belief := max h ∈ [ H ] max π ∈ Π L E P π ∥ b h ( τ h ) -b apx h ( z h ) ∥ 1 and Z := max h ∈ [ H ] |Z h | . Furthermore, if P is additionally γ -observable (c.f. Assumption 2.2), then π ⋆ is also an approximate optimal policy in the space of Π such that v P ( π ⋆ ) ≥ max π ∈ Π v P ( π ) -O ( ϵ + H 2 ϵ belief ) , as long as L ≥ ˜ Ω( γ -4 log( SH/ϵ )) .

5.2 Approximate Belief Learning

At a high level, our belief-learning algorithm first learns an approximate POMDP model ̂ P by explicitly exploring the state space. The main technical challenge here is that there may exist states that are reachable with very low probability, making it infeasible to collect enough samples to sufficiently explore them, thus potentially breaking the γ -observability property of the ground-truth model P . To circumvent this issue, we ignore such hard-to-visit states and redirect probabilities flowing to them to certain other states. Thus, in our truncated POMDP, where each state is sufficiently explored, we can approximate the transition and emission matrices to a desired accuracy uniformly across all the states and preserve the γ -observability property. This ensures that the learned approximate belief function in the truncated POMDP is sufficiently close to the actual belief function of the original POMDP P . Note that the key to achieving belief learning with both polynomial sample and time complexities is our explicit exploration in the state space, which relies on executing fully observable policies from an MDP learning subroutine. We remark that the belief function may also be learned even if the state space is only explored by partially observable policies, thus utilizing only hindsight observability may be sufficient for this purpose [44]. However, for such exploration to be computationally tractable , one requires to avoid using computationally intractable oracles for POMDP learning , which is in fact our final goal. We present the guarantees in the next theorem and postpone the proof to Appendix H. Theorem 5.2. Consider a γ -observable POMDP P (c.f. Assumption 2.2) and assume that L ≥ ˜ Ω( γ -4 log( SH/ϵ )) for an ϵ > 0 . Then, we can learn an approximate belief { b apx h } h ∈ [ H ] from Algorithm 4 using ˜ O ( S 2 AH 2 O + S 3 AH 2 ϵ 2 + S 4 A 2 H 6 O ϵγ 2 ) episodes in time POLY ( S, H, A, O, 1 γ , 1 ϵ , log ( 1 δ ) ) such that with probability at least 1 -δ , for any π ∈ Π L and h ∈ [ H ] , E P ∥ b h ( τ h ) -b apx ( z h ) ∥ 1 ≤ ϵ.

π h

Theorem 5.2 shows that an approximate belief can be learned with both polynomial samples and time, which, combined with Theorem 5.1, yields the final polynomial sample and quasi-polynomial time guarantee below. In contrast to the case without privileged information [25, 27], the sample complexity is reduced from quasi-polynomial to polynomial for γ -observable POMDPs. Note that the computational complexity remains quasi-polynomial, which is known to be unimprovable even for planning [27]. The key to such an improvement, as pointed out in Section 3.2, is the more practical update rule of actor-critic (in conjunction with our belief-weighted idea), which allows more computation at each iteration (instead of only performing computation at the sampled finite-memory). This allows the total computation to remain quasi-polynomial, while the overall sample complexity becomes polynomial. A detailed comparison can be found in Table 1.

0

20

40

60

80

100

Episodes

2.00

2.25

2.50

2.75

3.00

3.25

3.50

3.75

Rewards

Case 1

Belief-weighted optimistic AAC (ours)

Optimistic Asymmetric VI (ours)

Asymmetric Q

0

20

40

60

80

100

Episodes

4.5

5.0

5.5

6.0

6.5

7.0

Rewards

Case 2

Belief-weighted optimistic AAC (ours)

Optimistic Asymmetric VI (ours)

Asymmetric Q

0

20

40

60

80

100

Episodes

2.2

2.4

2.6

2.8

3.0

3.2

Rewards

Case 3

Belief-weighted optimistic AAC (ours)

Optimistic Asymmetric VI (ours)

Asymmetric Q

0

20

40

60

80

100

Episodes

4.5

5.0

5.5

6.0

6.5

Rewards

Case 4

Belief-weighted optimistic AAC (ours)

Optimistic Asymmetric VI (ours)

Asymmetric Q

Theorem 5.3. Let P be a γ -observable POMDP (c.f. Assumption 2.2), and consider L ≥ ˜ Ω( γ -4 log( SH/ϵ )) for an ϵ > 0 . With probability at least 1 -δ , Algorithm 2 can learn a policy π ∈ Π L such that v P ( π ) ≥ max π ′ ∈ Π v P ( π ′ ) -ϵ, using POLY ( S, H, 1 /ϵ, 1 /γ, log(1 /δ ) , O, A ) episodes and in time POLY ( S, H, 1 /ϵ, log(1 /δ ) , O L , A L ) .

6 Numerical Validation

We now provide some numerical results for both of our principled algorithms. Here we mainly compare with two baselines, the vanilla asymmetric actor-critic [68], and asymmetric Q -learning [7], on two settings, POMDP under the deterministic filter condition (c.f. Definition 3.2) and general POMDPs. We report the results in Table 2 and Figure 2, where our algorithms converge faster to higher rewards. We defer the implementation details and discussions to Appendix I.

7 Extension to Partially Observable MARL with Privileged Information

7.1 Privileged Policy Learning: Equilibrium Distillation

To understand how the deterministic filter condition may be extended for POSGs, we first note the following equivalent characterization of Definition 3.2, the proof of which is deferred to Appendix J.

Proposition 7.1. Definition 3.2 is equivalent to the following: for each h ∈ [ H ] , there exists an unknown function ϕ h : T h →S such that P P ( s h = ϕ h ( τ h ) | τ h ) = 1 for any reachable τ h ∈ T h .

Proposition 7.1 implies that at each step h , given the entire history information, the agent can uniquely decode the current underlying state s h . Thus, we generalize this condition to POSGs by requiring each agent to uniquely decode the current state s h given the information it has collected so far.

Definition 7.2 (Deterministic filter condition for POSGs) . We say a POSG G satisfies the deterministic filter condition if for each i ∈ [ n ] , h ∈ [ H ] , there exists an unknown function ϕ i,h : C h ×P i,h →S such that P G ( s h = ϕ i,h ( c h , p i,h ) | c h , p i,h ) = 1 for any reachable ( c h , p i,h ) .

Here we have required that each agent can decode the underlying state through their own information individually . Naturally, one may wonder whether one can relax it so that only the joint history information of all the agents can decode the underlying state. However, we point out in the following that it does not circumvent the computational hardness of POSG, the proof of which is deferred to Appendix J. Note that the computational hardness result can not be mitigated even with privileged state information, as the hardness we state here holds even for the planning problem with model knowledge, with which one can simulate the RL problems with privileged information.

Proposition 7.3. Computing CCE in POSGs that satisfy that for each step h ∈ [ H ] , there exists a function ϕ h : C h ×P h →S such that P G ( s h = ϕ h ( c h , p h ) | c h , p h ) = 1 for any reachable ( c h , p h ) is still PSPACE-hard .

Learning multi-agent individual decoding functions with unilateral exploration. Similar to our framework for POMDPs, our framework for POSGs is also decoupled into two steps: i) learning an expert equilibrium policy that is fully observable, ii) learning the decoding function , where the first step can be instantiated by any provable off-the-shelf algorithm of learning in Markov games. The major difference from the framework for POMDPs lies in how to learn the decoding function. In Theorem J.1, we prove that the difference of the NE/CE/CCE-gap between the expert policy and the distilled student policy is bounded by the decoding errors under policies from the unilateral deviation of the expert policy. Hence, given the expert policy π , the key algorithmic principle is to perform unilateral exploration for each agent i to make sure the decoding function is accurate under policies

( π ′ i , π -i ) for any π ′ i , keeping π -i fixed. We refer the detailed algorithm to Algorithm 5, and present below the guarantees for learning the decoding functions and the corresponding distilled policy for learning NE/CCE, while we defer the results for learning CE to Theorem J.6.

̸

Theorem 7.4 (Equilibria learning; Combining Theorem J.1 and Theorem J.4) . Under Assumption C.8 and conditions of Definition 7.2, given a ϵ 2 -NE/CCE π E for the associated Markov game of G , Algorithm 5 can learn decoding function { ̂ g i,h } i ∈ [ n ] ,h ∈ [ H ] such that with probability at least 1 -δ , it is guaranteed that max u i ∈ Π i ,j ∈ [ n ] P u i × π -i , G ( s h = ̂ g j,h ( c h , p j,h )) ≤ ϵ 4 nH 2 , for any i ∈ [ n ] , h ∈ [ H ] with both sample and computational complexities POLY ( S, A, H, O, 1 ϵ , log 1 δ ) . Consequently, policy π distilled from π E (c.f. Theorem J.1 for the formal distillation procedures) is an ϵ -NE/CCE of G .

7.2 Privileged Value Learning: Asymmetric MARL with Approximate Belief Learning

For POMDPs, we have used finite-memory policies for computational efficiency. We generalize to POSGs with information sharing by defining the compression of the common information.

Definition 7.5 (Compressed approximate common information [57, 79, 51]) . For each h ∈ [ H ] , given a set ̂ C h , we say Compress h is a compression function if Compress h ∈ { f : C h → ̂ C h } . For each c h ∈ C h , we denote ̂ c h := Compress h ( c h ) . We also require the compression function to satisfy the regularity condition that for each h ∈ [ H ] , there exists a function ̂ Λ h +1 such that ̂ c h +1 = ̂ Λ h +1 ( ̂ c h , ϖ h +1 ) , for any c h ∈ C h , ϖ h +1 ∈ Υ h +1 , where we recall c h +1 := c h ∪ ϖ h +1 and the definition of Υ h +1 in Assumption C.7.

Similar to the framework we developed for POMDPs in Section 5, we firstly develop the multi-agent RL algorithm based on some approximate belief, and then instantiate it with one provable approach for learning such an approximate belief.

Optimistic value iteration of POSGs with approximate belief. For POMDPs, the sufficient statistics for optimal decision-making is the posterior distribution over the state given history. However, for POSGs with information-sharing, as shown in [63, 62, 51], the sufficient statistics become the posterior distribution over the state and the private information given the common information, instead of only the state. Therefore, we consider the approximate belief in the form of ̂ P h : ̂ C h → ∆( P h ×S ) for each h ∈ [ H ] , where we define the error compared with the ground-truth belief to be ϵ belief := max h ∈ [ H ] max π ∈ Π E G π ∑ s h ,p h | P G ( s h , p h | c h ) -̂ P h ( s h , p h | ̂ c h ) | , i.e., the expected total variation distance from the true one. Note that both ̂ P h and thus ϵ belief have implicit dependencies on Compress h , as ̂ c h := Compress h ( c h ) . We outline our algorithm in Algorithm 7, which is conceptually similar to the algorithm for POMDPs (Algorithm 2), maintaining the asymmetric critic (i.e., value function), and performing the actor update (i.e., policy update) using the belief-weighted value function.

Theorem 7.6 (Equilibria learning; Combining Theorem J.15 and Theorem J.16) . Fix ϵ, δ ∈ (0 , 1) . Under Assumption C.8, with probability at least 1 -δ , Algorithm 7 can learn an ( ϵ + H 2 ϵ belief ) -NE if G is zero-sum and ( ϵ + H 2 ϵ belief ) -CE/CCE if G is general-sum with sample complexity O ( H 4 SAO log( SAHO/δ ) ϵ 2 ) and computational complexity POLY ( S, ( AO ) O ( γ -4 log( SH/ϵ )) , H, 1 ϵ , log 1 δ ) .

Learning approximate belief with model truncation. The belief learning algorithm we design for POSGs is conceptually similar to that we designed for POMDPs, where the key to achieving both polynomial sample and computational complexity is still to firstly learn approximate models, i.e., transitions and emissions, and then carefully truncate (as in Section 5.2) its transition and emission to build the approximate belief, where we defer the detailed algorithm to Algorithm 8. Next, we provide its provable guarantees, which leads to a final polynomial-sample and quasi-polynomial-time complexity result when combined with Theorem 7.6.

Theorem 7.7. For any ϵ > 0 , under Assumption 2.2, it holds that one can learn the approximate belief { ̂ P h : ̂ C h → ∆( S × P h ) } h ∈ [ H ] such that ϵ belief ≤ ϵ H 2 with both polynomial sample complexity and computational complexity POLY ( S, A, O, H, 1 γ , 1 ϵ , log 1 δ ) for all the examples in Appendix C.3. As a consequence, Algorithm 7 can learn an ϵ -NE if G is zero-sum and ϵ -CE/CCE if G is general-sum with sample complexity O ( H 4 SAO log( SAHO/δ ) ϵ 2 ) and computational complexity POLY ( S, ( AO ) O ( γ -4 log( SH/ϵ )) , H, 1 ϵ , log 1 δ ) .

References

A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. Optimality and approximation with policy gradient methods in Markov decision processes. In Conference on Learning Theory , pages 64-66, 2020.

E. Altman, V. Kambley, and A. Silva. Stochastic games with one step delay sharing information pattern with application to power control. In 2009 International Conference on Game Theory for Networks , pages 124-129. IEEE, 2009.

O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research , 39(1):3-20, 2020.

R. J. Aumann, M. Maschler, and R. E. Stearns. Repeated games with incomplete information . MIT press, 1995.

R. Avalos, F. Delgrange, A. Nowe, G. Perez, and D. M. Roijers. The wasserstein believer: Learning belief updates for partially observable environments through reliable latent space models. In The Twelfth International Conference on Learning Representations , 2023.

A. Baisero and C. Amato. Unbiased asymmetric reinforcement learning under partial observability. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems , 2022.

A. Baisero, B. Daley, and C. Amato. Asymmetric DQN for partially observable reinforcement learning. In Uncertainty in Artificial Intelligence , pages 107-117. PMLR, 2022.

N. Brukhim, D. Carmon, I. Dinur, S. Moran, and A. Yehudayoff. A characterization of multiclass learnability, 2022.

N. Brukhim, D. Carmon, I. Dinur, S. Moran, and A. Yehudayoff. A characterization of multiclass learnability. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS) , pages 943-955. IEEE, 2022.

S. Bubeck. Convex optimization: Algorithms and complexity. Found. Trends Mach. Learn. , 8(3-4):231-357, 2015.

Q. Cai, Z. Yang, C. Jin, and Z. Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning , pages 1283-1294. PMLR, 2020.

Q. Cai, Z. Yang, and Z. Wang. Reinforcement learning from partial observation: Linear function approximation with provable sample efficiency. In International Conference on Machine Learning , pages 2485-2522. PMLR, 2022.

C. L. Canonne. A short note on learning discrete distributions. arXiv preprint arXiv:2002.11457 , 2020.

D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl. Learning by cheating. In Conference on Robot Learning , pages 66-75. PMLR, 2020.

F. Chen, Y. Bai, and S. Mei. Partially observable RL with b-stability: Unified structural condition and sharp sample-efficient algorithms. In The Eleventh International Conference on Learning Representations , 2023.

X. Chen, Y. M. Mu, P. Luo, S. Li, and J. Chen. Flow-based recurrent belief state learning for pomdps. In International Conference on Machine Learning , pages 3444-3468. PMLR, 2022.

C. Dann, N. Jiang, A. Krishnamurthy, A. Agarwal, J. Langford, and R. E. Schapire. On oracleefficient pac rl with rich observations. Advances in neural information processing systems , 31, 2018.

S. Du, A. Krishnamurthy, N. Jiang, A. Agarwal, M. Dudik, and J. Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning , pages 1665-1674. PMLR, 2019.

Y. Efroni, C. Jin, A. Krishnamurthy, and S. Miryoosefi. Provable reinforcement learning with a short-term memory. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 5832-5850. PMLR, 2022.

E. Even-Dar, S. M. Kakade, and Y. Mansour. The value of observation for monitoring dynamic systems. In M. M. Veloso, editor, IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6-12, 2007 , pages 2474-2479, 2007.

J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 29, 2016.

J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

K. Fujii. Bayes correlated equilibria and no-regret dynamics. arXiv preprint arXiv:2304.05005 , 2023.

T. Gangwani, J. Lehman, Q. Liu, and J. Peng. Learning belief representations for imitation learning in pomdps. In uncertainty in artificial intelligence , pages 1061-1071. PMLR, 2020.

N. Golowich, A. Moitra, and D. Rohatgi. Learning in observable POMDPs, without computationally intractable oracles. In Advances in Neural Information Processing Systems , 2022.

N. Golowich, A. Moitra, and D. Rohatgi. Planning in observable pomdps in quasipolynomial time. arXiv preprint arXiv:2201.04735 , 2022.

N. Golowich, A. Moitra, and D. Rohatgi. Planning and learning in partially observable systems via filter stability. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages 349-362, 2023.

N. Golowich, A. Moitra, and D. Rohatgi. Exploration is harder than prediction: Cryptographically separating reinforcement learning from supervised learning. arXiv preprint arXiv:2404.03774 , 2024.

G. J. Gordon, A. Greenwald, and C. Marks. No-regret learning in convex games. In Proceedings of the 25th international conference on Machine learning , pages 360-367, 2008.

J. Guo, M. Chen, H. Wang, C. Xiong, M. Wang, and Y. Bai. Sample-efficient learning of pomdps with multiple observations in hindsight. In The Twelfth International Conference on Learning Representations , 2023.

A. Gupta, A. Nayyar, C. Langbort, and T. Basar. Common information based markov perfect equilibria for linear-gaussian games with asymmetric information. SIAM Journal on Control and Optimization , 52(5):3228-3260, 2014.

J. Hartline, V. Syrgkanis, and E. Tardos. No-regret learning in bayesian games. Advances in Neural Information Processing Systems , 28, 2015.

H. Hu, A. Lerer, N. Brown, and J. Foerster. Learned belief search: Efficiently improving policies in partially observable settings. arXiv preprint arXiv:2106.09086 , 2021.

N. Jiang. On value functions and the agent-environment boundary. arXiv preprint arXiv:1905.13341 , 2019.

N. Jiang, A. Krishnamurthy, A. Agarwal, J. Langford, and R. E. Schapire. Contextual decision processes with low bellman rank are pac-learnable. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 1704-1713. PMLR, 2017.

C. Jin, S. Kakade, A. Krishnamurthy, and Q. Liu. Sample-efficient reinforcement learning of undercomplete POMDPs. Advances in Neural Information Processing Systems , 33:1853018539, 2020.

C. Jin, A. Krishnamurthy, M. Simchowitz, and T. Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning , pages 4870-4879. PMLR, 2020.

C. Jin, Q. Liu, Y. Wang, and T. Yu. V-learning-a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint arXiv:2110.14555 , 2021.

S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning , volume 2, pages 267-274, 2002.

S. M. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems , pages 1531-1538, 2002.

B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2021.

V. R. Konda and J. N. Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems , pages 1008-1014, 2000.

A. Krishnamurthy, A. Agarwal, and J. Langford. Pac reinforcement learning with rich observations. Advances in Neural Information Processing Systems , 29, 2016.

J. Lee, A. Agarwal, C. Dann, and T. Zhang. Learning in pomdps is sample-efficient with hindsight observability. In International Conference on Machine Learning , pages 18733-18773. PMLR, 2023.

J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics , 5(47):eabc5986, 2020.

S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research , 17(1):1334-1373, 2016.

Q. Liu, A. Chung, C. Szepesvari, and C. Jin. When is partially observable reinforcement learning not scary? In Conference on Learning Theory , pages 5175-5220, 2022.

Q. Liu, P. Netrapalli, C. Szepesvari, and C. Jin. Optimistic MLE: A generic model-based algorithm for partially observable sequential decision making. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages 363-376, 2023.

Q. Liu, C. Szepesvári, and C. Jin. Sample-efficient reinforcement learning of partially observable Markov games. In Advances in Neural Information Processing Systems , 2022.

Q. Liu, T. Yu, Y. Bai, and C. Jin. A sharp analysis of model-based reinforcement learning with self-play. In International Conference on Machine Learning , pages 7001-7010. PMLR, 2021.

X. Liu and K. Zhang. Partially observable multi-agent RL with (quasi-)efficiency: the blessing of information sharing. In International Conference on Machine Learning , pages 22370-22419. PMLR, 2023.

X. Liu and K. Zhang. Partially observable multi-agent reinforcement learning with information sharing, 2024.

R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems , 30, 2017.

M. Lu, Y. Min, Z. Wang, and Z. Yang. Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes. In The Eleventh International Conference on Learning Representations , 2023.

X. Lyu, A. Baisero, Y. Xiao, and C. Amato. A deeper understanding of state-based critics in multi-agent reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 9396-9404, 2022.

X. Lyu, A. Baisero, Y. Xiao, B. Daley, and C. Amato. On centralized critics in multi-agent reinforcement learning. Journal of Artificial Intelligence Research , 77:295-354, 2023.

W. Mao, K. Zhang, E. Miehling, and T. Ba¸ sar. Information state embedding in partially observable cooperative multi-agent reinforcement learning. In 2020 59th IEEE Conference on Decision and Control (CDC) , pages 6124-6131. IEEE, 2020.

G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, S. bae Kim, and P. Agrawal. Learning to jump from pixels. In 5th Annual Conference on Robot Learning , 2021.

T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics , 7(62):eabk2822, 2022.

D. Misra, M. Henaff, A. Krishnamurthy, and J. Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. In International conference on machine learning , pages 6961-6971. PMLR, 2020.

P. Moreno, J. Humplik, G. Papamakarios, B. A. Pires, L. Buesing, N. Heess, and T. Weber. Neural belief states for partially observed domains. In NeurIPS 2018 Workshop on Reinforcement Learning under Partial Observability , 2018.

A. Nayyar, A. Gupta, C. Langbort, and T. Ba¸ sar. Common information based markov perfect equilibria for stochastic games with asymmetric information: Finite games. IEEE Transactions on Automatic Control , 59(3):555-570, 2013.

A. Nayyar, A. Mahajan, and D. Teneketzis. Decentralized stochastic control with partial history sharing: A common information approach. IEEE Transactions on Automatic Control , 58(7):1644-1658, 2013.

H. Nguyen, A. Baisero, D. Wang, C. Amato, and R. Platt. Leveraging fully observable policies for learning under partial observability. In Conference on Robot Learning , 2022.

H. Nguyen, B. Daley, X. Song, C. Amato, and R. Platt. Belief-grounded networks for accelerated robot learning under partial observability. In Conference on Robot Learning , pages 1640-1653. PMLR, 2021.

C. H. Papadimitriou and J. N. Tsitsiklis. The complexity of markov decision processes. Mathematics of operations research , 12(3):441-450, 1987.

A. Pathak, H. Pucha, Y. Zhang, Y. C. Hu, and Z. M. Mao. A measurement study of internet delay asymmetry. In Passive and Active Network Measurement: 9th International Conference, PAM 2008, Cleveland, OH, USA, April 29-30, 2008. Proceedings 9 , pages 182-191. Springer, 2008.

L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for image-based robot learning. Robotics: Science and Systems XIV , 2018.

S. Qiu, Z. Dai, H. Zhong, Z. Wang, Z. Yang, and T. Zhang. Posterior sampling for competitive rl: Function approximation and partial observation. Advances in Neural Information Processing Systems , 36, 2024.

T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine learning , pages 681-689, 2018.

T. Roughgarden. Algorithmic game theory. Communications of the ACM , 53(7):78-86, 2010.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

S. Shalev-Shwartz, S. Shammah, and A. Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295 , 2016.

L. Shani, Y. Efroni, A. Rosenberg, and S. Mannor. Optimistic policy optimization with bandit feedback. In International Conference on Machine Learning , pages 8604-8613. PMLR, 2020.

I. Shenfeld, Z.-W. Hong, A. Tamar, and P. Agrawal. Tgrl: An algorithm for teacher guided reinforcement learning. In International Conference on Machine Learning , pages 31077-31093. PMLR, 2023.

M. Shi, Y. Liang, and N. Shroff. Theoretical hardness and tractability of pomdps in rl with partial online state information, 2024.

S. M. Shortreed, E. Laber, D. J. Lizotte, T. S. Stroup, J. Pineau, and S. A. Murphy. Informing sequential clinical decision-making through reinforcement learning: an empirical study. Machine learning , 84:109-136, 2011.

Z. Song, S. Mei, and Y. Bai. When can we learn general-sum Markov games with a large number of players sample-efficiently? arXiv preprint arXiv:2110.04184 , 2021.

J. Subramanian, A. Sinha, R. Seraj, and A. Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. J. Mach. Learn. Res. , 23:12-1, 2022.

J. Tsitsiklis and M. Athans. On the complexity of decentralized decision making and detection problems. IEEE Transactions on Automatic Control , 30(5):440-446, 1985.

M. Uehara, A. Sekhari, J. D. Lee, N. Kallus, and W. Sun. Computationally efficient pac rl in pomdps with latent determinism and conditional embeddings. In International Conference on Machine Learning , pages 34615-34641. PMLR, 2023.

O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

A. Wang, A. C. Li, T. Q. Klassen, R. T. Icarte, and S. A. McIlraith. Learning belief representations for partially observable deep rl. In International Conference on Machine Learning , pages 35970-35988. PMLR, 2023.

L. Wang, Q. Cai, Z. Yang, and Z. Wang. Represent to control partially observed systems: Representation learning with provable sample efficiency. In The Eleventh International Conference on Learning Representations , 2022.

H. S. Witsenhausen. A counterexample in stochastic optimum control. SIAM Journal on Control , 6(1):131-147, 1968.

G. Yang, M. Liu, W. Hong, W. Zhang, F. Fang, G. Zeng, and Y. Lin. Perfectdou: Dominating doudizhu with perfect information distillation. Advances in Neural Information Processing Systems , 35:34954-34965, 2022.

Y. Yang, Y. Jiang, J. Chen, S. E. Li, Z. Gu, Y . Yin, Q. Zhang, and K. Yu. Belief state actor-critic algorithm from separation principle for POMDP. In 2023 American Control Conference (ACC) , pages 2560-2567. IEEE, 2023.

S. Young, M. Gaši´ c, B. Thomson, and J. D. Williams. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE , 101(5):1160-1179, 2013.

A. Zanette and E. Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning , pages 7304-7312. PMLR, 2019.

W. Zhan, M. Uehara, W. Sun, and J. D. Lee. PAC reinforcement learning for predictive state representations. In The Eleventh International Conference on Learning Representations , 2023.

Supplementary Materials for 'Provable Partially Observable Reinforcement Learning with Privileged Information'

Contents

A Societal Impact

Our work is theoretical by nature, and aimed at better understanding reinforcement learning under partial observability with privileged information. As such, we do not anticipate any direct positive or negative societal impact from this research.

B Related Work

Provable partial observable RL. While POMDPs are generally known to be both statistically hard [43] and computationally intractable [66], a productive line of research has identified several structured subclasses of POMDPs that can be efficiently solved. [43] introduced the class of POMDPs in the rich-observation setting, where the observation space can be large and fully reveal the underlying state, where sample-efficient RL becomes possible [35, 60]. [19] introduced k -step decodable POMDPs, where the last k observation-action pairs can uniquely determine the state, and proposed polynomial-sample complexity algorithms (assuming k is a small constant). Beyond settings where the underlying state can be exactly recovered, [36, 47] proposed weakly revealing POMDPs, where the observations are assumed to be informative enough. Under the weakly revealing condition (and its variant), there has been a fast-growing line of recent works developing sample-efficient RL algorithms for various settings, see e.g., [84, 15, 12, 54, 48, 90]. Notably, these algorithms are typically computationally inefficient, requiring access to an optimistic planning oracle for POMDPs. On a promising note, [26] showed that in observable POMDPs (see Assumption 2.2), one can have quasi-polynomial-time algorithms for planning the near-optimal policy, which further leads to provable RL algorithms [25, 27] with both quasi-polynomial samples and time.

RL under hindsight observability. The closest line of research to ours are the recent theoretical studies for Hindsight Observable Markov Decision Processes (HOMDPs) [44], where the underlying state is revealed at the end of the episode; see also subsequent related works in [30, 76] with different observation feedback models. These works focused purely on sample efficiency , and showed that polynomial sample complexity can be achieved without (or by further relaxing) aforementioned structural assumptions of the model (e.g., observability or decodability), in both tabular and/or function approximation settings. However, the algorithms (also) require an oracle for planning or even optimistic planning in a learned approximate POMDP, which are not computationally tractable in general. Indeed, without any structural assumption, learning the optimal policy in HOMDPs is computationally no easier than the planning problem, which thus remains PSPACE-hard . [66]. Meanwhile, even under the additional assumption of observability on the underlying POMDP model, it is still not clear if these algorithms can avoid computationally intractable oracles, since the approximate POMDP that [44] needs to do planning on at every iteration during learning can be quite different from the underlying model. For example, at the beginning of exploration when not enough samples are collected, or when there exist certain states that remain less explored during the entire learning process, the potentially misspecified emission (and transition) may break the observability (or other structural) assumptions made for the underlying POMDP. This makes that single iteration even computationally intractable. In contrast, our focus is on better understanding practically inspired algorithmic paradigms, without computationally intractable oracles, which in practice often do have and use the privileged state information during each episode (instead of only at the end) [68, 45, 14].

Most related empirical works. Privileged information has been widely used in empirical partially observable RL, with two main types of approaches based on privileged policy and privileged value learning, respectively. For the former, one prominent example is expert distillation [14, 64, 58], also known as teacher-student learning [45, 59, 75], as we analyze in Section 4. For the latter, asymmetric actor-critic [68] represents one of the well-known examples, with other studies in [7, 3]. Learning privileged value functions (to improve the policies) has also been widely used in multi-agent RL, featured in centralized-training-decentralized-execution, see e.g., [53, 22, 70, 86, 82]. Intriguingly, it was shown that if the privileged value function depends only on the state, the associated actor will cause bias [6, 55, 56]. This has thus necessitated the use of history/belief in asymmetric actor-critic, as in our Section 5. Notably, the empirical framework in [83] exactly matches ours, where they exploited the privileged state information in training for belief learning , followed by policy optimization over the learned belief states. Indeed, many empirical works explicitly separate the procedures of explicit

belief-state learning and planning [24, 65, 33, 16, 87] as we study in Section 5, oftentimes with privileged state information to supervise the belief learning procedure [61, 5].

C Additional Preliminaries

C.1 Additional Preliminaries on POMDPs

Belief and approximate belief. Although in a POMDP, the agent cannot see the underlying state directly, it can still form a belief over the underlying state by the historical observations and actions. Definition C.1 (Belief state update) . For each h ∈ [ H +1] , the Bayes operator B h : ∆( S ) ×O → ∆( S ) is defined for b ∈ ∆( S ) , and o ∈ O by:

for each x ∈ S . For each h ∈ [ H +1] , the belief update operator U h : ∆( S ) ×A×O→ ∆( S ) , is defined by

where T h ( a ) · b represents the matrix multiplication. We use the notation b h to denote the belief update function, which receives a sequence of actions and observations and outputs a distribution over states at the step h : the belief state at step h = 1 is defined as b 1 ( ∅ ) = µ 1 . For any 2 ≤ h ≤ H and any action-observation sequence ( a 1: h , o 1: h ) , we inductively define the belief state:

We also define the approximate belief update using the most recent L -step history. For 1 ≤ h ≤ H , we follow the notation of [26] and define

where D ∈ ∆( S ) is the prior for the approximate belief update. Then for any 1 ≤ h -L < h ≤ H and any action-observation sequence ( a h -L : h , o h -L +1: h ) , we inductively define

For the remainder of our paper, we will use an important initialization for the approximate belief, which are defined as b ′ h ( · ) := b apx , P h ( · ; Unif( S )) .

C.2 Additional Preliminaries for POSGs

Model. We use a general framework of partially observable stochastic games (POSGs) as the model for partially observable MARL. Formally, we define a POSG with n agents by a tuple G = ( H, S , {A i } n i =1 , {O i } n i =1 , T , O , µ 1 , { r i } n i =1 ) , where H denotes the length of each episode, S is the state space with |S| = S , A i denotes the action space for agent i with |A i | = A i . We denote by a := ( a 1 , · · · , a n ) the joint action of all the n agents, and by A = A 1 ×··· × A n the joint action space with |A| = A = ∏ n i =1 A i . We use T = { T h } h ∈ [ H ] to denote the collection of transition matrices, so that T h ( · | s, a ) ∈ ∆( S ) gives the probability of the next state if joint action a ∈ A is taken at state s ∈ S and step h . In the following discussions, for any given a , we treat T h ( a ) ∈ R |S|×|S| as a matrix, where each row gives the probability for the next state from different current states. We use µ 1 to denote the distribution of the initial state s 1 , and O i to denote the observation space for agent i with |O i | = O i . We denote by o := ( o 1 , . . . , o n ) the joint observation of all n agents, and by O := O 1 ×··· × O n with |O| = O = ∏ n i =1 O i . We use O = { O h } h ∈ [ H ] to denote the collection of the joint emission matrices, so that O h ( · | s ) ∈ ∆( O ) gives the emission distribution over the joint observation space O at state s and step h . For notational convenience, we will at times adopt the matrix convention, where O h is a matrix with rows O h ( · | s h ) . We also denote

O i,h ( · | s ) ∈ ∆( O i ) as the marginalized emission for agent i agent. Finally, r i = { r i,h } h ∈ [ H ] is a collection of reward functions, so that r i,h ( s h , a h ) is the reward of agent i agent given the state s h and joint action a h at step h .

Similar to a POMDP, in a POSG, the states are not observable to the agents, and each agent can only access its own individual observations. The game proceeds as follows. At the beginning of each episode, the environment samples s 1 from µ 1 . At each step h , each agent i observes its own observation o i,h , where o h := ( o 1 ,h , . . . , o n,h ) is sampled jointly from O h ( · | s h ) . Then each agent i takes the action a i,h and receives the reward r i,h ( s h , a h ) . After that the environment transitions to the next state s h +1 ∼ T h ( · | s h , a h ) . The current episode terminates once s H +1 is reached.

Information sharing, common and private information. Each agent i in the POSG maintains its own information, τ i,h , a collection of historical observations and actions at step h , namely, τ i,h ⊆ { o 1 , a 1 , o 2 , · · · , a h -1 , o h } , and the collection of such history at step h is denoted by T i,h .

In many practical examples, agents may share part of the history with each other, which may introduce some information structures of the game that may lead to both sample and computation efficiencies. The information sharing splits the history into common/shared and private information for each agent. The common information at step h is a subset of the joint history τ h = ( τ i,h ) i ∈ [ n ] : c h ⊆ { o 1 , a 1 , o 2 , · · · , a h -1 , o h } , which is available to all the agents in the system, and the collection of the common information is denoted as C h and we define C h = |C h | . Given the common information c h , each agent also has her private information p i,h = τ i,h \ c h , where the collection of the private information for agent i is denoted as P i,h and its cardinality as P i,h . The cardinality of the joint private information is P h = ∏ n i =1 P i,h . We allow c h or p i,h to take the special value ∅ when there is no common or private information. In particular, when C h = {∅} , the problem reduces to the general POSG without any favorable information structure; when P i,h = {∅} , every agent holds the same history, and it reduces to a POMDP when the agents share a common reward function, where the goal is usually to find the team-optimal solution.

Policies and value functions. We define a stochastic policy for agent i at step h as:

where Ω h is the random seed space, which is shared among agents and ω i,h ∈ Ω h is the random seed for agent i . The corresponding policy class is denoted as Π i,h . Hereafter, unless otherwise noted, when referring to policies , we mean the policies given in the form of (C.1), which maps the available information of agent i , i.e., the private information together with the common information, to the distribution over her actions.

We define π i as a sequence of policies for agent i at all steps h ∈ [ H ] , i.e., π i = ( π i, 1 , · · · , π i,H ) . We further denote Π i = × h ∈ [ H ] Π i,h as the policy space for agent i and Π = ∏ i ∈ [ n ] Π i as the joint policy space. As a special case, we define the space of deterministic policy as ˜ Π i , where ˜ π i ∈ ˜ Π i maps the private information and common information to a deterministic action for agent i and the joint space as ˜ Π = ∏ i ∈ [ n ] ˜ Π i .

A product policy is denoted as π = π 1 × π 2 · · · × π n ∈ Π if the distributions of drawing each seed ω i,h for different agents are independent, and a (potentially correlated) joint policy is denoted as π = π 1 ⊙ π 2 · · · ⊙ π n ∈ Π .

We are now ready to define the value function conditioned on the common information under our model of POSG with information sharing:

Definition C.2 (Value function with information sharing) . For each agent i ∈ [ n ] and step h ∈ [ H ] , given common information c h and joint policy π = ( π i ) n i =1 ∈ Π , the value function conditioned on the common information of agent i is defined as: V π, G i,h ( c h ) := E G π [ ∑ H h ′ = h r i,h ′ ( s h ′ , a h ′ ) ∣ ∣ c h ] , where the expectation is taken over the randomness from the model G , policy π , and the random seeds. For any c H +1 ∈ C H +1 : V π, G i,H +1 ( c H +1 ) := 0 . From now on, we will refer to it as value function for short.

Another key concept in our analysis is the belief about the state and the private information conditioned on the common information among agents. Formally, at step h , given policies from 1 to h -1 ,

we consider the common-information-based conditional belief P π 1: h -1 , G h ( s h , p h | c h ) . This belief not only infers the current underlying state s h , but also all agents' private information p h . With the common-information-based conditional belief, the value function given in Definition C.2 has the following recursive structure:

where the expectation is taken over the randomness of ( s h , p h , a h , o h +1 ) . With this relationship, we can define the prescription-value function correspondingly, a generalization of the action-value function in (fully observable) stochastic games and MDPs to POSGs with information sharing, as follows.

Definition C.3 (Prescription-value function with information sharing) . At step h , given the common information c h , joint policies π = ( π i ) n i =1 ∈ Π , and prescriptions ( γ i,h ) n i =1 ∈ Γ h , the prescriptionvalue function conditioned on the common information and joint prescription of agent i is defined as:

where prescription γ i,h ∈ ∆( A i ) P i,h replaces the partial function π i,h ( · | ω i,h , c h , · ) in the value function. From now on, we will refer to it as prescription-value function for short. With such a prescription-value function, agents can take actions purely based on their local private information [63, 62, 51].

This prescription-value function indicates the expected return for agent i when all the agents firstly adopt the prescriptions ( γ j,h ) j ∈ [ n ] and then follow the policy π .

Equilibrium notions. With the definition of value functions, we can accordingly define the solution concepts. Here we define the notions of ϵ -NE, ϵ -CCE, ϵ -CE, and ϵ -team optimum under the information-sharing framework as follows. For a joint policy ( π i ) n i =1 ∈ Π we denote the expected reward of agent i by v G i ( π ) = E P π [ ∑ H h =1 r i,h ( s h , a h )] .

Definition C.4 ( ϵ -approximate Nash equilibrium with information sharing) . For any ϵ ≥ 0 , a product policy π ⋆ ∈ Π is an ϵ -approximate Nash equilibrium of the POSG G with information sharing if:

Definition C.5 ( ϵ -approximate coarse correlated equilibrium with information sharing) . For any ϵ ≥ 0 , a joint policy π ⋆ ∈ Π is an ϵ -approximate coarse correlated equilibrium of the POSG G with information sharing if:

Definition C.6 ( ϵ -approximate correlated equilibrium with information sharing) . For any ϵ ≥ 0 , a joint policy π ⋆ ∈ Π is an ϵ -approximate correlated equilibrium of the POSG G with information sharing if:

where m i is called strategy modification for agent i , and m i = { m i,h,c h ,p i,h } h,c h ,p i,h , with each m i,h,c h ,p i,h : A i → A i being a mapping from the action set to itself. The space of m i is denoted as M i . The composition m i ⋄ π i is defined as follows: at step h , when agent i is given c h and p i,h , the joint action chosen to be ( a 1 ,h , · · · , a i,h , · · · , a n,h ) will be modified to ( a 1 ,h , · · · , m i,h,c h ,p i,h ( a i,h ) , · · · , a n,h ) . Note that this definition follows from that in [78, 50, 38, 49, 51] when there exists certain common information, and is a natural generalization of the definition in the normal-form game case [71]. We denote by M gen i the space of all possible strategy modifications m i if it conditions on any history information instead of only ( c h , p i,h ) . Similarly, we use M S ,i to denote the space of all possible strategy modifications m i if it only conditions on the current state e.g., a modification m i,h,s : A i →A i .

C.2.1 Evolution of the Common and Private Information

Assumption C.7 (Evolution of common and private information) . We assume that common information and private information evolve over time as follows:

Common information c h is non-decreasing over time, that is, c h ⊆ c h +1 for all h . Let ϖ h +1 = c h +1 \ c h . Thus, c h +1 = { c h , ϖ h +1 } . Further, we have

where χ h +1 is a fixed transformation. We use Υ h +1 to denote the collection of ϖ h +1 at step h .

Private information evolves according to:

where ξ i,h +1 is a fixed transformation.

Equation (C.3) states that the increment in the common information depends on the 'new" information ( a h , o h +1 ) generated between steps h and h +1 and part of the old information p h . The incremental common information can be obtained by certain sharing and communication protocols among the agents. Equation (C.4) implies that the evolution of private information only depends on the newly generated private information a i,h and o i,h +1 . These evolution rules are standard in the literature [62, 63], specifying the source of common information and private information. Based on such evolution rules, we define { f h } h ∈ [ H ] and { g h } h ∈ [ H ] , where f h : A h × O h → C h and g h : A h ×O h →P h for h ∈ [ H ] , as the mappings that map the joint history to common information and joint private information, respectively.

C.3 Strategy Independence of Belief and Examples

To solve a POSG without computationally intractable oracles, certain information-sharing is needed even under the observability assumption [51]. We thus make the following assumption as in [51].

Assumption C.8 (Strategy independence of beliefs [62, 31, 51]) . Consider any step h ∈ [ H ] , any policy π ∈ Π , and any realization of common information c h that has a non-zero probability under the trajectories generated by π 1: h -1 . Consider any other policies π ′ 1: h -1 , which also give a nonzero probability to c h . Then, we assume that: for any such c h ∈ C h , and any p h ∈ P h , s h ∈ S , P π 1: h -1 , G ( s h , p h | c h ) = P π ′ 1: h -1 , G ( s h , p h | c h ) .

We provide examples satisfying this assumption in Appendix C.2, which include the fully-sharing structure as in [27, 69] as a special case. Finally, we also assume that common information and private information evolve over time properly in Assumption C.7, as standard in [62, 63, 51], which covers the models considered in [27, 69, 49].

Here we take the examples from [52, 51] to illustrate the generality of the information-sharing framework.

Example C.9 (One-step delayed sharing) . At any step h ∈ [ H ] , the common and private information are given as c h = { o 2: h -1 , a 1: h -1 } and p i,h = { o i,h } , respectively. In other words, the players share all the action-observation history until the previous step h -1 , with only the new observation being the private information. This model has been shown useful for power control [2].

Example C.10 (State controlled by one controller with asymmetric delay sharing) . We assume there are 2 players for convenience. It extends naturally to n -player settings. Consider the case where the state dynamics are controlled by player 1 , i.e., T h ( · | s h , a 1 ,h , a 2 ,h ) = T h ( · | s h , a 1 ,h , a ′ 2 ,h ) for all ( s h , a 1 ,h , a 2 ,h , a ′ 2 ,h , h ) . There are two kinds of delay-sharing structures we could consider: Case A: the information structure is given as c h = { o 1 , 2: h , o 2 , 2: h -d , a 1 , 1: h -1 } , p 1 ,h = ∅ , p 2 ,h = { o 2 ,h -d +1: h } , i.e., player 1 's observations are available to player 2 instantly, while player 2 's observations are available to player 1 with a delay of d ≥ 1 time steps. Case B: similar to Case A but player 1 's observation is available to player 2 with a delay of 1 step. The information structure is given as c h = { o 1 , 2: h -1 , o 2 , 2: h -d , a 1 , 1: h -1 } , p 1 ,h = { o 1 ,h } , p 2 ,h = { o 2 ,h -d +1: h } , where d ≥ 1 . This kind of asymmetric sharing is common in network routing [67], where packages arrive at different hosts with different delays, leading to asymmetric delay sharing among hosts.

Example C.11 (Symmetric information game) . Consider the case when all observations and actions are available for all the agents, and there is no private information. Essentially, we have c h = { o 2: h , a 1: h -1 } and p i,h = ∅ . We will also denote this structure as fully sharing hereafter.

Example C.12 (Information sharing with one-directional-one-step delay) . Similar to the previous cases, we also assume there are 2 players for ease of exposition, and the case can be generalized to multi-player cases straightforwardly. Similar to the one-step delay case, we consider the situation where all observations of player 1 are available to player 2 , while the observations of player 2 are available to player 1 with one-step delay. All the past actions are available to both players. That is, in this case, c h = { o 1 , 2: h , o 2 , 2: h -1 , a 1: h -1 } , and player 1 has no private information, i.e., p 1 ,h = ∅ , and player 2 has private information p 2 ,h = { o 2 ,h } .

Example C.13 (Uncontrolled state process) . Consider the case where the state transition does not depend on the actions, that is, T h ( · | s h , a h ) = T h ( · | s h , a ′ h ) for any s h , a h , a ′ h , h . Note that the agents are still coupled through the joint reward. An example of this case is the information structure where controllers share their observations with a delay of d ≥ 1 time steps. In this case, the common information is c h = { o 2: h -d } and the private information is p i,h = { o i,h -d +1: h } . Such information structures can be used to model repeated games with incomplete information [4].

D Collection of Algorithms

Algorithm 1 Learning Decoding Function with Privileged Information

Require:

POMDP P ,

Expert policy π E ∈ Π S ,

Number of sampled episodes per step M .

Ensure: A decoding function for each step { g h } h ∈ [ H ] (see Theorem 4.6)

For the h = 1 step: Collect M state-observation pairs from the first-step ̂ D 1 = {( s ( i ) 1 , o ( i ) 1 )} i ∈ [ M ] on POMDP P and define the decoding function g 1 for the first step as:

for each step h ∈ [2 , H ] do

Collect M episodes {( s ( i ) 1: H +1 , o ( i ) 1: H , a ( i ) 1: H )} i ∈ [ M ] on POMDP P using policy π E and let:

Define the decoding function g h for step h as:

Algorithm 2 Belief-Weighted Optimistic Asymmetric Actor-Critic with Privileged Information

Require:

POMDP P ,

Subroutine T Q that given policy π ∈ Π L , outputs { ˜ Q } h ∈ [ H ] that approximates { Q π, P h } h ∈ [ H ] ,

Subroutine T b that outputs { b apx h } h ∈ [ H ] that approximate { b h } h ∈ [ H ] ,

Initial finite-memory policy π 0 = { π 0 h } h ∈ [ H ] ∈ Π L for the POMDP P , step-size η , and number of iterations T .

Ensure: A near-optimal policy

end for return A policy uniform at random sampled from set { π t } t ∈ [ T ]

Algorithm 3 Optimistic Q -function Estimation with Privileged Information

Require:

POMDP P , policy π 1: H ∈ Π L such that π h : S → ∆( A ) ,

Number of episodes M per step.

Ensure: Approximate Q -functions { Q } (see Lemma H.3)

˜ h h ∈ [ H ] Initialize:

for step h = H,.. . , 1 do

Collect M trajectories using policy π and let D h = { τ ( i ) } i ∈ [ M ] be the collected trajectories. Compute empirical counts and define empirical distributions:

for each memory-state pair ( z h , s h ) ∈ Z h ×S do

Algorithm 4 Approximate Belief Learning with Privileged Information via Model Truncation

Require:

POMDP P = ( H, S , A , O , { T h } h ∈ [ H ] , { O h } h ∈ [ H ] , µ 1 , { r h } h ∈ [ H ] ) ,

An MDP learning oracle MDP_Learning that efficiently learns an approximate optimal policy of an MDP,

Number of trajectories N ,

The threshold ϵ .

Ensure: Approximate belief { b apx h } h ∈ [ H ] (See Theorem H.5)

Collect N trajectories by executing policy Ψ( h, s h ) for the first h -1 steps then take action a h for each a h ∈ A deterministically and denote the dataset { ( s i h , o i h , a i h , s i h +1 ) } i ∈ [ NA ]

end for

end for

for h ∈ [ H ] do

end for

end for

Let

Algorithm 5 Learning Multi-Agent (Individual) Decoding Functions with Privileged Information (NE/CCE Version)

Require:

POSG G = ( H, S , A , O , { T h } h ∈ [ H ] , { O h } h ∈ [ H ] , µ 1 , { r i } i ∈ [ n ] ) ,

π ∈ Π S , controller set {I h ⊆ [ n ] } h ∈ [ H ] ,

Procedure MDP_Learning ( · , · ) that takes as input an MDP and a reward function and returns an approximate optimal policy,

Number of trajectories N .

Ensure: A decoding function for each agent and step { ̂ g j,h } j ∈ [ n ] ,h ∈ [ H ] (see Theorem J.4)

̂ r i,h ′ ( s ′ h , a ′ h ) ← ✶ [ h ′ = h, s ′ h = s h ] for any ( h ′ , s ′ h , a ′ h ) ∈ [ H ] ×S × A Define the Markov game M associated with G as M = ( H,S,A, { T h } h ∈ [ H ] , µ 1 ) , where we omit the specification for the reward functions and one can specify them arbitrarily

Define M ( π -i ) to be the MDP marginalized by π -i

end for

For each i ∈ [ n ] , a h ∈ A , collect N trajectories by executing policy Ψ i ( h, s h ) × π -i for the first h -1 steps then take action a h deterministically and denote the dataset { ( s k,i h , o k,i h , a k,i h , s k,i h +1 ) } k ∈ [ NA ]

end for

end for

Algorithm 6 Learning Multi-Agent (Individual) Decoding Functions with Privileged Information (CE Version)

Require:

POSG G = ( H, S , A , O , { T h } h ∈ [ H ] , { O h } h ∈ [ H ] , µ 1 , { r i } i ∈ [ n ] ) ,

π ∈ Π S , controller set {I h ⊆ [ n ] } ,

Procedure MDP_Learning ( · , · ) that takes as input an MDP and a reward function and returns an approximate optimal policy,

Number of trajectories N .

Ensure: A decoding function for each agent and step { ̂ g j,h } j ∈ [ n ] ,h ∈ [ H ] (see Theorem J.6)

end for

For each i ∈ [ n ] , a h ∈ A , collect N trajectories by executing policy Ψ i ( h, s h ) × π -i for the first h -1 steps then take action a h deterministically and denote the dataset { ( s k,i h , o k,i h , a k,i h , s k,i h +1 ) } k ∈ [ NA ]

end for

end for

Algorithm 7 Optimistic Common-Information-Based Value Iteration with Privileged Information

Require:

An approximate belief { ̂ P h : ̂ C h → ∆( S × P h ) } h ∈ [ H ] ,

Number of iterations K .

Ensure: An approximate equilibrium policy Initialize:

Define

Define

end for

end for

end for end for

return π k ⋆ for general-sum games or the marginalized policy of π k ⋆ for zero-sum games

Algorithm 8 Approximate Belief Learning for MARL with Privileged Information

Require:

POSG G = ( H, S , A , O , { T h } h ∈ [ H ] , { O h } h ∈ [ H ] , µ 1 , { r i,h } i ∈ [ n ] ,h ∈ [ H ] ) ,

Controller set {I h ⊆ [ n ] } h ∈ [ H ] ,

Procedure MDP_Learning ( · , · ) ,

Number of trajectories N ,

Accuracy ϵ .

Ensure:

An approximate belief

r h ′ ( s ′ h , a ′ h ) ← ✶ [ h ′ = h, s ′ h = s h ] for any ( h ′ , s ′ h , a ′ h ) ∈ [ H ] ×S × A . M← ( H, S , A , { T h } h ∈ [ H ] , µ 1 ) to be the MDP by ignoring the observation and emission of G , where we omit the specification for the reward functions and one can specify them arbitrarily Ψ( h, s h ) ← MDP_Learning ( M , r )

̂ Collect N trajectories by executing policy Ψ( h, s h ) for the first h -1 steps then take action a h for each a h ∈ A deterministically and denote the dataset { ( s i h , o i h , a i h , s i h +1 ) } i ∈ [ NA ]

end for end for

for h ∈ [ H ] do

end for

end for

Let

return { ̂ P h } h ∈ [ H ]

E Missing Details in Section 3

Proof of Proposition 3.1: We recall that b h ( · ) is the belief of the agent about the underlying state, see Appendix C for a detailed introduction. Note that Equation (3.1) can be written as

Therefore, for any h ∈ [ H ] and τ h such that P π ′ , P ( τ h ) > 0 , we can optimize π separately for each h ∈ [ H ] and τ h as:

Now we are ready to construct the counter-example of γ -observable POMDP P ϵ for some ϵ ∈ (0 , 1) with H = 1 , S = { s 1 , s 2 } , A = { a 1 , a 2 } , and O = { o 1 , o 2 } . We let µ 1 = ( 1 -γ 2 -γ , 1 2 -γ ) , O 1 ( o 1 | s 1 ) = 1 , and O 1 ( o 1 | s 2 ) = 1 -γ , O 1 ( o 2 | s 2 ) = γ . Therefore, it is direct to see that O 1 is exactly γ -observable. Most importantly, we choose r 1 ( s 1 , a 1 ) = 1 , r 1 ( s 1 , a 2 ) = 0 , and r 1 ( s 2 , a 1 ) = 0 , r 1 ( s 2 , a 2 ) = ϵ .

Therefore, given such a reward function, the fully observable expert policy is given by π ⋆ 1 ( a 1 | s 1 ) = 1 and π ⋆ 1 ( a 2 | s 2 ) = 1 , i.e., choosing a 1 at state s 1 and a 2 at state s 2 deterministically. Meanwhile, by our construction, one can compute that the belief given observation o 1 ensures b 1 ( o 1 ) = Unif( S ) . Hence, the corresponding 'distilled' partially observable policy under observation o 1 is given by

where the last step is due to q ∈ ∆( A ) . Now consider the function g ( x ) = xf (1 /x ) for x > 0 . It is direct to compute that g ′ ( x ) = f (1 /x ) -f ′ (1 /x ) x , and g ′′ ( x ) = f ′′ (1 /x ) x 3 ≥ 0 due to the convexity of the function f . Thus, we conclude that g is also convex. By Jensen's inequality, we have

where the equality holds when q ( a 1 ) = q ( a 2 ) = 1 2 . This indicates that ̂ π ⋆ 1 ( · | o 1 ) = Unif( A ) . On the other hand, combining the fact that b 1 ( o 1 ) = Unif( S ) with ϵ < 1 , it is direct to see that the optimal partially observable policy ˜ π ∈ arg max π ∈ Π v P ( π ) satisfies ˜ π 1 ( a 1 | o 1 ) = 1 . Now we are ready to evaluate the optimality gap between ˜ π and ̂ π ⋆ as follows

where the last step is due to the fact that ˜ π is the optimal policy, leading to the fact that V ˜ π, P ϵ 1 ( o 2 ) -V ̂ π ⋆ , P ϵ 1 ( o 2 ) ≥ 0 . Now it is not hard to compute that

Meanwhile, we can evaluate that

and correspondingly V ˜ π, P ϵ 1 ( o 1 ) -V ̂ π ⋆ , P ϵ 1 ( o 1 ) = 1 -ϵ 4 , implying that v P ϵ ( ˜ π ) -v P ϵ ( ̂ π ⋆ ) ≥ (1 -γ )(1 -ϵ ) 4 . This concludes our proof. ■

Note that another counterexample in a similar spirit was also constructed in [34], demonstrating that the expert policy for a poorly chosen agent-environment boundary can be useless in imitation learning, although the γ -observability property is not satisfied for the construction therein.

̸

Remark E.1. The counter-example P ϵ constructed above can be also used to demonstrate the bias of the state-only-based value function as an estimate of the history-based value function that appeared in the policy gradient formula for POMDPs, in the finite-horizon setting, i.e. E s h ∼ b h ( τ h ) [ V π, P ϵ h ( s h )] = V π, P ϵ h ( τ h ) (mirroring Theorem 4.2 of [6]). Specifically, in the counter-example above, we consider the policy π such that π 1 ( a 1 | o 1 ) = 1 and π 1 ( a 2 | o 2 ) = 1 . The state-only-based value function can be evaluated as

which implies that E s 1 ∼ b 1 ( o 1 ) [ V π, P ϵ 1 ( s 1 )] = V π, P ϵ 1 ( s 1 )+ V π, P ϵ 1 ( s 2 ) 2 = 1 -γ 2 -γ + γϵ 2 . On the other hand, it holds that V π, P ϵ 1 ( o 1 ) = 1+ ϵ 2 , which is not equal to E s 1 ∼ b 1 ( o 1 ) [ V π, P ϵ 1 ( s 1 )] , showing the bias of such a state-only value function.

Example E.2 (Deterministic POMDP [36, 81]) . We say a POMDP P is of deterministic transition if entries of matrices { T h } h ∈ [ H ] and the vector µ 1 are either 0 or 1 . Note that we do not make any assumptions on the emission matrices.

̸

Example E.3 (Block MDP [43, 18]) . We say a POMDP P is a block MDP if for any h ∈ [ H ] , s h , s ′ h ∈ S , it holds that supp( O h ( · | s h )) ∩ supp( O h ( · | s ′ h )) = ∅ when s h = s ′ h .

Example E.4 ( k -step decodable POMDP [19]) . We say a POMDP P is a k -step decodable POMDP if there exists an unknown decoder ϕ ⋆ = { ϕ ⋆ h : Z h → S} h ∈ [ H ] such that for any h ∈ [ H ] and reachable trajectory τ h , P P ( s h = ϕ ⋆ h ( z h ) | τ h ) = 1 , where Z h = ( O × A ) max { h -1 ,k -1 } × O , z h = (( o, a ) k ( h ): h -1 , o h ) , and k ( h ) = max { h -k +1 , 1 } .

Finally, to understand how our condition can extend beyond known examples in the literature, we show that one can indeed allow the decoding length of Example E.4 to be unknown and arbitrary (instead of being a small known constant as in [19] to have provably efficient algorithms).

Example E.5 (POMDP with arbitrary, unknown decodable length) . This example is similar to Example E.4, but the decoding length m is unknown and not necessarily a small constant.

In light of the pitfall in Proposition 3.1, we will analyze both the computational and statistical efficiencies of expert distillation in Section 4, under the condition in Definition 3.2.

Proof of Example E.2 & Example E.3 & Example E.4 & Example E.5: To see why those examples follow our Definition 3.2, it is indeed an immediate result of Proposition 7.1. ■

Proof of Proposition 3.3:

Here we evaluate the computational complexity and sample complexity of each iteration t as follows.

Sample complexity: The algorithm executes the policy π t -1 and collect K episodes, denoted as { s k 1: H +1 , o k 1: H , a k 1: H } k ∈ [ K ] . Thus, the sample complexity of each iteration is Θ( K ) .

Computational complexity for policy evaluation: The policy evaluation of the vanilla asymmetric actor-critic is done by minimizing the Bellman error. In the finite-horizon setting with tabular parameterization, it is equivalent to performing the following update for each h ∈ [ H ] in a backward way and each k ∈ [ K ] :

for some stepsize α ∈ (0 , 1) , where J ( τ k h , s k h , a k h ) := { j ∈ [ K ] | ( τ j h , s j h , a j h ) = ( τ k h , s k h , a k h ) } . Therefore, the computational complexity for this procedure is of POLY ( H,K ) .

Computational complexity for policy improvement: For tabular parameterization, computing ∇ log π t -1 h ( a k h | τ k h ) takes O (1) computation. Hence the policy update in Equation (3.2) performs POLY ( H,K ) computation.

Meanwhile, under the exponential time hypothesis, there is no polynomial time algorithm for even planning an ϵ -approximate optimal policy in γ -observable POMDPs [26]. This implies that the vanilla asymmetric actor-critic needs to take super-polynomial time to find an approximately optimal policy. This implies the corresponding sample complexity has to be super-polynomial.

Finally, we remark that even if we let the policy and the Q -function not depend on the entire history τ h but only the finite-memory z h , the proof still holds. The key is that whenever one only computes at the sampled history/finite-memories, i.e., updates the policy in an asynchronous way (in contrast to the synchronous one where the policies at all histories/finite-memories are updated), the sample and computational complexities will be coupled with the same order per iteration, which implies a super-polynomial sample complexity due to the super-polynomial computational complexity. This completes the proof. ■

Derivation for the closed-form update Equation (3.3). Note that the proximal policy optimization [72] update has the policy improvement as follows

where η is some learning rate and L t -1 ( π ) is a first-order approximation of the expected accumulated rewards at π t -1 :

By plugging L t -1 ( π ) into Equation (E.1), with simple algebric manipulations, we prove that:

F Missing Details in Section 4

Proof of Lemma 4.4: The proof follows by the assumption that the total cumulative reward is at most H ,

̸

which completes the proof.

■

̸

̸

̸

Proof of Theorem 4.5: For each step h ∈ [ H ] we define D h to be the distribution over the underlying state s h -1 at step h -1 , taken action a h -1 ∈ A based on π E , the underlying state transitioned to s h ∈ S , and the observation o h ∼ O h ( · | s h ) . We remind that we include at step zero, a dummy state-observation pair ( s 0 , o 0 ) , for notational convenience. Formally, D h is defined as D h ( s h -1 , a h -1 , o h , s h ) := P π E , P [ s h -1 , a h -1 , o h , s h ] .

We first use union bound to decompose the probability that we incorrectly decode,

̸

̸

̸

For each h ∈ [ H ] , we can use M episodes to collect M samples from the distribution D h . In addition, since state s H +1 is dummy, we need not to collect episodes for D H +1 . Denote the set of collected samples by ̂ D M h . We define the decoding g h for step h ∈ [ H ] as follows:

Observe that by Definition 3.2, { s h | ( s h -1 , a h -1 , o h , s h ) ∈ ̂ D M h } is either the empty set or contains only a single elements, in which case, it is true that g h ( s h -1 , a h -1 , o h ) = ψ h ( s h -1 , a h -1 , o h ) ( ψ is the real decoding function, see Definition 3.2). Moreover, we slightly abuse the notation and let ˜ D M h denote the empirical distribution induced by the samples in ̂ D M h . Thus, with probability at least 1 -δ H and setting M = Θ ( A · O · S +log( H/δ ) ϵ 2 ) for each step h ∈ [ H ] , we obtain the following by the result in [13]:

̸

Thus, by union bound, with probability at least 1 -δ , we have that for each step h ∈ [ H ] ,

̸

which in combination with Equation (F.1) concludes the proof. Finally, we note that we used a total of Θ ( H · A · O · S +log( H/δ ) ϵ 2 ) episodes from the POMDP, and the computational time was POLY ( H,A,O,S, 1 ϵ , log ( 1 δ ) ) . ■

G Provably Efficient Expert Policy Distillation with Function Approximation

We now turn our attention to the rich-observation setting under our deterministic filter condition. Definition 3.2 motivates us to consider only succinct policies that incorporate an auxiliary parameter representing the most recent state, as well as the most recent observations and actions. To handle the large observation space, we further assume that for each step h ∈ [ H ] , the agent selects a decoding function g h from a family of multi-class classifiers F h ⊂ {S × A × O → S} . For the function class F h , we make the standard realizability assumption. We formally summarize our assumptions in Assumption G.1.

Assumption G.1. We consider a POMDP that satisfies Definition 3.2. In addition, to derive learning algorithms that do not dependent on O , for each step h ∈ [ H ] , we assume that we have access to a class of functions F h : S × A × O → S such that the perfect decoding function ψ h ∈ F h .

We aim for our final bounds to depend on a complexity measure of the function class F = {F h } h ∈ [ H ] rather than the cardinality of the observation space O . We utilize the Daniely and Shalev-ShwartzDimension (DS Dimension) (Theorem G.2), which characterizes the PAC learnability for multi-class classification [8]. Defining the DS dimension is beyond the scope of our paper; we direct interested readers to [8] for further details. For intuition, readers can think of the DS Dimension as a certificate of PAC learnability without loss of intuition.

Theorem G.2 (Theorem 1 in [8]) . Consider a family of multi-class classifiers F that map features in space x ∈ X to labels in space y ∈ Y . Moreover, assume there is a joint probability distribution D over features in X and labels in Y , and that there exists g ∗ ∈ F such that for each ( x, y ) ∈ supp( D ) , g ∗ ( x ) = y . Given n samples from D , there exists an algorithm that with probability at least 1 -δ outputs ˜ g ∈ F such that

̸

where d DS ( F ) denotes the Daniely and Shalev-Shwartz-Dimension of the function class F .

We are now ready to present the main theorem of this section.

Theorem G.3. Consider a POMDP P that satisfies Definition 3.2, a policy π E ∈ Π S , and let {F h ⊆ {S × A × O → S}} h ∈ [ H ] be the decoding function class, and ψ h ∈ F h for each h ∈ [ H ] , i.e., {F h } h ∈ [ H ] is realizable. Then given access to the classification oracle of [9], there exists an algorithm learning the decoding function { g h } h ∈ [ H ] such that with probability at least 1 -δ , for each step h ∈ [ H ] :

̸

using O ( H 2 ( max h ∈ [ H ] d 3 / 2 DS ( F h )+log ( 1 δ ) ) ϵ ) episodes, where d DS ( F h ) is the Daniely and Shalev-

Shwartz-Dimension of F h [8].

Combining Theorem G.3 and Lemma 4.4, we obtain the final polynomial sample complexity in this function approximation setting, using classification (supervised learning) oracles (c.f. Table 1).

Proof of Theorem G.3:

For each step h ∈ [ H ] , we define D h to be the distribution over the underlying state s h -1 at step h -1 , taken action a h -1 ∈ A from π E , the underlying state transitioned to s h ∈ S , and the hallucinated observation o h ∼ O h ( · | s h ) (we remind readers that for step 0 , we use dummy state s 0 and action a 0 ). Formally, the probability that the sequence ( s h -1 , a h -1 , o h , s h ) is sampled from D h equals to

̸

̸

̸

For each h ∈ [ H ] , we can use ˜ O ( H ϵ · ( max h ∈ [ H ] d 3 / 2 DS ( F h ) + log ( 1 δ ) )) episodes to collect ˜ O ( H ϵ · ( max h ∈ [ H ] d 3 / 2 DS ( F h ) + log ( 1 δ ) )) samples from distribution D h . Hence, by Theorem G.2, with probability at least 1 -δ H , we have that

̸

Thus, by union bound, with probability at least 1 -δ , using a total of ˜ O ( H 2 ϵ · ( max h ∈ [ H ] d 3 / 2 DS ( F h ) + log ( 1 δ ) )) episodes we have that,

̸

which in combination with Equation (G.1) concludes the proof.

H Missing Details in Section 5

Proof of Theorem 5.1: Let π ∗ ∈ argmax π ∈ Π L E s 1 ∼ µ 1 [ V π 1 ( s 1 )] , where we define V π 1 ( s 1 ) := E o 1 ∼ O 1 ( · | s 1 ) ,a 1 ∼ π 1 ( · | o 1 ) [ Q π h ( z 1 = ( o 1 ) , s 1 , a 1 )] . We first note the following equation

Next, we make use of the performance difference lemma [39, 1, 74] on the extended space ∏ h ∈ [ H ] ( Z h ×S ) .

Definition H.1. Consider the class of policies Π PL such that at step h ∈ [ H ] , the policies in Π PL take an action based on finite memory up to this step and the use of the underlying state, e.g., for each policy π 1: H ∈ Π PL , π h : Z h ×S → ∆( A ) .

Observation 1. Note that Π L ⊆ Π PL .

Lemma H.2 (Performance difference Lemma [39, 1, 74]; see e.g., Lemma 1 in [74]) . For any pair of policies π = { π h } h ∈ [ H ] , π ′ = { π ′ h } h ∈ [ H ] ∈ Π PL , and approximation of the Q -function of policy π , we have that for each state s 1 ∈ supp( µ 1 ) :

where ˜ V π h ( z h , s h ) = E a h ∼ π h ( · | z h ) [ ˜ Q π h ( z h , s h , a h )] .

Setting π = π t ∈ Π L ⊆ Π LP , and π ′ = π ∗ ∈ Π L ⊆ Π LP , where we remind that π ∗ ∈ argmax π ∈ Π L V π 1 ( s 1 ) , and for each z h ∈ Z h and h ∈ [ H ] , we abuse the notation by letting π t h ( · | z h , s h ) = π t h ( · | z h ) and π ∗ h ( · | z h , s h ) = π ∗ h ( · | z h ) for all s h ∈ S . The above formulation is thus simplified to

■

where in the inequality above we used Lemma H.3. Since our policy does not depend on the realized underlying state s h ,

The last inequality follows by Lemma H.3. By averaging we get,

where the last inequality follows since for fixed h ∈ [ H ] and z h ∈ Z h , the agent updates her policy on memory z h according to MWU on feedback { E s h ∼ b apx ( z h ) [ ˜ Q π t h ( z h , s h , a ) ]} a ∈A , and thus, the accumulate regret is bounded by (Section 4.3 in [10]):

The proof follows by combining Equation (H.1) and the inequality above. Finally, to achieve the near optimality in the class of Π L , we bound the optimistic estimate using Equation (H.2) in Lemma H.3, and its global near-optimality for a large enough L under γ -observability is a direct consequence of Theorem 4.1 in [26]. ■

Lemma H.3 (Optimistic Q -function - adapted from [48]) . Given a policy π ∈ Π L , and a parameter M ∈ N , let { ˜ Q π h : Z h ×S×A→ [0 , H ] } h ∈ [ H ] be the output of Algorithm 3. Then, with probability at least 1 -δ : ∀ z h ∈ Z h , s h ∈ S , a h ∈ A

where V π 1 ( s 1 ) = E o 1 ∼ O 1 ( · | s 1 ) ,a 1 ∼ π 1 ( · | o 1 ) [ Q π h ( z 1 = ( o 1 ) , s 1 , a 1 )] , ˜ V π 1 ( s 1 ) = E o 1 ∼ O 1 ( · | s 1 ) ,a 1 ∼ π 1 ( · | o 1 ) [ ˜ Q π h ( z 1 = ( o 1 ) , s 1 , a 1 )] and ˜ V π h ( z h , s h ) = E a h ∼ π h ( · | z h ) [ ˜ Q π h ( z h , s h , a h )] . Moreover, Algorithm 3 needs a total of H · M episodes from POMDP P and runs in time POLY ( H,M,S,A L , O L ) .

Proof. For each step h ∈ [ H ] , collect M trajectories with states using policy π on POMDP P and let D h = { τ ( i ) } i ∈ [ M ] be those collected trajectories. Define the empirical transition, observation and reward distribution as follows:

Set δ 1 = δ 2 · S · ( A +1) . By [13], there exists a constant C > 0 such that for each step h ∈ [ H ] , state s ∈ S and action a ∈ A with probability at least 1 -δ 1 :

For the rest of the proof, we condition on this event. By union bound, this happens with probability at least 1 -δ 2 . We define the optimistic Q -function recursively as follows for a memory-state pair ( z h , s h ) ∈ Z h ×S :

where ˜ V π h +1 ( z h +1 , s h +1 ) = E a h +1 ∼ π h +1 ( · | z h +1 ) [ ˜ Q π h +1 ( z h +1 , s h +1 , a h +1 )] . Hence the time complexity of our algorithm is POLY ( H,M,S,A L , O L ) . To prove the first condition, we fix step h ∈ [ H ] , z h ∈ Z h , a h ∈ A and state s h ∈ S and consider the case where ˜ Q π h ( z h , s h , a h ) = H -h +1 . In this case, since by assumption on the POMDP P , r h ( s h , a h ) ≤ 1 , and by definition of ˜ Q π h +1 ( · , · , · ) ≤ H -h , we have:

̸

If ˜ Q π h ( z h , s h , a h ) = H -h +1 , observe that:

and hence, { ˜ Q π h } h ∈ [ H ] satisfies the first condition. Moreover, it holds that:

Thus, it holds that:

Thus, we conclude that

To finish the proof, we make use of the following lemma.

Lemma H.4 (Lemma 6 in [48]) . For each step h ∈ [ H ] , and state-action pair ( s h , a h ) ∈ S × A , with probability at least 1 -δ 2 :

By setting δ 2 = δ 2 · S · A · H , and taking union bound we have that with probability at least 1 -δ :

Proof of Theorem 5.2: The proof of Theorem 5.2 follows by combining Theorem H.5 and Theorem H.6 below. Theorem H.5 proves that we can approximately learn a POMDP model P computationally and sample efficiently, thanks to the privileged information.

Theorem H.5. Fix any ϵ, δ ∈ (0 , 1) . Algorithm 4 can learn the approximate POMDP model with transition ̂ T 1: H and emission ̂ O 1: H such that with probability at least 1 -δ , for any policy π ∈ Π gen and h ∈ [ H ]

using POLY ( S, A, H, O, 1 ϵ , log( 1 δ )) episodes in time POLY ( S, A, H, O, 1 ϵ , log( 1 δ )) .

Proof. Note that by Lemma H.11, it suffices to consider only π ∈ Π S as the optimal value for policies in Π gen can be achieved by those in Π S (by considering r h ( s h , a h ) := ∥ T h ( · | s h , a h ) -̂ T h ( · | s h , a h ) ∥ 1 + ∥ O h ( · | s h ) -̂ O h ( · | s h ) ∥ 1 ). For each h ∈ [ H ] and s h ∈ S , we define

Fix ϵ 1 > 0 , we define U ( h, ϵ 1 ) = { s h ∈ S | p h ( s h ) ≥ ϵ 1 } . By the guarantee of the EULER algorithm from [89, 37], one can learn a policy Ψ( h, s h ) with sample complexity ˜ O ( S 2 AH 4 ϵ 1 ) such that d Ψ( h,s h ) h ( s h ) ≥ p h ( s h ) 2 for each s h ∈ U ( h, ϵ 1 ) with probability 1 -δ 1 . Now we assume this event holds for any h ∈ [ H ] and s h ∈ U ( h, ϵ 1 ) . For each s h ∈ S and a h ∈ A , we have executed in Algorithm 4 the policy Ψ( h, s h ) followed by an action a h ∈ A for N episodes, and denote the number of episodes that s h and a h are visited as N h ( s h , a h ) . Then with probability 1 -e Nϵ 1 / 8 , N h ( s h , a h ) ≥ Np h ( s h ) 2 by Chernoff bound. Now conditioned on this event, we are ready to evaluate the following: for any π ∈ Π S

where the first inequality follows by [13] with probability at least 1 -δ 2 , and the second inequality uses d Ψ( h,s h ) h ( s h ) ≥ p h ( s h ) 2 for all s h ∈ U ( h, ϵ 1 ) . Similarly, for any π ∈ Π S

where similarly the first inequality follows by [13] with probability at least 1 -δ 2 , and the second inequality uses d Ψ( h,s h ) h ( s h ) ≥ p h ( s h ) 2 for all s h ∈ U ( h, ϵ 1 ) . Therefore, by a union bound, all the high probability events above hold with probability

Therefore, we can choose N = ˜ Θ( S 2 + SO ϵ 2 ) and ϵ 1 = Θ( ϵ S ) , leading to the total sample complexity

which completes the proof.

Now with such a model learned in a reward-free way, we are ready to present our main result for approximate belief learning.

Theorem H.6. Consider a γ -observable POMDP P (c.f. Assumption 2.2), an ϵ > 0 , approximate transition and emission { ̂ T h } h ∈ [ H ] and { ̂ O h } h ∈ [ H ] learned from Algorithm 4 that ensure that for any π ∈ Π gen and h ∈ [ H ] :

Then we can construct in time POLY ( H,A,S,O, 1 ϵ , log 1 δ ) a belief { b apx h : Z h → ∆( S ) } h ∈ [ H ] with no further samples. In addition, if the parameter in Algorithm 4 satisfies N = ˜ Θ( O log( SH/δ ) γ 2 ϵ 1 ) and our class of finite memory policies Π L satisfies L ≥ ˜ Ω( γ -4 log( S/ϵ ) , then for any π ∈ Π gen and h ∈ [ H ] :

Proof. We firstly consider the following simple while important fact: for the estimated emission ̂ O h , its observability can be evaluated as

for any b, b ′ ∈ ∆( S ) and ∥ ̂ O h -O h ∥ ∞ := max s h ∈S ∥ O h ( · | s h ) -̂ O h ( · | s h ) ∥ 1 . Therefore, if one can ensure that the emission at any state s h is learned accurately in the sense that ∥ O h ( · | s h ) -̂ O h ( · | s h ) ∥ 1 ≤ γ 2 , we can conclude that ̂ O h is also γ/ 2 -observable. However, the key challenge here is that there could exist some states s h that can only be visited with a low probability no matter what exploration policy is used . Therefore, emissions at such states may not be learned accurately. To address this issue, our key technique is to redirect the transition probability into states that cannot be explored sufficiently to some highly visited states, in a new POMDP that is close to the original one in generating the beliefs.

Specifically, first, for any ϵ 1 > 0 , we define

By Chernoff bound, with probability at least 1 -Se -Nϵ 1 / 8 , it holds that

where p h ( s h ) := max π ∈ Π S d π h ( s h ) . To see the reason, we notice that for any s h such that p h ( s h ) ≥ ϵ 1 , with probability 1 -e -Nϵ 1 / 8 , it holds that N h ( s h ) N ≥ ϵ 1 2 . Therefore, the last step is by taking a union bound for all s h . Now with S low h defined, we are ready to construct a truncated POMDP P trunc such that for each h ∈ [ H ] , we define the transition as

Meanwhile, for the initial state distribution, we define

For emission, we simply define

Finally, we define the rewards for P trunc arbitrarily. Now we examine the total variation distance between the trajectory distributions in P and P trunc . For any policy π ∈ Π gen , it is easy to see that

for any τ h ∈ T high h := { ( s 1: h , o 1: h , a 1: h -1 ) | s ′ h ∈ S high h ′ , ∀ h ′ ∈ [ h ] } , since some rarely visited states' probability has been redirected to the highly visited ones in P trunc . Meanwhile, it holds by a union bound that for any h ∈ [ H ]

Therefore, by noticing that P π, P trunc ( τ h ) = 0 for any τ h ̸∈ T high h and h ∈ [ H ] , the total variation distance between the trajectory distributions in P and P trunc can be bounded by

On the other hand, by Equation (H.9) of Lemma H.9, we have

With such an intermediate quantity P trunc , we define the transition of its approximate version ̂ P trunc as follows: we define the transition as

Meanwhile, for the initial state distribution, we define

For emission, we define

Now we define ̂ O sub h ∈ R |S high h |× O to be the sub-matrix of ̂ O trunc h , where we only keep those rows of ̂ O trunc h that correspond to the states in S high h . Similarly, we define O sub h ∈ R |S high h |× O to be the sub-matrix of O h , where we only keep those rows of O h that correspond to the states in S high h . It is direct to see that O sub is still γ -observable. Meanwhile, we notice that

where the first inequality is by Lemma J.9, and the second inequality is by the definition of S high h . Therefore, if we take

it is guaranteed that ∥ ̂ O sub h -O sub h ∥ ∞ ≤ γ 2 . Therefore, we conclude that ̂ O sub h is also γ/ 2 -observable.

Now we are ready to examine ̂ b ′ , trunc h . We firstly define the following POMDP ̂ P sub , which essentially deletes all states in S low h from the state space of ̂ P trunc at each step h , which does not affect the trajectory distribution as they were not reachable in ̂ P trunc . Notice that the emission of ̂ P sub is exactly ̂ O sub h , implying that ̂ P sub is a γ/ 2 -observable POMDP. Therefore, for policy class with finite memory Π L with L ≥ ˜ Ω( γ -4 log( S/ϵ ) , by Theorem 4.1 in [25], it is guaranteed that for any π ∈ Π ,

where ̂ b sub ( τ h ) , ̂ b ′ , sub ( z h ) ∈ ∆( S high ) . Now we claim that

where we define b apx h ( z h ) ∈ ∆( S ) by augmenting ̂ b ′ , sub h ( z h ) with 0 for states from S low h . To see the reason, we notice that simulating ̂ P trunc is exactly equivalent to simulating ̂ P sub , and that ̂ b trunc h ( τ h )( s h ) = 0 for s h ∈ S low h , ̂ b trunc h ( τ h )( s h ) = ̂ b sub h ( τ h )( s h ) for s h ∈ S high h .

For the total variation distance between the trajectory distributions in P trunc and ̂ P trunc , it holds that by Lemma H.9

where the last step is by Lemma H.7.

Now by Equation (H.4), we have

where the last step is by Theorem H.5. Hence, by Lemma H.9, it holds that

Finally, we are ready to prove

Therefore, by setting ϵ 1 = Θ ( ϵ H 2 S ) , we prove our lemma. Observe that our algorithm needed no further samples. The computational complexity follows by computing belief b apx h on POMDP ̂ P sub using finite-memory policies of size ˜ Θ( γ -4 log( S/ϵ )) . For the final sample complexity, we only need to ensure N = ˜ Θ( O log( SH/δ ) γ 2 ϵ 1 ) in Equation (H.3), thus concluding our proof.

We conclude the proof of Theorem 5.2 by combining Theorem H.5 and Theorem H.6.

H.1 Supporting Technical Lemmas

In the following, we provide some technical lemmas and their proofs.

Lemma H.7. Fix n > 0 and an index set S ⊆ [ n ] . For two sequences x 1: n and y 1: n such that x i , y i ∈ [0 , 1] for i ∈ [ n ] and ∑ i x i = 1 , ∑ i y i = 1 , we define

We define ̂ y 1: n similarly. Then, it holds that

■

Proof. Note that

which completes the proof.

Lemma H.8. Fix two finite sets X , Y and two joint distributions P 1 , P 2 ∈ ∆( X × Y ) . It holds that

Proof. For the second inequality, it holds that

Meanwhile, for the first inequality, it holds that

concluding our lemma.

Lemma H.9. Consider any two POMDP instances P and ̂ P and define the belief functions as b 1: H and ̂ b 1: H , respectively (see Appendix C for the definition of belief functions). It holds that for any π ∈ Π gen

where we remind readers that we denote by τ H = ( s 1: H , o 1: H , a 1: H -1 ) the trajectory with states from an episode of the POMDP.

Proof. The first inequality also appears in [44] and we provide a simplified proof here for completeness. By Lemma H.8, it holds that

where the last step is again from Lemma H.8. Therefore, by repeatedly unrolling the inequality, we proved the first result.

For the second result, we notice that by Lemma H.8, it holds that

Notice the fact that P π, P h ( s h | τ h ) does not depend on π since it is exactly the belief b h ( τ h )( s h ) , we conclude that

where the last step comes from the fact that after marginalization, the total variation distance will not increase. By combining it with the first result, we proved the second result.

Corollary H.10. Consider any two POSG instances G , ̂ G that satisfy Assumption C.8 and the corresponding belief functions in the form of P G , P ̂ G : C h → ∆( P h ×S ) for any h ∈ [ H ] , respectively. It holds that for any π ∈ Π gen

Proof. For the second result, we notice that by Lemma H.8, it holds that

Notice the fact that P π, G h ( s h , p h | c h ) , P π, ̂ G h ( s h , p h | c h ) do not depend on π due to Assumption C.8, we conclude that

where the last step again comes from the fact that after marginalization, the total variation distance will not increase. By combining it with Lemma H.9, we proved the second result.

Lemma H.11. For any reward function r 1: H of P with r h : S × A → R for any h ∈ [ H ] , it holds that

Proof. Denote π ⋆ ∈ Π S to be the optimal policy obtained by running value iteration only on the state space S for P . Now we are ready to prove the following argument for any π ∈ Π gen inductively:

It is easy to see the argument holds for h = H . Fix state-action pair ( s h , a h ) ∈ S × A and trajectory ( s 1: h -1 , o 1: h , a 1: h -1 ) , we note that:

where the first inequality comes from the inductive hypothesis.

I Missing Details in Section 6

POMDP under deterministic filter condition. We first evaluate our algorithms on POMDPs with certain structures, i.e., the deterministic conditions. In particular, we generate POMDPs, where either the transition dynamics are deterministic or the emission ensures decodability. We test three of our approaches, expert policy distillation, asymmetric optimistic natural policy gradient. We summarize our results in Table 2, where the four cases corresponds to POMDPs with ( S = A = 2 , O = 3 , H = 5) , ( S = A = 2 , O = 3 , H = 10) , ( S = 3 , A = 2 , O = 4 , H = 5) , ( S = 3 , A = 2 , O = 4 , H = 5) , and we can see that our approach based on expert distillation outperforms all the other methods, which is consistent with the fact that such methods have exploited the special structures of the POMDPs achieving both polynomial sample and computational complexity.

General POMDPs. Here we also evaluate our methods for general randomly generated POMDPs without any structures. Hence, we compare the baselines with asymmetric optimistic natural policy gradient and asymmetric optimistic value iteration (i.e., the single-agent version of Algorithm 5). In Figure 2, we show the performance of different algorithms in POMDPs of different size, where the four cases corresponds to POMDPs with ( S = A = O = 2 , H = 5) , ( S = A = O = 2 , H = 10) , ( S = O = 3 , A = 2 , H = 10) , ( S = A = 3 , O = 2 , H = 10) , and our approaches achieves the highest rewards with small number of episodes.

Implementation details. For each problem setting, we generated 20 POMDPs randomly and report the average performance and its standard deviation for each algorithms. For our algorithms based on privileged value learning methods, we find that using a finite memory of 3 already provides strong performance. For our algorithms based privileged policy learning, we instantiate the MDP learning algorithm with the fully observable optimistic natural policy gradient algorithm [74]. Meanwhile, for both the decoder learning and belief learning, we find that just utilizing all the historic trajectories gives us reasonable performance without additional samples. For baselines, the hyperparameters α for Q -value update and step size for the policy update are tuned by grid search, where α controls the update of temporal difference learning (recall the update rule of temporal difference learning as

Q ← (1 -α ) Q + αQ target ). For asymmetric Q -learning, we use ϵ -greedy exploration, where we use the seminal decreasing rate ϵ t = H +1 H + t . Finally, all simulations are conducted on a personal laptop with Apple M1 CPU and 16 GB memory.

Empirical insights and interpretation of the experimental results. To understand intuitively why our approach outperforms those baseline algorithms, we notice the key difference in the value and policy update style between our approaches and vanilla asymmetric actor critic and asymmetric Q -learning. The baselines often roll-out the polices, collect trajectories, and only update the value and the policies on the states/history the trajectories have visited. Therefore, ideally, to learn a good policy for baselines, the number of trajectories to collect is at least as large as the history size, which is indeed exponential in the horizon H . This is known as curse of history for partially observable RL. In contrast, in our algorithms, we explicitly estimate the empirical transition and emissions, which is indeed of polynomial size. Thus, the sample complexity avoids suffering from the potential exponential dependency of horizon or the length of the finite memory. Finally, we acknowledge that the baselines are developed to handle complex, high-dimensional deep RL problems, while scaling our methods to deep RL benchmarks requires non-trivial engineering efforts.

J Missing Details in Section 7

Proof of Proposition 7.1:

Given the condition of Definition 3.2, the function ϕ h that satisfies the condition of Proposition 7.1 can be constructed recursively as follows for any reachable τ h

and ϕ 1 ( o 1 ) := ψ 1 ( o 1 ) . Therefore, we can prove by induction that by belief update rule

where the last step is by the definition of our ϕ h . Therefore, we have P P ( s h = ψ h ( ϕ h -1 ( τ h -1 ) , a h -1 , o h ) | τ h ) = 1 .

For the other direction, it is similar to the proof of Lemma C.1 in [19] for m -step decodable POMDP. Here we prove it for completeness. For any reachable trajectory τ h ∈ T h , it holds by the belief update and induction that

Meanwhile, since we know P P ( · | τ h ) is a one-hot vector, we can construct ψ h such that ψ h ( s h -1 , a h -1 , o h ) is the unique s h that makes P P ( s h | τ h ) > 0 with s h -1 = ϕ h -1 ( τ h -1 ) . If this procedure does not complete the definition of ψ for some ( s h -1 , a h -1 , o h ) , it implies that either s h -1 is not reachable or o h is not reachable given s h -1 , i.e., P P ( o h | s h -1 , a ′ h -1 ) = 0 for any a ′ h -1 ∈ A , thus recovering the conditions of Definition 3.2. ■

Proof of Proposition 7.3:

Note that for any given problem instance of a POMDP, we can construct a POSG by adding a dummy agent that has the observation being the exact state at each time step, and has only one dummy action that does not affect the transition or reward. Therefore, even the local private information p i,h of the dummy agent can decode the underlying state, and hence ( c h , p h ) reveals the underlying state. Therefore, the corresponding POSG with the dummy agent satisfies the condition in Proposition 7.3. Meanwhile, it is direct to see that any CCE of the POSG is an optimal policy for the original POMDP. Now by the PSPACE-hardness of planning for POMDPs [66], we proved our proposition. ■

Theorem J.1. Suppose the POSG G satisfies Definition 7.2. For any π ∈ Π S and (potentially stochastic) decoding functions ̂ g = { ̂ g i,h } i ∈ [ n ] ,h ∈ [ H ] with ̂ g i,h : C h × P i,h → ∆( S ) for each i ∈ [ n ] , h ∈ [ H ] , it holds that

̸

̸

where π ̂ g is the distilled policy of π through the decoding functions ̂ g , where at step h , each agent i firstly individually decodes the state by sampling s h ∼ ̂ g i,h ( · | c h , p i,h ) , and then acts according to the expert π i,h , where in the following discussions, we slightly abuse the notation and regard ̂ g i,h ( c h , p i,h ) as a random variable following the distribution of ̂ g i,h ( · | c h , p i,h ) . In other words, the decoding process does not need correlations among the agents.

Proof. For notational simplicity, we write v i instead of v G i when the underlying model is clear from the context. Firstly, we consider any deterministic decoding function ̂ ϕ = { ̂ ϕ i,h } i ∈ [ n ] ,h ∈ [ H ] with ̂ ϕ i,h : C h ×P i,h →S for each i ∈ [ n ] , h ∈ [ H ] , and note the following that for any i ∈ [ n ] ,

̸

✶ ✶ where we define R := ∑ h r i,h ( s h , a h ) , and the last step is by the definition of π ̂ ϕ

̸

̸

Therefore, by noticing the fact that ̂ g is equivalent to a mixture of deterministic decoding functions, where at the beginning of each episode, one can firstly independently sample the outcome for each ( c h , p j,h ) for j ∈ [ n ] and h ∈ [ H ] , we conclude that

̸

̸

Nowweprove our result for NE and CCE first. Due to similar arguments for evaluating v i ( π ) -v i ( π ̂ ϕ ) , for any u i ∈ Π i ∪ Π S ,i , it holds that

̸

We notice the following fact that

where we define T H ( ̂ ϕ ) := { τ H ∈ T H | ∀ j ∈ [ n ] \ { i } , h ∈ [ H ] : s h = ̂ ϕ j,h ( c h , p j,h ) } . By definition of π ̂ ϕ , it holds that

̸

Therefore, we have

Correspondingly, it holds that

which implies that

̸

̸

̸

̸

̸

̸

Again by the fact that ̂ g is equivalent to a mixture of deterministic decoding functions, it holds

̸

Now we are ready to evaluate the NE/CCE-gap of policy π ̂ g as follows:

̸

Now we notice that max u i ∈ Π S ,i v i ( u i × π -i ) = max u i ∈ Π i v i ( u i × π -i ) since Π S ,i ⊆ Π i by the deterministic filter condition Definition 3.2 and π -i is a Markov policy. Therefore, we conclude that

̸

̸

where the second last step is by a union bound, thus proving our result for NE and CCE.

For CE, consider any strategy modification m i ∈ M i ∪ M S ,i , it holds that

̸

Now we notice that max m i ∈M S ,i v i (( m i ⋄ π i ) ⊙ π -i ) = max m i ∈M S ,i v i (( m i ⋄ π i ) ⊙ π -i ) since M S ,i ⊆ M i by the deterministic filter condition Definition 7.2 and Lemma J.2. Therefore, we conclude that

̸

̸

where the third step is due to the same reason as Equation (J.1).

Lemma J.2. For any π ∈ Π S , it holds for any reward function and i ∈ [ n ] that

̸

̸

̸

̸

̸

̸

̸

̸

̸

Proof. Denote m ⋆ i ∈ argmax m i ∈M gen i v i (( m i ⋄ π i ) ⊙ π -i ) and ̂ m ⋆ i ∈ argmax m i ∈M S ,i v i (( m i ⋄ π i ) ⊙ π -i ) . Now we shall prove that V ( m ⋆ i ⋄ π i ) ⊙ π -i , G i,h ( s 1: h , o 1: h , a 1: h -1 ) ≤ V ( ̂ m ⋆ i ⋄ π i ) ⊙ π -i , G i,h ( s h ) inductively for each h ∈ [ H ] . Note that it holds for h = H +1 . Now we consider the following

where the second inequality follows from the inductive hypothesis and the third step is by the definition of ̂ m ⋆ i ∈ argmax m i ∈M S ,i v i (( m i ⋄ π i ) ⊙ π -i ) .

Lemma J.3. Given an approximate POSG ̂ G that satisfies Assumption C.8 with approximate transitions and emissions being { ̂ T h , ̂ O h } h ∈ [ H ] , we define the approximate decoding function ̂ g to be

for each h ∈ [ H ] , s h ∈ S , c h ∈ C h , p i,h ∈ P i,h . Then it holds that for any π ∈ Π S ,

̸

and

̸

Proof. Note for any i ∈ [ n ] , u i ∈ Π i , j ∈ [ n ] , h ∈ [ H ] , it holds

̸

due to the condition in Definition 7.2. Meanwhile,

where the first inequality is by the first inequality in Lemma J.7, the second and third inequalities are due to the fact that TV distance does not increase after marginalization, and the last inequality is by Lemma H.9. Since π -i is a fixed and fully-observable Markov policy, by Lemma H.11, we have

thus proving the first result of our lemma.

For the second result of our lemma, it can be proved similarly that for any i ∈ [ n ] , m i ∈ M i , j ∈ [ n ] , h ∈ [ H ] ,

̸

By Lemma J.2, we proved the second result.

Theorem J.4. Fix any ϵ, δ ∈ (0 , 1) and π ∈ Π S . Algorithm 5 can learn a decoding function ̂ g such that with probability 1 -δ

̸

with total sample complexity ˜ O ( nS 2 AHO + nS 3 AH ϵ 2 + S 4 A 2 H 5 ϵ ) and computational complexity POLY ( S, A, H, O, 1 ϵ ) , log 1 δ .

Proof. With the help of Lemma J.3, it suffices to prove

The following proof procedure follows similarly to that of Theorem H.5. For each h ∈ [ H ] and s h ∈ S , we define

Fix ϵ 1 , δ 1 > 0 , we define U ( h, ϵ 1 ) = { s h ∈ S | p h ( s h ) ≥ ϵ 1 } . By [37], one can learn a policy { Ψ i ( h, s h ) } i ∈ [ n ] with sample complexity ˜ O ( S 2 A i H 4 ϵ 1 ) such that max i ∈ [ n ] d Ψ i ( h,s h ) × π -i h ( s h ) ≥

p h ( s h ) 2 for each s h ∈ U ( h, ϵ 1 ) with probability 1 -n · δ 1 . Now we assume this event holds for any h ∈ [ H ] and s h ∈ U ( h, ϵ 1 ) . For each s h ∈ S and a h ∈ A , we have executed each policy { Ψ i ( h, s h ) × π -i } i ∈ [ n ] for the first h -1 steps followed by an action a h ∈ A for N episodes and denote the total number of episodes that s h and a h are visited as N h ( s h , a h ) , and N h ( s h ) = ∑ a ∈A N h ( s h , a ) . Then, with probability 1 -e -Nϵ 1 / 8 , we have N h ( s h , a h ) ≥ Np h ( s h ) 2 by Chernoff bound. Now conditioned on this event, we are ready to evaluate the following for any i ∈ [ n ] , and u i ∈ Π S ,i :

where the second step is by Lemma J.8, and the last step is by Cauchy-Schwarz inequality. Similarly,

where the second step is by Lemma J.9, and the last step is by Cauchy-Schwarz inequality. Therefore, by a union bound, all high probability events hold with probability

Therefore, we can choose N = ˜ Θ( S 2 + SO ϵ 2 ) and ϵ 1 = Θ( ϵ S ) , leading to the total sample complexity

which completes the proof.

Note that although our Algorithm 5 and Theorem J.4 are stated for NE/CCE, it can also handle CE with simple modifications, where the key observation is that the strategy modification m i ∈ M S ,i can also be regarded as a Markov policy in an extended MDP marginalized by π -i as defined below.

Definition J.5. Fix π ∈ Π S . We define M extended i ( π ) to be an MDP for agent i , where for each h ∈ [ H ] , the state is ( s h , a i,h ) , the action is some modified action a ′ i,h , the transition is defined as T extended h ( s h +1 , a i,h +1 | s h , a i,h , a ′ i,h ) :=

E a -i,h ∼ π h ( · | s h ,a i,h ) [ T h ( s h +1 | s h , a ′ i,h , a -i,h ) π h +1 ( a i,h +1 | s h +1 )] , where we slightly abuse the notation of π h ( a -i,h | s h , a i,h ) and π h ( a i,h | s h ) by defining them as the posterior and marginal distributions induced by the joint distribution π h ( a h | s h ) . Similarly, the reward is given by r extended h ( s h , a i,h , a ′ i,h ) := E a -i,h ∼ π h ( · | s h ,a i,h ) [ r h ( s h , a ′ i,h , a -i,h )] .

With the help of such an extended MDP, we can develop Algorithm 6, which is a CE version of Algorithm 5 with the following guarantees.

Theorem J.6. Fix any ϵ, δ ∈ (0 , 1) and π ∈ Π S . Algorithm 6 can learn a decoding function ̂ g such that

̸

with total sample complexity ˜ O ( nS 2 A 3 HO + nS 3 A 4 H ϵ 2 + S 4 A 6 H 5 ϵ ) and computational complexity POLY ( S, A, H, O, 1 ϵ ) .

Proof. Due to the construction of M extended i ( π ) , the proof of Theorem J.4 readily applies, where the only difference is that the state space of M extended i ( π ) is now SA i , larger than that of M ( π -i ) by a factor of A i thus proving our theorem.

We next introduce and prove several supporting lemmas used before.

Lemma J.7. Suppose we can sample from a joint distribution P ∈ ∆( X × Y ) for some finite X , Y i.i.d. Then we can learn an approximate distribution Q ∈ ∆( X × Y ) with sample complexity Θ ( |X||Y| +log 1 /δ ϵ 2 ) such that

with probability 1 -δ

Proof. Note the following holds

Therefore, we have

By the sample complexity of learning discrete distributions [13], we can learn Q such that ∑ x ∈X ,y ∈Y | P ( x, y ) -Q ( x, y ) | ≤ ϵ in sample complexity Θ ( |X||Y| +log 1 /δ ϵ 2 ) with probability 1 -δ . Thus, we proved our lemma.

Lemma J.8 (Concentration on transition) . Fix δ > 0 and dataset { τ k H } k ∈ [ N ] sampled from P (or G in the multi-agent setting) under policy π ∈ Π gen . We define for each h ∈ [ H ] , ( s h , a h , s h +1 ) ∈ S × A × S

for some absolute constant C 1 > 0 , where we define ̂ T h ( s h +1 | s h , a h ) = N h ( s h ,a h ,s h +1 ) max { N h ( s h ,a h ) , 1 } .

Proof. This is done by firstly bounding ∥ T h ( · | s h , a h ) -̂ T h ( · | s h , a h ) ∥ 1 for specific k, h, s h , a h according to [13], and then taking union bound for all k ∈ [ K ] , h ∈ [ H ] , s h ∈ S , a h ∈ A .

Lemma J.9 (Concentration on emission) . Fix δ > 0 and dataset { τ k H } k ∈ [ N ] sampled from P (or G in the multi-agent setting) under some policy π ∈ Π gen . We define for each h ∈ [ H ] , ( s h , o h ) ∈ S × O

Then, with probability at least 1 -δ , it holds that

for some absolute constant C 2 > 0 , where we define ̂ O h ( o h | s h ) = N h ( s h ,o h ) max { N h ( s h ) , 1 } .

Proof. This is done by firstly bounding ∥ O h ( · | s h ) -̂ O h ( · | s h ) ∥ 1 for specific k, h, s h according to [13], and then taking union bound for all k ∈ [ K ] , h ∈ [ H ] , s h ∈ S .

Now we switch to proving the guarantees for Algorithm 7.

Lemma J.10. Fix δ > 0 . With probability 1 -δ , it holds that for any k ∈ [ K ] , h ∈ [ H ] , s h ∈ S :

where ̂ J k h is defined in Algorithm 7.

Proof. This is done by firstly bounding ∑ o h +1 ∣ ∣ ∣ P G ( o h +1 | s h , a h ) -̂ J k h ( o h +1 | s h , a h ) ∣ ∣ ∣ for specific k, h, s h , a h according to [13], and then taking union bound for all k ∈ [ K ] , h ∈ [ H ] , s h ∈ S , a h ∈ A .

From now on, we shall use the bonus

in Algorithm 7, for some absolute constant C 3 > 0 .

Before presenting our technical analysis, we define the following notation for the ease of presentation. We define the following approximate value functions for any policy π ∈ Π in a backward way for h ∈ [ H ] when given some approximate belief in the form of { ̂ P h : ̂ C h → ∆( P h × S ) } h ∈ [ H ] as discussed in Section 7.2:

for each ( i, c h ) ∈ [ n ] ×C h and γ h ∈ Γ h , where we define ̂ V π, G i,H +1 ( c H +1 ) = 0 .

Intuitively, this definition of ̂ V π, G i,h ( c h ) mimics the Bellman equation of the ground-truth value function V π, G i,h ( c h ) by replacing the ground-truth belief P G ( s h , p h | c h ) by ̂ P h ( s h , p h | ̂ c h ) . Next, we point out the following quantitative bound when using ̂ V π, G i,h ( c h ) to approximate V π, G i,h ( c h ) .

Lemma J.11. For any π ′ , π ∈ Π , it holds that

where

Proof. It follows directly by combining Lemma 4 and Lemma 8 of [51].

Meanwhile, note that although in Algorithm 7, the value functions we maintain have input ̂ c h instead of c h for computational efficiency, we extend the definitions of those value functions to also accept c h as inputs as follows (with a slight abuse of notation):

where we recall that ̂ c h = Compress h ( c h ) .

Lemma J.12 (Optimism 1 for NE/CCE) . With probability 1 -δ , for any k ∈ [ K ] , for Algorithm 7, it holds that for any i ∈ [ n ] , π ′ i ∈ Π i , h ∈ [ H ]

where we recall that ̂ c h = Compress h ( c h ) .

Proof. We will prove by backward induction. Obviously, it holds for h = H +1 . Now we assume the lemma holds for h +1 , then by definition

where the last step is by inductive hypothesis. Now note that for any ( s h , p h , a h ) , we have

where we notice P G ( o h +1 | s h , a h ) = ∑ s h +1 O h +1 ( o h +1 | s h +1 ) T h ( s h +1 | s h , a h ) for the first inequality, and the second inequality comes from the construction of our bonus b k -1 h ( s h , a h ) in Equation (J.3) and Lemma J.10. Meanwhile, by the definition of value functions, it holds that

Now we conclude

By definition, we have Q high ,k i,h ( c h , γ h ) = Q high ,k i,h ( ̂ c h , γ h ) , thus proving Q high ,k i,h ( ̂ c h , γ h ) ≥ ̂ Q π ′ i × π k -i , G i,h ( c h , γ h ) . Now for the value function, note that

where the first step is by the property of Bayesian CCE, and the second step is by Q high ,k i,h ( c h , γ h ) ≥ ̂ Q π ′ i × π k -i , G i,h ( c h , γ h ) for any γ h ∈ Γ h as proved above. Again by definition, we proved V high ,k i,h ( ̂ c h ) = V high ,k i,h ( c h ) ≥ ̂ V π ′ i × π k -i , G i,h ( c h ) .

Lemma J.13 (Optimism 1 for CE) . With probability 1 -δ , for any k ∈ [ K ] , for Algorithm 7, it holds that for any i ∈ [ n ] , m i ∈ M i , h ∈ [ H ]

Proof. We will prove by backward induction. Obviously, it holds for h = H +1 . Now we assume the lemma holds for h +1 . Now we notice that by definition,

where the last step is by inductive hypothesis. Now note that for any s h , p h , a h , we have

where we notice P G ( o h +1 | s h , a h ) = ∑ s h +1 O h +1 ( o h +1 | s h +1 ) T h ( s h +1 | s h , a h ) for the first inequality, and the second inequality comes from the construction of our bonus b k -1 h ( s h , a h ) in Equation (J.3) and Lemma J.10. Meanwhile, by the definition of value functions, it holds that

Now we conclude

By definition, we have Q high ,k i,h ( c h , γ h ) = Q high ,k i,h ( ̂ c h , γ h ) , thus proving Q high ,k i,h ( ̂ c h , γ h ) ≥ ̂ Q ( m i ⋄ π k i ) ⊙ π k -i , G i,h ( c h , γ h ) . Now for the value function, note that

where the first step is by the property of Bayesian CE, and the second step is by Q high ,k i,h ( c h , γ h ) ≥ ̂ Q ( m i ⋄ π k i ) ⊙ π k -i , G i,h ( c h , γ h ) for any γ h ∈ Γ h . Again by definition, we proved V high ,k i,h ( ̂ c h ) = V high ,k i,h ( c h ) ≥ ̂ V ( m i ⋄ π k i ) ⊙ π k -i , G i,h ( c h ) .

Lemma J.14 (Pessimism) . With probability 1 -δ , for any k ∈ [ K ] , for Algorithm 7, it holds that for any i ∈ [ n ] , h ∈ [ H ]

Proof. We prove by backward induction on h . Obviously, the lemma holds for h = H +1 . Now we assume the lemma holds for h +1 . Similar to the proof of the previous lemma, we note by inductive hypothesis

where for any s h , p h , a h , we have

where the last step again comes from the construction of our bonus in Equation (J.3) and Lemma J.10. Therefore, we conclude

Similarly, for value function, it holds that

thus proving our lemma.

Theorem J.15 (NE/CCE version) . With probability 1 -δ , Algorithm 7 enjoys the regret guarantee of

Correspondingly, this implies that one can learn an ( ϵ + H 2 ϵ belief ) -NE if G is zero-sum and ( ϵ + H 2 ϵ belief ) -CCE if G is general-sum with sample complexity O ( H 4 SAO log( SAHO/δ ) ϵ 2 ) and computation complexity POLY ( S, max h ∈ [ H ] | ̂ C h | , max h ∈ [ H ] |P h | , H, 1 ϵ , log 1 δ ) .

Proof. Note for any given i ∈ [ n ] and π ′ i ∈ Π i , by Lemma J.12 and Lemma J.14, it holds

Therefore, it suffices to bound V high ,k i,h ( c k h ) -V low ,k i,h ( c k h ) :

where we define the two Martingale difference sequences as follows

and the error of the belief is defined as

Since | Z 1 k,h | ≤ H , | Z 2 k,h | ≤ H , and ϵ h ( c k h ) ≤ 2 , by Azuma-Hoeffding bound, we conclude with probability 1 -3 δ , the following holds

Meanwhile, by the pigeonhole principle, it holds that

Now by Lemma J.12 and Lemma J.14 and putting everything together, we conclude

Now by Lemma J.11, we proved the regret guarantees as follows

For the PAC guarantees, since we define k ⋆ ∈ arg min i ∈ [ n ] ,k ∈ [ K ] V high ,k i, 1 ( c k 1 ) -V low ,k i, 1 ( c k 1 ) , we have

Finally, for two-player zero-sum games, we denote ̂ π k ⋆ to be the marginalized policy of π k ⋆ . Then we have

thus concluding our theorem.

Theorem J.16 (CE version) . With probability 1 -δ , Algorithm 7 enjoys the regret guarantee of

Correspondingly, this implies that one can learn an ( ϵ + H 2 ϵ belief ) -CE with sample complexity O ( H 4 SAO log( SAHO/δ ) ϵ 2 ) and computation complexity POLY ( S, max h ∈ [ H ] | ̂ C h | , max h ∈ [ H ] |P h | , H, 1 ϵ , log 1 δ )

Proof. Then proof follows as that of Theorem J.15, where we only need to change the first step of the proof as

by Lemma J.13 and Lemma J.14, and the remaining steps are exactly the same.

Lemma J.17 (Adapted from Theorem H.5) . Algorithm 8 can learn the approximate POMDP with transition ̂ T 1: H and emission ̂ O 1: H such that for any policy π ∈ Π gen and h ∈ [ H ]

using sample complexity ˜ O ( S 2 AHO + S 3 AH ϵ 2 + S 4 A 2 H 5 ϵ ) with probability 1 -δ .

Proof. Note that Algorithm 8 is essentially treating the POSG G as a centralized MDP and running Algorithm 4, where the only modifications we make in Algorithm 8 is that we take the controller set (see some examples of the controller set in Appendix C.3) into considerations when learning the models. Specifically, for the transition ̂ T h , what we estimate is only ̂ T h ( s h +1 | s h , a I h ,h ) instead of ̂ T h ( s h +1 | s h , a h ) , where T h ⊆ [ n ] is the controller set. Therefore, the sample complexity of Algorithm 8 will not be worse than that of Algorithm 4.

Proof of Theorem 7.7:

Note that the proof idea essentially resembles that of Theorem H.6, where we construct the model G trunc for G in exactly the same way as constructing P trunc for P . Therefore, by Corollary H.10, we have

Meanwhile, we can construct ̂ G trunc and ̂ G sub using exactly the same way as for ̂ P trunc and ̂ P sub , where ̂ G sub is a γ/ 2 -observable POSG.

Now, according to [51], for all the examples in Appendix C.3, there exists a compression function that maps c h to ̂ c h such that the size of the compressed common information is quasi-polynomial, i.e., ̂ C h ≤ ( AO ) Cγ -4 log SH ϵ 2 for some absolute constant C and ϵ 2 ∈ (0 , 1) , and the corresponding approximate belief { ̂ P h : ̂ C h → ∆( S × P h ) } h ∈ [ H ] satisfies that

Therefore, we can do the same augmentation for ˜ P h on states from S low h to construct the approximate belief ̂ P h as in the proof of Theorem H.6, and the remaining steps follow from the proof of Theorem H.6. This will lead to a total of polynomial-time and polynomial-sample complexities. ■

J.1 Background on Bayesian Games

The Bayesian game is a generalization of normal-form games in partially observable settings. Specifically, a Bayesian game is specified as ( n, {A i } i ∈ [ n ] , { Θ i } i ∈ [ n ] , { r i } i ∈ [ n ] , µ ) , where n is the number of players, A i is the actor space, Θ i is the type space, r i : × i ∈ [ n ] (Θ i ×A i ) → [0 , 1] is the reward function, and µ is the prior distribution of the joint type. At the beginning of the game, a type θ = ( θ i ) i ∈ [ n ] is drawn from the prior distribution µ ∈ ∆(Θ) . Then each agent i gets its own type θ i and takes the action a i . We define a pure strategy of an agent as s i ∈ ST i := { Θ i →A i } . We define J i ( s i , s -i ) to be the expected rewards for agent i , given the pure joint strategy ( s i , s -i ) .

By definition, J i ( s i , s -i ) can be evaluated as

Bayesian NE. We define γ ⋆ ∈ × i ∈ [ n ] ∆( ST i ) is an ϵ -NE is it satisfies that

Bayesian CCE. We say a distribution of joint strategies γ ⋆ ∈ ∆( × i ∈ [ n ] ST i ) to be a ϵ -Bayesian CCE if it satisfies

(Agent-form) Bayesian CE. We say a distribution of joint strategies γ ⋆ ∈ ∆( × i ∈ [ n ] ST i ) to be an ϵ -agent-form Bayesian CE if it satisfies

where M i = { Θ i × A i → A i } is the space for strategy modification, where m i modifies s i as follows: given current type θ i and the recommended action a i , the strategy modification changes the action to the another action m i ( θ i , a i ) .

Note that Bayesian NE for zero-sum games, and (agent-form) Bayesian CE/CCE are all tractable solution concepts and can be computed with polynomial computational complexity, e.g., [29, 32, 23].

K Concluding Remarks and Limitations

In this paper, we aim to understand the provable benefits of privileged information for partially observable RL problems under two empirically successful paradigms, expert distillation [14, 64, 58] and asymmetric actor-critic [68, 7, 3], which represent privileged policy and privileged value learning, respectively, with an emphasis on studying both the computational and sample efficiencies of the algorithms. Our results (as summarized in Table 1) showed that privileged information does improve learning efficiency in a series of known POMDP subclasses. One potential limitation of our work is that we only focused on the case with exact state information. It remains to explore whether such an assumption can be further relaxed, e.g., when privileged state information may be biased, partially observable, or delayed, as usually happens in practice, and how our theoretical results may be affected. Meanwhile, as an initial theoretical study, we have been primarily focusing on the tabular settings (except Appendix G), and it would be interesting to extend the results to function-approximation settings to handle massively large state, action, and observation spaces in practice.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract indeed accurately summarizes out paper's contribution and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We clearly outline all of our assumption needed to derive our theoretical results.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We prove every theorem or lemma in this paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, detailed are introduced in Appendix I

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have included detailed introductions on the algorithms and environment to reproduce our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have reported the number of repeated experiments in Appendix I and error/variance bars in Figure 2 and Table 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provides all details in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: It does.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We address the societal impacts of our paper in Appendix A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: No external codes, data, models are used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No such experiments were involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No such approval was needed.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers valuable insights into the use of privileged information in partially observable reinforcement learning (POMDP), presenting theoretical advancements and practical algorithms. However, challenges remain in ensuring the effectiveness of expert distillation, applicability to real-world problems, and bridging the gap between training and test environments. Addressing these limitations is essential for advancing the practical implementation of Privileged Policy Learning.\n\nSTRENGTHS:\n1. The paper formalizes expert distillation, highlighting its limitations in finding near-optimal policies in observable POMDPs. This formalization provides a structured approach to understanding and improving existing empirical methods.\n\n2. The introduction of the deterministic filter condition broadens the scope of tractable POMDP models and ensures that expert distillation can achieve polynomial sample and computational complexities.\n\n3. The investigation into multi-agent RL (MARL) frameworks with privileged information and the proposed algorithms ensure polynomial complexities.\n\nWEAKNESSES:\n1.\tExpert policies in pratical problems may not be optimal [1-3]. This raises concerns about the effectiveness of Privileged Policy Learning. If the expert policies are suboptimal, the student policies distilled from them may inherit these inefficiencies, potentially leading to suboptimal performance in practical applications. \n\n2.\tA significant limitation highlighted is the dependency on access to the state of the POMDP environment. In most practical scenarios, the internal states are not observable, restricting the application of the proposed methods. The deterministic filter condition, while theoretically valuable, may not always be applicable in real-world problems where state observability is a significant challenge.\n\n3.\tThe paper discusses the difficulty of obtaining the internal state in actual problems, creating a gap between the training environment (where privileged information is available) and the test environment (where it is not). This discrepancy can lead to performance degradation when the policies trained with privileged information are deployed in real-world scenarios where such information is unavailable. \n\n[1] Wu, Yueh-Hua, et al. \"Imitation learning from imperfect demonstration.\" International Conference on Machine Learning. PMLR, 2019.\n\n[2] Tian, Yuandong, Qucheng Gong, and Yu Jiang. \"Joint policy search for multi-agent collaboration with imperfect information.\" Advances in neural information processing systems 33 (2020): 19931-19942.\n\n[3] Tang, Yu, et al. \"Hierarchical reinforcement learning from imperfect demonstrations through reachable coverage-based subgoal filtering.\" Knowledge-Based Systems 294 (2024): 111736.\n\nQUESTIONS:\nThe paper does not discuss the tightness of the derived bounds or whether they meet the needs of practical problems.\n\nLIMITATIONS:\nLack of sufficient experimental verification", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper has several contributions. The authors examine using privileged information for partial observability in RL. The paper looks at two empirical paradigms, expert distillation and asymmetric actor-critic, analyzing their computational and sample efficiency under specific conditions.

In addition, the authors introduce a belief-weighted optimistic asymmetric actor-critic algorithm, providing insights into MARL with CTDE.

STRENGTHS:
- There is a considerable amount of summarization and formalization, such as for the expert distillation paradigm, which is appreciated.
- Theoretical results with the proposed deterministic filter condition is a novel contribution.
- The belief-state learning contribution could be interesting for future works.

WEAKNESSES:
- Like most theoretical papers, examination of existing empirical paradigms  and evaluation of proposed method is rather lacking, while understandable, is still a weakness.
- The deterministic filter condition, if I understand correctly, is weaker than previous conditions but still rather restrictive compared to real-world tasks.
- More toy examples could be helpful along side theory work.

QUESTIONS:
- What is the purpose of developing the deterministic filter condition? Can you give a motivating example?

LIMITATIONS:
The authors are upfront about their assumptions; however, I would like to mention that reliance on specific assumptions for computational tractability might limit the generalizability of the results.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel theoretical characterization of certain kinds of\nPOMDP's which admit efficient learning. First, related characterizations are\nexplored and theoretical results show that these classes of POMDP's suffer from\ncertain drawbacks when trying to learn policies. Based on this analysis, a new\nclass of POMDP's is defined using the \"deterministic filter condition\".\nThis characterization of POMDP's is used to develop a novel learning algorithm\nbased on a decomposition of POMDP policy learning into belief learning and a\nfully-observable policy learning step. Theoretical results show that policy\nlearning in POMDP's satisfying the deterministic filter condition condition is\ntractable given access to privileged information at training time. The paper\nalso presents an extension of the proposed algorithm to handle partially\nobservable multi-agent RL.\n\nSTRENGTHS:\n- This paper considers a problem of practical importance, namely how to learn\n  policies when there is more information available at training time than test\n  time.\n- The extension to multi-agent RL is a useful inclusion.\n- Giving some theoretical failings of existing approaches helps motivate the\n  work.\n- The theoretical analysis of the proposed algorithm is extensive.\n\nWEAKNESSES:\n- Some important sections are deferred to the appendix (related work,\n  conclusion, and the discussion of experiments). I understand that space in the\n  main paper is quite limited but I think it's important to include these\n  sections. Maybe instead some theoretical results could be deferred to the\n  appendix, especially those in section 3 concerning the shortcomings of\n  existing approaches.\n- More generally, so much of the discussion in the paper is in the appendix that\n  the main body of the paper is quite hard to read.\n- The central definition 3.2 could be explained a bit more (see questions).\n- The legends on the graphs in the experimental section cover a lot of the\n  relevant information in the graph.\n\nQUESTIONS:\nI'm not sure that my understanding of 3.2 is correct. The intuition given in the\npaper is that it corresponds to allowing $\\gamma$ to go to 1 in assumption C.8.\nBut to my understanding C.8 encodes a certain kind of approximate observability\nand letting $\\gamma$ go to one recovers a fully-observable environment. (I\nbelieve this is related to the restriction that $b^s$ is a one-hot vector in\ndefinition 3.2). But if the environment is observable then there's no need to\nconsider POMDP's anymore, regular MDP's should be sufficient. Could you clarify\nthis definition a bit more, and especially how it is distinct from assuming that\nthe emission function can be inverted?\n\nLIMITATIONS:\nLimitations of the proposed approach are discussed, although this discussion is left to the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission addresses both statistically and computationally efficient reinforcement learning (RL) in Partially Observable Markov Decision Processes (POMDPs). The training phase has access to hidden states, while the goal is to learn the optimal policy at test time without such access. The authors propose a sample and computation-efficient algorithm under a deterministic filter stability condition.\n\nSTRENGTHS:\n- The proposed setting is interesting and has significant potential for practical applications.\n\n- The paper makes a commendable effort to cover various potential cases, enhancing the applicability of the problem settings.\n\nWEAKNESSES:\n**High-Level Critic**\n\n- More space should have been dedicated to elaborating on the real technical question that the paper aims to resolve. With access to hidden states, the statistical hardness is virtually eliminated, so the focus could have been purely on the computational aspect. The poly-sample result is not surprising, and with the $\\gamma$-observable assumption of (Golowich et al., 2022), quasi-polynomial complexity does not seem surprising either.\n\n- The privileged information combined with the deterministic filter condition is very strong. It is unclear why Definition 3.2 was needed given the strong assumption of privileged information.\n\n- The proposed algorithm does not seem as practical as advertised. Beyond the tabular setting, it is unclear how to implement it with general function classes, which are common in deep RL.\n\n\nThe current manuscript needs significant improvement in terms of writing, as detailed below.\n\n- At the beginning of the introduction, it is confusing whether the paper is about RL in POMDPs with more information or about multi-agent RL in POMDPs. The two components investigated in this paper are orthogonal to each other, and the main result seems to be more about the former.\n\n- The contribution of the paper needs to be more clearly articulated in the introduction. My understanding is that the paper is about (1) an algorithm that achieves both sample and computational efficiency with some new technical conditions, and (2) an extension of the proposed condition and algorithm to the multi-agent setting.\n\n- The sentence in Lines 64-65 is confusing. What do we aim to achieve with actor-critic?\n\n- Proposition 3.1 pertains to a specific algorithm, not to the general hardness of the problem.\n\n- Jargon such as privileged, expert distillation, or teacher-student teaching may not help much in motivating the setting of this paper, as these are not the applications the paper pursued. It would be simpler if the problem were motivated from a theoretical perspective, focusing on technical challenges in existing works and some relevance to practice in the context of actor-critic implementation.\nThe decoding function $g$ in Lemma 4.3 is unclear. Definition 3.2 defines a posterior probability given $(s_{h-1}, a_{h-1}, o_h)$, but Lemma 4.3 assumes the posterior is almost deterministic. This could have been clearly stated during the problem setup. This assumption is essentially similar to the block MDP.\n\nQUESTIONS:\n- Definition 3.2: Does $\\gamma$-observability or $\\gamma$-separated PSR imply Definition 3.2?\n\n- When (Golowich et al., 2022) already achieved quasi-computational complexity for planning, what does Theorem 5.3 improve upon it? How are the previous results in Section 4 related to this result?\n\nLIMITATIONS:\nThere are no experiments on practical benchmarks other than synthetic small tabular POMDPs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}