Provable Partially Observable Reinforcement Learning with Privileged Information
Authors: Yang Cai
Xiangyu Liu
Argyris Oikonomou
Kaiqing Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: reinforcement learning
pomdp
partial observability
computational
privileged information
expert distillation
teacher-student learning
Primary area: reinforcement_learning
TL;DR: We study the provable efficiency of partially observable RL with privileged information to the latent state during training for both single-agent and multi-player settings.

Abstract: Partial observability of the underlying states generally presents significant challenges for reinforcement learning (RL). In practice, certain *privileged information* , e.g., the access to states from simulators, has been exploited in training and achieved prominent empirical successes. To better understand the benefits of privileged information, we revisit and examine several simple and practically used paradigms in this setting, with both computation and sample efficiency analyses. Specifically, we first formalize the empirical paradigm of *expert distillation* (also known as  *teacher-student* learning), demonstrating its pitfall in finding near-optimal policies. We then identify a condition of the partially observable environment, the deterministic filter condition, under which expert distillation achieves sample and computational complexities that are *both* polynomial. Furthermore, we investigate another successful empirical paradigm of *asymmetric actor-critic*, and focus on the more challenging setting of observable partially observable Markov decision processes. We develop a belief-weighted optimistic asymmetric actor-critic algorithm with polynomial sample and quasi-polynomial computational complexities, where one key component is a new provable oracle for learning belief states that preserve *filter stability* under a misspecified model, which may be of independent interest. Finally, we also investigate the provable efficiency of partially observable multi-agent RL (MARL) with privileged information. We develop algorithms with the feature of centralized-training-with-decentralized-execution, a popular framework in empirical MARL, with polynomial sample and (quasi-)polynomial computational complexity in both paradigms above. Compared with a few recent related theoretical studies, our focus is on understanding practically inspired algorithmic paradigms, without computationally intractable oracles.

Provable Partially Observable Reinforcement Learning with Privileged Information
Yang Cai
Xiangyu Liu
Argyris Oikonomou'
Kaiqing Zhang?
Yale University
University of Maryland, College Park
yang. cai@yale.... edu, xyliu9990umd, edu argyris owkonomou@yale....mmmoo.o..ooaa. edu, kaiqingound. edu
Abstract
Partial observability of the underlying states generally presents significant chal- lenges for reinforcement learning (RL). practice, certain privileged information, e.g., the access to states from simulators, has been exploited in training and has achieved prominent empirical successes. To better understand the benefits of priv- ileged information, paradigms in this setting. Specifically, we first formalize the empirical paradigm of expert distillation (also known teacher-studen learning), demonstrating its pitfall finding near-optimal policies. We then identify a condition of the partially observable environment, the deterministic filter condition, under which expert dis- tillation achieves sample and computational complexities that are both polynomial. Furthermore, investigate another successful empirical paradigm of asymmetric actor-critic, and focus on the more challenging setting of observable partially observable Markov decision processes. We develop welief-weighhed asymmetric actor-critic algorithm with polynomial sample and quasi- mausi-polynomial computational complexities, which one key component is new provable oracle for learning belief states that preserves filter stability under a misspecified model, which may be of independent interest. Finally, we also investigate the provable efficiency of partially observable multi-agent RL (MARL) with privileged information. We develop algorithms featuring centralized-trod -ewehiinn-wit--eeeeneee -xxecutionnnnnni a popular framework in empirical MARL, with polynomial sample and (quasi- ppolynomial computational complexities both paradigms above. Compared with few recent related theoretical studies, our focus is understanding practically inspired algorithmic paradigms, without computationally intractable oracles.
revisit and examine several simple and practically used
Introduction
In most real-world applications of reinforcement learning (RL), c.g., perception- based robot learning [46, 3], autonomous driving [73, 41], dialogue systems [88]. and clinical trials [77], only partial observations of the environment state are available for sequential decision-making. Such partial observability presents significant challenges for efficient decision-makin and learning, with known computational [66] and statistical [43, 36] barriers under the general model of partially observable Markov decision processes (POMDPs). The curse of partial observability becomes severer when multiple RL agents interact, where not only the environment state, but also other agents' information, are not fully-observable in decision-making [85, 80].
On the other hand, a flurry of empirical paradigms has made partially observable (multi-agent) RL promising in practice. One notable example is to exploit the privileged information that may be available (only) during training. The privileged information usually includes direct access to the underlying states, as well as access to other agents' observationssaccions in multi-agent RL (MARL), due the use of simulators and/or gege-preeision sensors for training. The latter is also known as
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the centralized- mnntralized-raiinngg-ae sithin--ttt--deeetraaieedddd become prevalent in practice [53, 70, 22, 82]. These approaches can be mainly categorized into two types: i) privileged policy learning, where an expert/teacher policy is trained with privileged information, and then distilled into student partially observable policy. This expert distillation, also known as teacher-student learning, approach has been the key to some empirical successes in robotic locomotion [45, 59] and autonomous driving [14]; ii) privileged value learning, where a value function is trained conditioned on privileged information, and used to improve a partially observable policy. is typically instantiated as the asymmetric actor-critic algorithm [68], and serves as the backbone of some
(CTDE) framework in deep MARL, and has
successes in robotic manipulation [46, 3] and MARL [53, 82].
Despite the remarkable empirical successes, theoretical understandings of partially observable RL with privileged information have been rather limited, except for a few recent prominent advances in RI. with hindsight observability [44, 30] (see Appendix B for detailed discussion). However, most of these theoretically sound algorithms are different from those used in practice, and require computationally intractable oracles to achieve provable sample efficiency. The soundness and efficiency of the aforementioned paradigms used in practice remain elusive. In this work, we examine both paradigms of expert distillation and asymmetric actor-critic, with foresight privileged information as in these empirical works. In contrast to [44, 30J, which purely focused sample efficiency, we aim to understand the benefits of privileged information by examining these practically inspired paradigms under several POMDP models, without computationally intractable oracles. We defer a detailed literature review to Appendix B, and summarize our contribution as follows.
Contributions. We first formalize the empirical paradigm of expert distillation, and demonstrate its pitfall in distilling near-optimal policies even in observable POMDPs, a model class that was recently shown to allow provable partially observable RL without computationally intractable oracles [25]. We then identify a new condition for POMDPs, the deterministic filter condition, and establish sample and computational complexities that are both polynomial for expert distillation. The new condition is weaker and thus encompasses several known (statistically) tractable POMDP models (see Figure 1 for summary). Further, revisit the asymmetric actor-critic paradigm and analyze its efficiency under the more challenging setting of observable POMDPs above (where expert distillation fails). Identifying the inefficiency of vanilla asymmetric actor-critic, and inspired by the empirical success in belief-stute-learniig critic, with polynomial-sample and guaai-polynomial-ting belief-state learning oracle that preserves filter stability under a misspecified model, which may be of independent interest. Finally, we also investigate the provable efficiency of partially observable multi-agent RL with privileged information, by studying algorithms under the CTDE framework, with polynomial-sample and (quasi-ppolynomial- complexities in both paradigms above.
we develop a new belief-weightee version of asymmetric actor- complexities. Key to the results is new
2 Preliminaries
2.1 Partially Observable RL (with Privileged Information)
r). where H denotes the length of each episode, S is the state space with |8| = S. A denotes the action space with A = A. We use T = (TTe)e to denote the collection of transition matrices, so that (- E A(S) gives the probability of the next state if action a is taken at state s and step h. In the following discussions, for any given a, we treat T (a) E R S] (S] as a matrix, where each row gives the probability for reaching each next state from different current states. We
Model. Consider a POMDP characterized by a tuple P = A,,,,,
μu1 to denote the distribution of the initial state 81, and O to denote the observation space with |0] = =0. We use O (0a) ) he[er] to denote the collection of emission matrices, so that O (- s) E AOO gives the emission distribution over the observation space O at state s and step h. For notational convenience, we will times adopt the matrix convention, where Oj is matrix with rows O2, (- {)) for each E S. Finally, r = {ra]NetH) is a collection of reward functions, so that (s, a) E [0, 1] is the reward given the state and action a at step h. When privileged information is available, the agent can observe the underlying state E S directly during training (only). We thus denote the trajectory until step h with states as Fn (s1:h; 01:k: 01:h-1). the one without states as Th (01:h: 01:A-1), and its space as Tr. Finally, we use (7x)) to denote the posterior distribution over the underlying state at step given history The which is known as the belief state (c.f. Appendix C.I 1. for more details).
Policy and value function.
We define stochastic policy at step h as:
Th 0³ x hh + A(A),
(2.1)

---

where the agent bases on the entire (partially observable) history for decision-makinn The cor- responding policy class is denoted as I1, We further denote II Xhe[m)lIa. We also define I1 : ( Ty : Shh x Oh x AAh11 - A(A) for h E [H]} to be the most general policy space in partially observable RL with privileged state information, which can potentially depend all historical states, observations, and actions. It can be seen that I[[[ We may also use policies that only receive a finite memory instead of the whole history inputs: fix an integer L > O, we define the policy space IT to be the space of all possible policies π = T1:H == (TT))E [R) such that Th ZA. + ((A) with Zn := Omiu(L.h) X Ammin(L.h) for each h E [M]. Finally, we define the space of state-based policies as IIs, i.e., for any π = #1. E lls, TTy : + (AA) for all E [M].
Given the POMDP model P. we write P. $1: 11 aI: (o.= to denote the event & when (81.H+1, #1:H1 01:H) ) is drawn as a trajectory following the policy æ in the model P. We will also use the shorthand notation (-) if (51:H+1, H1:H, 01:H) H) is evident. We write 1E to denote the ex- pectation similarly. We define the value function at step h as V (yh) == E [Ei re(#,, ae) [n]- denoting the expected accumulated rewards from step h. where yh S (s1.hh O1ch, a1:h-1), and we slightly abuse the notation by treating as a set the sequence of states $1sh, the sequence of observations o1 and the sequence of actions 01:A-1 up to time h, which are the available information to the agent at step h. We say Uh is reachable if there exists some policy π E I such that IT (Un) > O. For h = 1. we adopt the simplified notation v. (Tz) E E Th (sp, a. Meanwhile, we also define 0% (Uh, an) == E re(ss, a+) y/,, ax.] We denote the occupancy measure on the state space d.. (sx) = pa.P (st). The goal of learning in POMDPs is to find the optimal policy that maximizes the expected accumulated reward over the policies that take Th as input at each step h E [H], i.e., those E 11. Formally, we define:
H
Definition 2.1 (e-optimal policy). Given e <> 0, policy x² E is e-optimal, if o (TT*) N
ma vº (TT) 6.
Learning with privileged information. Common RL algorithms for POMDPs deal with the scenario where during both the training and test time, the agent can only observe its historical observations and actions Tr at step h, while the states are not accessible. In other words, the agent can only utilize policies from II to interact with the environment. In contrast, in settings with privileged information, c.g., training in simulators and/or using sensors with higher precision, the underlying state can be used in training. Thus, the agent is allowed to utilize policies from the class I1 during training. Meanwhile, the objective is still to find the optimal history-dependent policy in the space of II, since at test time, the agent cannot access the state information anymore, and it is the performance for such policies that matters eventually. For simplicity, we assume the reward function is known since under our privileged information setting, learning the reward function is much easier than learning the transition and emission, and the sample/computational complexity for the former is dominated by that for the latter. This assumption has also been made for learning in POMDPs without privileged information [36, 47, 48].
2.2 Partially Observable Multi-agent RL with Information Sharing
Partially observable stochastic games (POSGs) are a natural generalization of POMDPs with multiple agents of potentially independent interests. We define a POSG with agents by tuple G (H, 1/ y T, O, μ1, (-)) where each agent has its individual action space A,, observation space O., and reward function r; {Te.,e Me with ri (s, a) [0, 1] denoting the reward given state 8 and joint action a for agent i at step h. An episode of POSG proceeds as follows: each step h and state 8h, a joint observation is drawn from (0s,n)<e(x) ~ 0 [s)), and each agent receives its own observation og,h+ takes the corresponding action di,h, obtains the reward rr, (8h, ox). where aj := (st))) and then the system transitions to the next state as $/+1 ~ Tx (- sh, ax). Notably, each agent i may not only know its local information (o,,1.h; dt,1::-1) but also information from some other agents. Therefore, we denote the information available to each agent i at step h also as Tr, C (Diih, 11) and define the common information as cr = Oie(n) Tr.h and private information as Pi,k = Ti.k Ch. We denote the space for common information and private information as Cn and Pi.,. for each agent i and step h. The joint private information at step h is denoted as pt. = (pt,))e(()) where the collection of the joint private information is given by Pa. = Pi,A x... x Pr,,. We refer more examples of this setting of POSG with information-ssaring to Appendix C.2 (and also (62, 63, 51].. Correspondingly, the policy each agent deploys at test time takes the form of Tih Sn x Cn PLA + A(A.), where 22 is the space of random seeds. We denote the policy
3

---

With STD:
Without PI
With Pl (Ours)
"DDggrmnismm
Transition
Detemmnnattc POMDP
Without adtirional recumngs
Orackeetflcient
60)
Tabular: Poly sample + time
poly sample puly time
(puly sample, puly limes)
Detemminitic
Compuissimally harder than SL. (28)
A-drowaable
Expesemiai-it--
poly sampla quasi-poly time
POMDP
sample time [19]
FA:
Without WSF.:
Poly sample Classification (SL) oraele
ledecodabee POMDP
POMPP
Suttstically hand [177
With
unknown
poly sample
Only
Dok small oxnates))
Poly sample [36]
(poly aimple
Observable
det. niter
POSG with
POMDP
Quasi-poly sample time
N/A
Poly sampll + time
Poly sample + Quaci poly
known setting
mnsoown
Weakly revedling
Obsexable
POMOP
Block MEP MDP
Übservation
POSG
Observable
[21]
POMDP
Mnformativenees
Observablity:
Table 1: Comparison of the theoretical guarantees with and without privileged information. Pl: priv- ileged information; STD: structural assumptions on transition dynamics, e.g., deterministic transi- tion or reachability of all states; SL: supervised learning; FA: function approximation; WSE: well- separated emission.
(poly sample, quasi-poly time)
Figure 1: A landscape of POMDP models that partially observable RL with privileged informa- tion can/cannot address. The axes denote the "re- strictiveness" of the assumptions, on the emission channels and transition dynamics, respectively.
space for agent as IT. If Tish takes the state 8y instead of (Ch: Pi.A) as input, we denote its policy space as 5s.,,, e.g., for each agent i, and policy T1:H1 ITs we have Ti,h S - A(A:) for each step h E []] Similar to the POMDP setting, we define 11 to be the most general policy space, i.c., I : T1-H Ty : Sh O² x A + ((A) for h E } Note that this model covers several recent POSG models studied for partially observable MARL, e.g., [49, 27]. For example, at each step h, if there is no shared information, then e₁ 0. and all history information is shared, then inforeaation-bssed learning, the training algorithm may exploit not only the underlying state information, but also the observations and actions of other agents.
Pi.s 0 for all i E [n]] In
Solution concepts. The solution concepts for POSGs are usually the equilibria, particularly Nash equilibrium (NE) for two-player zero-sum games (i.e.. when = 2 and Fi,A + 12,4 1),' and correlated equilibrium (CE) or coarse correlated equilibrium (CCE) for general-sum games. We defer the formal definitions of these standard solution concepts to Appendix C.2.
2.3 Technical Assumptions for Computational Tractability
key technical assumption is that the POMDPs/POSGs we consider satisfy an observability assump- tion, outlined below. This observability assumption allows us to short memory policies to approximate the optimal policy, and yields mnasi-polynomill
complexity for both planning and learning in POMDPs/POSGs [26, 25, 51]. Meanwhile, we defer an additional assumption to ensure the traceability for solving POSGs to Appendix C.3.
Assumption 2.2 (y-observability [20, 26, 25]). Let > O. For h E [H) we say that the matrix Og satisfies the y-observabilityy assumption for each E [H), for any b, D' E (S)), |0o - O b'1, >
y ||6 6' A POMDP/POSG satisfies y-observability if all its Og for h E [M] do so.
3 Revisiting Empirical Paradigms of RL with Privileged Information
Most empirical paradigms of RL with privileged information can be categorized into two types: i) privileged policy learning, where the policy in training is conditioned on the privileged information, and the trained policy is then distilled to policy that does not take the privileged information as input. This is usually referred to as either expert distillation [14, 64, 58] or teacher-student learning [45, 59, 75] in the literature; ii) privileged value learning, where the value function conditioned on the privileged information, and is then used to directly output a policy that takes partial observation (history) as input. One prominent example of ii) is asymmetric-actor-co noting that asymmetric-actor for multi-agent RL, centralized- -treiddtrnninn--iiee weweentalized--eeeeeeeee instantiated under the actor-critic framework, with the critic taking privileged information input
[68, 3]. is worth is also closely related to one of the most successful paradigms
[53, 86, 21], which is usually
Note that we require F1,A + 12,h to be I instead of 0 to be consistent with our assumption that riA4 E [0, 1] for each i E j0, 1J. and this requirement does not lose optimality as one can always subtract the constant-sum. offset to attain zero-sur sum reward structure.

---

in training. Here we formalize and revisit the potential pitfalls of these two paradigms, and further develop theoretical guarantees under certain additional conditions and/or algorithm variants.
3.1 Privileged Policy Learning: Expert Policy Distillation
The motivation behind expert policy distillation is that learning an optimal fully observable policy in MDPs is a much easier and better-studied problem with many known efficient algorithms. The (expected) distillation objective can be formalized as follows:
H Dy (T (4) (-
#* arg min
B
(3.1)
where x' E I is some given behavior policy to collect exploratory trajectories, π² E arg max (T) denotes the optimal fully observable policy, and D, denotes the general f- divergence to measure the discrepancy between n²" and т.
Such formulation looks promising since it essentially circumvents the challenging issue of ex- ploration in partially observable environments, by directly mimicking an expert policy that can be obtained from any off-the- MDP learning algorithm. However, we point out in the following proposition that even if the POMDP satisfies Assumption 2.2, the distilled policy can still be strictly suboptimal even with infinite data, i.e., by solving the expected objective Equation (3.1) completely. We postpone the proof of Proposition 3.1 to Appendix E.
Proposition 3.1 (Pitfall of expert policy distillation). For any 7 E (0, 1), there exists a 7-observable POMDP P with D in Equation (3.1), it holds that u (f*) S max
S = O A 2 such that for any behavior policy x' E II and choice of
(T) (1=))1-7)
The key reason Equation (3.1) fails is that in general, the underlying state can remain highly uncertain even given the full history. Thus, the distilled policy may not be able to mimic the state-based expert policy well at different states 8n the associated n 8%.) differs significantly across 8). To see how may rule out such an issue, notice that if
(note that according to Assumption 2.2, we have y is at most since y S |0,, loc I 1), implying that the observation can decode the underlying state, the bound in Proposition 3.1 becomes vacuous. Inspired by this, we propose the following condition that incorporates this case of ? = 1, and will be shown to suffice to make expert distillation effective.
Definition 3.2 (Deterministic filter condition). We say a POMDP P satisfies the deterministic filter condition for each h > 2, the belief update operator under P satisfies that there exists an unknown function 1h S x A x O + S such that for any reachable 8h-1 E S, O% E O, ah-1 E A, Un (b 0h--1 on) = (.0))))) where we define for any 8 E S, 1/ E A(S) and (s)) 1 is a one-hot vector.
addition, for = 1, there exists a function 1/ O + S such that for any reachable 01, Bi (1) (01(0)) where (b; := ~ (o)))) A(S).
Un(b; an-1, on) == PR. P app-1, -1 e (SS) are the belief update operators under the Bayes rule for any b E A(S), for which we defer the formal introduction to Appendix C.I.
Notably, this condition is weaker than and thus covers several known tractable classes of POMDPs with sample and computation efficiency guarantees including Block MDP, deterministic POMDP, k-decodable POMDP as well as a new setting we have identified and existing literature cannot handle. We refer the formal introduction to Appendix and Figure 1 for an illustration.
In light of the pitfall in Proposition 3.1, we will analyze both the computational and statistical efficiencies of expert distillation in Section under the condition in Definition 3.2.
3.2 Privileged Value Learning: Asymmetric Actor-Critic
Asymmetric actor-critic [68] iterates between two procedures as in standard actor-critic algorithms [42]: policy improvement and policy evaluation. As the name suggests, its key difference from the standard actor-critic is that the algorithm maintains Q-value functions (the critic) based on the state/privileged information, while the policy receives only the (partially observable) history input.
Policy evaluation. At iteration t - 1. given the policy -'-1, the algorithm estimates Qufuncionss in the form of (Q (Th, Sh, an) )e(H.. where we adopt the "unbiased" version [6] such that Q-
5

---

functions are conditioned on both the history and the states. One key to achieving sample efficiency by adding some bonus terms in policy evaluation to encourage exploration. i.e., obtaining some optimistic Q-function estimates, similarly as in the fully-observable MDP setring, see e.g., [11, 74], for which we defer the detailed introduction to Section 4.
Policy improvement. At each iteration I, given the critic 1 8h, an H for x'-1, the vanilla asymmetric actor-critic algorithm updates the policy according to the sample-based gradient estimation via K trajectories {st.H+1. oi. af.H (k) sampled from x²-1
f PROIU
+
Σ REJK] NE
Σ
Vx oo
- "(h Th 2 (rh.. shi att
(3.2)
where X is the step-size and PROJ is the projection operator onto the space of II, which corresponds to projecting onto the simplex of (-A) for each h E [H]. Here we point out the potential drawback of the vanilla algorithm as in [68, 6]. where the key insight is that for each iteration of policy evaluation and improvement, one roughly only performs the computation of order O(K H), while needing to collect K new episodes of samples. Thus, the sample complexity will scale in the same order as the computational complexity when the algorithm converges after some iterations to e-optimal solution, which will be super-polynomial even for 7-observable POMDPs [27]. Proof of the result is deferred to Appendix E.
Proposition 3.3 (Inefficiency of vanilla asymmetric actor-critic). Under the tabular parameteri- zation for both the policy and the value function, the vanilla asymmetric actor-critic algorithm (Equation (3.2)) suffers from super-pole super-polynomial sample complexity for y-observable POMDPs under standard hardness assumptions.
To address such an issue, one may need to perform more computation per iteration, so that although the total computational complexity (iteration number x per-iteration computational complexity) is poprrmpllynomia, the total iteration number can be lower such that the total sample complexity may be lower as well. This desideratum is hard to achieve if one computes policy update only on the sampled trajectories Tı per iteration, i.e., update asynchronously, since this will couple the scales of computational and sample complexities similarly as Equation (3.2). In contrast, we first propose to update all trajectories per iteration in synchronous way, with the following proximal-policy optimization-typa [72] policy improvement update with the state-history depeedee- Q- ffffucctins (Q) (Th, 8h; (n)) (N)))
*/( Th) x o - (: (x)) exp (N(Ee)-bu(T)) 1 Sh, ()),))),, vh E [H), Th E Th
(3.3)
where we recall by (Tr) E A(S) denotes the belief state and 1) > O is the learning rate. This update rule also reduces to the natural policy gradient (NPG) [40] update under the softmax policy parameterization in the oully-observable case [1], when updated for each state 8n separately [74]. We defer the detailed derivation of Equation (3.3) to Appendix E.
However, such an update presents two challenges: (1) It requires enumerating all possible Th, whose number scales exponentially with the horizon, making it still computationally intractable; (2) An explicit belief function b,, is needed. Motivated by these two challenges. we propose to consider finite- memory-based policy and assume access to an approximate belief function {6" : Zn + (SS))ee(m (the learning for which will be made clear later). Correspondingly, the policy update is modified as:
< x T/ - (-I=a) exp (B) 1 Sh. sha :1]) Vh E [M], E Zn.
Then we develop and analyze one possible approach to learning such an approximate belief efficiently (c.f. Section 5). It is worth noting that the policy optimization algorithm we aim to develop and analyze does not depend on the specific algorithm approximate belief learning. Such a decoupling enables more modular algorithm design framework, and can potentially incorporate the rich literature on learning approximate belief's in practice, see e.g., [24, 65, 16, 87, 83], which has mostly not been theoretically analyzed before. We will thus analyze such oracle in Section 5.
2 As pointed out in [6]. the original asymmetric actor-critic [68], where the value function was only conditioned on the state, is biased estimate of the actual distory-conditionas value function that appears the policy gradient in the horiiie- discounted setting. We verify that such a state-based value function is indeed also biased under our finite-horizon setting, see Remark E for an example. We will thus use the unbiased value function estimate conditioned on both the state and the history throughout.

---

4 Provably Efficient Expert Policy Distillation
We now focus on the provable correctness and efficiency of expert policy distillation, under the deterministic filter condition in Definition 3.2. We will defer all the proofs in this section to Appendix F. Definition 3.2 motivates us to consider only succinct policies that incorporate an auxiliary parameter representing the most recent state, well the most recent observations and actions. We consider policies that are the composition of two functions: at step h a function gh : S x A x O S that decodes the state based on the previous state, the most recent action, and the most recent observation, and a policy n. E IIs that takes as input the current (decoded) underlying state and outputs distribution over actions.
Definition 4.1. We define a policy class ITP as:
Ir = { (gh (5, (x-1, 9h S Sx A(A) nE(H]
where T. stands for an arbitrary expert policy, and II stands for the distilled policy class, and for h = 1, ag. se are some fixed dummy action and state. Intuitively, the distilled policy π E II exccutes follows: it firstly decodes the underlying states by applying {gt bein) recursively along the history, and then takes actions using xh based on the decoded states.
Our goal is to learn the two functions independently, that is, we want to learn an approximately optimal policy F= E ITs with respect the MDP M derived from POMDP P by omitting the observations and observing the underlying state (see Definition 4.2 for a formal definition), and for each step h E [H] a decoding function gh (5 1,- 0%) such that the probability of incorrectly decoding a state-action cate-action-oossevation triplet over the trajectories induced by the policy x is low.
Definition 4.2 (POMDP-induced MDP). Given POMDP P (H, S, A, O, (4:1777) r), we define its associated Markov Decision Process (MDP) M as M = (,,,,, M4, r) without observations. Definition 4.3. Consider a POMDP P that satisfies Definition 3.2, and let vb {vA be the promised set of functions that always correctly decode a state-action-observaate underlying state. Consider policy π (v).)) S Axx A(A))e(M E II.. We slightly abuse the notation and simply denote by 2 (T v (TT).
triplet into an
Lemma 4.4. Let P = (H, O, Ir, O, r) be a POMDP that satisfies Definition 3.2, set of decoding functions {gh heE(M)
and consider policy x= E lls. Consider
such that, pa FP
[3h [H] 9h (sh-1, -, 0%) 7 sn] I C. Consider the policy π [Th (9x())) 8 x A O + A(A)) ле[H] on the POMDP P, then: v²(TT) 2 (- He.
We can use any off-the-shelf algorithm to learn an approximate optimal policy T F for the associated MDP M (see Definition 4.2). Thus, in the rest of the section, we focus on learning the decoding function {9h Jae To efficiently learn the decoding function, we model the access to the underlying state by keeping track of the most recent pair of the action and the observation, as well as the two most recent states. We summarize the algorithm of decoding-functio
learning in Algorithm 1.
Theorem 4.5, Consider a POMDP P that satisfies Definition 3.2, a policy TF E IIs, and let Then, with probability at {9h Jo)E be the output of Algorithm I with M = Aoo+log(H/8)
2²
least - 8, for each step h E [H]: PP ,P [3h E [H] 9h (s) a on) # sn] I e, using POLY(H, O, S, L log (t)) episodes in time POLY A, S, log (!)).
The following is an immediate consequence of Lemma 4.4 and Theorem 4.5. Note that both the sample and computation complexities are polynomial, which is in stark contrast the k-decodable POMDP case [19] (a special one covered by our Definition 3.2). for which the sample complexity is necessarily exponential in k when there is no privileged information [19]. In fact, thanks to privileged information, the complexities are only polynomial in horizon H even when the decodable length is unknown. For the benefits of using privileged information in several other subclasses of problems, we refer to Table 1 for more details.
Theorem 4.6. Let P satisfy Definition 3.2 and consider any policy T IIs. Using POLY A, O, S, e log (1)) episodes and in time POLY A, O, 1 log ( (1)) we can compute a policy x E IP (see Definition 4.1) such that with probability at least 1 - 8, v (T)) 2 (m
Extension to the case with general function approximation. Due to the modularity of our algorithmic framework and its compatibility with supervised learning oracles, it can be readily generalized to the function approximation setting to handle large observation spaces. We defer the corresponding results to Appendix G.
7

---

5 Provable Asymmetric Actor-Critic with Approximate Belief Learning
Unlike most existing theoretical studies on provably sample-efficient partially observable RI. [36, 25, 47], which directly learn an approximate POMDP model for planning near-optimal policies, we consider general framework with two steps: firstly learning an approximate belief functionn, followed by adopting fully observable RL subroutine on the belief state space.
5.1 Weiih--e Optimistic Asymmetric Actor-Critic
We now introduce our main algorithmic contribution to the privileged policy learning setting. Our algorithm is conceptually similar to the natural policy gradient methods [40, 1, 74] in the fully- observable setting, with additional weighting over the states using some learned belief states, to handle the additional depeedeeee in the asymmetric critic. The overall algorithm is presented in Algorithm 2. The algorithm requires a belief-learning subroutine that takes the stored memory as input and outputs belief about the underlying state (c.f. (6% ) (ee Additionally, similar to the fully observable setting, we include subroutine to estimate the Q-function, which introduces additional challenges due to partial observability (see Appendix H). We establish the performance guarantee of Algorithm 2 in the following theorem. We defer the proof to Appendix H.
Theorem 5.1 (Near-optimal policy). Fix e, E (0, 1). Given a POMDP P and an approximate belief' (b% Z,_ + ((S))e)(M) with probability at least 1 - 6, Algorithm 2 can learn an approximate opti- mal policy r* P in the space of IT such that uP (m*) 2 max v²(m) (T)
-O + H² Heeeeet with sample complexity POLY(S, A, O, H, 3. looo }) and time complexity POLY(S, O, H, H, Z, 2 1o }). where ébelief is the belief-learning error defined as éhelief == [a max E (T,()T)) 6ATh (za)lla and Z == max Zzl.. Furthermore, if P is additionally 7-observable (c.f. As- sumption 2.2), then x² is also an approximate optimal policy in the space of II such that P (x*)= 2 maxxen v (T) O(e + ehelier) as long as L 2 log(S/))))
5.2 Approximate Belief' Learning
At high level, our belief-learning algorithm first learns approximate POMDP model P by explicitly exploring the state space. The main technical challenge here is that there may exist states that reachable with very low probability, making it infeasible to collect enough samples to sufficiently explore them, thus potentially breaking the --observability property of the ground-truth model P. To cireumvent this issue, we ignore such hard-to-visit states and redirect probabilities flowing to them to certain other states. Thus, in our truncated POMDP, where each state is sufficiently explored, we can approximate the transition and emission matrices desired accuracy uniformly across all the states and preserve the 7-observability property. This ensures that the learned approximate belief function in the truncated POMDP is sufficiently close to the actual belief function of the original POMDP P. Note that the key to achieving belief learning with both polynomial sample and time complexities is our explicit exploration in the state space, which relies on executing, fully observable policies from an MDP learning subroutine. We remark that the belief function may also be learned even if the state space is only explored by partially observable policies, thus utilizing only hindsight observability may be sufficient for this purpose [44]. However, for such exploration to be computationally tractable, one requires to avoid using computationally intractable oracles for POMDP learning. which in fact our final goal. We present the guarantees in the next theorem and postpone the proof to Appendix H.
Theorem 5.2. Consider a --observable POMDP P (c.f. Assumption 2.2) and assume that L 2 21 log(SH/c) for an e > 0. Then, can gorithm 4 using Ög SHAH"O AH² + S*A"H"0) episodes in time POLY (S, H, A, 1 log (t) such that with probability at least 1 1-66 8, for any π E I and h E [H), EP b,, (T%) bh (2+) S c.
)E[H)] from Al- can learn an approximate belief (b)
oq²
7'
Theorem 5.2 shows that an approximate belief can be learned with both polynomial samples and time, which, combined with Theorem 5. yields the final polynomial sample and quasi-polynomin time guarantee below. In contrast to the case without privileged information [25, 27]. the sample complexity is reduced from quasi-polynomia to polynomial for --bbeervable POMDPs. Note that the computational complexity remains quasi-poli polynomal,, which is known to be unimprovable even for planning [27]. The key to such an improvement, as pointed out in Section 3.2, is the more practical update rule of actor-critic (in conjunction with our belief-weighted idea), which allows more computation at each iteration (instead of only performing computation at the sampled finite-mmmory) while the overall sample complexity
This allows the total computation to remain quasi-polynone becomes polynomial. A detailed comparison can be found in Table 1.
8

---

Case
Case 2
Case
Case
Cyrakdi.oreeat Vitoes)
Optiiite
Nopreniio
doyoraniig Va AAC
Acponsiic
Rewards
Werll MCC
Rewards
Rewards
Rewards
AaILAAC
22
Episodes
Episodes
Episodes
Episodes
Figure 2: Results for POMDPs of different sizes, where our methods achieve the best performance with the lowest sample complexity (VI: value iteration; AAC: asymmetric actor-critic).
Theorem 5.3. Let P be a 7-observable POMDP (c.f. Assumption 2.2), and consider I, > ( (o)) for an > 0. With probability at least 8, Algorithm 2 can learn pol- icy π E II such that x² (x) 2 max (xx')) e, using POLY(S, H, 1/,, 1/7, log(1/6), 0, episodes and in time PSLY(S, H. log(//), 0 A²..
6 Numerical Validation
We now provide some numerical results for both of our principled algorithms. Here mainly compare with two baselines, the vanilla asymmetric actor-critic [68], and asymmetric Q-learning [7], on two settings, POMDP under the deterministic filter condition (c.f. Detinition 3.2) and general POMDPs. We report the results in Table 2 and Figure 2, where our algorithms converge faster to higher rewards. We defer the implementation details and discussions to Appendix I.
7 Extension to Partially Observable MARL with Privileged Information
7.1 Privileged Policy Learning: Equilibrium Distillation
To understand how the deterministic filter condition may be extended for POSGs, we first note the following equivalent characterization of Definition 3.2, the proof of which is deferred Appendix J.
Proposition 7.1. Definition 3.2 is equivalent to the following: for each h [H]. there exists unknown function Φn Th + S such that (sh = dh (Tx) mh) = for any reachable Th E TT
Proposition 7.1 implies that at each step h, given the entire history information, the agent can uniquely decode the current underlying state $h- Thus, we generalize this condition to POSGs by requiring each agent to uniquely decode the current state 8/ given the information it has collected so far.
Definition 7.2 (Deterministic filter condition for POSGs). We say POSG G satisfies the determinis- tic filter condition for each E [m), / E [M]. there exists an unknown function Φi.A C₁ x PP,A + S such that TP0 (sn Φi.h (Cr. Pi,A) | Ch. Pi. = for any reachable Ps.h).
Here have required that each agent can decode the underlying state through their own information individually. Naturally, one may wonder whether one can relax it so that only the joint history information of all the agents can decode the underlying state. However, we point in the following that it does not circumvent the computational hardness of POSG, the proof of which is deferred to Appendix J. Note that the computational hardness result can not be mitigated even with privileged state information, as the hardness we state here holds even for the planning problem with model knowledge, with which one can simulate the RL problems with privileged information.
Proposition 7.3. Computing CCE in POSGs that satisfy that for each step h E [M], there exists function da : C₁. x Ph, + S such that pP8 (8h- = (n(CC, Ph) Cr, Ph.) = 1 for any reachable (Ch,Ph) still hard
Learning multi-agent individual decoding functions with unilateral exploration. Similar to our framework for POMDPs, our framework for POSGs also decoupled into two steps: i) learning an expert equilibrium policy that is fully observable, ii) learning the decoding function, where the first step can be instantiated by any provable fff-the-shee shelf algorithm of learning in Markov games. The major difference from the framework for POMDPs lies in how to learn the decoding function. In Theorem I, we prove that the difference of the NE/CE/CCE-gan between the expert policy and the distilled student policy is bounded by the decoding errors under policies from the unilateral deviation of the expert policy. Hence, given the expert policy H, the key algorithmic principle to perform unilateral exploration for each agent i to make sure the decoding function is accurate under policies

---

(T;; for any ग.. keeping x-; fixed. We refer the detailed algorithm to Algorithm 5, and present below the guarantees for learning the decoding functions and the corresponding distilled policy for learning NE/CCE, while we defer the results for learning CE to Theorem J.6.
Theorem 7.4 (Equilibria learning; Combining Theorem J.1 and Theorem J.4). Under Assumption C.8 and conditions of Definition 7.2, given 2 S-NE/CCE T for the associated Markov game of g. Algorithm 5 can learn decoding function (ūich Ne(()),EE))) such that with probability at least - 5, it is guaranteed that eT.,jet Pu.x*-..9 (s). # 95 (CA. Pj,A)) S anira for any i E [ni], h E [H] with both sample and computational complexities POLY(S, A, H, O, 1. log }) Consequently, policy distilled from n² (c.f. Theorem J.1 for the formal distillation procedures) is an c-NE/CCE of G.
7.2 Privileged Value Learning: Asymmetric MARL with Approximate Belief Learning
For POMDPs, we have used finite-emmory policies for computational efficiency. We generalize to POSGs with information sharing by defining the compression of the common information.
Definition 7.5 (Compressed approximate common information [57, 79, 5ID. For each h E [H]
given set Cr. we say Compress, is compression function if Compress, E {f Cn - ). For each C₁ E Cr, we denote Cn == Compress, (cn). We also require the compression function to satisfy the regularity condition that for each h E [f], there exists a function A4+1 such that Gh+1 = A (Ch, wh+1), for any ca E Cr, Wh+1 E Y where we recall Ca+1 == ch U Wl+1 and the definition of YA+1 in Assumption C.7.
Similar to the framework we developed for POMDPs in Section 5, we firstly develop the multi-agent RI algorithm based on some approximate belief, and then instantiate it with one provable approach for learning such approximate belief.
Optimistic value iteration of POSGs with approximate belief. For POMDPs, the sufficient statistics for optimal decision-making is the posterior distribution over the state given history. However, for POSGs with information-sharing. as shown in [63, 62, 51], the sufficient statistics become the posterior distribution over the state and the private information given the common information, instead of only the state. Therefore, we consider the approximate belief in the form of Pr. Cr + A(Pr x S) for each h E [H], where we define the error compared with the ground-trui -tuth belief to be ébelief == maXNE[H] m Bg Een.ps. [P (sh, Ph. ca.) Pr. (sh. Ph (.)), i.e., the ex-
pected total variation distance from the true one. Note that both Pn. and thus Cbeliet have implicit dependencies on Compress,, as G, == Compress, (ca... We outline our algorithm Algorithm 7, which is conceptually similar to the algorithm for POMDPs (Algorithm 2), maintaining the asym- metric critic (i.e., value function), and performing the actor update (i.e., policy update) using the beliht-weighted value function.
Theorem 7.6 (Equilibria learning: Combining Theorem J.15 and Theorem J.16). Fix 6, E 1). Under Assumption C.S, with probability at least 1 1- 8. Algorithm 7 can learn an (c + Hebette)--EE if 9 is zero-sum and (c + H² (Peeeer) if g is general-sum with sample complexity O AA' PSLY(S, (A0) (og(SH)e)) H, : log }).
and computational complexity
Learning approximate belief with model truncation. The belief learning algorithm we design for POSGs is conceptually similar to that we designed for POMDPs, where the key to achieving both polynomial sample and computational complexity is still to firstly learn approximate models, i.e., transitions and emissions, and then carefully truncate (as in Section 5.2) its transition and emission to build the approximate belief, where we defer the detailed algorithm to Algorithm 8. Next, we provide its provable guarantees, which leads to final polynomial-sampe and quasi-polynomial-imm complexity result when combined with Theorem 7.6.
Theorem 7.7. For any >> 0, under Assumption 2.2, it holds that can learn the approxi- mate belief (Pr, Cn + A(S Ph)]ne(m such that ebetiet S _ complexity and computational complexity POLY A, O, H, 4, 7, log 4) for all the examples in Appendix C.3. As a consequence, Algorithm 7 can lear an e-NE if G is zero-sum and e-CE/CCE is general-sum with sample complexity O H! SAO (SHO//) POLY(S, (A0)06) (og(SH)))), (00(5)))))(()))) H, log })-
H with both polynomial sample
and computational complexity
e
10

---

Acknowledgement
The authors would like to thank the anonymous reviewers and area chair from NeurlPS 2024 for the valuable feedback. Y.C. acknowledges the support from the NSF Awards CCF-1942583 (CAREER) and CCF-2342642. X.L. and K.Z. acknowledge the support from Army Research Laboratory (ARL) Grant W91 1NF-24-1--008 K.Z. also acknowledges the support from Simons-Berkeles Research Fellowship. A.O. acknowledges financial support from a Meta PhD fellowship. a Sloan Foundation Research Fellowship and the NSF Award CCF- 194-5992583 (CAREER). Part of this work was done while the authors were visiting the Simons Institute for the Theory of Computing.
References
[1] A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. Optimality and approximation with policy gradient methods Markov decision processes. In Conference on Learning Theory, pages 64 66, 2020.
[2] E. Altman, V. Kambley, and A. Silva. Stochastic games with one step delay sharing information pattern with application to power control. 2009 International Conference on Game Theory for Networks, pages 124-129. IEEE, 2009.
[3] O. M. Andrychowicz, B. Baker, M. Chociej. R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, al. Learning dexterous in-hand manipulation. The Interna- tional Journal of Robotics Research, 39(1):3-20) 2020.
[4] R. J. Aumann, M. Maschler, and R. E. Stearns. Repeated games with incomplete information. MIT press, 1995.
[5] R. Avalos, F. Delgrange, A. Nowe, G. Perez, and D. M. Roijers. The wasserstein believer: Learning belief updates for partially observable environments through reliable latent space models. In The Twelfth International Conference Learning Representations, 2023.
[6] A. Baisero and C. Amato. Unbiased asymmetric reinforcement learning under partial observabil- ity. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, 2022.
[7] A. Baisero, B. Daley, and C. Amato. Asymmetric DQN for partially observable reinforcement learning. In Uncertainty in Artificial Intelligence, pages 107-117. PMLR, 2022.
[8] N. Brukhim, D. Carmon, I. Dinur, Moran, and A. Yehudayoff. characterization of multiclass learnability, 2022.
(9] N. Brukhim, D. Carmon, I. Dinur, S. Moran, and A. Yehudayoff. A characterization of multiclass learnability. In 2022 IEEE 63rd Annual Symposium Foundations of Computer Science (FOCS), pages 943-955. IEEE, 2022.
[10] S. Bubeck. Convex optimization: Algorithms and complexity. Found. Trends Mach. Learn., 8(3-4):231-357, 2015.
(11) Q. Cai, Z. Yang, C. Jin, and Z. Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning, pages 1283-1294. PMLR, 2020.
[12] Q. Cai, Z. Yang, and Z. Wang. Reinforcement learning from partial observation: Linear function approximation with provable sample efficiency. In International Conference on Machine Learning, pages 2485-2522. PMLR, 2022.
[13] C. L. Canonne. short note on learning discrete distributions. arXiv preprint arXiv:2002.11453 2020.
[14] D. Chen, B. Zhou, Koltun, and P. Krähenbühl. Learning by cheating. In Conference on Robot Learning, pages 66-75. PMLR, 2020.
[15] F. Chen, Y. Bai, and S. Mei. Partially observable RL with b-stability: Unified structural condition and sharp sample-efficient algorithms. In The Eleventh International Conference on Learning Representations, 2023.
11

---

[16] X. Chen, Y. M. Mu, P. Luo, S. Li, and J. Chen. Flow-based recurrent belief state learning for pomdps. In International Conference on Machine Learning. pages 3444-3468 PMLR, 2022.
[17] C. Dann, N. Jiang, A. Krishnamurthy, A. Agarwal, J. Langford, and R. E. Schapire. On oracle- efficient pac rl with rich observations. Advances in neural information processing systems, 31, 2018.
[18] S. Du, A. Krishnamurthy, N. Jiang, A. Agarwal, M. Dudik, and J. Langford. Provably efficient with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1665-1674 PMLR. 2019.
[19] Y. Efroni, C. Jin, Krishnamurthy, and S. Miryoosefi. Provable reinforcement learning with short-term memory. In K. Chaudburi, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5832-5850. PMLR, 2022.
[20] E. Even-Dar, S. M. Kakade, and Y. Mansour. The value of observation for monitoring dynamic systems. In M. M. Veloso, editor, IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6- 12, 2007, pages 2474-2479, 2007.
[21] J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 29, 2016.
[22] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[23] K. Fujii. Bayes correlated equilibria and no-regret dynamics. arXiv preprint arXiv:2304.05005. 2023.
[24] T. Gangwani, J. Lehman, Q. Liu, and J. Peng. Learning belief representations for imitation learning in pomdps. uncertainty artificial intelligence, pages 1061-1071. PMLR, 2020.
[25] N. Golowich, A. Moitra, and D. Rohatgi. Learning in observable POMDPs, without computa- tionally intractable oracles. In Advances Neural Information Processing Systems,
[26] N. Golowich, A. Moitra, and D. Rohatgi. Planning in observable pomdps in quasipolynomial time. arXiv preprint arXiv:2201.0473: 2022.
[27] N. Golowich, A. Moitra, and D. Rohatgi. Planning and learning partially observable systems via filter stability. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing. pages 142, 2023.
[28] N. Golowich, A. Moitra, and D. Rohatgi. Exploration is harder than prediction: Cryp- tographically separating reinforcement learning from supervised learning. arXiv preprint arXiv:2404.03774 2024.
[29] G. J. Gordon, A. Greenwald, and C. Marks. No-regret learning in convex games. In Proceedings of the 25th international conference on Machine learning, pages 360-367, 2008
[30] J. Guo, M. Chen, H. Wang, C. Xiong, M. Wang. and Y. Bai. Sample-efficient learning of pomdps with multiple observations in hindsight. In The Twelfth International Conference on Learning Representations, 2023.
[31] A. Gupta, A. Nayyar, C. Langbort, and T. Basar. Common information based markov perfect equilibria for linear-gaussian games with asymmetric information. SIAM Journal on Control and Optimization, 52(5):3228-322 3260, 2014.
[32] Hartline, V. Syrgkanis, and E. Tardos. No-regret learning bayesian games. Advances in Neural Information Processing Systems, 28, 2015.
12

---

[33] H. Hu, A. Lerer, N. Brown, and J. Foerster. Learned belief search: Efficiently improving policies in partially observable settings. arXiv preprint arXiv:2106.09088 09086, 2021.
[34] N. Jiang. On value functions and the agent-environment boundary. arXiv preprint
htth::995..3344 2019.
[35] N. Jiang, A. Krishnamurthy, A. Agarwal, Langford, and R. Schapire. Contextual decision processes with low bellman rank are pac-learnable. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW. Australia, 6-11 11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1704-1713. PMLR, 2017.
[36] C. Jin, S. Kakade, A. Krishnamurthy, and Q. Liu. Sample-efficient reinforcement learning of undercomplete POMDPs. Advances in Neural Information Processing Systems, 33:18530- 18539, 2020.
[37] C. Jin, A. Krishnamurthy, M. Simchowitz, and T. Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning, pages 4870-4879, PMLR, 2020.
[38] C. Jin, Q. Liu, Y. Wang, and T. Yu. V-learning-a a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint
2021.
[39] S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, volume 2, pages 267-274, 2002.
[40] S. M. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, pages 1531-1538, 2002.
[41] B. R. Kiran, Sobh, Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions Intelligent Trunsportation Systems, 23(6):4909-4926, 2021.
[42] V. R. Konda and J. N. Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems, pages 1008-1014, 2000.
[43] A. Krishnamurthy, Agarwal, and Langford. Pac reinforcement learning with rich observa- tions. Advances in Neural Information Processing Systems, 29, 2016.
[44] J. Lee, A. Agarwal, C. Dann, and T. Zhang. Learning in pomdps is sample-efficient with hindsight observability, In International Conference on Machine Learning, pages 18733-18773. PMLR, 2023.
[45] J. Lee, Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5996, 2020.
[46] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Muchine Learning Research, 17(1):1334-137) 2016.
[47] Q. Liu, A. Chung, C. Szepesvari, and C. Jin. When is partially observable reinforcement learning not scary? In Conference on Learning Theory, pages 51 5175-5220, 2022.
[48] Q. Liu, Netrapalli, C. Szepesvari, and C. Jin. Optimistic MLE: generic model-based algorithm for partially observable sequential decision making. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pages 363-3 376,, 2023.
[49] Q Liu, C. Szepesvári, and C. Jin. Sample-efficient reinforcement learning of partially observable Markov games. In Advances in Neural Information Processing Systems, 2022.
[50] Q Liu, Yu. Y. Bai, and C. Jin A sharp analysis of model-based reinforcement learning with self-play. In International Conference on Machine Learning, pages 7001-7010. PMLR, 2021.
[51] X. Liu and K. Zhang. Partially observable multi-agent RL with (quasi-)efficience: the blessing of information sharing. In International Conference on Machine Learning, pages 22370-22419. PMLR, 2023,
13

---

[52] X. Liu and K. Zhang. Partially observable multi-agent reinforcement learning with information sharing, 2024.
[53] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-compettivv environments. Advances in Neural Information Processing Systems, 30, 2017.
[54] M. Lu, Min, Z. Wang, and Yang. Pessimism the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes. In The Eleventh International Conference on Learning Representations, 2023.
[55] X. Lyu, A. Baisero, Y. Xiao, and C. Amato. A deeper understanding of state-based crities in multi-agent reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 9396-9404, 2022.
[56] X. Lyu, A. Baisero, Y. Xiao, B. Dalcy, and C. Amato. On centralized critics in multi-agent reinforcement learning, Journal of Artificial Intelligence Research, 77:295-354, 2023.
[57] W. Mao, K. Zhang, E. Michling, and T. Başar. Information state embedding in partially observable cooperative multi-agent reinforcement learning. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 6124-6131. IEEE, 2020.
[58] G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, S. bae Kim, and P. Agrawal. Learning to jump from pixels. In 5th Annual Conference on Robot Learning, 2021.
[59] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62):eabk2822, 2022.
[60] D. Misra, M. Henaff, A. Krishnamurthy, and J. Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. In International conference machine learning. pages 6961-6971. PMLR, 2020.
[61] P. Moreno, J. Humplik, G. Papamakarios, B. A. Pires, L. Buesing, N. Heess, and T. Weber. Neu- ral belief states for partially observed domains. In NeurIPS 2018 Workshop on Reinforcement Learning under Partial Observability, 2018
[62] A. Nayyar, A. Gupta, C. Langbort, and T. Başar. Common information based markov perfect equilibria for stochastic games with asymmetric information: Finite games. IEEE Transactions on Automatic Control, 59(3):555-570, 2013.
[63] A. Nayyar, A. Mahajan, and D. Teneketzis. Decentralized stochastic control with partial history sharing: A common information approach. IEEE Transactions on Automatic Control, 58(7):1644-1652 2013.
[64] H. Nguyen, A. Baisero, D. Wang. C. Amato, and R. Platt. Leveraging fully observable policies for learning under partial observability. In Conference on Robot Learning, 2022.
[65] H. Nguyen, B. Daley, X. Song, C. Amato, and R. Platt. Belief- griindgg networks for accelerated robot learning under partial observability. In Conference on Robot Learning, pages 1640-1653. PMLR, 2021.
[66] C. H. Papadimitriou and J. N. Tsitsiklis. The complexity of markov decision processes. Mathe- maticy of operations research, 42(3):441-450, 1987.
[67] A. Pathak, H. Pucha, Y. Zhang, Y. C. Hu, and Z. M. Mao. A measurement study of internet delay asymmetry. In Passive and Active Network Measurement: 9th International Conference, PAM 2008, Cleveland, OH, USA, April 29-30, 2008. Proceedings 9, pages 182-191. Springer, 2008.
[68] I.. Pinto, M. Andrychowicz, Welinder, W. Zaremba, and Abbeel. Asymmetric actor critic for image- based robot learning. Robotics: Science and Systems XIV, 2018.
14

---

[69] S. Qiu, Z. Dai, H. Zhong, Z. Wang, Z. Yang. and T. Zhang. Posterior sampling for competitive rl: Function approximation and partial observation. Advances in Neural Information Processing Systems, 36, 2024.
[70] Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. QMIX: Monotonic value function factorisation for deep multi-agen reinforcement learning. In Interna- tional Conference on Machine learning, pages 68 681-689, 2018.
[71] T. Roughgarden. Algorithmic game theory. Communications of the ACM, 53(7):78-86, 2010.
[72] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1T0.:::777.064777 2017.
[73] S. Shalev-Shwartz, Shammah, and A. Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv: 2016.
[74] I.. Shani, Y. Efroni, A. Rosenberg, and S. Mannor. Optimistic policy optimization with bandit feedback. In International Conference Machine Learning, pages
PMLR, 2020.
[75] I. Shenfeld, Z.-W. Hong, A. Tamar, and P. Agrawal. Tgrl: An algorithm for teacher guided reinforcement learning. In International Conference Machine Learning, pages 31077-31093. PMLR, 2023.
[76] M. Shi, Y. Liang, and N. Shroff. Theoretical hardness and tractability of pomdps in with partial online state information, 2024.
[77] M. Shortreed, E. Laber, D. J. Lizotte, T. Stroup, Pineau, and S. Murphy. Informing se- quential clinical decision-making through reinforcement learning: an empirical study. Machine learning. 84:109-136, 2011.
[78] Z. Song, S. Mei, and Y. Bai. When can we learn general-sum Markov games with a large number players sample-effiiennly) arXiv preprint arXiv:21 2021.
[79] J. Subramanian, A. Sinha, R. Seraj, and A. Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. J. Mach. Learn. Res., 23:12-1, 2022.
[80] Tsitsiklis and M. Athans. On the complexity of decentralized decision making and detection problems. IEEE Transactions Automatic Control, 3005):440-446, 1985.
[81] M. Uehara, A. Sekhari, J. D. Lee, N. Kallus, and W. Sun. Computationally efficient pac rl in pomdps with latent determinism and conditional embeddings. In International Conference Machine Learning, pages 1-64114441. PMLR, 2023.
[82] O. Vinyals, I. Babuschkin, W. M. Czameekki M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782)350-334, 2019.
[83] A. Wang, A. C. Li, T. Q. Klassen, R. T. Icarte, and S. A. Mcllraith. Learning belief representa- tions for partially observable deep rl. In International Conference Machine Learning, pages 35970-35988, PMLR, 2023.
[84] L. Wang, Q. Cai, Z. Yang, and Wang. Represent to control partially observed systems: Rep- resentation learning with provable sample efficiency. In The Eleventh International Conference on Learning Representations, 2022.
[85] H. S. Witsenhausen. A counterexample in stochastic optimum control. SIAM Journal on Control, 6(1):131-147, 1968.
[86] G. Yang, M. Liu, W. Hong, W. Zhang, F. Fang, G. Zeng, and Y. Lin. Perfectdou: Dominating doudizhu with perfect information distillation. Advances in Neural Information Processing Systems, 35:34954-34965. 2022.
15

---

[87] Y. Yang, Y. Jiang, J. Chen, S. E. Li, Z. Gu, Y. Yin, Q. Zhang, and K. Yu. Belief state actor- critic algorithm from separation principle for POMDP. In 2023 American Control Conference (ACC), pages 2560-2567. IEEE, 2023.
[88] S. Young, M. Gašié, B. Thomson, and J. D. Williams. Pondp-based statistical spoken dialog systems: review. Proceedings of the IEEE, 100 1179,, 2013.
[89] A. Zanette and E. Brunskill. Tighter problem-dependen regret bounds reinforcement learning without domain knowledge using value function bounds. In International Conference Machine Learning, pages 7304-7312. PMLR, 2019.
[90] W. Zhan, M. Uehara, W. Sun, and J. D. Lee. PAC reinforcement learning for predictive state representations, In The Eleventh International Conference on Learning Representations, 2023.
16

---

Supplementary Materials for "Provable Partially Observable Reinforcement Learning with Privileged Information"
Contents
Introduction
2 Preliminaries
2.1
Partially Observable RL (with Privileged Information)
2.2 Partially Observable Multi-agent RI. with Information Sharing
2.3
Technical Assumptions for Computational Tractability
3
Revisiting Empirical Paradigms RL with Privileged Information
3.1 Privileged Policy Learning: Expert Policy Distillation
3.2 Privileged Value Learning: Asymmetric Actor-Critic
4
Provably Efficient Expert Policy Distillation
5
Provable Asymmetric Actor-Critic with Approximate Belief Learning
5.1
Belief-Weighted Optimistic Asymmetric Actor-Critic
5.2 Approximate Belief Learning..
Numerical Validation
7 Extension to Partially Observable MARL with Privileged Information
7.1 Privileged Policy Learning: Equilibrium Distillation.
7.2 Privileged Value Learning: Asymmetric MARL with Approximate Belief Learning
10
A Societal Impact
19
Related Work
19
C Additional Preliminaries
20
C.1 Additional Preliminaries on POMDPs
20
C.2 Additional Preliminaries for POSGs
20
C.2.1 Evolution of the Common and Private Information ................................
23
C,3 Strategy Independence of Belief and Examples
23
Collection of Algorithms
24
Missing Details in Section 3
Missing Details in Section 4
33
17

---

G Provably Efficient Expert Policy Distillation with Function Approximation
34
H Missing Details in Section 5
36
H.1 Supporting Technical Lemmas
44
I Missing Details in Section 6
47
J Missing Details in Section 7
48
J.1 Background on Bayesian Games
61
Concluding Remarks and Limitations
62
18

---

A
Societal Impact
Our work is theoretical by nature, and aimed at better understanding reinforcement learning under partial observability with privileged information. As such, we do not anticipate any direct positive or negative societal impact from this research.
B Related Work
Provable partial observable RL. While POMDPs are generally known to be both statistically hard [43] and computationally intractable [66], productive line of research has identified several structured subclasses of POMDPs that can be efficiently solved. [43] introduced the class of POMDPs in the rich-observation setting, where the observation space can be large and fully reveal the underlying state, where samplc-efficient RL becomes possible [35, 60]. [19] introduced k-step decodable POMDPs, where the last k observation-action pairs can uniquely determine the state, and proposed polymo complexity algorithms (assuming k is a small constant). Beyond settings where the underlying state can be exactly recovered, [36, 47] proposed weakly revealing POMDPs, where the observations are assumed to be informative enough. Under the weakly revealing condition (and its variant), there has been frswiggowing line of recent works developing sample-efficient RL algorithms for various settings, see e.g., [84, 15, 12, 54, 48, 90]. Notably, these algorithms are typically computationally inefficient, requiring access to an optimistic planning oracle for POMDPs. On a promising note, [26] showed that in observable POMDPs (see Assumption 2.2), one can have quasi colasiiooooooma-iiinii algorithms for planning the near-optimal policy, which further leads to provable RL algorithms [25, 27] with both quasi-polynomial samples and time.
RL under hindsight observability. The closest line of research to ours are the recent theoretical studies for Hindsight Observable Markov Decision Processes (HOMDPs) [44], where the underlying state is revealed at the end of the episode; see also subsequent related works in [30, 76) with different observation feedback models. These works focused purely on sample efficiency, and showed that
polynomial sample complexity can be achieved without (or by further relaxing) aforementioned structural assumptions of the model (e.g.. observability or decodability), in both tabular and/or function approximation settings. However, the algorithms (also) require an oracle for planning or even optimistic planning in a learned approximate POMDP, which are not computationally tractable in general. Indeed, without any structural assumption, learning the optimal policy in HOMDPs computationally no easier than the planning problem, which thus remains PSPACE- hardd [66]. Meanwhile, even under the additional assumption of observability on the underlying POMDP model, is still not clear if these algorithms can avoid computationally intractable oracles, since the approximate POMDP that [44] needs to do planning on at every iteration during learning can be quite different from the underlying model. For example, at the beginning of exploration when not enough samples are collected, or when there exist certain states that remain less explored during the entire learning process, the potentially misspecified emission (and transition) may break the observability (or other structural) assumptions made for the underlying POMDP. This makes that single iteration even computationally intractable. In contrast, our focus is on better understanding practically inspired algorithmic paradigms, without computationally intractable oracles, which in practice often do have and use the privileged state information during each episode (instead of only at the end) [68, 45, 14].
Most related empirical works. Privileged information has been widely used in empirical partially observable RI., with two main types of approaches based on privileged policy and privileged value learning, respectively. For the former, one prominent example expert distillation [14, 64, 58], also known as teacher-student learning [45, 59, 75], as we analyze in Section 4. For the latter, asymmetric actor-critic [68] represents one of the well-known examples, with other studies in [7, 31. Learning privileged value functions (to improve the policies) has also been widely used in multi-agent RL, featured in centralized werddd-ttrnnnneeeeee boccentralized-exeuutte was shown that if the privileged value function depends only on the state, the associated actor will cause bias [6, 55, 56). This has thus necessitated the use of history/belief in asymmetric actor-critic, as in our Section 5. Notably, the empirical framework in [83] exactly matches ours, where they exploited the privileged state information in training for belief learning, followed by policy optimization over the learned belief states. Indeed, many empirical works explicitly separate the procedures of explicit
see e.g., [53, 22, 70, 86, 82]. Intriguingly.
19

---

belief-state learning and planning [24, 65, 33, 16, 87] as we study in Section 5, oftentimes with privileged state information to supervise the belief learning procedure [61, 5].
C
Additional Preliminaries
C.1 Additional Preliminaries on POMDPs
Belief and approximate belief. Although in a POMDP, the agent cannot see the underlying state directly, it can still form a belief over the underlying state by the historical observations and actions.
Definition C.I (Belief state update). For each h E [H + 1], the Bayes operator В₁ :(S) x x (SS) defined for b (SS), and o by:
()(012)((2)
(n(b;o)(z)) Exes Onlo =))()²)
for each x S. For each [Mf + 1J. the belief update operator Un A(S) x A(5), defined by
(0; o) = -Ba+1 (Ta(a) 0)
where Tr, (a) represents the matrix multiplication. We use the notation b, to denote the belief update function, which receives sequence of actions and observations and outputs distribution over states at the step h; the belief state at step h = defined as b (9) = fu For any 2 S S H and any action-observation sequence (ura; 01.5), we inductively define the belief state:
(1 01AA) = TT (ax) ) (41
by (a 1, 01.) B₁ (b, (a1:h-1, 11:hh-1,00::1111 01:h-1);
We also define the approximate belief update using the most recent L-step history. For 1 S h < H, we follow the notation [26] and define
apx, (0; D) = μn by
if
D otherwise,
where D E A(S) is the prior for the approximate belief update. Then for any 1 1 h - I, h H and any action-observation sequence (0,- L:h, Oa- 1:hh), we inductively define
b
papx h+1
(an- 9h 1+1 D) = Tn (ax) b%h (an- (an-Lsh-1r 1, Oh
papx,P h
(ak-L:h- - 1 1+1s D) B,,
(b, (b, (0h-Lsh-14 Oh-L+1 D);
For the remainder of our paper, will an important initialization for the approximate belief, which are defined b,,()) = by
apx,P Un).
C.2 Additional Preliminaries for POSGs
Model. We use a general framework of partially observable stochastic games (POSGs) as the model for partially observable MARL. Formally, we define POSG with agents by tuple g = (H, [A:] (0)).=1 T, O, Pl, (r:]i_)), where H denotes the length of each episode, S is the state space with SS] S, A; denotes the action space for agent i with [A, A; We denote by a == (a1,... a,,) the joint action of all the n agents, and by A = Ai x ... x A., the joint action space with [A] A I A.. We T (Ta)eej to denote the collection of transition matrices, that T- a) ASS) gives the probability of the next state if joint action a A is taken at state E and step h. In the following discussions, for any given a, we treat (a)a) E s|S|||s] as matrix, where each row gives the probability for the next state from different current states. We use μn to denote the distribution of the initial state 31, and O; to denote the observation space for agent i with |O,| O,, We denote by 0 == (0l,...,O0. ...) the joint observation of all n agents, and by 0 O x to denote the collection of the joint emission matrices, so that O (- E A(O) gives the emission distribution over the joint observation space O at state and step h. notational convenience, we will at times adopt the matrix convention, where 0, is matrix with rows O- (-)ss)) We also denote
O. with O O I - O,- We use ! (0, he([F)]
20

---

where Qt is the random seed space, which is shared among agents and Wi,h E 22 is the random seed for agent i. The corresponding policy class is denoted as a Hereafter, unless otherwise noted, when referring to policies, we mean the policies given in the form of (C.1), which maps the available (rn(-1s) E (O,)) as the marginalized emission for agent agent. Finally, ri = frum Ine(m is collection of reward functions, so that (sh, ax) is the reward of agent agent given the state Sh. and joint action Og at step h.
Similar to a POMDP, in a POSG, the states are not observable to the agents, and each agent can only access its own individual observations. The game proceeds as follows. At the beginning of each episode, the environment samples $1 from μ41 At each step h, each agent i observes its own observation 0i where o₁ == (01.h..... ................................ 0n,h) is sampled jointly from (- [sh)- Then each agent i takes the action 0i, and receives the reward ri,k r (Sh, an). After that the environment transitions to the next state 14+1 Ta (- 8h, an). The current episode terminates once 8H++1 is reached.
Information sharing, common and private information. Each agent i in the POSG maintains its own information, Ti a collection of historical observations and actions at step h, namely, TiA S {or,a) di; .2,-' ,0 j. and the collection of such history at step h is denoted by Ti,
In many practical examples, agents may share part of the history with each other, which may introduce some information structures of the game that may lead to both sample and computation efficiencies. The information sharing splits the history into common/shared and private information for each agent. The common information at step h is a subset of the joint history тh (Tr.))ie(m]
ON S {o,, 01 02,- ................................ 1,-1, ). which is available to all the agents in the system, and the collection of the common information denoted as Co and we define Cj. |Cal. Given the common information Ch, each agent also has her private information PLA = Ti,k \ Cb. where the collection of the private information for agent is denoted Path and its cardinality
Pi.-- The cardinality of the joint private information is Ps. II==1 P.A.- We allow O₁ or P.A. to take the special value @ when there is no common or private information. In particular, when C._ = {0). the problem reduces the general POSG without any favorable information structure; when P.,a = {0}, every agent holds the same history, and it reduces to a POMDP when the agents share a common reward function, where the goal is usually to find the team-optimal solution.
Policies and value functions.
We define a stochastic policy for agent at step h as:
Teh : S2 X Pe,k x C. + A(A),
(C.1)
information of agent i.e., the private information together with the common information, to the distribution over her actions.
We define as sequence of policies for agent at all steps h E [H], i.e., (Ti,1, Ti. We further denote II, = X XNe[H] I,, as the policy space for agent i and II = =Ihee(m I1, as the joint
policy space. As a special case, we define the space of deterministic policy as II, where R, E IT; maps the private information and common information to deterministic action for agent and the joint space II [[se[1] II,
A product policy is denoted as = X ... ... X T. E TT if the distributions of drawing each seed Wi,h for different agents are independent, and a (potentially correlated) joint policy is denoted as Ti C 722
Tr E II.
We are now ready to define the value function conditioned on the common information under our model of POSG with information sharing:
Definition C.2 (Value function with information sharing). For each agent i E [n]] and step h E [H], given common information on and joint policy π (Tt) E 11, the value function conditioned on the common information of agent i is defined as: V. (cr) E IH Enrm (Sw. where the expectation is taken over the randomness from the model G. policy ग,, and the random seeds. For any cH+1 E CH+1 VIT. (cH+1) == 0. From now on, we will refer to it as value function for short.
ch.
Another key concept our analysis is the belief about the state and the private information condi- tioned on the common information among agents. Formally, step h, given policies from 1 to h 1,
21

---

we consider the common-information conditional belief 1 (8h, Ph ca.. This belief not only infers the current underlying state sh, but also all agents' private information ps. With the common-information- based conditional belief, the value function given in Definition C.2 has the following recursive structure:
(cx) = (r,, an) + V. i,h+1 (cN+1) (cl,,
(C.2)
where the expectation taken over the randomness (8), Ph, Oh 0h+1). With this relationship, we can define the prescription-valie function correspondingly. generalization of the action-value function in (fully observable) stochastic games and MDPs to POSGs with information sharing, as follows.
Definition C.3 (Prescription-vaiee value function with information sharing). At step h, given the common information Ch. joint policies क = (r+) E II, and prescriptions (7i,)==1 E Tı, the prescription- value function conditioned on the common information and joint prescription of agent is defined as:
0 (Ch. (75iN))E(()) = ES (sn, an) + V.N4 (9h+)) c/ (19i))ss()]
6.h41
where prescription Vit E (AA,) replaces the partial function Tt Wi,ks -) in the value function. From now on, will refer to it prescription-valu function for short. With such prescription-valie function, agents can take actions purely based on their local private information [63, 62. 51].
This prescription-value value function indicates the expected return for agent i when all the agents firstly adopt the prescriptions (1j,n)je(n) and then follow the policy π.
accordingly define the lution concepts. Here we define the notions of c-NE, c-CCE, c-CE, and c-team optimum under the information-sharing framework as follows. For joint policy (TT)) II we denote the expected
Equilibrium notions. With the definition of value functions,
reward of agent i by uff (x) F ->= Ta, (*k. ax)]-
Definition C.4 (e-approximate Nash equilibrium with information sharing). For any 2 a product policy x² E IT is an c-approximate Nash equilibrium of the POSG G with information sharing if:
NE-gap(T') max
max of (m x mi,) of (1*) S.
Definition C.5 (e-approximate coarse correlated equilibrium with information sharing). For any < 0, a joint policy x² II is an c-approximate coarse correlated equilibrium of the POSG g with information sharing if:
CCE-gap(#*) == max
max of ( x m'.) - -0((T') S e. <lell,
Definition C.6 (e-approximate correlated equilibrium with information sharing). For any e 2 0, joint policy xt e II is an e-approximate correlated equilibrium of the POSG G with information sharing if:
CE- CE-gap(rT") == max max ((m, m;)) O x=z,)= (g(()) - Se, 6:
where m, is called strategy modification for agent and m, {mi,h, PP. Hhen with each m.p..c..p... A, + A, being a mapping from the action set to itself. The space of m; denoted as M,. The composition m; o ग. is defined as follows: at step h, when agent i is given and Pi,s. the joint action chosen to be (01,h. .a.h..... an,h) ) will be modi- fied to (ai.h..... mijhees.P... (as,)),... an,t). Note that this definition follows from that in [78, 50, 38, 49, 51] when there exists certain common information, and is natural generaliza- tion the definition in the normal-form game case [71]. We denote by A the space of all possible strategy modifications
if it conditions on any history information instead of only (Ca, Pi,A). Simi- larly, we use Ms. to denote the space of all possible strategy modifications m, it only conditions on the current state e.g., modification mi,hs A; A..
22

---

C.2.1 Evolution of the Common and Private Information
Assumption C.7 (Evolution of common and private information). We assume that common informa- tion and private information evolve over time as follows:
. Common information cp is non-decreasing over time, that is, ch S ch+1 for all h. Let #h+1 Ch+1 \ Ch. Thus, ch+1 {ch, #h+1). Further, we have
h++1 Xh+1 (Phs oh, on+1);
(C.3)
where Xh+1 is fixed transformation. We use TA41 to denote the collection of Wh+1 at step h.
Private information evolves according to:
P E (Pi, Oi,hs 0c141),
(C.4)
where f1,4+1 is a fixed transformation.
Equation (C.3) states that the increment in the common information depends the "new" information (ah, 0h+1) generated between steps h and + 1 and part of the old information Pą- The incremental common information can be obtained by certain sharing and communication protocols among the agents. Equation (C.4) implics that the evolution of private information only depends on the newly generated private information di and 0,,4+1- These evolution rules standard in the literature [62, 63], specifying the source of common information and private information. Based on such evolution rules, define {fn he][F)] and {gA, he] where fn AA O + C.. and gh : AAh Oh + Pa for [H), as the mappings that map the joint history common information and joint private information, respectively.
C.3 Strategy Independence of Belief and Examples
To solve POSG without computationally intractable oracles, certain information-sharing is needed even under the observability assumption [51]. We thus make the following assumption as in [51].
Assumption C.8 (Strategy independence of beliefs [62, 31, 51D. Consider any step [H] any policy π E II. and any realization of common information Ca that has a non-zero probability under the trajectories generated by -1 Consider any other policies Ti=h-1 which also give a non- zero probability to C₁. Then, we assume that: for any such C₁. E Cr, and any ph Pa,, S,
P. 1 (sh, Ph cx) = 1P 1 (sh, Ph. cn).
We provide examples satisfying this assumption in Appendix C.2, which include the fully-sharing structure as in [27, 69] as a special case. Finally, we also assume that common information and private information evolve over time properly in Assumption C.7, as standard in [62, 63, 51], which covers the models considered in [27, 69, 49].
Here we take the examples from [52, 51] illustrate the generality of the information-sharing framework.
Example C.9 (One-step delayed sharing). At any step h E [H). the common and private information are given as (02:n-1.01.. and Pi.A for,a) J. respectively. In other words, the players share all the action-observation history until the previous step h 1, with only the new observation being the private information. This model has been shown useful for power control [2].
We assume there 2 players for convenience. It extends naturally n-player settings. Consider the where the state dynamics are controlled by player 1, i.e., (. 8h, a1,h: a2,,) = (- Sh, a1,n: alg,)) for all (sh, a1,,, 02.h, d2,n.:h h). There are two kinds of delay-sharing structures we could con- sider: Case A: the information structure is given ch {01,2:A, 02,2:h-d 01,1:h-1}, P1,A = 0, P2.h = (02Ah-d1:hh }. i.e. player I's observations are available player 2 instantly, while player 2's observations are available to player with a delay of d 2 time steps. Case B: similar to Case A but player I's observation is available to player 2 with delay of step. The information structure is given as с = {01,2%-11 02,2.1-4 11,1:h-1). P1.4
Example C.10 (State controlled by one controller with asymmetric delay sharing).
(01.1) P2.6 = {02,h-d+18 ). where d > 1. This kind of asymmetric sharing is common network routing [67], where packages arrive at different
hosts with different delays, leading to asymmetric delay sharing among hosts.
23

---

Example C.11 (Symmetric information game). Consider the case when all observations and actions are available for all the agents, and there is no private information. Essentially, we have ch (02:h; 01:h-1111 and Pi.A D. We will also denote this structure as fully sharing hereafter.
Example C.12 (Information sharing with one-directionallonn--tte delay). Similar to the previous cases, we also assume there are 2 players for ease of exposition, and the case can be generalized to multi-player cases straightforwardly. Similar to the one-step delay case. we consider the situation where all observations of player 1 are available player 2, while the observations of player 2 are available to player 1 with one-step delay. All the past actions are available to both players. That is, in this case, C% (01,2:h- 02,2: 01:h-1); and player has no private information, i.e., P1.A 0, and player 2 has private information P2,h = {02,1)
Example C.13 (Uncontrolled state process). Consider the case where the state transition does not depend the actions, that is, (x Sh; ah) T- 8h, d(.) for any Sh, ah dg. h. Note that the agents are still coupled through the joint reward. An example of this case is the information structure where controllers share their observations with delay of d time steps. In this case, the common information is C₁ {02:h-a) and the private information is P {osh-d+1:hhf J. Such information structures can be used to model repeated games with incomplete information [4].
D Collection of Algorithms
Algorithm Learning Decoding Function with Privileged Information
Require:
POODPP P.
Expert policy x
Ils.
Number of sampled episodes per step M.
Ensure: A decoding function for each step {9x heeja (see Theorem 4.6)
For the h 1 step: Collect M state-observation pairs from the first-step D = {1f )
(
on POMDP P and define the decoding function Φ1 for the first step as:
(01) {sy (81, E D
for each step h E [2] do Collect M episodes
{ {(
(i) (1:H+1 I. 01:H SE(M)
(8)
POMDP P using policy n and let:
D = 1- 1 (
(
(1)
KE[M]
Define the decoding function Oh. for step h as:
0h-1, ) = {sh :
(5h-1, sh)) Dr
end for return [EEA)
24

---

Algorithm 2 Weighte Optimistic Asymmetric Actor-Critic with Privileged Information
Require:
POMDP P,
Subroutine Tq that given policy II outputs (Qen that approximates {0r S.
Subroutine Ti that outputs {6% ]ejn that approximate (ba)
Initial finite- memory policy x² [*]]ee[H] E IT for the POMDP P, step-size f, and number of iterations T'
Ensure: A near-optimal policy (b% ]иE]H]
Tp(P)
for Iterations / = T
(², }>e'"" + (P,
Update the policy for each an A, 2h E 2n as
(an 2n) x nh - 24) exp (y (sw)
[6h - Sh an))
Denote = = Ti. end for return A policy uniform at random sampled from set { heel
Algorithm 3 Optimistic Q-function Estimation with Privileged Information
Require:
POMDP P, policy TI.H E _1 such that Ta : + A(A),
Number of episodes M per step.
Ensure: Approximate Q-functions {Qn he[M (see Lemma H.3) Initialize: A2H+1 E 2H+1, SH+1 E S, au+1 E A
1
QU+I 80t1, 0,
A
for step H...., do
Collect M trajectories using policy and let D,, (7(0 Nee[M] be the collected trajectories. Compute empirical counts and define empirical distributions:
(n Sh, an) = [T D₁ (8%, 0/1 8//42) (sh, ahe 8h+1))
H[ Dr (5/ a/))
(sn, an)H
[T E Dr (8/, o0h = (sh, on)
On(on |8h)
17 E Dh. s% snY
for each memory- pair (2h, sn))
Zn S do
(Zh,, Sh,
Sh, an) = min
И h + 1,
en T (nn) on41~ (: [88+1)
(V+ (2+41,
Slog(1/61)
r(sh, + H. min 2,C
Mixx(Na(>h, an), 1)
log((108)
+ sh A min
2,
mNx(Nn-1(S
where Vn41(2n41, 8h+1) E Bansronn se ( 8h41, end for end for return (2, (rEm)
1641, 01+1)]
25

---

Algorithm 4 Approximate Belief Learning with Privileged Information via Model Truncation
Require:
POMDP = (H, (TH, (eTH) {O*)ee;H) M4, frx (ra))
An MDP learning oracle MDP Learning that efficiently learns an approximate optimal policy of an MDP,
Number of trajectories N,
The threshold e.
Ensure: Approximate belief (b% heEH (See Theorem H.5)
for h [H] St S do Pr 0g ) <- 1 [[ = h s/ 8N for any (h', sh af)) E [H] x S x A M <- (H, S, (Tx (M,,,, (Ph) (E)) be the MDP associated with P (rh,, 5n) + MDP Learning(M))
Collect N trajectories by executing policy (h, sn) for the first h - steps then take action an for each an A deterministically and denote the dataset {(), 0 aj He
for (on, ah, Sh+1) O x A 8 do (sn) f Eie[NA] [ss. = sn]]
(s an) EiE[NA] 1[s/, Sh, d anp
(sh, an, sp+1) + EiE[NA) [ss) 8h. aj = ah sh = Sh+1]
(A on) + DiEiNA 1(3h Sh o 0hl
(S 5h; an) (50,0)
(on 8p) f- Na(an.on)
Na(en)
end for end for
for E [H] do
Slow
8/1 S
Na(en)
I
NA
shigh
E
Nk(sa) NA
end for
for (h, sh, sh+1) E [H] shiat
x x A x shigh
do
(a as)
Timine ( 9/ (hh an) t (s s/y an)
D (on 5t) t On sn))
77 7rrne (s1) (81) Eegeotm (11)
Vs1 E shigh
1s
end for
Let
Puub pub (H, (Sihh J [N((H] A, O, (Trmn: (NEHH), {ōr
)ne(H))777 MAT (rn)ne(a))
Define (brsub, Zhp A(St JRETHn to be the approximate belief w.r.t. Pouu
Define [biP0 2n A(S)) such that byth (sn) By,""0 (2x)(s)) for 84 E Shigh and 0
otherwise.
return (6),)) JNe[H]
26

---

Algorithm Learning Multi-Agent (Individual) Decoding Functions with Privileged Information (NE/CCE Version)
Require:
(Tnhnem ( (rahie(m))
и E ITs, controller set (I, S [n]]){())
Procedure MDP_Learning an approximate optimal policy,
that takes input MDP and reward function and returns
Number of trajectories N.
Ensure: A decoding function for each agent and step (eghheenw)ee(m (see Theorem J.4)
for [H].s. Sh, S do
for E [n] do
Fi. (s), ob,) < 1/h h,
= sne] for any (h', s/ a(,)) [M]] Sx
Define the Markov game M associated with G as M (H, S, A, & v1), where we omit the specification for the reward functions and one can specify them arbitrarily
Define M( to be the MDP marginalized by π-
(h, 8n) + MDP_ Learning(M((T-)))) end for
For each The for the first - steps then take action aş deterministically and denote the dataset ki k. k. ( ohh a (441 (NE(NA)
[n], ap A, collect N trajectories by executing policy V. 8h) x
for (on, an, 8h+1) E A
(sh) EME[NTAEIN) 1[s ssh]
(sh, aT 8h+1)= + EKEINI,EEM) 1[s) Ait
Ri 8h: T.T. aT 8 hi
sh+1
(n on) EXEINT,EEM 1/5
Sh: On
onl
TA(SH41 8h, (TT,) + Nu
Na(en.on) (on sn) Na(an)
end for end for Define G = A, (Trhnem, (Onhne(m) ( H1, (riheem))
Define Ti (8% Ch, Pi.A)) = p6 (sn Ch P..)) for each j e [n]. h E [H], en C.. Pi, E Pij return ([m,))][[,,
27

---

Algorithm Learning Multi-Agent (Individual) Decoding Functions with Privileged Information (CE Version)
Require:
- (Tehaa (O ui, (ra)een))
E 1ls, controller set (In S [n]]]
Procedure MDP Learning(::.) that takes input MDP and reward function and returns an approximate optimal policy,
Number of trajectories N.
Ensure: A decoding function for each agent and step bew,)eee) (see Theorem J.6)
for [H) 8h S do
for i E [n] do
Fr, (5/2 ob,,) t 1hh h, = sn] for any (h', s/2, a/)) [H] S A. Define Mestendod (zz) be the extended MDP, which is defined in Definition J.5.
(h, sn) MDP Learning( (Mertendd (T). Ft)
end for
For each i [m). as A, collect N trajectories by executing policy (i sh) x T- for the first - steps then take action as deterministically and denote the dataset ki k. k.s
(ssh oh" ah sht1 [KE[NA
for (on, ah, sh+1) E O Ax
(sN) + EME[N]AEMN 1 ki
=sn]
(nh,n, (1ls,aTT..hr Mi 3h+1) + EKEININEIM) 1[s) =
= 5h: a hTa Ri sh+1
(n on) + EnE(N),EE()) 1ish Ret
sh; On
onl
To (8h4 8h; (T..) + Na(onaaT,,h)
(on sn) x Na(snoo))
Na(on)
end for end for
Define G (H, S, A, A, (T,, (Oxhne H1, (ri)ee(n)
Define (sh Chi Pj..)) = pd (sn ch. P..n) for each E [n], h [H] cæ Cu. P
return (9g, (i(
28

---

Algorithm 7 Optimistic iiainon-iffrmmatinnn Based Value Iteration with Privileged Information
Require:
POSG G (H, (Tx)nemm) fOx (0nhee frahee)))
An approximate belief (Ph C₁ - A(S Ph)lnem.
Number of iterations K.
Ensure: An approximate equilibrium policy Initialize:
(sn, an) - 0, (sh, an. on) + 0n) [ (004111s,0000
8h, an) 0
for l E [K] do
for +- H, H 1, .....
for En E Cn do
1.h (Cn, Ph: an)) + minn (ra an) + b%. (sn. + E To - (clonen)
Filich s,+1 (Cn (Cn-1) H - h 13 for [m)
2 (ōns Ph. Sh. ah.
max (ra((sn an) (8n, an) + E., 2 ( (aaam)) [vlm (cati)) o) for e [m]
Define
high (Ch, 7h) = sn. Bla (R.wWeee Chhichi (Ch.Ph. Sh, an) for e [m]
Define
Q (Ch, 7b) = E 3n.Pa Phly (1r1 B (as m (1..))De T (Gn, Ph. sh, and for e [n]]
(mft (.,1,,1,,, ,,,, DJen Bayesian-CE//CC high, (Tn, Demm)) (c.f. Appendix J.1)
k (Ch) + Ewn Qhiihh (En, { wwyn Cn. ))smm) for i e [n]
low (2h) Ewn
Q (Zn, {x (: Wij. Tne }emm) for E [m]]
end for end for
Execute nh and get trajectory (st..r- ah of o1. for E [H] sp. E S.a, an E A, 01t1 E O do
NY (Shh < EIERK] 1 1s// = 8h, d/ = anl
NÝ (sh, an oh+1) - CiElH] 1 = 8h, at = on o/41 ox+1l
Jh (on+1 1h, an) + Nº (n..n.on11)))))
Ng(en.as)
end for end for k*
high. (ct) Vlaw (cf)
arg min Vi,1 return n²² for general-sum games or the marginalized policy of *² for zero-sum games
29

---

Algorithm 8 Approximate Belief Learning for MARL with Privileged Information
Require:
POSG G (H, (Tea) (Onhew, (ra.n (eaa)e))))))))))..
Controller set {In. S [n]]at[
Procedure MDP JDDP_carning(11
Number of trajectories N,
Accuracy 6.
Ensure: An approximate belief
for h E [H), 8t. E S do
Tie (5 d,))
1 [h' = h s/ sn] for any (/',),,a)) a/.) E [H] Sx
M + (H, S, A, (T helHH, (41) to be the MDP by ignoring the observation and emission of
G. where we omit the specification for the reward functions and one can specify them arbitrarily (h,, 8n) + MDP_ _aP__Lernigg(M,
Collect N trajectories by executing policy V (h, sn) for the first h - steps then take action an for each an A deterministically and denote the dataset {s), 02 a HE
for (on, an, Sh+1) E O x A x S do
(sn) f Lie(NA] [s = sn]
(sn, ()) + Eie[NA] [ss 8h, T'r.,, h =aT,.h1
Nú TTn,hs 8h+1) ΣiE[NA] 1 10 8h, a = (Tr,h: 8/41 8441]
(n on) + Eie(Na] [ Sh, o onl
(s Sh: (T.A) (s
(o 8x) <- Na(sn.0))
Na(sn)
end for end for
for h E [H] do
Slow
8/ S
(ah)))
I
NA
Shigh
E S
Ni((s))) NA
end for for (h, Shi On ahi sh+1) 1) [H] Shigh O x Ax shigh do
T 8h, an) -- (sn+1 sh, an) + Enhu cattr ( as))a)) n+1
Tnnme (on 81)) -- Onh (8))
7T (s1) = (11)) + Exgeom (x1) (sti
Vss E shizh
end for Let
guub = (H, (S
shigg
]NEnH] O, { Tiuuue
H (0 M
ctuune (raR)vETN)AATH))
Define { Pr, C, + A(S)i x Pw)lnemm to be the approximate belief w.r.t. Gunh
Define ( (P C. + A(S Pa)heej such that Pa (Sh, Ph (.) (aasn, Ph (w)) for sh E Shigh
and 0 otherwise return (e]H
30

---

E Missing Details in Section 3
Proof Proposition 3.1: We recall that (-) is the belief of the agent about the underlying state, see Appendix C for a detailed introduction. Note that Equation (3.1) can be written as
N
arg min E
Σ
B
Nr Beands(T)) (Ta) [- [=
xell
Therefore, for any h E [H] and The such that px (Th) > O, we can optimize л separately for each [H] and Ty as:
T Th) E argmin Be
(EA(A)
Now we are ready to construct the counter-example of --obeervable POMDP P for some c E (0, 1)
with H 1, S 85 A {a²,a²}. and O = {or,0²}. We let in ( )ー
On (o" |s') Or is exactly 7-observable. Most importantly, we choose ri"s", a²) = 1, rs a²) 0, and ri(s d²) = 0, (.?, =.
1, and 0, | 1 (1 s2) = 7. Therefore,
direct to see that
Therefore, given such a reward function, the fully observable expert policy is given by ス (a (t = and कं (a² 3²)
i... choosing a state s and a² state s² deterministically. Meanwhile, by our construction, one can compute that the belief given observation o ensures b (o = Unif(S). Hence, the corresponding "distilled" partially observable policy under observation o is given by
(-'0") = argmin Esr ( [())|()) [4)
gEA(A) D)( +
argmin (EA(A)
f(L/g(a)))g(a)) + f(0)g(a²) ( +(0)()))
argmin
(EA(AA)
+ + (()(a((()(((aaa))) Da(a") 1(//((²))aaa
argmmin (EA(A)
where the last step is due to E ((A). Now consider the function g(x) = x (1/xx) for x > 0. Is direct to compute that g'xx) f(1/z) f'(/a), and g²(x) ["0/=] > 0 due to the convexity of the function f. Thus, we conclude that g also convex. By Jensen's inequality, we have
[(1/g(a (g(aa)) + 1(1/g(a2))
f(2/(g(a²) + q(a (a()))
2
where the equality holds when q(a') q(a²) 1. This indicates that 7- Uni On the other hand, combining the fact that b, (o') Unif(S) with < 1, is direct that the optimal partially observable policy т E arg max u (Tr) satisfies गा (a+ |o*) 1. Now we are ready to evaluate the optimality gap between and R* follows
2
(
PP (o' (O')(V PC (o') V p +P'
(o (o))(V (0²) V PP
(o²))
2 PP" (0 (V (o') V P (o')),
where the last step is due to the fact that से is the optimal policy, leading the fact that V (02)
v p' (0²) > 0. Now it is not hard to compute that
PP (0')21-7
Meanwhile, can evaluate that
V.tP' (0t)
1+6 2' V²a pr (o²)
4
and correspondingly Va.P! - This concludes our proof.
V/'P'' pp P(to) -- implying that " (7) ()*)(((*))) (1-)(1-0)
31

---

Note that another counterexample in a similar spirit was also constructed in [34], demonstrating that the expert policy for poorly chosen agent-environment boundary can be useless in imitation learning, although the 7-observability property is not satisfied for the construction therein.
Remark E.1. The counter-example p constructed above can be also used to demonstrate the bias of the state-only- value function as an estimate of the history-based value function that appeared in the policy gradient formula for POMDPs, in the maite-horizon setting, i.e. (Tx) [ P" (sn)) +
V. (Th) (mirroring Theorem 4.2 of [6]). Specifically, the counter- eaunte above, we consider the policy π such that Th (a1 | 01) = 1 and कı (a2 02) = 1. The state-only-base value function can be evaluated
Vr (51) = 1-7
VT. (22) = 7Ee
2 7
which implies that ~b (ot) [VT (51)) = v.
(1))+
+
On the other hand, (s1)), showing the bias of such
2
holds that V.T.P' (01) 1ts, which is not equal to (o.) a state-only value function.
(V
Example E.2 (Deterministic POMDP [36, 81]. We say a POMDP P is of deterministic transition entries of matrices [Tn]ee(m and the vector bl are either 1. Note that we do not make any assumptions on the emission matrices.
Example E.3 (Block MDP [43, 18)). We say a POMDP P is a block MDP if for any h E [H], 84,5% 5/ S, it holds that supp (- (s))) n (u ()) when sn. + s/
Example E.4 (k-step decodable POMDP [19]). We say POMDP is k-step decodable POMDP
if there exists an unknown decoder o* = (0) Z, + S SeE such that for any h [H] and
reachable trajectory The IP? (sh o (zx) Th)
where 3n (O A)max(h-1A-1) x O,
24 = ((o, a ())(A))h_1;0)) 0,), and k:(h) = max (h- k 10++,1 1,1).
Finally, to understand how our condition can extend beyond known examples in the literature, we show that can indeed allow the decoding length of Example E.4 to be unknown and arbitrary (instead of being a small known constant as in [19] to have provably efficient algorithms).
Example E.5 (POMDP with arbitrary, unknown decodable length). This example is similar to Example E.4, but the decoding length m is unknown and not necessarily small constant.
In light of the pitfall in Proposition 3.1, we will analyze both the computational and statistical efficiencies of expert distillation in Section 4, under the condition in Definition 3.2.
Proof of Example E.2 & Example E.3 & Example E.4 & Example E.5: To see why those examples follow our Definition 3.2, it is indeed immediate result of Proposition 7.1.
Proof Proposition 3.3:
Here we evaluate the computational complexity and sample complexity of each iteration t as follows.
Sample complexity: The algorithm executes the policy -t-1 and collect episodes, denoted as (si.H41, of. [ Thus, the sample complexity of each iteration is ((K).
Computational complexity for policy evaluation: The policy evaluation of the vanilla asymmetric actor-critic is done by minimizing the Bellman error. In the fmite-horizon setting with tabular parameterization. it is equivalent to performing the following update for each h E [H] in backward way and each E [K]
Q A at) - 1-)aath sh a
Chei Sh shai))
+ rh(sh att + [T( s/ at))
Σ
jEs F (
for some stepsize a E (0,1), where J(Th, s/ at) = (j E Therefore, the computational complexity for this procedure is of POLY POLY(H,
(T), s
(rt ot (t.))))))) at)).
32

---

Computational complexity for policy improvement: For tabular parameterization, computing v log (a + takes ((1) computation. Hence the policy update Equation (3.2) performs POLY (H, K) computation.
Meanwhile, under the exponential time hypothesis, there is no polynomial time algorithm for even planning e-approximate optimal policy in 7-observable POMDPs [26]. This implies that the vanilla asymmetric actor-critic needs to take super- super-polynomial time to find an approximately optimal policy. This implies the corresponding sample complexity has to be super-polyy super-polynomia..
Finally, we remark that even if we the policy and the Q-function not depend on the entire history Th but only the finite-memory Eh. the proof still holds. The key is that whenever one only computes the sampled history/inite-memorres,, i.e., updates the policy in an asynchronous way (in contrast to the synchronous one where the policies at all sitorris/fniitemmmrrr and computational complexities will be coupled with the same order per iteration, which implies a super-polynomall sample complexity due to the super-polynomial computational complexity. This completes the proof.
are updated), the sample
Derivation for the closed-form update Equation (3.3). Note that the proximal policy optimization [72] update has the policy improvement as follows
π 6- arg max
L
(T)
'B.
Σ AE(u)
KL(TT. (-1 | (.1mm))
m.))
(E.1)
where is some learning rate and (TT) is first-order approximation of the expected accumulated rewards at 111:
L (T) = v² (1 E
Σ hE
(Th, ( Sh. 3, m- - = 2 (-1m))
By plugging (Tr) into Equation (E,1), with simple algebric manipulations, we prove that:
t (- TT) x - Th) exp (nE.)>bu(r)) 10: (Tn, Shy .)])
Missing Details in Section 4
Proof of Lemma 4.4: The proof follows by the assumption that the total cumulative reward is at most H,
u
(T) >
E
Σ Th hE(m
1[th :E [H] Ih (sh-1, ah-1, on) = sn]
EP
I[Ah:E [M] yn 188-1,,h-1,,0 on) =sn]
Th
hE(m
+ E.
Σ hE[M]
1 :E [H] (8h-1, * snl
EP
Σ Th hE(H]
13 :E [H] 9h (h--1, s8)
E
Σ Th NE[H]
-EP
Σ rh
15h :E [H] 9h 4h-1, + snl
KE(H]
(T F)
Hp .P
[3h :E [H] : Ih (sh-1, ah-1 f sn)
p
He,
which completes the proof.
33

---

Proof of Theorem 4.5: For each step h E [H]] we define Dj to be the distribution over the underlying state Sh-1 at step h 1, taken action a/-1 based on x the underlying state transitioned to 8n. E S, and the observation Oh ~ On(-I 8h). We remind that we include at step zero, a dummy state-observation pair (ND, 0o), for notational convenience. Formally, D, is defined as
Da 44-1, oh $h,)= == F= :P [sh-1, a4-1- Oh, 5%1
We first use union bound decompose the probability that we incorreetly decode,
p _ P
[3/ E [H] gh (8h-1, 1, on) o 8n] -
Σ
p .P
Ign (8h-1, 0h- 0n) # 8n)
he
Σ NEW
Deennbaa lg (8h 0h, + sh)
(F.1)
For each h E [H] we can use M episodes to collect M samples from the distribution DA. In addition, since state $ 41 is dummy, we need not to collect episodes for DH+1. Denote the of collected samples by D. We define the decoding gh. for step h E [M] as follows:
9h (sh 9n = ={ (sh-1, 4h-1 On sn) E D
Observe that by Definition 3.2, {sh (5h-1, ah 0h; 8A) E D } is either the empty set or contains only a single elements, in which case, is true that gh (8h-1, ah-1 0h) = thu ($h-1, O4-1, 9b) (e) is the real decoding function, Definition 3.2). Moreover, we slightly abuse the notation and let D denote the empirical distribution induced by the samples in D. Thus, with probability least (A... (-.0.--log lox(H/01
8h)
1 4 and setting M θ
for each step h E [H]. we obtain the following by the
result in [13]:
p Ion (8h-1, 1, a4-1, 0n) F sn] _p= P (gh (8h- ah-1, on) = ()
(Sk-11 1 on, 8n) & DM
Σ
PONDD lu= w'l supp Di:1)
(Esupp(Dn))
Sdrv (D,. DM
Se.
Thus, by union bound, with probability at least 8, we have that for each step h [H].
P 11 [on (8h-1, Ah on) # sn) S
which in combination with Equation (F.1) concludes the proof. Finally, we note that we used total of θ H A-0-8-Joz(H/0) episodes from the POMDP, and the computational time was (H, log (1)).
G Provably Efficient Expert Policy Distillation with Function Approximation
We now turn attention to the rich-observation setting under deterministic filter condition. Definition 3.2 motivates us to consider only succinct policies that incorporate an auxiliary parameter representing the most recent state, as well as the most recent observations and actions. To handle the large observation space, we further assume that for each step à E [H] the agent selects a decoding function 9h. from family of multi-class classifiers F, (S x A x O - S}. For the function class Fr, we make the standard realizability assumption. We formally summarize our assumptions in Assumption G.
Assumption G.I. We consider a POMDP that satisfies Definition 3.2. In addition, to derive learning algorithms that not dependent on O, for each step à E HH), we assume that we have access to class of functions FA S Ax O S such that the perfect decoding function thn. Fr.
34

---

We aim for our final bounds to depend on complexity measure of the function class F= {Fi hee(m rather than the cardinality of the observation space O. We utilize the Daniely and Shalev-Shward Dimension (DS Dimension) (Theorem G.2), which characterizes the PAC learnability for multi-class classification [8]. Defining the DS dimension beyond the scope of our paper; we direct interested readers to [8] for further details. For intuition, readers can think of the DS Dimension certificate of PAC lcarnability without loss of intuition.
Theorem G.2 (Theorem 1 in [8]), Consider a family of multi-class classifiers F that map features in space X to labels in space y é y. Moreover, assume there is joint probability distribution D over features in X' and labels in y, and that there exists g E J such that for each (z, y) E supp(D), g" (x) = y. Given samples from D, there exists an algorithm that with probability least 1
outputs JF such that
( + log (1)
P PLay)- (T() * OO
n
where (F) denotes the Daniely and
Dimanssoo of the function class J.
We are now ready to present the main theorem of this section.
Theorem G.3. Consider a POMDP P that satisfies Definition 3.2, a policy x E Ils, and let (Fn C (S A x O S) NETH be the decoding function class, and thy, E Fr. for each h [H] ie.. (FrJhejm) is realizable. Then given access to the classification oracle of [9], there exists an algorithm learning the decoding function {gh heje such that with probability at least 1 6, for each step h E [
[H] gh 1hh-1,a--1-11 on) * * S, H2a e m) 45 (Fh)++oo)g(
p
Bh
using O
episodes, where dps(Fh) is the Daniely and Shalev-
Shwartz- Dwwartz-Dimension of Fr, [8].
Combining Theorem G.3 and Lemma 4.4, we obtain the final polynomial sample complexity in this function approximation setting, using classification (supervised learning) oracles (c.f. Table 1).
Proof of Theorem G.3:
For each step [H], we define D, to be the distribution over the underlying state 84-1 at step 1, taken action d4-1 1 E A from RF the underlying state transitioned to Sn E S, and the hallucinated observation oß ~ O, ( (-18)) (we remind readers that for step O, we use dummy state so and action ag). Formally, the probability that the sequence (5h-1, Ah-1, Oh sh) is sampled from Dy equals to
D (s -1 Oh $n) = p E [s-- 1n-1: Ob 8n.
We first use union bound to decompose the misclassification error,
p P B3h
[H]
9h (Sh- -1-1: on) + snl s
Σ p=".P Igh (5h-1, 1h-1, on) # spl
NET
= Σ
[Th (8) 1..1...-111 11 1,, # sn]
hE H
(G.1)
For each E [H) we can use Ö 0( (maxne[H) (F,) + log (t)) episodes to collect
δ # [aX 3/2 (F)) + log (t)) samples from distribution Dj. Hence, by Theorem G.2 with probability at least 5 H.. we have that
PP Teh-1.eh1..0.oau
Th (8K-1, a1, on) * sn] S H
Thus, by union bound, with probability at least 1
using total of
u1a maxxeeH 3/2 (F)) + log (1))) episodes we have that,
Σ P P( Rhh (8 (8h-11 9h-1 on) * sn) Se,
h
35

---

which in combination with Equation (G.1) concludes the proof.
H Missing Details in Section
Proof of Theorem 5.1: Let n² argmax Beiwan (s)), where define Vf((s)
Bor M))ll (0 = (01), 81 ai)]. We first note the following equation
1 B S [V
T EE[T]
EEronil [V2 (s1)] +
1
Σ
E. Bsrapi (vr (81) Vi" (51))
1 Σ
Barus
(VT' (ir((s)) (s1)
T tE|T
LE(T)
(H.I)
Next, we make use of the performance difference lemma [39, 1, 74] on the extended space [ (Zn S).
Definition H.1. Consider the class of policies I1 such that at step h E [M] the policies in II take action based finite memory up to this step and the of the underlying state, e.g., for each policy TI E PT Th 2n x S + (AA).
Observation 1. Note that II CIP
Lemma H.2 (Performance difference Lemma [39, 1, 74]; see e.g., Lemma in [74]). For any pair of policies (Th )e( n' we have that for each state E supp(μ1):
II",, and approximation of the Q-function of policy
VIT (21, VT (11,
Σ
EFANTI
(0T sn, T. Th, 8h) - x 2r. snn)
hE(H)
+
E ETANTl
@Tl 8h, an)
onti (-10))n)) (na and + (itz( [4t1)]
hE[H]
o441 Dp [841)
where 1 (2h, sh)) (. ( sn, sp. an)l.
Setting x' E ITL C ITLP and n' n² E ITL C ITLP, where remind that x²
argmax,ent V (81), and for each 2n E Zn and h E [H) we abuse the notation by letting o (-
2h. 81) n/. 24) and m.( 2h, 8h) m.( 2n) for all 8y E S. The above formulation is thus simplified to
Esgnei (s1( VT (s1)
Σ Er
(@x (En, 8h, )), m/ ( En, 8n) - m.( zb; on))
bE[H]
Σ ETANT (Th. sp and
(ra(sn, an) + (3441,
SntInTT smaan on obs (8412)
hE[H]
Σ EFa ((0T (En, m Tth
Eh, Fh, 8n) m (. Z/, 8h)) on))
36

---

where in the inequality above we used Lemma H.3. Since our policy does not depend on the realized underlying state Sh.
Ees Bernun
[VT ($1) (VT((s1)
Σ BrAnn
((0T% (Ens sn. T Zh, 81) - Th Z/: (nn)
hE[H]
Σ he(mf]
E BIT
( (Bea
@ (En. sne ahl. 2n) T//l 2m))
Σ hE[H]
(Bea (ou)(z))
(2 (Zhr (Zn, sh: ( 2m)) -./( zn))
E Erinat K (Eerndeto (0h (En, sh: (on @ (en, sh. :) Thi znd -/ (mm)))
Σ he(H]
2 Σ
R ~o* [(Resmomccn) @ (Enr on. .) (t(( =n) - - =n)?]
bE[H]
Σ Eraws ([EEss @hh (zn, sh: - Esg @ (E St: ->) 111]]]
he[M]
2 Σ
E E (Beanor
@h (Z
sh. (zh,
(t (-1 =N) - -m/( ((((-12n)] - 2. H
Σ
Erywar (dTv((b(TT)) (=x))
ME
he(M
The last inequality follows by Lemma H.3. By averaging we get,
1
Es VV' (81)
T
EE[T]
2 :BermilVy (VT (s1)] +
1
Σ Ena
Σ
(om((n)
@T (Z (2e :)] . an) 7/(.1 zn))
T
hE(M
LE(T]
-2 H. Σ E Enaw [drv (bu(tT)) byt (2n)))
hE(m)
2 E Barmil [VT (s1) + T
H max E, ~r he(m
Σ FE(T)
(Benobe(ow)
[K (EN, sn, .) v 2n) - xx(. 1m)))))))
- -2.H2
max B ~x- [dTv (ba(T))) (Zn))) kE(H)
2 Exrrnil (VT (s1)] 2H / H log (I-A))
2 H2
max ET,,OT (dr (zx)) he(m]
VT
where the last inequality follows since for fixed h [H] and 2n, Zno the agent updates her policy on memory 2h according to MWU on feedback { (b,,,)))(
@ (E sh. a)) JaeA and thus, the
accumulate regret is bounded by (Section 4.3 in [10]):
(Q (Zn. sn. .) T 2n) - 10))m((( 2m))
Een (oui(n)))
1E[2]
log (1-41)
+ 17- T Q (zh, (zh.sh. Sh.
1) log (IAA))
+n T H = 2 log (IAI).
The proof follows by combining Equation (H.I) and the inequality above. Finally, to achieve the near optimality in the class of 4T4, we bound the optimistic estimate using Equation (H.2) in Lemma H.3, and global oeat-optimality for large enough L. under 7-observability is a direct consequence of Theorem 4.1 in [26].
37

---

Lemma H.3 (Optimistic Q-function adapted from [48]). Given policy л E II,, and a parameter M E N. let 1Q7 Zn x x A [0, H. hejm be the output of Algorithm 3. Then, with probability at least 1 8: Vza E Zn, sp ES,ar an
H h+12 (= Sh. an)
E Sh41~Tal1 (asen)) o++1 01 (111ss++)))
an) an) + Vi+ (26+1,
max(O, (OD)
S.A S.
S.A
M S A- A-H. H
BarNi [s1] E, E (v) [ 50 o H²
log
log
M
(H.2)
where (s1) = (11-)()(11))))mm
= (1, (T((s)) =
$1,
~o (-1 #1 aa).a (- [os)
1 (21 (01), an)l and V (2h, = EuL saa 10) (za, Sh, an)l Moreover, Algorithm needs a total of M episodes from POMDP P and runs in time POLY(H, S, 04,04.
Proof. For each step E [H]. collect M trajectories with states using policy x on POMDP P and let Dn = [ [e[[M] be those collected trajectories. Define the empirical transition, observation and reward distribution as follows:
NA (Sh, ahs sh+1) =|{T= = (s 01 11, .i.., s oh o/ r/h) Dn.
(9h, Cr. sh+1) = (5 a sh 1)
Na(sh, an) = Σ
(n aa(snaan,aa 80+1):
Eh+1ES
Nu(sh) Σ Nn(sn,an) an),
EA (s 0n) = -|T = (5'2, o1 ai ...,, s/ o/ o/ r/)) E Dn (8h, on) = (5% o/)b//
NN Ohs sh+1)
(sh+1 8h, an))
No. (sh. an)
Na(SE, on)
(on 8h)
(sn)
Set 61 = 2-5-(a+1) By [13), there exists a constant C > such that for each step E [M], state
E S and action E A with probability at least 1 61:
Slog(1/51)
[ 8h, un) Ta(.1 (. 8h, ax)ll min
2,C
max (8h: (1)h:a)):) an);
O 1lg(11/1) (nax(NN,ss))..))
[. sn) O.( sn)ll S min
2,C
For the rest of the proof, we condition on this event. By union bound, this happens with probability at least 5- We define the optimistic Q-function recursively as follows for a memory-state pair (Zn, 8h)
2n x S:
(H =0,
VZH+1 E 2141,8H41 E
(T Sh, an) min
h
(V (2641: Sh41)l )) + r(sn, ap)
8 Ls (n)an.a)).
In 1...6n111111 1 (111111)
S Slog(1/51)
+ min
2,C'
max(NN(s, an),
O Olog(1/61)
+ E + (- H. min 2, V max(Nu+1 (8xx4))))
38

---

Eag 1 11f a a Q1 (2n+1 Sh+1, ah+1)]. Hence the time com- plexity of our algorithm is POLY(H, A,, 0²). To prove the first condition, we fix step [H],2, 222 an.an and state εд S and consider the case where Q% (2 Sh. ah. = h++1. In this case, since by assumption on the POMDP P, Th (sh, a4. I 1, and by definition of Qh+1l, H h, have:
where (2
(h,, 8h: an) = 1 + H h E ex (9nos)) [ra(sh; ah) + ( 81+1)]
on 01 (811))
Q7 (Zh, 8h, ah) # h observe that:
S Slog(1/61)
Qh( Sh, (2,
an) =B
Met1 (2h+1, (h+11l + r(sh, an) + H min 2, 2,
ont T (onion) o a mat1)
max an),
O Olog(1/81)
E Sh 11 (s H min 2,C V max(Nn+ (89441))))))) ),
B Ta (-1ss.a)) (na an) + Vi+t (Rh+1s Sh+1]], +1 s41))
and hence, 12 heE satisfies the first condition. Moreover, it holds that:
s (log(1/81)
sa TT [TT (s),an) (ra(s:: an) + V (#t+1, 9h+1)1+ 2H min 2,C (7 8h. an) <
V
max(NN an),
on Grtt (+44)))))
2 E 2 (n) H min 2,C V
O Olog(1/61)
maxx(N+t (8x+1 19441);11 1),
S Slog(1/61)
<B snti~Tal (-10),a)) (n an) Vitta (Zh41 S4+1) 664 min 2,C V
max(NNn an),
ob 0 en+1)
O Olog(1/61)
+ E. E Ma H min 2,C
maxx(N (1
Thus, it holds that:
VI (k), Vi (Zhy 8h)
S (log(1/51)
< E
on (: (-11)))
[V+ ( 8h+1) V (1 C. H H-E ma
(120) (112))))
max(NNn(sx) an), 1)
5441~aa ss.)))) o6 1 (-1 sh11)
O Olo(11/5))
+ 2 H.
ag ( (x)) Sh a (-1 ka.ak)
V max max(Nn+1 (9h+1,
Thus, we conclude that
max(S, log(11/)))) max(NNx(s,, an), 1)
Ea (T:) - E [VT (V E, (1
Σ AE(M]
8.H. C.H.C. C.
8 Mmxx(OO S) 11g(1/))) C-
Σ E, (11 (...)) max (N an),
RE[H]
To finish the proof, we make use the following lemma.
Lemma H.4 (Lemma 6 in [48]). For each step h E [H], and state-action pair (sh, un) E S A.
with probability at least dz:
1
-0
(v/
S. A log log(M/55) MM/82)
max(Nx(s. an).
M
39

---

By setting 82 2.S.A-H' and taking union bound we have that with probability at least 1 - 5:
5
E [s) - Ba (ff((s)))
[V
=8ymax(O,5)) log( 1/61) C. Σ
B (n....
max (N (sh,)))))))
ke(m]
00 H? ma)( S. A
S.A)
M MM A. 11
-log
log
M
Proof of Theorem 5.2: The proof of Theorem 5.2 follows by combining Theorem H.S and Theo- rem H.6 below. Theorem H.S proves that we can approximately learn a POMDP model P computa- tionally and sample efficiently, thanks the privileged information.
Theorem H.S. Fix any e, E (0, 1). Algorithm 4 can learn the approximate POMDP model with transition TH. and emission 01.H such that with probability at least 1 8, for any policy E Ie and h E [M]
EP ( 8h, an) (. sh, an) + ||Ox 0..( sw)llaa $ O(c).
using POLY(S, A, H, 1 ())(()) episodes in time AH, log( (!)).
Proof. Note that by Lemma H.11. suffices consider only π IIs as the optimal value for policies in _[ can be achieved by those in IIs (by considering (3), as) == [ 8h, ax)
Tx a1)ll + [0 (s)) On(- 11). For each h [H] and s S, we define
(SA) max di (sn). sElls
Fix > O, we define u(h, = {sΛ E S (p)) 2 e1}. By the guarantee of the EULER algorithm from [89, 37], can learn a policy (((h,s)) 9n) with sample complexity Ö 82 A such that d% (h (sh) 2 pa(*)) for each sy E *U(h, 61) with probability 1 81. Now we assume this A, we have executed
event holds for any h E [H] and E e(). For each E S and in Algorithm 4 the policy v/ (h, sn) followed by an action а₁ E for N episodes, and denote the number of episodes that 8), and on are visited as (M az). Then with probability 1 /
Na (*t, an) 2 (s.. by Chernoff bound. Now conditioned on this event, we are ready to evaluate the following: for any IIs
E [T (..1.118 Sh, and Tn(-1sh,aa)ll: 2 Σ
ax)l1
(Sh) (- 8h, an) Tx 1 sh. an)lla
<
<Sc
+ 1
Σ
d (an (sw)
2S ASSlog11/6) (1/52)
2
N Npn (sh)
<Sc +
Σ
dn
(hhsa)
(sn) sh)
25 log(1/82)
(Pn(sn)
5n (ld
< SSt + Σ
V Va (h.wo) (on))
25 log(1/52)
N
Sn.
(log(1/82)
< SSc + S
N
40

---

where the first inequality follows by [13] with probability at least - 62, and the second inequality uses dn (hh,sh)
(sh) Pa (5n) for all sn E (k, e1). Similarly, for any 1Is
E (1((((118)) - -n(. s1)ll S S61 + 1
Σ
di (8N))
20 log(1/62)
Npn(sh)
An (n((h))))
SSc +
E
dp
(h.
(SN))
10 (sn(ss)
Sca
E an Eu(h,,))
Val (h
20 log (122)
N
250 (ogg(2)))))
< SE1 +
N
where similarly the first inequality follows by [13] with probability at least - 82, and the second inequality uses d% (H(hss))ss) 2 pa(sa) for all 8s. (1) Therefore, by union bound, all the high probability events above hold with probability
1 - SH0 SH Ae TeTAe--NeJ -N1/8 - 2SAH62
Therefore, we can choose N = -=055=500 and 61 = (G)), leading to the total sample complexity
)) = 6 (
5 AH
SAAHO + SH
S*A2H
SHA N θ
(H.3)
22
which completes the proof.
Now with such a model learned in reward-free way, we are ready to present our main result for approximate belief learning.
Theorem H.6. Consider a 7-observable POMDP (c.f. Assumption 2.2), an e > O, approximate transition and emission [TT and (0% learned from Algorithm 4 that ensure that for any π E I[ and E [H]:
EP
[ sh, an) - Tx(- sh, an) (-O((1 On(: sn)ll1s (..(:1ss)li1 S S0 (
Then we can construct in time POLY(H, A, S, O, log )) belief (b Zn + A(S))eE with no further samples. In addition, the parameter in Algorithm satisfies N θ 7tti
and
our class finite memory policies I satisfies L 2 h [H]
then for any æ E I and
E (y (2+))l < O(e).
Proof. We firstly consider the following simple while important fact: for the estimated emission On. its observability can be evaluated as
10 (b 11)l1 2 I10 (b 8')l1 - 11 - 0 - (011--1111- -b--
2 ( -10 Onlloo) - 8' I11-
for any b, E (SS) and |00 O1 max,, O0 (. (n) O% (.1s))))) Therefore, if one can ensure that the emission at any state 8s, is learned accurately in the sense that ||O,( (-| 8h)
can conclude that Õ, is also 7/ 7/2-observable. However, the key challenge here that there could exist some states 8). that can only be visited with a low probability no matter what exploration policy used. Therefore, emissions at such states may not be learned accurately. To address this issue, our key technique is to redirect the transition probability into states that cannot be explored sufficiently to some highly visited states, in new POMDP that close the original one in generating the beliefs.
sx)lli (-1)))) 3.
Specifically, first, for any é O, we define
h : 8h ES
Na(sn)
<
é1
S = S S/
N
2
41

---

By Chernoff bound, with probability at least 1 Se --eSeNe Ne1/8 it holds that
S C {sh E S (s( < <1h,
where pa (sk) == maX (8x). To see reason, we notice that for any 8h. such that pa (sn)) 2 61, with probability
e-Ne/ it holds that N,(ss)) 2 - Therefore, the last step is by taking union bound for all s4. Now with Si: defined, are ready to construct truncated POMDP pinune such that for each h E [H], we define the transition as
5h, an) Eej Eghn E55M TA (1/+1
TA
Thrun (sh+1 Sh. an) = (s(5++1 9h, an) +
Vsn S, sh+1 Shigh ap E A,
TA (8h+1 8h, an) = O, Vss E E 8h+1 E Sh an E A.
Meanwhile, for the initial state distribution, we define
11 trume (81) == (1())) + Enjest (1((1)
V81 Shigh
1
111 (81) = 0, tsa E S
For emission, we simply define
O% Omnn (on 84)) = On 18h) 4h E [H], E E E EO
Finally, we define the rewards for prmune arbitrarily. Now examine the total variation distance between the trajectory distributions in P and piuuee" For any policy æ E I[ it is easy to see that
P (Ta) S p. (Tn),
for any Fa E TT := ((8 O1:h: (1:A-1) 1) 8/ E Shirh wh' E [M]], since some rarely visited states' probability has been redirected to the highly visited ones pirune Meanwhile, it holds by union bound that for any h E [H]
P P
(T & Thish
Σ N'E(h]
P E S S HS
Therefore, by noticing that pa parze (Th) = for any Ta & Thigh and h E [H]. the total variation distance between the trajectory distributions in and pirune be bounded by
Σ IPT.P(TN) pappiin ( 2HS
(H.4)
On the other hand, by Equation (H.9) of Lemma H.9, we have
E by (Tr) trune (Th 11) <
Σ 1
porrp (Th)l 4HH
(H.S)
FA
With such an intermediate quantity puruee we define the transition of its approximate version Paunc as follows: we define the transition
Tymen (8h+1 Sh, an) = (s Sh, an) + EslurESiin EStir Th (n sh (h))a)) an)
high Ash S, 8h+1
1411 ah E A,
Sh+t1
8h+1 E Sho an A. (ymu"((s44) Bh: an) = O, Vsa E S, Meanwhile, for the initial state distribution, we define
E Ex (i(1))
[41 (81) == Jrn (81)
Asl
shigh
(
high
( (s1) := =., Vs E Slow
42

---

For emission, we define
Og (on sn.) == (on 8h), vh E [H], E S. E On.
Now we define 0 E | x to be the sub-matrix of On where we only keep those rows
of O that correspond the states in shi Similarly, we define 0g gl to be the sub-matrix of Ch, where we only keep those rows of On that correspond to the states in shigh is direct to see that Q is still 7-observable. Meanwhile, we notice that
O Olgg(S (og(SH/8) (A()
20 (og(SH/0)
1 10% = max E.
|[ (0((-18)) 8%) O (188))ll S
max E.S.
S max
E.E.S
No1
where the first inequality is by Lemma J.9, and the second inequality by the definition of S Therefore, if we take
80 log(SH/0)
--11
it is guaranteed that Oraih -0 S 2. Therefore, we conclude that Oryi is also 7/2 r/2-observable.
Now ready examine Bi, We firstly define the following POMDP Poub, which essentially deletes all states in S from the state space of Pimine at each step h, which does not affect the trajectory distribution as they were not reachable in Pirmne Notice that the emission of puuu is exactly Orghh implying that Push is 7/ 2-bbbebbee POMDP. Therefore, for policy class with finite memory 11 with 2 52(7 log(S/e). by Theorem 4. in [25], is guaranteed that for any æ E I1,
Pusk 1 (Tr) - 3, 111 6,
where (Th), Br (zx) (zn) E A(S
Now we claim that
E Time
6% (2n)li Sc
(H.6)
where we define b (2n) E A(S) by augmenting B),tt (zn) with for states from S/ To
see the reason, we notice that simulating Pinune is exactly equivalent to simulating Puub, and that Byume (#x) 0 for 8% Slow Bj (Tx:) 0g (7x) for 8n shigh
For the total variation distance between the trajectory distributions pirnne and Prrine it holds that by Lemma H.9
Σ TM
I pinu (TH) puueu Soux (Tx)l
< B.pu paue
| (- (188,
- -Tr.une (1181, an))aa + +100 (.18n) -)ninn (.18n)ll
E
[ (a..:18h,0a) (8, an)) T an)ll (1Da ( 0x (.((11)))l]]
kE(m]
where the last step is by Lemma H.7.
Now by Equation (H.4), we have
Σ
sh, an)ll + (-Ox(-1s)) (+1sm\l)) [- sh. an) - -
Σ
SHSEi + EP ||TA(-Ism (Tn(-Ish,ap) an) (-1 sh an)l + (-0 (-1 (k) - (=)(-s)))ll
h
C
2
3
43

---

where the last step is by Theorem H.5. Hence, by Lemma H.9, it holds that
+8H2Ses Sc1,
(H.7)
E p
16 (Tn)
Birune (Tn)lli - 3
2e
+ 16H SeT
(H.8)
Finally, we are ready prove
E ba (Th)
(=x))
(1x))lz
E (Th)
by grme
(Th)ll1
E [
(( (Tr)
Bruue (Tr) + E..1 1| 18 (Tr) (zh)) 11
E ((b((T)) bh bi,une (Th)ll1
E paue l10% mume (Th)
Bi (Th)
(Th) 1 + EE 100, (Tr) (2n))))))
+4 - -p
11 2|| p
poie
p Puue
1
< S(11 Oee).
Therefore, by setting c O (HA5), ) prove our lemma. Observe that algorithm needed no further samples. The computational complexity follows by computing belief by on POMDP Pouu using unite-memory policies of size O(x for--loogg log(S/c)). For the final sample complexity, only need to ensure = O 0 (6)(5H in Equation (H.3), thus concluding our proof.
We conclude the proof of Theorem 5.2 by combining Theorem H.5 and Theorem H.6.
H.1 Supporting Technical Lemmas
In the following, provide some technical lemmas and their proofs.
Lemma H.7. Fix 0 and index set S C [n]. For two sequences Tis and yio! such that Li E [0, 1] for E [n] and Σ, Iz = =1, E, y 1, we define
G x2 E5es Dz
Vi V
3 =0 E S.
S
We define Pr: similarly. Then, it holds that
x yil 2 Σ
3 Tzl-
Proof. Note that
Ejes Ij
y/1 Ejes Bj S
Σ 2
Dil
Σ
[ Di Σ
Iú +
|S|
igS
Σ jES 4
Ejes Mj |S|
< (n
|S|) -1S)
|. Del
S
ig
Vel,
which completes the proof.
Lemma H.8. Fix two finite sets X, 3 and two joint distributions Pı, Pg E A(X' x Y). It holds that
-Ep,(0) Σ (P((v - Pa(ulag x)) < Σ P2
Pe(a, v) Σ PP)(x) P2(x)
J.y
Ep,(0) (x)
Σ p))) [)(]
44

---

Proof. For the second inequality, it holds that
Σ
y) - P2(x,u)] = Σ P1 v) - ((o))) x) P2(2,
2.9
J.y
Σ V) - -( ( ()) P2(2.V)
a.y
Σ ((()) P2(v1 =))] P2 (P) P2(x))
= Ep (a)
Σ P((vl)) - P2(112)) + Σ -P(()))P(('))
Meanwhile, for the first inequality, it holds that
Σ
- - Σ
p1 - x) + a) P2(2,))
=
2.8
F.y
( +0) - P2(2,))
2.
Σ (PP)())(P)())))))) x²) - P2(ul x)) + P2 (P) (z) - Pa(x))
(x)(a) Σ Pi [2) - P2(1=)) + Σ -
concluding our lemma.
Lemma H.9. Consider any two POMDP instances P and P and define the belief functions as b
and by respectively (see Appendix C for the definition of belief functions). It holds that for any
T [|
I.p.P pTp ||1
p (T - P (FH)
=
S
1/111 Milli
+ R
P
Σ
[- 8n, az) - (- sp: an)lla
hE(H
E
Σ
|0 8b) - 0n 18)111
hE[H]
E (T, -TN(7)) = 21|p1 - fulll l + 2
Σ
| sny (n) Tx (-1 sh an)ll2
h 11]
2E
Σ
[:0 (n) - 0. sn)ll,,
AE
where we remind readers that we denote by Tи = (S1.H, 01:H -1:H-1) the trajectory with states from an episode of the POMDP
Proof. The first inequality also appears in [44] and we provide simplified proof here for complete- ness. By Lemma H.8, it holds that
p (TH) pr.. (7n)l
p (T 1) - P (Fu-1)]+ E
Σ
TH TH (a FH Tn-n)Tu--(sH 10) |8H)
T
eg 9g-1--H-0
-H (ax FH TT T (SH SH 0 (HH)
p (FH - pr. (H ) K [ITH= -0--n(1ss---.0 an + [[On(|IsH) Gw [sw))]]
FH-1
45

---

where the last step is again from Lemma H.8. Therefore, by repeatedly unrolling the inequality, we proved the first result.
For the second result, we notice that by Lemma H.8, it holds that
Σ 1 (Th, P (Th, shh) Σ P (Th) P (T+)+EE'
Σ P ( Tm) Px Tx)
Notice the fact that Px (sN Th) does not depend on since it exactly the belief bn(T)) we conclude that
E b (Th) -n s
Σ p
(Th, sh.)
-P (Th, sn)l + Σ (TT) -P (Th))
Σ FH
p (TH)
-PT.P(TH))
(H.9)
where the last step comes from the fact that after marginalization, the total variation distance will not increase. By combining it with the first result, we proved the second result.
Corollary H.10. Consider any two POSG instances G, & that satisfy Assumption C.8 and the corresponding belief functions the form of 1o p C₁ A(P,, x8) for any E [H]. respectively. It holds that for any Ie
B ||T (s, (c,)
(
p (,, (x)lli
< 21|m1 Pull1 + 2
Σ
[-T((-Ish,a,
T- 8h, an)ll + 2B
Σ
[-A (n)) On(. (n(-1s))1 18))
ME
hE(M)
Proof. For the second result, we notice that by Lemma H.8, it holds that
Σ ( Ph, 8n) p (ch, Ph; sn)l
Σ I (ca) (cx))
Ch
E +
|P
Ps ph cn) - PT (sn. [c)]
ShcPh
Notice the fact that p. (sh,Ph] en), & (sh, (sn,, Ph (n) do not depend on æ due to Assumption C.8, we conclude that
E [19 (cc)) - -P (. <
- (ch, Ph. Sn) -P (ch, sn) Σ p. (ch) Pr. (cN)
ca.Phes
(H.10)
FH
where the last step again comes from the fact that after marginalization, the total variation distance will not increase. By combining with Lemma H.9, we proved the second result.
Lemma H.11. For any reward function f1:H of P with ra S x A + R for any h E [H). it holds that
7 (T)
(((T) S max o (T).
max TETT
=ells
Proof. Denote n¹ Ils be the optimal policy obtained by running value iteration only on the state space S for P. Now we are ready to prove the following argument for any E I[ inductively:
Q P (Sp, ap) 2 Q (S Ol.h, a..))
46

---

Asymmetric optimistic NPG
Expert policy distillation
Asymmetric Q-learning
Vanilla AAC
Case
3.32 0.66
3.33 0.62
3.04 0.58
3.25 + 0.65
Deterministic POMDP
Case 2
7.1 + 0.48
7.26 0.68
6.15 + 0.85
6.41 + 0.91
Case
3.04 0.23
3.25 0.333
3.09 0.39
3.1 = 0.38
Case 4
6.51 + 0.6
6.54
6.16 + 0.48
5.87 + 0.58
Case
3.31 + 0.46
3.37
3.03 0.4
3.08 + 0.43
Case 2
6.36 + 0.52
6.67
5.74 0.43
5.7 = 0.46
Block MDP
Case 3
3.2 + 0.26
3.37 = 0.22
3.14 0.31
2.97 = 0.32
Case 4
6.01 + 0.32
6.44 0.36
5.58 0.32
5.33 = 0.19
Table 2: Rewards of different approaches for POMDPs under the deterministic filter condition.
It is easy to see the argument holds for h = H. Fix state-action pair (sh, an) E S x A and trajectory ($1:h-1. P1:h. 01:h-1), we note that:
Q% (sh. an) = (sh, on) + ESHTI~Ta( (s..aa)
max (sh+1, (sh+1,a=+1) 0ht1
2 (sh, on) + E Eatin an) oh+1 (1
max 2 ($1.h-1: 014+1 a1-b-1) ah+1
(s an) + Bent ony oh+1 (:s1000=
E B (aw.m 11t. O14h (1411
Q (s1 01sh, a1.)),
where the first inequality comes from the inductive hypothesis.
I Missing Details in Section 6
POMDP under deterministic filter condition. We first evaluate algorithms POMDPs with certain structures, i.e., the deterministic conditions. In particular, we generate POMDPs. where either deterministic the emission ensures decodability. We test three of our approaches, expert policy distillation, asymmetric optimistic natural policy gradient. We summarize our results in Table 2, where the four cases corresponds POMDPs with
the transition dynamics
A A= 5). (S A = 2, O = 3, H = 10). (S = 3, A = 2, 4, H = 33 and we can see that our approach based on expert distillation outperforms all the other methods, which is consistent with the fact that such methods have exploited the special structures of the POMDPs achieving both polynomial sample and computational complexity.
3,
General POMDPs. Here also evaluate methods for general randomly generated POMDPs
without any structures. Hence, we compare the baselines with asymmetric optimistic natural policy gradient and asymmetric optimistic value iteration (i.c., the single-agent version of Algorithm 5). In Figure 2, we show the performance different algorithms POMDPs of different size, where the four cases corresponds
POMDPs with (S - 0 2, H 5), (S = 0 = 2, H 10), (S 0 = 3, A 2, H = 10), (S = A = 3, = 2, H = 10), and our approaches achieves the highest rewards with small number of episodes.
Implementation details. For each problem setting, we generated 20 POMDPs randomly and report the average performance and its standard deviation for each algorithms. For our algorithms based on privileged value learning methods, we find that using a finite memory of 3 already provides strong performance. For our algorithms based privileged policy learning, instantiate the MDP learning algorithm with the fully observable optimistic natural policy gradient algorithm [74]. Meanwhile, for both the decoder learning and belief learning, we find that just utilizing all the historic trajectories gives reasonable performance without additional samples. For baselines, the hyperparameters for Q-value update and step size for the policy update are tuned by grid search, where a controls the update of temporal difference learning (recall the update rule of temporal difference learning as
47

---

Q + aQuage). For asymmetric Q-learning, we use e-greedy exploration, where we use the seminal decreasing rate é, = #+1 M+1 Finally, all simulations are conducted on personal laptop with Apple MI CPU and 16 GB memory.
Empirical insights and interpretation of the experimental results. To understand intuitively why our approach outperforms those baseline algorithms, we notice the key difference in the value and policy update style between our approaches and vanilla asymmetric actor critic and asymmetric Q-learning. The baselines often roll-out the polices, collect trajectories, and only update the value and the policies on the states/history the trajectories have visited. Therefore, ideally, learn good policy for baselines, the number of trajectories to collect is at least as large as the history size, which indeed exponential in the horizon H. This is known as curse of history for partially observable RL. In contrast, in our algorithms. we explicitly estimate the empirical transition and emissions. which is indeed of polynomial size. Thus, the sample complexity avoids suffering from the potential exponential dependency of horizon or the length of the finite memory. Finally, we acknowledge that the baselines are developed to handle complex, high-dimensional deep RI. problems, while scaling our methods to deep RL benchmarks requires non-trivial engineering efforts.
J Missing Details in Section 7
Proof of Proposition 7.1:
Given the condition of Definition 3.2, the function on that satisfies the condition of Proposition 7.1 can be constructed recursively as follows for any reachable Th
(TT)T) == (h(cn- 1 (Th-1) 1), 1 on),
and Φi (01) (01). Therefore, we can prove by induction that by belief update rule
pp pp(sn Th) = Un (1) a 0h) (
where the last step is by the definition of our di- Therefore, v/n, (7 on)
we have 1PP(sH
For the other direction, it is similar to the proof of Lemma C.I in [19] for m--tep decodable POMDP. Here we prove it for completeness. For any reachable trajectory Th Th. it holds by the belief update and induction that
pp (SK TH)) (bdh -n-1, 0n) = P 8h - (T-- a). -
Meanwhile, since we know (T) is a one-hot vector, we can construct vh such that vin (8h-1, (h) is the unique 8h that makes IP (sh. 7h) >> 0 with 8h-1 = Φh-1 (7k-1). If this procedure does not complete the defimition of s for some (sh-1, a4-1, it implies that either 8h-1 is not reachable or Oß is not reachable given 8h-1. i.e., 1p' (on 9h-1,0h-11 0 for any d/- E A, thus recovering the conditions of Definition 3.2.
Proof of Proposition 7.3:
Note that for any given problem instance of a POMDP, we can construct a POSG by adding dummy agent that has the observation being the exact state each time step, and has only one dummy action that does not affect the transition or reward. Therefore, even the local private information Ps. of
the dummy agent can decode the underlying state, and hence Ph) reveals the underlying state. Therefore, the corresponding POSG with the dummy agent satisfies the condition in Proposition 7.3. Meanwhile, it is direct to see that any CCE of the POSG an optimal policy for the original POMDP Now by the
of planning for POMDPs [66], we proved our proposition.
Theorem J.1. Suppose the POSG G satisfies Definition 7.2, For any π E lls and (potentially stochastic) decoding functions ğ ( Jiem,,ie with Ginh C.. PL, A(S) for each i [n], [H], holds that
NE/CCE-gap(7*) - NE/CCE-gap(T) S 2n!1² max max max P - (sn + gin Pi.)) (E jE(M
CE-gap(=9) CE-gap(T) S 2nH2 max max max p( (sh $ Fi.n (ch, Pf.)))
iE(n] m:eM; (E(M)
48

---

where T is the distilled policy of т through the decoding functions g. where at step h, each agent i firstly individually decodes the state by sampling Sn ~ Is,al- Ch. Pt.)), and then acts according to the expert Ti where in the following discussions, we slightly abuse the notation and regard Di.k (ce. Pt,s) as a random variable following the distribution of gi.n Cb. Pt,h). In other words, the decoding process does not need correlations among the agents.
Proof. For notational simplicity, we write vs instead of of when the underlying model is clear from the context. Firstly, we consider any deterministic decoding function 3 = { hie (n), with Hih : Cn PLAA + S for each i [n], h E [M] and note the following that for any i [n].
(x( - Us (TT ) E R|
E + [
[RI E [n]] / E [M] : on = S (Ch, PYAN)] - E 1 [Ri E [n], h E [M] : 8h =
+ - R1 jjj E [m]. h E [M] 8h + 3.. (2.h) R R1 Fij E [<)] h E [H] : sh #
= [ E [n], / E [H] on # و (ch. Pj. )]]
E
[ E E [n), / E [M] on # (s.((cn
where we define R = Σn ( on), and the last step is by the definition of π
(x(T)) ( & HP (3j E [m]. h E [H] Sn. +
Therefore, by noticing the fact that ف is equivalent to a mixture of deterministic decoding functions, where at the beginning of each episode, one can firstly independently sample the outcome for each P..t)) for j E [n]] and h E
[H]. we conclude that
(T(TT (x(7)) = (T( - E G (1((TT) S HE
G- Paj pa. [jj E [n], E [M] 8h + (Ch.P.h.)..)))
-P (Fj E [n]. h E [H] 8h # (chh(c.Pp.)))
Now we prove our result for NE and CCE first. Due to similar arguments for evaluating for any u I1 U IIs, it holds that
(1 x π - - ve (u x +-1) RG ue
[3j E [<] \ (1), E (M] : 8h +
Hipu HIpux 9 (3j E [M] (1), E [H] sh + T.
We notice the following fact that pn puax=... [Vjj E [n] (i h E [M] : 5h = 33.n(cn.PPN)) = Σ
pa, xa (FH),
where we define (0) = (TH E THIViE E |n] \ (i), E [M] sh = psnh(enpphh). By
definition of x 3. it holds that
VTH E (H()) par 0 = , (TH))
Therefore, we have
pu & (Vj E [x] (0) [M] : 8h = = (V) E [M] \ : sh = -g.
= ppe
Correspondingly, it holds that
p44*= . [n] \ (i E [M] sn & @,,n(ch, = 1px 1p= 2 (3j E |n| \ (1), [M]: sn # #aan(cn,
which implies that
eu x x ve(u x T < 1I24x=2,.. (3j [n] \ (i], h E [M] : St. + ش (c),,))
HPpu (3j E [m] (1), h E [ Sh. + 2 (Cn.P(()))))))
49

---

Again by the fact that ② is equivalent to mixture of deterministic decoding functions, it holds
(u, x m (u x --) = B Bang lils _ ( x T--)
< HB G pu pupx (3) E [m] \ (i), h E [H] + (ck.(c))))))))
HP x & (3j [m] (i) h [M] sh # Ten(cnn
(J.1)
Now we are ready to evaluate the NE/CCE-gap policy T as follows: NE/CCE-gaa(TT) NE/CCE-gap(TT
max vi(u; x n HP [n]. E [H] sn + @nn(cn. u, Elll
N;ETT S <ax max (u; 2,2 .E[n]
Now we notice that & or(u; π-1) = max (z(u; x T--) since Is, S ΠI by the deterministic filter condition Definition 3.2 and T-; is a Markov policy. Therefore, we conclude that
NE/CCCF-gap(T") - NE/CCF-gap(T)
HP ( [H] on + 9g.(ce. max (u x 22 max vilus X T-1) INElT;
max 1E[M]
wIlll
(u x vi x < HPT E [H] on + (.a(en.P..)))
max kE[M]
max INETT;
max HP (3j E [n]] (1), / E [H] Sn + (((((n) + HT [r]. [M] sn # Tsn
max
<E[n]
W.ETT;
S 2H
max EE!
pa X (sn # (c.h(chPP..il
JE(m)
2nH2
p (sh Fz.h Pj.n),
where the second last step is by union bound, thus proving our result for NE and CCE.
For CE, consider any strategy modification m, M, U.ss, it holds that
CF-gap(=99 CF-gap(T)
< max va((m; o n. 2.) m,EM.
max vi((m o TI) T-1) HP m. M.EMs,.
M (Jje [n], E [H] : sh # Tinn Pin)).
Now notice that m v; (mm, o T,) E T-)) max,, E.M.,, (m(mm T.) T since
Ms.; S .M by the deterministic filter condition Definition 7.2 and Lemma J.2. Therefore, we conclude that
CE-gap(=9) CE-gap(TT
$ max max c:((m; T *E|n] m.EM;
max um(mm o.) + HP H (3j [m], [H] sh. # (ch. m.EM;
< max max *E[n] m.EM; (1:((m; n
2 _ v ((m; o n.) m + HP (3j E (m), [H] : on # (an((c)Pj.)))
< max max se|n] m.EM;
Hp J0 (3j [n] {i}, h E [M] sh. + Gan(ch.
Sh. Ijnn (ch: Pj.n)) HP E in], H
<2H
max telp),m,EM.
p(m:on;) , (sn + G..h Ps.))
JEIN h
< 2nHl
plm (sh + (en(Cn, (C..P.))))
where the third step is due to the same reason as Equation (I.1).
Lemma J.2. For any स E Iss, holds for any reward function and E [m]] that
max vi(mm; Tz) x-)) m,EMssi m.EM?
va((m; m:) @ m-).
50

---

Proof. Denote m E E.M, Ci((m o mT) @ π--) and my E argmaxm;eMs. Di((m;
Ti) n--)- Now shall prove that V mmmooooo :.C (81:t) 01ch; 01
y(m oo ::0 (s%)
Vin
inductively for each h [H]. Note that holds for h + 1. Now we consider the following
(mn Jo 4
(81:k) O1cho (1:A-1)
E
Ta( shame an-oTh (8a)
ra(Sh, mj,n ($1 Ord a13-11 Oc.h), a.ih))
hou 0441~0o41(- (h+1)
(m]oo)]00n11 1,41
($1:b111 014+1 01:h-11 m o1d 014-1, an))
E
rh(s), ( O 01.--1 a1,),,
V (mmoom) v.9 (9nt1)
(11n: marale v.l 0141=0841(111 sh+1)
E
(- sn))
The (h(s m, a.,)), + 6h+11
(0 (( (sn+1)
* Tapi ( (aise,,aa
014100421 (8841)
(0 (sh),
1,
where the second inequality follows from the inductive hypothesis and the third step is by the definition of m argmaxm;e v;((m; TT)) @ -)).
Lemma J.3. Given an approximate POSG & that satisfies Assumption C.8 with approximate tran- sitions and emissions being nTn, Oneeet we define the approximate decoding function ğ to be
(sn (he pp (sh Ch. Pt,h)
for each h [H]. Sh. E cn a Ch. Pib Pi Then it holds that for any Ils,
puxx=-10 (sh + Ii. (ch.
max
JEL
E& uy xx-
Σ hE(H-1
s4, ax) -- sh ax)lla +
Σ
|0x(-1sh)) On (- (s1s))
max
ullss..
hE[m]
and
plm:on:)@ (s # (g.,(ch.
max
max m.M
EG (m;
Σ
||T sh, and sh. an)112
Σ
[00((1ss) - Dn((1s))lla
hE(H]
Proof. Note for any i E [n], us [I E [n], h E [H], it holds
p (sn + (chn(CrP55.)) =
E
Σ
ps (su Ch. P..))) - -P8 cb. prn)
M4 x
8b
51

---

due to the condition in Definition 7.2. Meanwhile,
1 B9
(sh Pj.a.) - pp (sh, ch., Pj.A)]
2
-
eh
Σ
puix (h,
Ch. PjAn) puxx= (sh, Ch. Pp.k))
*/ CA PP.A
p x0 (Th) -Pu.xx_00 (Tx)
x (Tn)
pu pu.xx_.. (TH))
TH
16 My x
[[T([1] Sh, an)) - -TA(t (-1 Sh, an)ll + NE[H-1]
Σ
Σ 110((118) (A) O- (-1 9h)ll1 NE(M
where the first inequality is by the first inequality in Lemma J.7, the second and third inequalities due to the fact that TV distance does not increase after marginalization, and the last inequality is by Lemma H.9. Since nh is fixed and fully-observable Markov policy, by Lemma H.11, we have
Bg
Σ h
[Ta(- sh, an) - [- St: anlll1 +
Σ heim
110x(-1 |
< max
EG u, Xm-
Σ
[Tx(- sn, an) - T. sh,
an)ll +
Σ
[10(1110
(-1
w.Ells.
AE[
he(m
thus proving the first result of our lemma.
For second result of our lemma, it can be proved similarly that for any E [n]. my EM..j E [m]. h E [H].
p Jom
(sh + Fe,h
< EG
Σ
[T 8h. an) - T- 1- an)
Σ 10 (%) -.,(-1s)lla
(m
H-
hE[H]
By Lemma J.2, we proved the second result.
Theorem J.4. Fix any e, ô E 10,1) and Iss. Algorithm 5 can learn decoding function ğ such that with probability 8-8
max JEl
XM (sh + (an(C..Pj.))) S,
with total sample complexity Ö Ō( S AH + S*A2#") and computational complexity PSLY(S, A, H,O, !), log
Proof. With the help of Lemma J.3, it suffices to prove
E9 PuXXT-
Σ
[Ta( sh, (h, an) Tn(- Fr((l1sp sn, an)lli + 1[0x (x))))))))))))))))))<)))))))(( (n)-ls Se.
max Ell.s.
KE[H]
The following proof procedure follows similarly to that of Theorem H.5. For each h [M] and 8h. E S, we define
Ph = max veln),u.elss
Fix θ1 > O, we define U(h,e) e1) = {sh S (pn(ss)) 2 e1). By [37], one can learn a pol-
icy (V, sn) hee()) with sample complexity Ö SAA, such that max d% (h. (sx)) 2
52

---

2 for each Sh E c1) with probability 1 - 81. Now we assume this event holds for any E [M] and 8h e1). For each 8h E and on E A, we have executed each policy (V.(h, 81) x T-itve(n) for the first h - steps followed by an action an E A for N episodes and denote the total number of episodes that Sp, and a₃ are visited as (sh. (4), and (8)8))=>a (a(sn, a). Then, with probability 1 -Ne//8 we have (s,; (h)) (Pp().) by Chernoff bound. Now conditioned on this event, we are ready to evaluate the following for any i [n]] and U Ils,
E9 _ [TT( sh, an) Tn S/. ab)ll1
Σ Shid
( x T=<)aaaa 5h) ( sh; on) - -- (
sh an)
s (log(1/82)
Sc. +
Σ
de -4x== ((s)]us x Ilaaaan sun)
Nn. (sh. an)
25 25log(1/52) (pn
2 Sc1 +
Σ
h
duix= (Mx x -lalaaan V
22 St1 + Σ Eval 1u [
25 log (1)(
(
N
2 2log(1/02)
2 Sc1 + +SV
N
where the second step is by Lemma J.8, and the last step is by Cauchy-Schwarz Similarly,
EC u, xx_, 10 8b)
On(: (6)11
Σ do x (s))
(m)) (-((1ss)))
O log(1102) (n(sn)
2 +
20 log(1/82)
2 S61
Σ
dp d= V(SN)
Npn (sn)
Sy (
20 log (1/2))
< S2-Se1 +
Σ
dn (sn)
N
(
25O 1og(1/02)
2 S61 S61 +
where the second step is by Lemma J.9, and the last step is by Cauchy-Schwarz inequality. Therefore, by union bound, all high probability events hold with probability
1 - SHHn, - SHAE -Na/8 - ASAH1
Therefore, we can choose N = 5 5500 and = O leading to the total sample complexity
S AHH*
))
θ St AHO n AH
+ S
SHA nN +
2²
which completes the proof.
Note that although our Algorithm and Theorem J.4 are stated for NE/CCE, it can also handle CE with simple modifications, where the key observation is that the strategy modification m; E Ms. can also be regarded as Markov policy in an extended MDP marginalized by T as defined below.
Definition J.5. Fix æ E IIs. We define Mextended (T). to be an MDP for agent where for each h E [H], the state is a..)) the action is some mod- ified action d' the transition is defined Th Texiended (5h+1, 01 Shy ai, d(,))
53

---

E, s.mmmafl ss,as,)) (T Sh: di,. a. (01,4+1 |1++]]] where we slightly abuse the no- tation of (a-aii 8h; (2,6)) and Th (as.h (8h) by defining them as the posterior and marginal distributions induced by the joint distribution TA (ap, 1h). Similarly, the reward is given by pestended (5h, di,b; (,,n) ;= : Ba- (- (s.eca)(ra(ss , a-i,n]]. h
With the help of such an extended MDP, we can develop Algorithm 6, which is a CE version of Algorithm 5 with the following guarantees.
Theorem J.6. Fix any e, E (0, 1) and π E Ils. Algorithm can learn a decoding function ğ such that
max Me.jl... p(m,ox,)ox-..0/ (sh + (ch, PAA)) S.
with total sample complexity Ö HO s**"#") and computational complexity A,,
Proof. Due the construction of (āslendel(T)) (TT), the proof Theorem J.4 readily applies, where the only difference is that the state space of Mpxtended (IT) is now SA,, larger than that of MM by factor of A, thus proving our theorem.
We next introduce and prove several supporting lemmas used before.
Lemma J.7. Suppose we can sample from a joint distribution P E A(X x y) for some finite X, ) i..d. Then can learn upproximate distribution E A(X x 3) with sample complexity such that
θ tlo 1/6)
E Σ
Q(yl)))
Σ xEX.EE
(P(z,)) Q(x, Se
yEY
with probability 5. 5.
Proof. Note the following holds
PP v) ((=,
Σ
IP 3) -()))))))) - Q(z,v)
BEN
Σ TEX
P(a,y) -(=))((11)) Q(x.0))
Therefore, we have
Egzrr
Σ p(yla) -((l))))
Q(1l))s
Σ
PP(..)) Q(r,0))+++
Σ
Q (p)l)))P(>) )(x))
yEy
wEX
VEX.EEy
2 Σ
P(x.y)-(((z,) Q(x,v))
(J.2)
ZEN.EEY
By the sample complexity of learning discrete distributions [13], we can learn Q such that
EzEx.yEy P( y) Q(x y)1 < c in sample complexity (x1]))+ 1/6 with probability 1-6 Thus, we proved our lemma.
2²
Lemma J.8 (Concentration on transition). Fix d >> 0 and dataset ( kEINI sampled from P (or G in the multi-agent setting) under policy I[gn We define for each h [H), (sh, an, s1+1) E
SxAx5
(s an)
Σ 1|s sh. ai = ax), (a an, 8h+1) = Σ
1 1ss = sh, at = an sha1 = sn]-
RE[N]
REIN]
Then, with probability at least holds that for any E [K], h E [H] E E. E A:
S Slog(SAHK/D
[ 8h, un) -xa (h, sh, ax)ll a
Na ax); 1}
for some absolute constant C > 0. where we define T (8h+1 18h an) Na (sn. mox(Natt
54

---

Proof. This is done by firstly bounding [- sh, an) [- (7.-1s%. sh. an an)ll2 for specific k, h, S/, an according to [13], and then taking union bound for all [K]. h E [H], Sh. S, ag A.
Lemma J.9 (Concentration on emission). Fix 5 > 0 and dataset ( kEIND sampled from P (or 9 in the multi- multi-agent agent setting) under some policy π E Il We define for each h E [H] (sh, on) E O
(sh, on)
1 1ss 8/. of onl kE(N]
Σ
(N(sh)
Σ kEIN]
1ss sn).
Then, with probability at least
holds that
Olog(S (SHK/))
||O(-1 (k) - (-(-1s)))i s Cop
max (sb).,
for some absolute constant C₂ > O, where we define (n,on s1)) = max (Nx
(s..on)))))
Proof. This is done by firstly bounding (-) 9p.) - (-1 (1-ss))l for specific k. h, sn according to [13], and then taking union bound for all [K]. h [H], 8h E S.
Now we switch to proving the guarantees for Algorithm 7.
Lemma J.10. Fix 8 O. With probability 1 - 6, it holds that for any & E [K], E [H), 8h E S:
Olog(SHKA/A
Σ
po (on+1 18h, 10041110,000 an) Jhk (on+1 (p. S C33
(sh, an)
where }4 is defined in Algorithm 7.
Proof. This is done by firstly bounding Eon pp (on+1 88, an) - Jh (on+1 8h. an) for specific
k, h, Sh, ag according to [13], and then taking union bound for all k E [K], h E [H], sh E S, ap e A.
From now on, we shall use the bonus
by (8h, an) min { (a(H-
O (og(SAHH/0) max(NV ax), 2(H h)
(J.3)
in Algorithm 7, for some absolute constant C3 > 0.
Before presenting our technical analysis, we define the following notation for the ease of presentation.
We define the following approximate value functions for any policy E 11 in backward way for
h [H] when given some approximate belief in the form of {P, C. A(Pn S)heem as discussed in Section 7.2:
n (cx) F.
%0. Bale (8x)) (a
Ea 1- (111)1)))))))) (1- (11 (TsA(s)) + V.T.G ((tti(eatt)]
Q (ck, 76) E
20.PA Enli (8x) E Plasa (pp.a)se
Eeg Mann (...,, 1 (r)((s)) + V (
for each ()) E [n] x Cn and 7h E Th. where we define V (cH+1) 0.
Intuitively, this definition of D.T.G (cx) mimics the Bellman equation the ground -truh value function v (cx) by replacing the ground-truth belief TP (Sk, Ph ca) by Pa(sn, Ph (h). Next, point the following quantitative bound when using V (cx) to approximate Vith (cx).
55

---

Lemma J.11. For any n', T E II, holds that
B VTL (cn)
1 (c+)) S (H (H-h) - + Ehelies
Cheliefs
in
1.h
where
max max By (s Pal: 18h) he(M] well
Cbllief
Proof. It follows directly by combining Lemma 4 and Lemma of [51].
Meanwhile, note that although in Algorithm 7, the value functions maintain have input Tn instead of en for computational efficiency, we extend the definitions of those value functions to also accept on imputs as follows (with slight abuse of notation):
hhishik (ch: 7h) = @ i.
(Cn, nh), g (ch: 7hh) = 2 (n):
Qhinh (Ca Ph. Sh, an) == hiihh. (Chs Ph: Sh, an);
highh (cn) = h-ggh (Ch)
Q.on (ch. Phi St: an) = o (Ch: Ph: Sh, an),
Vloowkk (cn) = Volow, (or), 1,
where we recall that Cj. Compress, (c.).
Lemma J.12 (Optimism 1 for NE/CCE). With probability 1 8, for any k E [K], for Algorithm 7, holds that for any i E [n]. x; E EII, [H]
Qhigh (Op: (h)
Q x (chs
(highh((Cr)) (Gr)
high
p (cn),
i.
in
where we recall that č₁ Compress, (ck).
Proof. We will prove by backward induction. Obviously, holds for = + 1. Now we assume the lemma holds for h + 1, then by definition
Q mh
Ba BaC 18 Blagnnn Dpsa [O (ch, Ph, Sho an)
F. mn Pack ax) E Bleaa a 6 l p min frinrran(nn and + b an)
Wiich (ca+2)) H h +
+ E Bo t lanoww V
sn Paler an) Pregammau (p5,))))e(()
Dajxa" ir (catt)) , H h }1
min (1s( on) by " an)
B B ((1sma)))
where the last step is by inductive hypothesis. Now note that for any (sh, Ph, an), we have
x (C (catt)
b/ 1-11 (sh,, ar) + E Bot I3 ((1en)aw)
Datixii.0 (catt)
1/ (8-, un) (H- (8, (pp))ppppp (p8(11 (0,saan)ll + (ot. (nts))
(-1eats) pprixti (cnti))
where we notice pp (on+1 8h, an) = Esner O1+1 (0++1 (h41)TTx 8h, an) for the first in- equality, and the second inequality comes from the construction of our bonus 6 (sk ax) in Equation (J.3) and Lemma 1.10. Mcanwhile, by the definition of value functions, it holds that
xpaxx=1.0 1 (cn+1) S ---h Therefore, we have ih+1
E an (n+1))
an) min rinlsn, and + U/ (sh, an) + E Pon41~3-116 -
Vi
x (cati))
H - 1
7,--1 (cntt)
2 )+ + Ea
56

---

Now we conclude
high. (ch: 7h)
2 E Pa~ Ph (o Plogammaat maNlee Beas N ( shaas)o aan (sh.
+ p (ch+1)
= - (on, mh)-
By definition, we have Qih (ch, 7h)
Qhichh
high. (ōh, mh) thus proving Qhigh (Ōns (k) 2
0 (ch, m.). Now for the value function, note that
high (cn) = Ewn Quih (ch, ATH (-1wj, Cae ()em)m)
2 Eug Eus, high (cn, T',a whi Ch )) (Ty Wy,nr Cn. (EEM)11))
00 2 Eug Bus, 0 "mix=";00 (ch, o (w Ch. ) ( Wggn One Dare:annn.--D/eeaa... DEET))))
x (cr),
where the first step is by the property of Bayesian CCE, and the second step is by Q (ch, 7h) 2
Q xx :.C (ch, 7h) for any 7h E T as proved above. Again by definition, we proved Vrligh, (Ch) =
3,4
Viigh (ch) i.h
2 Vin
(c.).
Lemma J.13 (Optimism 1 for CE). With probability - 8, for any & E [K], for Algorithm 7, it holds that for any E [n], m E M, [H]
(m,,ott)ont..
Qih
(Gn, 7h) 2 QiA
(ch+ 7hh
hhigh (Cr.) 1,
(m,oo)))x (ct))
Proof. We will prove by backward induction. Obviously, holds for H 1. Now we assume the lemma holds for h + 1. Now we notice that by definition,
ghich (en, Tb)= E. p. B. (6,) Eo a Go (p..))seM To (cn, Phu shs su.an)
su,
E P. BB (aaaa 2 Paa)lyen min ( an) + (sm, an)
E
B (.laman))
thish (cn+1) 1I h h+ (iti
2 E.n.DA (v FF Paagaayzat pywn)sett))
min (an an) bh "(n,, an) on - (.1am
(ommort)ent 1h+1
(cm11) H - -H-h-1p
where the last step is by inductive hypothesis. Now note that for any 8h, Phy ahe we have
b (st, an) o4 22 (b))))
(an-i) P& s. an)li
by (sn, and (H -h11sh 1 (.
pomionttont (cati)
+ EERLII-TTC on)on -ontill sA41
p (cut1) sh+1
2 B (1 1944))
where we notice p (on+1 8h, an) = Emao Op+1(0n+1 88+1 T (8N+1 8h. an) for the first in- equality, and the second inequality comes from the construction of our bonus 1/- (sh, az) in Equation (J.3) and Lemma J.10. Meanwhile, by the definition of value functions, it holds that
57

---

1 (0m hh+1
(cn+a) S H h. Therefore, we have
Esg (oggiTaalle (1 11 sn
min(ran(sh: an) br as)
io (cnt1)
E -r
' (vr (a))
ih+1
H h 13
pommootronti (outi)) sb+t
2 an) + B1 (oan)
Now we conclude
Niigh (ca. 7hh)
B na Pule 12x) Beaga Ipg.aaa
BentinTal ho (-lene)) Peu and
ahtt
= (m 00 (cr, yh).
By definition, we have highh. (Ch. 7hh) = =Qha (Cn, nh) thus proving Q
high.
high (hh 2
(m, ) 1 (Ch, 7h). Now for the value function, note that
hiigh (cn) (cn) = Ewn Qihh high. Q (ch, (Tghh (-1 Wyj. Che ))se(m)
z Eu hhish (cn, f (- (wj Tne ))ee))))))) (min o wi Wise Cn.
2 Ewn Qimion (m 1 (cn, (Tf. |wgn Toe )ESEAM (mi, at ( wi,hs Tne ))
(miox (xx jox" (cn),
where the first step is by the property of Bayesian CE, and the second step is by Qhighh (cn. 7h) 2
(ompoo) )am" (ch, (ch, Ih) for any 7h E Ги- Again by definition, we proved thigg (2x) pofm:ont)on (ck)- i,
high. (cx) 2 4
Lemma J.14 (Pessimism). With probability 1 - 5, for any [K], for Algorithm 7, it holds that for any E [n), h E [H]
Qlom- 7h) S Q (cn,
low i
(Gn)
V (cx).
Proof. We prove by backward induction on h. Obviously, the lemma holds for h = H + 1. Now we assume the lemma holds for h 1. Similar the proof of the previous lemma, we note by inductive hypothesis
Qo (n) S sn.Pa Pal-.- (2.) (as, 7aa (Ps.)))snn))
max { (r.n an) 1 (sh, an)
sn s I (6 (anaa g
E
where for any Sh. Ph. ane have
b/ (sp: ap) + B Jh (nla)))))
[ ( (cn=1)
< bb k- (sh, an) + (H - h)]]][ (-1sh, p.(.)
sh. an)ll
o (
+ Esg ~~ (: sn. 12 11)1))))
pa ihti (cx+t))
< Ben (e..eeee...n))).. on.an)on 1~0nn 1f- 1 11)
58

---

where the last step again comes from the construction of our bonus in Equation (J.3) and Lemma J.10. Therefore, we conclude
Q (ch. 7h) S E Pa.pa-Patt 1&a) ((aga (pswweee D
E Bn ( (ol (ran(sn, and + 1,411 ((44((941))))
& (ch, 7n). in
Similarly, for value function, holds that
Volow (ct)) Ewn (o (ch, a wann Cno (Emm))
i.h
Stuw Ti (ch, (mt, Wegh Ch ))s(()) = D .9 (cn):
thus proving our lemma.
Theorem J.15 (NE/CCE version). With probability 1 1-6, Algorithm enjoys the regret guarantee of
Σ kE[K]
S O(KII Ebelief 112 11 SPOKK (o5(AAK1))) IT' /loSA/Olog(SAHKK)) x (cf) Vol (cf) (
max T/Em,
111
Correspondingly, this implies that one can Iearn an (e + Ebelet if & is zero-sum and (c -CCe is general-sum with sample complexity O H H''SO (og(SAHO//)) and computation complexity PSLY(S, max [Cx] maxne(e] [Pa], H, 1 log 4
Proof. Note for any given E [n]] and T/ E HI, by Lemma J.12 and Lemma J.14, holds
V xxm= 2.f (et) Vi.h . (ch) S V-high. (ch) V-low,k (c%)
max *(Ell, i.
Therefore, it suffices to bound highh (ctt
low, (cft):
high, (ch) (ch))
Bunn Bunn ( Coaeiii (ch, ( , oh. beta) -. (Tfaat1 (1, (,,h ))em))
= =B ~ Pacts 122,
en - 1 Bunn (oiiih Erff 1, Memm))
Ol (- zh. )) + (II h+ 1)en(ch)
& B, Ba - BBu Cagaa matt & Iusa (t
Quicr ( (ch: sh, an)
Qua (ch ph. Shy an) + (H - 11 1)cn(ch)
I Zhen + Qhigh (ch ph, sh alt " (ch. ph. the an) + (H 4 ()ca(ch)
S 7l., + Bonting 20.1 (hat)
Whishi (cx+1) Vittt (ca+1)
(H - + 1)ea (cfi) 2b (sh,
S 7 + +B BBotIII (1)
Vhiihh (ch+1) - ihttl (cn+1) + (H h (c + al)
< Z + zi., + 1,t (c - V
(HI - h + (e)x ( ah))
where we define the two Martingale difference sequences as follows
Quic (ch: Ph: 8h, an) Joo (ch Phi 8h, an)
ZkA E Pa. ot Besn Th Easan (t.. Ql (cis ph shi att)
Quich (ct vā. shi at)
2i,, = E. Bo (11
high (c++1)
low, iit1 (co+1)
(
hight i,h41 (ch+z) -v-lvo v/
and the error of the belief is defined as
(c = 1 pp - 11
59

---

Since |2 - H, A H, and (x - 2, by Azuma- Huema-Hoeffding bound, we conclude with
probability 36, the following holds
Σ k, OOH k.h
HK lolgog 3)
E S S01H HK log 1 z).
Σ (n(ch)
RG
Σ (n(ow)
+ +0 + HKl 1 S K E + Oly HK log z).
x
k,A
Meanwhile, by the pigeonhole principle, holds that
Σ
(s a W H H/Olog(SAHK// - h/
> k.h v max N (s x
k,h
0 H (o +HSAA)))))))))
Now by Lemma J.12 and Lemma J.14 and putting everything together, we conclude
Σ max m
max
D (aljxa--ae 1.0 (ct) (ct)
HHH Ehenae SogO +
(SH
kE[K]
Now by Lemma J.1I, we proved the regret guarantees as follows
Σ max FEIM]
max FlEL,
vixxx... (ct)
V1
Vir (ct)
kE|
<
O(KH² Fbeliet + H² SSAKSAK log(K + S SA (0005(555)))///
For the PAC guarantees, since we define k* E arg minig[n],AE[K]]] high, (ct)
Vilan
(ct), we have
CCR-gap(#**) O(H2 (beeiir)
max iE(n)
htigh, (cf Vloow Rt" (ct
O Ebelier)
Eheetei)) +
high, (ct)
Viuu, (ct)
KE(K
H Finally, for two-player zero-sum games, we denote *** to be the marginalized policy of *²" Then have
OHH Ebelict H² SAO (SAKK///
(olg(SAAKK/))
NE-gap(7**) S CCE-gap(7***,
thus concluding our theorem.
Theorem J.16 (CE version). With probability
Algorithm enjoys the regret guarantee of
Σ
(mfoxt)) (cf) - V. (cf)
max m/EM,
1,1
FE[M]
kE[R]
S O(KH2 Pbetiet + H² (SAKK/
Correspondingly, this implies that one can with
learn (c +
H Heeteerer) -CE
O SAO (og(SAH//))
sample
complexity [C,,, max
and
computation
complexity
PSLY(S, maxkeeH
[Pa], H, log )
Proof. Then proof follows as that of Theorem J.15, where we only need to change the first step of the proof
paixn (1x)) (ct)
Vih
V
(c))
high Vish
(c
by Lemma J.13 and Lemma J.14, and the remaining steps exactly the same.
60

---

Lemma J.17 (Adapted from Theorem H.5), Algorithm 8 can learn the approximate POMDP with transition Ti. and emission 01.H such that for any policy 11smh and h E [M]
E
[ Sh, an.) -- sh, an)ll | (-1()))))) sh)) (n(s-1s)lll] s O(t),
using sample complexity Ö 5( A AH s* with probability 1 8.
Proof. Note that Algorithm 8 essentially treating the POSG Algorithm 4, where the only modifications we make in Algorithm 8 is that we take the controller set (see some examples of the controller set in Appendix C.3) into considerations when learning the models. Specifically, for the transition The what we estimate is only (sn+1 sh, AI,. instead of TA (8n+1 8h, ah), where To S [n] is the controller set. Therefore, the sample complexity of Algorithm will not be worse than that Algorithm 4.
centralized MDP and running
Proof of Theorem 7.7:
Note that the proof idea essentially resembles that of Theorem H.6, where we construct the model Glune for G in exactly the same way as constructing piuee for P. Therefore, by Corollary H.10, have
E ||| (,, (cx)
Tp (cwlll)
Σ Tи
p (T
P (T < 4HSc
Meanwhile, we can construct Grune and Glub using exactly the same way for pinune and Puub, where Guub is 722-obbervablle POSG.
Now, according to [51], for the examples in Appendix C.3, there exists compression function that maps ca to C₁ such that the size of the compressed common information is quasi-polynomial.
22l i.e., Cr < (AO) CT log 22 for some absolute constant and 62 E (0, 1), and the corresponding
approximate belief (Pr Cr A(S x Ph)]RE(H] satisfies that
E
|
(c.1cc))
Therefore, we can do the same augmentation for PA. states from S construct the approxi- mate belief PA as in the proof of Theorem H.6, and the remaining steps follow from the proof of Theorem H.6. This will lead to total of polynomial-timg time and polynomial-same complexities.
J.1 Background on Bayesian Games
The Bayesian game is generalization of normal-form games in partially observable settings. Specifi- cally, Bayesian game specified as (n,, (A:)vet)), (0))ee (ra)eem)) ps), where is the number of players, A; is the actor space, θ, is the type space, r, X ( x A.)) [0, 1] is the reward function, and yu is the prior distribution of the joint type. At the beginning of the game, a type 0 (0)) is drawn from the prior distribution μ E A()). Then each agent i gets its own type A, and takes the action a; We define a pure strategy of agent as s; ST; := {O; - Ai}. We define (s, s-;) to be the expected rewards for agent i, given the pure joint strategy (s,, 8-)).
By definition, Je(s; s-:) can be evaluated as
J((s;, s-)) = Ee-nr ri s..
Bayesian NE,
We define 7- E A(S,)) e-NE it satisfies that - Vi [m], ST.
E Bany [[(s)
Bayesian CCE. CCE if it satisfies
We say a distribution of joint strategies 7* E A(Xsejn)ST) to be a e-Bayesian
Benyy Js(s) 2 E J(( -e, Vi [n], E ST,
61

---

(Agent-form) Bayesian CE. We say distribution of joint strategies y E ((Xie(m) to be an e-agent-form Bayesian CE it satisfies
E. Exnry Ji(s) Egw Ji o S1, 8-2) -, Vi [n], E Mi:
where M; = (O: x A; + A:} is the space for strategy modification, where m, modifies as follows: given current type 0, and the recommended action 0,, the strategy modification changes the action to the another action m.)0,,a))
Note that Bayesian NE for zero-sum games, and (agent-form) Bayesian CE/CCE are all tractable solution concepts and can be computed with polynomial computational complexity, e.g., [29, 32, 23).
K Concluding Remarks and Limitations
In this paper, we aim to understand the provable benefits of privileged information for partially observable RL problems under two empirically successful paradigms, expert distillation [14, 64, 58] and asymmetric actor-critic [68, 7. 3], which represent privileged policy and privileged value learning, respectively, with an emphasis on studying both the computational and sample efficiencies of the algorithms. Our results (as summarized in Table 1) showed that privileged information does improve learning efficiency in series of known POMDP subclasses. One potential limitation of our work is that we only focused on the case with exact state information. It remains to explore whether such an assumption be further relaxed, e.g., when privileged state information may be biased, partially observable, or delayed, as usually happens in practice, and how our theoretical results may be affected. Meanwhile, an initial theoretical study, have been primarily focusing the tabular settings (except Appendix G), and would be interesting to extend the results to aunction-approximen settings handle massively large state, action, and observation spaces in practice.
62

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract indeed accurately summarizes out paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We clearly outline all of our assumption needed to derive our theoretical
results.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
63

---

Justification:
We prove every theorem or lemma in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, detailed are introduced in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
64

---

Answer:
[Yes]
Justification: We have included detailed introductions on the algorithms and environment to reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
httpc/guiides/oeeuuuniiie son for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CCodeSubmiis
httppp.ccppubiic///ecc / ssi asionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have reported the number of repeated experiments in Appendix I and error/variance bars in Figure 2 and Table 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
65

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provides all details Appendix I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp///////uis...//e/// http:///ewrrp.cppppceeeeeeeeee/e httppp/bccethhccccueeeeeee..eeee ctwuidhlelllllleeccccccc ines?
Answer: [Yes]
Justification: It does.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We address the societal impacts of our paper in Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
66

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: No external codes, data, models are used.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
67

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No such experiments were involved.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No such approval was needed.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
68


Reviews:
[{"review_id": "4aBgtp2RK0", "reviewer": "Reviewer_zFhw", "summary": "The submission addresses both statistically and computationally efficient reinforcement learning (RL) in Partially Observable Markov Decision Processes (POMDPs). The training phase has access to hidden states, while the goal is to learn the optimal policy at test time without such access. The authors propose a sample and computation-efficient algorithm under a deterministic filter stability condition.", "questions": "- Definition 3.2: Does $\\gamma$-observability or $\\gamma$-separated PSR imply Definition 3.2?\n\n- When (Golowich et al., 2022) already achieved quasi-computational complexity for planning, what does Theorem 5.3 improve upon it? How are the previous results in Section 4 related to this result?", "limitations": "There are no experiments on practical benchmarks other than synthetic small tabular POMDPs.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The proposed setting is interesting and has significant potential for practical applications.\n\n- The paper makes a commendable effort to cover various potential cases, enhancing the applicability of the problem settings.", "weaknesses": "**High-Level Critic**\n\n- More space should have been dedicated to elaborating on the real technical question that the paper aims to resolve. With access to hidden states, the statistical hardness is virtually eliminated, so the focus could have been purely on the computational aspect. The poly-sample result is not surprising, and with the $\\gamma$-observable assumption of (Golowich et al., 2022), quasi-polynomial complexity does not seem surprising either.\n\n- The privileged information combined with the deterministic filter condition is very strong. It is unclear why Definition 3.2 was needed given the strong assumption of privileged information.\n\n- The proposed algorithm does not seem as practical as advertised. Beyond the tabular setting, it is unclear how to implement it with general function classes, which are common in deep RL.\n\n\nThe current manuscript needs significant improvement in terms of writing, as detailed below.\n\n- At the beginning of the introduction, it is confusing whether the paper is about RL in POMDPs with more information or about multi-agent RL in POMDPs. The two components investigated in this paper are orthogonal to each other, and the main result seems to be more about the former.\n\n- The contribution of the paper needs to be more clearly articulated in the introduction. My understanding is that the paper is about (1) an algorithm that achieves both sample and computational efficiency with some new technical conditions, and (2) an extension of the proposed condition and algorithm to the multi-agent setting.\n\n- The sentence in Lines 64-65 is confusing. What do we aim to achieve with actor-critic?\n\n- Proposition 3.1 pertains to a specific algorithm, not to the general hardness of the problem.\n\n- Jargon such as privileged, expert distillation, or teacher-student teaching may not help much in motivating the setting of this paper, as these are not the applications the paper pursued. It would be simpler if the problem were motivated from a theoretical perspective, focusing on technical challenges in existing works and some relevance to practice in the context of actor-critic implementation.\nThe decoding function $g$ in Lemma 4.3 is unclear. Definition 3.2 defines a posterior probability given $(s_{h-1}, a_{h-1}, o_h)$, but Lemma 4.3 assumes the posterior is almost deterministic. This could have been clearly stated during the problem setup. This assumption is essentially similar to the block MDP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback, and noticed that there were several important misunderstandings we would like to clarify.\n\n## Response to Point 1 of high-level critic:\n**For sample complexity, we respectfully disagree with the claim that with privileged information, statistical hardness is virtually eliminated.** This was exactly the point and technical contributions of the references [36,26,67] that only addressed the “sample complexity” improvement. **The key challenge is that the “output” of the learning process is still a “partially observable” policy (with partial observations as input), not one from the MDPs**. \n\n**For computation complexity, our results are not a direct combination of the poly sample learning result + quasi-poly time planning result.** Having a quasi-poly-time planning algorithm with model knowledge certainly does **NOT** necessarily imply that learning without model knowledge can also be made in quasi-poly time. In fact, the **time/computational complexity** of **learning** without model knowledge and **purely planning** with model knowledge can have **strict gaps** sometimes even in simpler models. For example, recently [24] showed that learning in block MDPs (without privileged information) is computationally (strictly) harder than supervised learning problems, despite that planning with model knowledge in block MDP is computationally easy. **Intuitively, it is the sampling errors in the learning setting that can make the computation nature of the problem fundamentally harder.** This is in the same spirit as the famous problem of *learning parity with noise*, which is also computationally hard, although solving this problem given model knowledge is trivial. \n\n**Back to our setting, access to privileged information never \"assumes this challenge of sampling error away\".** Therefore, our positive results on the computational complexity are **not** a direct consequence of [21, 22]. In particular, **both** the **algorithms** and **techniques** for showing \"poly-sample complexities\" in our paper are fundamentally different from those in [21] when there is no privileged information.\n\n## Response to Point 2 of high-level critic:\n\n**Why deterministic filter + privileged information?** \n\n- **Empirical motivations.** Our work aims to understand practical privileged-information-based algorithms, leading to the identification of the Deterministic Filter condition after recognizing the limitations of Expert Distillation. \n- **Privileged information alone imposes no assumptions on the POMDP model itself, implying its PSPACE-hardness computationally (inherited from planning).** This is because **planning with model knowledge** is **easier** than **learning with privileged information**, where one can essentially **simulate** the problem of learning with privileged information when knowing the model knowledge.\n- **Unifications of known problem classes (see Appendix E).** \n\n## Response to Point 3 of high-level critic:\n\nFirstly, we did **not** claim our algorithms are highly practical. Our focus was to understand existing practical paradigms, Expert Distillation, and Asymmetric Actor-Critic, in the most fundamental tabular setting, with some abstractions for theoretical analysis.\n\n**More importantly, we did have function approximation results (lines 275-277, and full results in Appendix G), where we only relied on a simple classification oracle, which is indeed quite compatible with current Deep RL implementations.**\n\n## Response to improvements in terms of writing:\n\n**Clarifications for contributions (single-agent v.s. multi-agent):** Our main contribution is understanding empirical paradigms using privileged information and then addressing their limitations. **More importantly, results for single-agent POMDP and multi-agent POSG are both under this privileged information setup, following the same algorithmic principle.** Therefore, they are **NOT** orthogonal.\n\n**Line 45-46 (goal of actor-critic):** Asymmetric Actor-Critic is already a strong empirical algorithm, and our goal is to firstly **identify its limitations** (Prop 3.3) and then further develop both computationally and statistically efficient versions of it (Sec 5).\n\n**Regarding Proposition 3.1:** We hope to firstly **understand** the limitation of existing empirical paradigms through Proposition 3.1. Then we further use it to **motivate** a (unifying) subclass of POMDPs (Def. 3.2) under which we **managed to prove** that the paradigm of Expert Policy Distillation **can be provably efficient** (Sec 4).\n\n**Response to the last point:** The benefits of privileged information are well-studied empirically. **Those terminologies are not what we invented but are already quite standard in empirical works.**\n\nFinally, we **strongly disagree** that this assumption is \"essentially similar to the block MDP\". **We have provided several examples in lines 919-929, including block MDP, but significantly going beyond it.** In fact, one example of Def 3.2 can include the POMDP with latent deterministic transition **without any assumptions on the emission**.\n\n## Response to Questions:\n\n**Firstly, neither observability or well-conditioned/regular/separated PSR condition implies Def 3.2.** To the best of our knowledge, such conditions instantiated to POMDP mainly rule out the case of *uninformative* observations. In contrast, Def 3.2 in extreme cases, can require **no assumptions on emission**.\n\n**Secondly, Thm. 5.3 improves (Golowich et al., 2022) by achieving both polynomial sample and quasi-polynomial time complexity (see more details in our response to Point 1).**\n\n**Finally, results in Sec 4 are \"in parallel\" with those in Sec 5.** In Sec 4, we have studied another class of POMDPs, i.e., those satisfying Def 3.2, **instead of assuming observability as in Sec. 5**. Since Def 3.2 and $\\gamma$-observability **do not imply each other**, the results of Sec 4 and Sec 5 **do not imply each other, either**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n76gomIdBP", "reviewer": "Reviewer_AXXx", "summary": "This paper presents a novel theoretical characterization of certain kinds of\nPOMDP's which admit efficient learning. First, related characterizations are\nexplored and theoretical results show that these classes of POMDP's suffer from\ncertain drawbacks when trying to learn policies. Based on this analysis, a new\nclass of POMDP's is defined using the \"deterministic filter condition\".\nThis characterization of POMDP's is used to develop a novel learning algorithm\nbased on a decomposition of POMDP policy learning into belief learning and a\nfully-observable policy learning step. Theoretical results show that policy\nlearning in POMDP's satisfying the deterministic filter condition condition is\ntractable given access to privileged information at training time. The paper\nalso presents an extension of the proposed algorithm to handle partially\nobservable multi-agent RL.", "questions": "I'm not sure that my understanding of 3.2 is correct. The intuition given in the\npaper is that it corresponds to allowing $\\gamma$ to go to 1 in assumption C.8.\nBut to my understanding C.8 encodes a certain kind of approximate observability\nand letting $\\gamma$ go to one recovers a fully-observable environment. (I\nbelieve this is related to the restriction that $b^s$ is a one-hot vector in\ndefinition 3.2). But if the environment is observable then there's no need to\nconsider POMDP's anymore, regular MDP's should be sufficient. Could you clarify\nthis definition a bit more, and especially how it is distinct from assuming that\nthe emission function can be inverted?", "limitations": "Limitations of the proposed approach are discussed, although this discussion is left to the appendix.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- This paper considers a problem of practical importance, namely how to learn\n  policies when there is more information available at training time than test\n  time.\n- The extension to multi-agent RL is a useful inclusion.\n- Giving some theoretical failings of existing approaches helps motivate the\n  work.\n- The theoretical analysis of the proposed algorithm is extensive.", "weaknesses": "- Some important sections are deferred to the appendix (related work,\n  conclusion, and the discussion of experiments). I understand that space in the\n  main paper is quite limited but I think it's important to include these\n  sections. Maybe instead some theoretical results could be deferred to the\n  appendix, especially those in section 3 concerning the shortcomings of\n  existing approaches.\n- More generally, so much of the discussion in the paper is in the appendix that\n  the main body of the paper is quite hard to read.\n- The central definition 3.2 could be explained a bit more (see questions).\n- The legends on the graphs in the experimental section cover a lot of the\n  relevant information in the graph.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We believe there are several important misunderstandings we want to clarify. Please see our responses below:\n\n---\n\n## Response to Point 1 of weakness\nWe are thankful for the suggestions on the organization of the paper. We will re-organize the paper by moving some motivation parts to the appendix, and moving back some more related work and experiment discussions. \n\n---\n\n## Response to Point 2 of weakness\nWe shall explain more about Definition 3.2 in the following responses to the questions. Meanwhile, we will add more discussions on Definition 3.2 in the main paper.\n\n---\n\n## Response to Point 3 of weakness\nWe have replotted the figures for better readability. Please see the attached PDF in our global author rebuttal. \n\n---\n\n## Response to Questions\nFirstly, the reviewer is correct that when $\\gamma$ tends to one, the condition implies a reversible emission matrix. However, **we only use this extreme example as a motivation.** This only corresponds to one extreme case of us, i.e., Block MDP (Example E.2). \n\n\n**More importantly, our condition (Def 3.2) is much more general than this relatively trivial case.** For example, we can also impose **no assumptions** on the emission **by allowing the emission to be totally uninformative/unobservable ($\\gamma=0$)**, while only requiring the latent state transition to be deterministic (see Example E.1). In general, our (newly identified) condition represents a **joint effect** of transition determinism and observation informativeness, which is **of independent interest** in the POMDP literature. The $\\gamma=1$ case and the deterministic latent state transition case just serve as *two extreme examples* of our condition. There are certainly more examples in between, e.g., Example E.3 and the red shade area in Fig. 1.\n\n\n**Intuitive explanations about the deterministic filter.** Firstly, the terminology of a *filter* refers to the process of *estimating the underlying unknown state using a sequence of actions and observation* (recursively). Therefore, our condition only assumes that if at step $h-1$, the *estimation* of the current states is not random, then the state estimation for the next step $h$ after taking the new action $a_{h-1}$ and receiving the new observation $o_h$ is also non-random. **Therefore, it is direct to see that requiring $o_h$ to be informative such that it can be inverted to/decode the state at step $h$ is only sufficient, but not necessary.**\n\n---\n\nWe hope our responses have addressed the reviewer’s concerns, and would be more than happy to answer any other questions the reviewer may have. Please do not hesitate to let us know if there are any other explanations/updates needed that may help re-evaluate our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7s0lS4ZcJt", "reviewer": "Reviewer_9C6t", "summary": "This paper has several contributions. The authors examine using privileged information for partial observability in RL. The paper looks at two empirical paradigms, expert distillation and asymmetric actor-critic, analyzing their computational and sample efficiency under specific conditions.\n\nIn addition, the authors introduce a belief-weighted optimistic asymmetric actor-critic algorithm, providing insights into MARL with CTDE.", "questions": "- What is the purpose of developing the deterministic filter condition? Can you give a motivating example?", "limitations": "The authors are upfront about their assumptions; however, I would like to mention that reliance on specific assumptions for computational tractability might limit the generalizability of the results.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- There is a considerable amount of summarization and formalization, such as for the expert distillation paradigm, which is appreciated.\n- Theoretical results with the proposed deterministic filter condition is a novel contribution.\n- The belief-state learning contribution could be interesting for future works.", "weaknesses": "- Like most theoretical papers, examination of existing empirical paradigms  and evaluation of proposed method is rather lacking, while understandable, is still a weakness.\n- The deterministic filter condition, if I understand correctly, is weaker than previous conditions but still rather restrictive compared to real-world tasks.\n- More toy examples could be helpful along side theory work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. Please see our responses below:\n\n\n## Regarding the examination of existing empirical paradigms\n\n\nWe agree with the reviewer that understanding existing empirical paradigms is important for theory-oriented research. We believe our paper indeed examined the theoretical perspectives of the **two most important empirical paradigms that utilized privileged information** in RL, by first identifying its suboptimality (Prop. 3.1), inefficiency (Prop. 3.3), and then proposing **new** provable algorithms (Sec 4 and Sec 5). Meanwhile, examining the experimental perspectives of those empirical paradigms is indeed not the main focus of our work but is an important future work we believe.\n\n---\n\n## Regarding the deterministic filter condition\nTo further clarify, our deterministic filter condition (Definition 3.2) serves two key purposes:\n- **Unification** of existing tractable POMDP models: It unifies several important models of tractable POMDPs, including deterministic POMDPs, Block MDPs, and k-decodable POMDPs (see Appendix E for details and novel examples).\n\n\n- Criteria for the empirical paradigm of **Expert Policy Distillation**: It represents a general class of structured POMDPs with provable efficiency for the empirical paradigm of *Expert Distillation*.\n\n---\n\n## Regarding more experimental results\n\n\nWe have conducted more extensive experimental evaluations on POMDPs of larger sizes. The pdf results are attached to the global author rebuttal\n\n---\n\nWe hope our responses have addressed the reviewer’s concerns, and would be more than happy to answer any other questions the reviewer may have. Please do not hesitate to let us know if there are any other explanations/updates needed that may help re-evaluate our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tIhL8XcRgh", "reviewer": "Reviewer_tn5m", "summary": "This paper offers valuable insights into the use of privileged information in partially observable reinforcement learning (POMDP), presenting theoretical advancements and practical algorithms. However, challenges remain in ensuring the effectiveness of expert distillation, applicability to real-world problems, and bridging the gap between training and test environments. Addressing these limitations is essential for advancing the practical implementation of Privileged Policy Learning.", "questions": "The paper does not discuss the tightness of the derived bounds or whether they meet the needs of practical problems.", "limitations": "Lack of sufficient experimental verification", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper formalizes expert distillation, highlighting its limitations in finding near-optimal policies in observable POMDPs. This formalization provides a structured approach to understanding and improving existing empirical methods.\n\n2. The introduction of the deterministic filter condition broadens the scope of tractable POMDP models and ensures that expert distillation can achieve polynomial sample and computational complexities.\n\n3. The investigation into multi-agent RL (MARL) frameworks with privileged information and the proposed algorithms ensure polynomial complexities.", "weaknesses": "1.\tExpert policies in pratical problems may not be optimal [1-3]. This raises concerns about the effectiveness of Privileged Policy Learning. If the expert policies are suboptimal, the student policies distilled from them may inherit these inefficiencies, potentially leading to suboptimal performance in practical applications. \n\n2.\tA significant limitation highlighted is the dependency on access to the state of the POMDP environment. In most practical scenarios, the internal states are not observable, restricting the application of the proposed methods. The deterministic filter condition, while theoretically valuable, may not always be applicable in real-world problems where state observability is a significant challenge.\n\n3.\tThe paper discusses the difficulty of obtaining the internal state in actual problems, creating a gap between the training environment (where privileged information is available) and the test environment (where it is not). This discrepancy can lead to performance degradation when the policies trained with privileged information are deployed in real-world scenarios where such information is unavailable. \n\n[1] Wu, Yueh-Hua, et al. \"Imitation learning from imperfect demonstration.\" International Conference on Machine Learning. PMLR, 2019.\n\n[2] Tian, Yuandong, Qucheng Gong, and Yu Jiang. \"Joint policy search for multi-agent collaboration with imperfect information.\" Advances in neural information processing systems 33 (2020): 19931-19942.\n\n[3] Tang, Yu, et al. \"Hierarchical reinforcement learning from imperfect demonstrations through reachable coverage-based subgoal filtering.\" Knowledge-Based Systems 294 (2024): 111736.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and believe there are several important misunderstandings we would like to clarify. We address the reviewer's concerns and questions below:\n\n---\n\n## Regarding the potential sub-optimality of the expert\n**Firstly, the focus of our paper is to study how to distill a student policy from a given (**even the perfect**) expert policy provably.** We agree that in practice, not having a perfect expert might be an issue. But we believe this is some perspective **orthogonal** to the focus of our paper. **In fact, training a better/optimal expert policy is the focus of RL in MDPs, which has been studied extensively with many existing works.** In contrast, we studied RL in POMDPs with privileged information. Among the **first attempts** at theoretically understanding privileged information, we started with two most commonly used algorithmic paradigms: Expert Distillation and Asymmetric Actor-Critic, with many empirical successes as justifications (see Appendix B for a detailed discussion), and **in the most fundamental and basic setting**. We believe this will serve as the foundation for further studying the extended setting mentioned by the reviewer. We do believe extensions to the case without “optimal” expert policy is an important future work to explore. We will make sure to refer to these papers when motivating and discussing such generalizations.\n\n---\n\n**More importantly, our analyses and results can be readily  generalized to the case when the expert policy is sub-optimal.** Specifically, for any $\\epsilon>0$, if the expert is only $\\epsilon$-optimal, i.e., the performance difference between the expert and the optimal policy is $\\epsilon$, the distilled student policy's performance gap compared with the optimal policy will also only increases by at most $\\epsilon$, through a direct triangle inequality argument. If needed, we can add this corollary in the next version of our paper.\n\n---\n\n## Regarding the access to the state\n**Depending on the access to the state is exactly the setting that we wanted to understand better theoretically, and the setting that has been widely used in practice.** This was referred to as settings with \"privileged information\" in the literature and our paper. The cases when such a state is *not accessible* are *not* the theme of our work here, and will have to suffer from the known computational and statistical hardness in general. \n\nMeanwhile, the deterministic filter condition is **weaker** than many existing “statistically tractable” POMDP models, and also allows us to incorporate “computationally” considerations when carefully incorporated with privileged information, i.e., our main results. Empirically, it can also serve as a good criteria for us to determine whether the empirical paradigm of expert distillation will suffer from sub-optimality or not. We will add clarifications in the revision to make these points clearer.\n\n---\n\n## Regarding the gap between training and testing\nWe firstly acknowledge that the performance *gap* between training and testing is the major challenging of RL in POMDPs with privileged information, which is **exactly we hope and managed to address in our paper**. In other words, all the guarantees (in particular, Theorem 4.5, 5.3) of our algorithms are showing that the **deployed policy** is an optimal one for the POMDP (with **partial observations** as input), instead of the training policy (allowing **privileged state information** as input). Hence, our results exactly showed that **such a performance gap can be conquered with our more carefully algorithm design and analyses** (while the naive versions of the algorithms from existing empirical works can fail, see our Sec 3). \n\n---\n\n## Regarding the tightness of the bound\nWe thank the reviewer for bring up this important question and discuss the tightness of our bound here.\n- For our results in Sec 4, we actually achieve **polynomial** sample complexity, which matches the best results for sample complexity [31, 39]. For the time/computational complexity, we also achieve the **polynomial** complexity, thus not improvable either (note that even planning in MDP is a P-complete problem).\n- For our results in Sec 5, we still achieve **polynomial** sample complexity, thus matching the best sample complexity as before. More importantly, we also achieve **quasi-polynomial** time/computational complexity, which is shown to be tight for the *observable POMDP* setting we are considering (see [22]).\n\n---\n\n## Regarding the experimental evaluation\n\n\nWe **have provided new experiments** for more instances of POMDPs of larger sizes. We refer to the uploaded pdf for detailed results. \n\n---\n\n## Regarding the additional references\n\n\nWe thank the reviewer for bringing more related literature [1, 2, 3] to our attention. We will definitely include them in our related work section in our revision.\n\n---\n\nWe hope our responses have addressed the reviewer’s concerns, and would be more than happy to answer any other questions the reviewer may have. Please do not hesitate to let us know if there are any other explanations/updates needed that may help re-evaluate our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Additional experimental results

In response to the reviewers, to make our experimental evaluation more sufficient, we **have added new results** by testing our algorithms on more POMDP problems of larger size than the original problems in the paper. Meanwhile, we also addressed the problem of overlapped legends in some figures. We hope those additional experimental evaluations and re-plotted figures will address the concerns regarding the empirical evaluation of our algorithms.

Meta-review: This paper presents novel analysis of partially observable RL in the case when privileged information is available during training (ability to observe ground truth states during training). In particular, the two most common privileged settings are considered---(fully observable) expert distillation and asymmetric actor-critic. Theoretical analysis of standard methods in these frameworks is conducted and a more efficient novel algorithm is proposed for the later case. Extensions to the MARL case are considered and some experiments are provided. 

While partially observable RL (RL in POMDPs) is popular, there is lack of theoretical analysis of many of the methods. Similarly, many algorithms focus on the privileged setting in an attempt to improve performance by using additional information from a simulator during training. A better understanding of these algorithms as well as improved algorithms would be beneficial to the community. 

The paper makes an important contribution but it is very dense and the significance of the results is somewhat unclear. Because there are multiple different types of privileged information considered as well as the multiagent (POSG) case, there are many topics covered in the paper, making it hard to fit everything in clearly. In terms of significance, there some question about how practical the analysis and algorithms are. The analysis makes some strong assumptions, such as the deterministic filter condition, but these assumptions still fit with a large portion of POMDPs (including several of those used in papers). The proposed algorithms are unlikely to scale well in their current form but they have the potential to inform future algorithms in the field. The author response and discussion was helpful for clarifying some of the confusions so the paper should be updated to include versions of this text. The additional results were also helpful to better understand the benefits of the approach. The paper would be strengthened by including more discussion about how future work could build off of the theory and algorithms in this paper to develop scalable methods that still outperform current approaches.
Decision: Accept (poster)