Title: To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable RL
Venue: NeurIPS 2025
Accepted: yes
Keywords: Partially Observable MDP

privileged expert distillation

learning to cheat

image-based RL

RL theory
Abstract: Partial observability is a notorious challenge in reinforcement learning (RL), due to the need to learn complex, history-dependent 
policies. Recent empirical successes have used *privileged expert distillation* -- which leverages availability of latent state information during training (e.g., from a simulator) to learn and imitate the optimal latent, Markovian policy -- to disentangle the task of ''learning to see'' from ''learning to act''. While expert distillation is more computationally efficient than RL without latent state information, it also has well-documented failure modes. In this paper -- through a simple but instructive theoretical model called the *perturbed Block MDP*, and controlled experiments on challenging simulated locomotion tasks -- we investigate the algorithmic trade-off between privileged expert distillation and standard RL without privileged information. Our main findings are: **(1)** The trade-off empirically hinges on the *stochasticity* of the latent dynamics, as theoretically predicted by contrasting *approximate decodability* with *belief contraction* in the perturbed Block MDP; and **(2)** The optimal latent policy is not always the best latent policy to distill. 
Our results suggest new guidelines for effectively exploiting privileged information, potentially advancing the efficiency of policy learning across many practical partially observable domains.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the theoretical properties of privileged expert distillation in partially observable RL. It begins by arguing that perfect decodability is unrealistic in practice and introduces the notions of approximate decodability and the perturbed Block MDP as a more practical model. The paper then systematically compares the theoretical guarantees and empirical performance of expert distillation versus standard RL in both deterministic and stochastic dynamics. Finally, it suggests that injecting stochasticity into the latent MDP can improve the effectiveness of distillation by yielding smoother experts.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper addresses an important and timely question: when and why expert distillation is beneficial, especially in the context of robot learning. This has significant practical relevance.\n\n- The focus on settings where decodability fails but observations remain informative is both realistic and more practical than prior assumptions of perfect decodability.\n\n- It presents a comprehensive and thoughtful literature review.\n\n- The theoretical results are well-developed, with formal proofs that support the key claims.\n\n- Experimental results are carefully designed and align well with the theoretical predictions.\n\n- The idea of smoothing the expert policy to improve distillation performance is well-motivated.\n\n**Weakness**\n\n1. Assumption about perfect decodability may be premature:\n\nThe paper's rejection of perfect decodability as unrealistic is foundational to its entire framing, and should be argued more rigorously. For instance:\n  - The presence of large initial prediction error is not necessarily evidence against decodability; even in decodable settings, early predictions may be inaccurate due to limited history\n  - Prediction errors could be caused by irrelevant or redundant latent variables (e.g., absolute coordinates) that are not essential for the task and naturally hard to infer without a reference frame.\n  - Figure 5 lacks clarity — it's difficult to determine whether the final error is substantively large or just numerically small due to scale. A more illustrative example showing what is being mispredicted and how much could better support the claim.\n\n2.  Lack of comparison to asymmetric RL:\n\nPrivileged expert distillation is often compared to asymmetric RL, which also utilizes privileged information during training. However, this paper only compares expert distillation to standard RL without privileged information. If latent states are available, then asymmetric RL would be a more practical and competitive baseline.\n\n3. Limited task diversity:\n\nWhile the experimental evaluation includes three DeepMind control suite tasks, all of them are locomotion-based. The lack of coverage of other types of partial observability — such as multi-agent coordination, strategy games, or visual navigation — limits the generality of the conclusions.\n\nQUESTIONS:\n1. Is there a theoretical or empirical distinction between injecting noise into actions, observations, or latent state transitions? How do these different sources of stochasticity affect decodability and distillation?\n\n2. In the stochastic setting, is the privileged expert trained with noise injected into the latent MDP, or is it trained on the original (noise-free) latent MDP? If the latter, doesn't that imply the expert and student are being trained on different MDPs, which could explain the degradation in performance?\n\n3. What is the relationship between environment noise and expert smoothing noise? For instance, does higher environment noise require a higher level of expert smoothing? Could expert smoothing still help in environments with no noise?\n\n4. Can the authors comment on whether the deterministic filter condition from [1] relates to their assumptions in Section 4? Is it possible to extend the results for deterministic dynamics to this broader condition?\n\n5. Could the authors provide some discussions on the paradigm introduced in [2]?\n\n[1] Cai, Y, et al. Provable partially observable reinforcement learning with privileged information. \n\n[2] Li, Y, et al. Guided Policy Optimization under Partial Observability.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

comments: SUMMARY:
This paper investigates the trade-off between expert distillation and standard RL in POMDPs, finding that the stochasticity of the latent dynamics governs their relative performance. The authors provide a theoretical framework based on approximate decodability and belief contraction to explain this trade-off. They also show empirically that distilling a smoother expert, trained with noise, can be a more effective strategy than distilling the technically optimal one.

STRENGTHS AND WEAKNESSES:
Strengths
- The paper provides a strong theoretical framework for the distillation vs. RL trade-off by introducing the 'perturbed Block MDP' model and using it to contrast the core conditions of 'approximate decodability' and 'belief contraction'.
- The paper effectively connects theory to practice, not only by validating its theoretical predictions with experiments (Fig. 3) but also by uncovering a significant and practical insight: distilling a 'smoother' expert can be more effective than distilling the optimal one (Fig. 4).

Weaknesses
- The provided theoretical bounds, which appear to be worst-case guarantees, do not always align perfectly with the empirical outcomes. 
- The finding that smoother experts can be better teachers is one of the most exciting directions in the paper, and the authors provide a compelling proof-of-concept by injecting motor noise. It would be valuable to see it explored further. For instance, a more systematic study on how to tune the 'optimal' amount of noise for a given task, or a deeper theoretical dive into how this smoothing technique impacts the action-prediction error bound, could significantly build upon this novel contribution.

QUESTIONS:
Please see weaknesses section.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies RL vs imitation learning in POMDPs. It firstly verifies that perfect L-decodable POMDP may not hold using experiments and then considered a setting call $\\delta$-perturbed Block MDP. Then the paper proposed provable algorithms based on the approximate $L$-decodability induced by the $\\delta$-perturbed Block MDP assumption. Finally, the paper also discussed how to improve expert distillation by smoothness.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n1. The paper combines many theoretical claims with experimental justifications, where provide valuable insights.\n2. The argument that latent expert might not be the best target and smoothness might help is novel.\n\nWeakness:\n\n1. I find the paper oftentimes confuses the concept of decodable and $L$-decodable. Technically speaking, I am not aware of any references that use the nomenclature of ``decodable POMDP''. Although the condition by [6] should be equivalent to decodable by $H$ steps. However, it seems to me what the paper has relaxed is the perfect $L$-decodable condition with small $L$ (Efroni et al 2022) instead of the condition in [6] as claimed in the intro. Therefore, to substantiate a meaningful relaxation over [6], the paper should either: (a) establish provable guarantees in the regime of large $L$, or (b) clearly discuss this distinction in the main text to avoid conceptual confusion.\n\n2. Secondly, it seems the technical assumption that really enables approximate $L$-decodability is perturbed block MDP + deterministic transition. In terms of theoretical contributions, this is arguably quite strong to have structural assumptions on emission and transition at the same time.  \n\n3. Meanwhile, in the experimental validation, the paper is only using $L\\in [5]$ to verify the decodability. This again tries to only verifies whether $L$-decodability holds with small $L$. Moreover, I understand in practice ppl often only use observation history. However, action history is also important in theory and in terms of verifying the theoretical conditions faithfully.\n\n4. As claimed, one important goal of the paper is to understand expert distillation vs RL. However, the message is that with deterministic transition, expert distillation match RL bounds. With stochastic transition, expert distillation is worse. Therefore, it seems the theory itself does not reveal the benefits of distillation.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the trade-off between privileged expert distillation—where a latent policy is trained using privileged information and then distilled into an executable policy via online behavior cloning—and reinforcement learning (RL) without privileged information, which uses frame-stacking, in partially observable Markov decision processes (POMDPs).\n\nAfter formalizing the sources of error in POMDPs in terms of decodability and belief contraction, the authors introduce a theoretical model called the perturbed block MDP. This model extends the block MDP framework by incorporating some irreducible error in the emission distribution. Using this instructive model, the authors demonstrate that expert distillation can be both computationally and sample efficient in deterministic environments, where the belief state becomes concentrated over transitions. However, they also show that in the worst case, expert distillation may lead to irreducible sub-optimality.\n\nAdditionally, the authors present a series of controlled experiments on locomotion tasks (including humanoid, dog, and walker) to empirically illustrate their theoretical findings\n\nSTRENGTHS AND WEAKNESSES:\n- The paper is well-motivated and clearly structured.\n- The topic of expert distillation versus reinforcement learning is highly relevant and of significant practical interest.\n- Although the paper is mathematically and notationally dense, the authors consistently provide intuitive explanations and high-level interpretations.\n- The authors openly discuss the limitations of their theoretical model and acknowledge the presence of other confounding factors not accounted for.\nI-  particularly appreciate the seamless integration of theorems with empirical validation, which helps to concretize the theoretical findings.\n- Due to time constraints and limited background in POMDP theory, I was unable to thoroughly review the proofs in the appendix and therefore cannot fully assess the correctness of the theorems.\n\nQUESTIONS:\n- in section 5.1, what do you mean by uniformly mixing dynamic? does it mean that we can teleport to any latent state from any latent state  at each step? does this mean that the belief state is uniform over the state space?\n- in the case of uniformly mixing dynamic, do you think we ever need to do frame-stacking, since previous observations don;t inform us about the next state ? \n- in section 5.2, I am curious what happens if we let expert distillation to run for more time and samples. does it recover the RL performance asymptotically ?  I am wondering if it is about having more samples or there are real fundamental gap between the two method in these environments.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}