Guided Policy Optimization under Partial Observability
Authors: Yueheng Li
Guangming Xie
Zongqing Lu
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
teacher-student learning
policy distillation
POMDPs
policy gradient
Primary area: reinforcement learning

Abstract: Reinforcement Learning (RL) in partially observable environments poses significant challenges due to the complexity of learning under uncertainty. While additional information, such as that available in simulations, can enhance training, effectively leveraging it remains an open problem. To address this, we introduce Guided Policy Optimization (GPO), a framework that co-trains a guider and a learner. The guider takes advantage of privileged information while ensuring alignment with the learner's policy that is primarily trained via imitation learning. We theoretically demonstrate that this learning scheme achieves optimality comparable to direct RL, thereby overcoming key limitations inherent in existing approaches. Empirical evaluations show strong performance of GPO across various tasks, including continuous control with partial observability and noise, and memory-based challenges, significantly outperforming existing methods.

Published as a conference paper at ICLR 2026
GUIDED POLICY OPTIMIZATION UNDER PARTIAL OB- SERVABILITY
Yueheng Li', Guangming Xie' Zongqing L
1 Sccooo of Advanced Manufacturing and Robotics, Peking University "cchool of Computer Science, Peking University (1 iliyuhheng, xiegming, zongging lu) @pku. edu. cn
ABSTRACT
Reinforcement Learning (RL) in partially observable environments poses sig- nificant challenges due to the complexity of learning under uncertainty. While additional information, such as that available simulations, can enhance training, effectively leveraging it remains an open problem. To address this, we introduce Guided Policy Optimization (GPO), a framework that co-trains a guider and a learner. The guider takes advantage of privileged information while ensuring align- ment with the learner's policy that is primarily trained via imitation learning. We theoretically demonstrate that this learning scheme achieves optimality comparable to direct RL, thereby overcoming key limitations inherent in existing approaches. Empirical evaluations show strong performance of GPO across various tasks, in- cluding continuous control with partial observability and noise, and memory- based challenges, significantly outperforming existing methods.
INTRODUCTION
Many real-world tasks can be formulated as sequential decision-making problems where agents take actions in an environment to achieve specific goals over time (Puterman) 2014). Reinforcement Learning (RL) has emerged as a powerful tool for solving such tasks, leveraging trial-and-error learning to optimize long-term rewards (Sutton & Barto 2018). Despite its success, RL encounters significant hurdles in complex and partially observable environments, where agents often operate with limited noisy information (Madani al. 1999). However, during training, we often have access to extra information that could significantly enhance learning efficiency and performance (Lee) al., 2020a Chen al., 2022). For instance, in robotics, while real-world sensor data may be noisy or incomplete, simulation environments typically provide full state observability.
Despite the potential of such privileged information, effectively leveraging in practice remains major challenge. One popular strategy to utilize this information is through methods like Imitation Learning (TSL), or policy distillation (Czar) necki et al. 2019). In these approaches, a teacher, equipped with privileged information, provides supervision to guide the student's 's learning process. However, this strategy introduces its own set of challenges: a teacher with privileged information may impose unrealistically high-performance standard, making it difficult for the student to effectively imitate. This issue, known as the "impossibly good" teacher (Walsman al. 2023) or imitation gap (Weihs et al., 2024), can hinder learning and degrade performance. To address this, previous work has sought to integrate environmental rewards into the learning process of the student. One approach is to combine RL with IL. (Weihs al.) 2024; Shenfeld et al., [2023a] Nguyen et al., 2023), switching to RL-based training when the teacher becomes inimitable. Another approach modifies environmental rewards based the teacher through policy distillation (Czarnecki el al., 2019 Walsman et al., 2023). However, such methods diminish the utility of privileged information, often resulting in inefficient use of the teacher's knowledge.
Learning (IL) (Hussein et al.) 2017). Teacher-Stude
To better exploit available information, we propose training a "possibly good" teacher, i.c., teacher whose policy remains within the learner's imitable region. Inspired by Guided Policy Search (GPS) Levvine & Koltun 2013, Montgomery & Levine 2016), we introduce Guided Policy Optimization
"Corresoonding authors.

---

Published as a conference paper at ICLR 2026
(GPO), novel framework that trains both the teacher and the student simultaneously while ensuring that the teacher's ' policy remains aligned with that of the student. The key insight behind GPO is that by leveraging privileged information during training, the teacher can be trained more effectively while ensuring that its performance is "possibly good," thus facilitating easier imitation by the student. Theoretically, we show that the student can achieve optimality similar direct RL. training. mitigating the suboptimality and imitation gaps that often arise from purely teacher-based supervision. We empirically validate our algorithm across various tasks, including didactic examples, challenging continuous control tasks in partially observable, noisy environments within the Brax (Freeman et al. 202I) domain, and in memory- -bssed tasks from the POPGym (Morad et al.) (023) benchmark. GPO shows consistent and significant improvements, underscoring its ability to exploit extra information and deliver robust performance across diverse domains.
2 BACKGROUND
We consider Partially Observable Markov Decision Process (POMDP) (Kaelbling et al. 1998), which characterized by the tuple (S, r, P. O,7). S represents the set of states, the set of actions, r the reward function, P the transition probability function, O the partial observation function and y the discount factor. At each time step t, the agent receives a partial observation or O(-1s;) for current state 8, E S. The agent then selects an action de E A according to or or its action-observation history Tr. : {on, ao; 01, aI..., 0r). The state transitions to the next state 81-11 according to P( (1,, az). and the agent receives reward r2- The goal for the agent is to find the optimal policy n² - ((A) that maximizes the return, expressed as π = arg max, Vg, where V₂r = BI yrre| [ represents cumulative rewards. When full state information s is available during training,
may also define policy μ 8 + ((A) based on privileged information. For clarity, throughout this paper we refer to such privileged training inputs simply the state though in practice they could take other forms. Likewise, we refer to partial observations simply as o, though in practice they may include histories or other derived features.
Finally, we emphasize that in the remainder of this paper, the term "optimal" refers to the stu- dent's optimal policy under partial observability-oot the teacher's optimal policy under privileged information, which is generally unattainable for the student.
2.1 TTACHER-STUDRRE LEARNING
Since we consider both training the teacher and student, in this paper, we use the term Teacher-Student Student Learning (TSL. to broadly refer to Imitation Learning (IL. (Hussein et al. 2017), policy distillation (Czarnecki et al. 2019), and related approaches, as there is no fundamental distinction between them. In TSL, the teacher policy is typically pre-trained using RL or derived from other methods such as a classical controller, which is assumed to effectively accomplish the desired task. The goal is for the teacher to somehow provide supervision to the student in learning to solve the same task.
A straightforward approach to training the agent is to directly supervise the student's policy using the teacher's ' policy μ., similar to Behavioral Cloning (BC) (Pomerlcau [991] Torabi et al. 2018):
min Eg,d, (x(((s)
where d, is the distribution of states under the teacher's 's policy, and DKL is the Kullback-Leiller (KI.) divergence. This objective encourages the student's policy to mimic the teacher's policy for the observed states. However, when the teacher's 's policy is based on privileged information, the student can only learn the statistical average of the teacher's actions (Warrington et al.. 20200) Weihs) et al. 2024), and be strictly suboptimal (Cai et al. 2024). In this paper, we refer to such a teacher as inimitable, and we highlight this limitation through two illustrative examples in the next subsection.
2.2 DIDACTIC EXAMPLES
DoorrDoor. In the classic TigerDoor problem (Littman et al., 1995), there are two doors with a tiger hidden behind one of them. The possible state sz (tiger behind the left door) and sæ (tiger behind the right door), with equal probabilities for each, form S Sn). The action set is an,01) ).), where az and an denote opening the left and right doors, respectively, and a
A (aL, an
2

---

Published as a conference paper at ICLR 2026
action
action
a
RR
01
a
Qtt
state
state
8L
-0.1
$L
0
SR
-0.1
SR
Table 1: TigerDoor problem
Table 2: TigerDoor-alt problem
denotes listening to determine the tiger's location. The teacher knows the tiger's location whereas the student can only ascertain it after choosing ag. The payoff matrix is shown in [Tb The optimal policy for the teacher is always choose the correct door without listening, whereas the student's optimal strategy involves first listening to locate the tiger. Consequently, the student cannot learn the optimal policy through supervision from the teacher, the teacher never chooses dg. Under the teacher's s supervision, the student will only Iearn to randomly select between and aR, resulting in an expected reward of 0.5. This scenario poses challenges for the supervised student, as the teacher fails to explore and gather essential information for the learner.
TigerDoo We also introduce an alternative version of the problem, TigerDoor-alt (Table 2p which further illustrates the imitation gap, even when no exploratory actions are required. In this scenario, the listening action ar is removed, and the reward for correctly selecting the left door increased to 2. Similarly, the teacher continues to select the correct door, while the student learns to randomly choose between the two doors, yielding an expected reward 0.75. However, the optimal policy for the student is to always choose the left door, which provides expected reward of 1. This discrepancy arises from the loss of information when converting the reward bwward-based objective into a policy-supervised objective.
The current solution to this issue to incorporate rewards into the student's ' learning process. To effectively utilize teacher supervision, there are two kinds of approaches. The first type dynamically adjusts the weight of the supervision loss between the teacher and pure RL training. This allows the algorithm to switch pure RL when the teacher is deemed inimitable (Weihs et al.) 2024] Shenfeld itt al., b).3abb). However, such approaches fail to fully utilize privileged information and may waste the valuable, often expensive, pre-trained teacher. The second kind incorporates teacher supervision into the reward signal, for instance, by using reward shaping via the teacher's 's value function (Walsman al. 2023). However, this supervision indirect and may require additional learning. Crucially, to the best of our knowledge, none of the existing methods provide theoretical guarantees that teacher supervision will actually beneficial.
Another research direction attempts to reconstruct privileged information from partial observations. However, such methods require the MDP to be decodable (Efroni al.] 2022), which clearly infeasible in the TigerDoor setting. For more detailed discussion of related work, see Appendix A
METHOD
We present our Guided Policy Optimization (GPO) framework, which co-trains two entities: the guider and the learner, which we use to differentiate from existing TSL methods. GPO iteratively updates both policies to ensure alignment. We then explore both the theoretical properties and practical implementation of GPO, introducing two variants: GPO-penalty and GPO-clip.
3.1 FROM GPS TO GPO
Unlike direct policy search methods, GPS (Levine & Koltun 2013 Montgomery & Levine 2016 does not optimize policy parameters directly. Instead, introduces an intermediate agent (guider) and policy, which is then used to employs trajectory optimization to learn a vareing learning. Although GPS is a model-based train a neural network policy (learner) through supervised method and is not directly applicable in setting, its idea of introducing intermediate agent to guide policy learning can be extended to an RL algorithm the context of POMDPs.
Specifically, since the guider is only used during training, it access any type of privileged information. The key requirement to ensure that the guider is imitable by the learner, which motivates us introduce the GPO framework, which operates through the following four steps:
3

---

Published as a conference paper at ICLR 2026
Teacher-Student Learning
Guided Policy Optimization
Supervise
Supervise
Teacher ((lss)
Student πI Ta)
Guider p(als)
Constrain
Learner T(alo)
Unable to Learn
Guarantee to Learn
Environment
Global State
Partial Obs
Single Interaction with Environment
Figure 1: The comparison between TSL and GPO.
Data Collection: Collect trajectories by executing the guider's policy, denoted as μ (1)),
Guider Training: Update the guider (A to p((*) according to RL objective Vy
Learner Training: Update the learner to (b-11) by minimizing the distance D( (k))).
Guider Backtracking: Set Set (+(++1))+ss))) - (-(o) for all states before the next iteration.
In the learner training step, D(T, (u) can be any Bregman divergence. For this work, we use the KL divergence, weighted by the state distribution dy. GPO iterates these steps until convergence, applying standard RL to train the guider while the learner seeks to mimic the guider's behavior. If the learner struggles due to discrepancies in observation spaces, the backtracking step adjusts the guider's 's policy to mitigate the imitation gap.
The comparison between TSL and GPO illustrated Fig. Several differences between the two frameworks exist. First, the teacher in TSL is typically provided or trained independently from the student, while GPO, the guider and learner are trained together. Second, TSL typically allows the student to interact with the environment, whereas GPO only uses guider, enabling more effective trajectory collection due to the behavioral policy being conditioned on privileged information. Lastly, and most importantly, TSL does not use the student constrain the teacher. This means that if the teacher too advanced for the student, the student will struggle learn from the teacher.
In contrast, GPO utilizes backtracking to guarantee the learner can effectively learn from the guider.
This is demonstrated by the following proposition:
Proposition If the guider's 's policy is updated using policy mirror descent in each GPO iteration:
ja arg min(-1p (VV (1) + D (1e, ()))
where Thk is the step size. Then the learner's policy update follows a constrained policy mirror descent: π (++)) arg minf (VV(
(T, x
#ell
Proof. See Appendix
Here, we assume that the guider y has access to an unlimited policy class, while the learner æ is constrained to limited policy class II for simplicity. Policy mirror descent (Tommr et al.) 2020 Xiao 2022) is general family of algorithms that encompasses a wide range fundamental methods in RL, including trust-region algorithms like TRPO (Schulman et al. 2015a) and PPO (Schulman et al.)
2017). This proposition shows that, despite the learner not directly interacting with the environment, standard RL update. Specifically, if we trust- region RL algorithms for the guider, the update for the learner's 's policy inherits the key properties, such policy improvement (Schulman et al. 2015a). This suggests that GPO can effectively address
the GPO update for the learner be viewed
4

---

Published as a conference paper at ICLR 2026
challenges in TSL, such as dealing with suboptimal teacher or the imitation gap, while still framing the learner's s policy as being supervised by the guider. In Appendix [ we provide an intuitive example illustrating how GPO can achieve optimal in the TigerDoor-al alt problem.
Given that GPO mirrors direct RL for the learner, one may ask: What are GPO's key advantages? The main benefit lies in leveraging additional information while simplifying learning. Since policy gradients suffer from high variance especially under partial observability, GPO splits learning into two phases: the guider with privileged information handles complex RL gradients, while the partial observable learner is trained via an easier supervised learning, reducing variance and complexity. For instance, to train robustness to noisy observations, GPO can train the guider on clean inputs and supervise the learner with noisy ones, resulting in a more stable and effective learning process.
3.2 GPO-PENALTY
This section introduces a straightforward implementation of the GPO framework using KL-divergence as penalty for the guider, which we refer to as GPO-penalty. Specifically, in step 2 of GPO, we use PPO as the underlying trust-region algorithm. The corresponding loss for the guider's policy is as follow
Porp ((s (s..))
(1( -E min
where p" = yu(a [[(]s)//((lss). Pltip = clip(p". 1rippp"11-ce0 advantage A a) is estimated using the Generalized Advantage Estimation (GAE) (Schulman et al. 2015b) with the value function V(s) trained via discounted reward-to-ge
- c, + e) and B denotes the behavioral policy. The
In step 3, since finding the exact minimizer of the distance measure is computationally prohibitive, we use gradient descent to minimize the BC objective:
C2(x) = E. Dxx. (n(11s), T (-l)))]-
Similarly, in step 4, we backtrack the guider's policy using the same BC loss:
(1) = E [ (=)11s),T((1)))
A key insight in GPO is that exact backtracking of the guider's 's policy is unnecessary it's sufficient to keep the guider within an imitable region relative to the learner. The learner may fail to follow the guider either because the guider is inimitable or because the guider learns faster, the latter being common due to inexact gradient updates. In such cases, aggressive backtracking can be harmful. Keeping the guider slightly ahead also allows it to collect better trajectories, as discussed in Section To maintain this balance, we introduce a coefficient a that modulates the guider's 's loss as
((p) (1()) + aLs(())
(5)
where is adapted based the distance T.3 ((4) relative to a threshold d, using constant scaling factor k:
a a= ka if Ca(n) > kd, a/k if Ca(N) < d/k.
(6)
This scheme is analogous to the KL.-penalty adjustment in PPO-penalty (Schulman et al., 2017). where the penalty coefficient adjusts based on the relationship between the KI. divergence and predefined threshold.
Another aspect is compensating for the leamerr' 's policy improvement, as we replace strict backtracking with a KL constraint. While it is possible to set a very small dungg this would inefficiently inflate a, hindering the guider's 's training. Notably, Proposition iimplies that applying GPO with PPO is effectively equivalent to applying PPO directly to the learner. Consequently, we can concurrently train the learner's 's policy using PPO during the GPO iterations. As a result, we introduce additional objective for the learner's policy:
(8..))) }
(a(T)) = -E min
where p" = ((alo)/B(ass). Considering that the behavioral policy is from the guider, to validate this update, we introduce the following proposition:
We omit subscripts for expectations in the remainder of the paper, all samples are drawn from the distribution induced by the hehavioral policy / = Mold-
5

---

Published as a conference paper at ICLR 2026
Proposition 2. For policy n, pl, 8 and all states 8, suppose Drr(u(-1s),3(11b) & c/2, then we have
Ea- [|] - p² (s, S S v 22targ
Proof. See Appendix
B
The assumption on total variation distance justified by the PPO update of the guider's policy (Appendix B). This proposition implies that when duny is small, the behavioral policy closely matches the learner's policy, allowing valid sample reuse for learner training.
Finally, we define the merged learner objective for the learner as:
(8) where the coefficient a from equation applied to the RL term. This mechanism compensates when the learner struggles to follow the guider. If the learner is able to fully track the guider, a approaches zero, allowing the guider to directly lead the learner to the optimal policy without requiring an additional RL objective. When the learner cannot keep pace, the RL. objective aids in the learner's training.
= = aC((T) + C2(T),
3.3 GPO-CLIP
In this section, we introduce a slightly modified implementation of the GPO framework, which we refer as GPO-clip. The key principle is that an effective guider should remain at the boundary of the learner's imitable region: if the guider is too far ahead, the learner struggles to follow; if too close, the guider's ssss ability provide effective supervision and better trajectory diminishes. To achieve this balance, the guider should halt updates when it moves too far ahead and avoid backtracking when it already sufficiently close.
We propose two modifications to the GPO-penalty algorithm introduced in the previous subsection. First, inspired by PPO-clip, we replace the clip function Platip in equation with the following double-clip function:
Peip clip clip( ((als) (aao))
A(alo) 3(a|s) -,, c,1+
8,1+
This formulation introduces additional inner clipping step. which halts the guider's updates under two conditions: (1) A (s, a) > 0 and p(als) > (a + 5), (2) (s. < and u(als) < (1 8). Considering that the positive (negative) advantage indicates that p(als) is set to increase (decrease), the double-clip function prevents further movement away from i when already distant.
It is important to note that, unlike PPO where PPO-clip can completely replace the KL-penalty term, this is the case in GPO. In PPO, the ratio p² (s, a) starts at the beginning of each epoch, ensuring that the clipped ratio keeps π near the behavioral policy. In GPO, however, the gap between (als) and μ(alo) may accumulate over multiple updates if the learner fails to keep up with the guider. The double-clip function galationnaaoone is insufficient to bring (alo) back into the 8 region once it has strayed too far. To address this, we introduce mask on the backtracking loss, defined as:
m(s a) = u(ass) a(alo) $ (1-8,1+6))
(10)
where I is the indicator function. This mask replaces the adaptive coefficient a of GPO-penalty, selectively applying the backtracking penalty only when (aa)oo drifts outside the ò region. Policies that remain close each other are left unaffected, preventing unnecessary backtracking.
Additionally, given that both the guider and learner are solving the same task, their policies should exhibit structural similarities. To leverage this, allow the guider and learner to share a single policy network. To distinguish between guider and learner inputs, we define unified input format: the input the guider's 'ss policy is defined as 0g = [s, o, 1). where s the state, is the partial observation, and the scalar serves as an indicator; the learner's 's input is defined as or = [ō. o, Oj. where Ō a zero vector with the same dimensionality 8, indicating that the learner has access only to the partial

---

Published as a conference paper at ICLR 2026
observation This approach is applied to both GeO-ttealty and GPO-clip, and the update for the shared policy network with parameters & is as follows:
pepnaly penalty (0)
min (pto A (0g: a), P A (0g: a) a Dua (()((()))) mT(-lor))
(11)
a min
a), Poii A (o,.
+ DEL ( no m
IGGPO.cli(0) (0) = I2 min (pro A a) Pelp
pa.56 A (0g,
+ m(s, a D (10 =z((o)))
min (og, Palsp A (o a))
+ DKL (ug (10g) mo (11)))
(12)
where á denotes nop-gradient operation on the parameters, and for GPO-clip a fixed parameter. The detailed algorithms are summarized in Appendix
4 EXPERIMENTS
In this section, we evaluate the empirical performance of GPO across various domains. For baselines, we consider two types of approaches for utilizing teacher supervision. The first type involves training both the teacher and student simultaneously. A summary of their
main characteristics is provided in Table 3 Among these, GPO-naive refers to GPO- penalty without the RI. auxiliary loss. PPO-asym directly trains the learner using PPO,
with the learner's s value function receiving Og input. PPO+BC trains the teacher with PPO, while the learner is trained via direct BC from the teacher. ADVISOR- co is a modification of ADVISOR (Weihs) al. 2024), and A2D is based the work by [Wariintonn et al., 2020). The second type involves training the teacher first, fol- lowed by the application of TSL methods.
Table 3: Co- Co-training algorithms.
Behavioral policy
Value function
Algorithm
Train μ
Train
PPO PPO-asym PPO+BC
(alo;
PPO
V(o)
{aao)
PPO
V(og)
PPO
m(alog)
BC
V
These include DA gger (Ross et 2011). PPO+BC-t, ADVISOR, ELF (Walsman ctt al. 2023). and E-F-as,m, where ELF a policy distillation method that utilizes reward shaping to provide supervision to the student, and ELF-asym is variant that uses an asymmetric value function. Further details about these algorithms can be found in Appendix E.
A2D
PPO
(alo
BC
V
ADVISOR-co
PPO
{(ao)
BC+PPO
V(oi)
GPO-naive
PPO
o(a
BC
V(og)
GPO-penalty
PPO
u(alog)
BC+PPO V
GPO-clip
PPO
p(a
BC+PPO V(og)
GPO-ablation
PPO
plalog))
PPO V(og)
4.1 DIDACTIC TASKS
GPO-penalty
GPO-clip
GPO-naive
PPO+BC
0.9
1.0
Reward
0.7
0.9
0.5
0.8
0
1M
2M
1M
2M
Steps
Figure 2: Results TigerDoor (left) and TigerDoor-alt (right).
We begin by evaluating our algorithm on two didactic problems introduced in Section 2.2 As shown in Fig. 2 direct cloning of the guider's policy converges suboptimal solution, expected. In
7

---

Published as a conference paper at ICLR 2026
C- 0.0
d 0.1
g=
e- 0.3
Ant
Halfchestah
HumansidStandup
Humanoid
Swimmer
We ke/dd
Nommalzee Baturn
Mastinnn
Hellonn
Figure 3: Comparison of GPO and baselines the Brax domain, where represents the scale of Gaussian noise added to the observations. The performance on each task is normalized to [0, 1 using the performance of the corresponding pre-trained teacher reference. Algorithms highlighted in red rely on a separately pre-trained privileged teacher.
contrast, all variants of GPO achieve optimal performance on these tasks. Although applying RL directly to the learner easily leads optimal solutions, is important to note that GPO-naive achieves optimality purely through supervised learning. This result verifies the optimality guarantee of the GPO framework described in Proposition [ suggesting that a guider constrained within the learner's imitable region can provide effective supervision, even with asymmetric information. Moreover, comparing GPO-naive to GPO-penalty and GPO-clip reveals that the introduction of direct RL training for the learner accelerates learning.
4.2 CONTINUOUS CONTROL TASKS IN BRAX
In this subsection, we present the results of our algorithms and baselines on several continuous control tasks in the Brax domain. To transform these tasks into a POMDP setting, we remove the velocity information of all joints, and add varying levels of noise to the observations. The guider has access to full, noiseless information, while the learner operates with partial and noisy inputs. For more details, please refer to Appendix E.
The results are shown in Fig. where the performance hierarchy is generally: GPO-clip > GPO- penalty > PPO-asym > GPO-naive > other baselines. important to note that, even without factoring in the cost of training the teacher (which is comparable to training GPO from scratch), methods that rely on a pre-trained privileged teacher perform well only in Halfcheetah and Swimmer tasks. Furthermore, the performance of these methods declines rapidly as the noise scale increases. This occurs because, when the pre- pre-traines trainrdnee teacher becomes too skilled for the student, provides little to no useful supervision, and may even have negative impact on learning.
For co-training approaches, we have the following observations: First, the superior performance of GPO-clip and effectively utilize additional information during training to facilitate the learner training. Second, comparing GPO-naive to GPO-penalty and GPO-clip, we see that introducing RL training for the learner improves performance. Third, the comparison between PPO+BC and GPO-naive highlights the necessity of backtracking. the guider is not constrained the learner, the guider's 's supervision may negatively influence the performance. Last, other baselines such as ADVISOR failed to utilize the privileged teacher such that it degenerates into pure PPO.
compared to the base algorithm PPO shows that this framework can
In summary, our method consistently outperforms the baselines, demonstrating its effectiveness in solving noisy and partially observable continuous control tasks. Additional experiments including L2T-RL (Wu et al. 2024), TGRL (Shenfeld et al., (2023)) and RMA (Kumar et al. (021) are provided Appendix
4.3
HEMORY-BASER TASKS IN POPGYM
Since using memory models to deal with POMDP is a common practice, we evaluate GPO in POPGym to show whether the algorithm effectively address memory-based tasks. The tasks
8

---

Published as a conference paper at ICLR 2026
include card and board games where agents must recall previous observations to extract useful information for decision-making critical information needed to remember, theoretically minimizing the imitation gap as long the memory model can store the necessary information. Although in practice, memory models struggle retain all information, especially in complex tasks, this setup allows us or clipping parameter, enabling the guider to explore further and provide more valuable supervision. Further details on the experimental settings are provided in Appendix
For these tasks, the guider's observation is designed to include the
larger KI. threshold
GPO penalty
GPO-cip
-00-asym
PPO
Aute
Dal any
CoumRecalliwy
Cartholeaayy
-0.00
-0.23
1.00
-0.00
0.34
0.00
0.70
0.39
Episode Reward
CountRoca Medium
Co CortFoleModuum um
Rep Rrviissrriis
-0.30
0.33
0.30
0.65
0.02
4.30
0.44
0.0
0.25
0.50
-0.30
0.40
-3.55
-0.11
0:30
-0.50
-0.48
0.20
1.00
200
100
200
100
200
Epoch
Figure 4: The results of GPO-clip, GPO PPO-asym, and PPO on 15 POPGym tasks.
Fig. shows the results on 15 POPGym tasks, where we compare GPO-penalty and GPO-clip to PPO-asym and PPO. The general conclusion mirrors the results from the previous subsection, where GPO-clip typically outperforms GPO-penalty, followed by PPO-asym and PPO. Key insights include: First, the superior performance of PePO-penalty indicates that the ability of the guider explore further without diverging too much from the learner proves valuable in these memory-based tasks. Second, while PPO-asym outperforms PPO, its performance improvement is less pronounced here than in the Brax domain, suggesting that asymmetric value function may not be very helpful for memory tasks. Third, although neither GPO-penalty nor GPO-clip exhibits superior performance in tasks like BattleshipMedium and CountRecallHard, this is due to the fact that we use the same parameter across all tasks, and performance could be improved as we show in the next section.
Overall, our methods demonstrate strong performance across the majority of tasks, providing an effective solution for memory- -based problems.
4.4
ABLATIONS AND DISCUSSIONS
In this section, we dive deeper into GPO's performance through ablations and further discussions.
Ant (full observable)
Ant (partial abserveble)
2e3
Humanold
AutoencodeEasy
Epizode Roward
Epicode Reward
Episode Reward
GPO stration
GrO-clp
pro-asym
ADVISOR
-0.3
PPC++ IC
PPO-BC
GP0-1p GPO-aalation
teacher
PPO ssym
0
100
2.0
Step
147
Step
Epoch
Figure 5: ADVISOR and PPO+BC with a pre- trained teacher.
Figure 6: Ablation studies.
Why does training a teacher first and applying TSL methods often fail? A representative example is the TogerDoor problem, where pre- trained teacher provides minimal no effective supervision for the student. Recent TSL approaches, such as ADVISOR and TGRL, address the challenge of an overly optimal teacher by reverting pure RI. thereby bypassing uninformative or misleading
9

---

Published as a conference paper at ICLR 2026
supervision. As shown in Fig. although ADVISOR and PPO+BC perform well in the fully observable Aut task where the teacher is trained, degenerates into PPO in the partially observable Ant task since the teacher is found inimitable.
Why do GPO outperform other baselines? We attribute the superior performance to two factors: effective RL training of the learner, and effective supervision from the guider. The benefit of RL training is shown in Fig. [left), where GPO-ablation (GPO-penalty without supervision, as described in Table 3) outperforms PPO-asym on the Humanoid task. Although both use similar objectives, GPO-ablation uses data collected by the guider, indicating that a better behavior policy improves learning efficiency. The effectiveness of the supervision comes from the guider being constrained to the imitable region while still learning rapidly. In Fig. [right), with the learner trained purely by supervision (GPO-clip with RI. disabled), GPO-clip outperforms GPO-ablation, PPO+BC. and PPO-asym. This shows that in memory- memory-intensive tasks, supervision is more beneficial than RI. Since PPO+BC performs poorly in noisy tasks in Section 4.2 but comparably to PPO-asym here, we can also infer that supervision plays particularly important role in these tasks. Moreover, GPO-clip's both using pure
strong performance over PPO+BC- of constraining the guider to policy the learner follow.
the importance
CountRecallEary
CountRecallHard
BattleshipEasy
BattieshipMedium
Epboie Reward
Epbcdde Reward
d= 0.003
0.0
-0.4
0:01
-0.
-03
0.3
d- 02
-0.8
PPO
-0.4
385
250
200
303
600
350
500
Epoch
Epoch
Epoch
Epach
(a) GPO-penalty with different KL-threshold.
(b) GPO-clip with different clip parameters.
Figure 7: The results of GPO-penalty and GPO-clip with different hyperparameters. The clip parameter p is defined in Appendix E.2
When does GPO fail? GPO can fail when the guider learns too slowly, often due to inadequate infor- mation. Another failure mode arises from poorly tuned KL thresholds (clip parameters). For instance, in the CountRecallHard task from POPGym, both GPO variants underperform compared to PPO and PPO-asym. As shown in Fig- 7 larger KL. thresholds help in simple tasks like CountRecallEasy and Batteshhipasy, but hurt performance in harder ones like CountRealllHard and BattleshipMediug This is because challenging tasks strain memory models like information, the learner cannot follow the guider. In such cases, a large KL threshold pushes the guider beyond the learner's reachable region, causing an unrecoverable imitation gap.
GRU fails retain key
CONCLUSION AND FUTURE WORK
In this paper, we introduce GPO, a method designed to leverage additional information in POMDPs during training. Our experimental results demonstrate that the proposed algorithm effectively ad- dresses noisy and memory-base partially observable tasks, offering a novel approach to utilizing auxiliary information for more efficient learning. Future work could explore extending guided policy optimization to the multi-agent setting, where agents often have access to global information during training but are constrained to local observations during execution.
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China [grant numbers U22A2062, U23B2037, 12272008, 62450001, 62476008).
REFERENCES
OpenAl: Marcin Andrychowicz, Bowen Baker, Macick Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray. Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning
10

---

Published as a conference paper at ICLR 2026
dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-23 -0, 2020. doi: 10.177/0273344919877777
Andrea Baisero and Christopher Amato. Unbiased asymmetric reinforcement learning under partial observability. arXiv preprint arXiv:2105. 16744, 2021.
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03077 2018.
Mohak Bhardwaj, Sanjiban Choudhury, and Sebastian Scherer. Learning heuristic search via imitation. In Conference on Robot Learning, pp. 271-280. PMLR, 2017.
Yang Cai, Xiangyu Liu, Argyris Oikonomou, and Kaiqing Zhang. Provable partially ob- servable reinforcement learning with privileged information. In A. Globerson, L. Mackey, D. Belgrave. A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neu- ral Information Processing Systems, volume 37, pp. 63790-63857. Curran Associates, Inc., 2024. URL htttps://proceeedingg neur cw/paperppppcc.cceeecce ff fttp_ffeaeeeeeee////// Conference pd
Kai-Wei Chang, Akshay Krishnamurthy, Hal Daumé, III, and John Langford. Learning to search better than your teacher. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2058 2066, Lille, France, 07-09 Jul 2015. PMLR. htRp:tt:::/// hattpss:/pprcceednngg. mllr press/v37 htt.///hhaggbl.ttttmmmm
Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 297-307. PMLR, 08-11 Nov 2022. httthhttt//////otttt
ngs m r.w.ss.prssse.. 64 / chwn22a. html
Sanjiban Choudhury, Ashish Kapoor, Gireeja Ranade, Sebastian Scherer, and Debadeepta Dey. Adaptive information gathering via imitation learning, arXiv preprint arXiv: 2017.
Wojciech M. Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1331-1340. PMLR, 16-18 Apr 2019, URL
ml http://nww......ee//////8///// wtteess/v9//caarreoe
19a. htm
Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Advances in neural information processing systems, 32, 2019.
Yonathan Efroni, Chi Jin, Akshay Krishnamurthy, and Sobhan Miryoosefi. Provable reinforcement learning with short-term memory. In International Conference Machine Learming, pp. 5832- 5850. PMLR, 2022.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-s differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.1 13281, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Ahmed Husscin, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps. In International conference on machine learning, pp. 2117- 2126. PMLR, 2018.
11

---

Published as a conference paper at ICLR 2026
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in
partially observable stochastic domains. Artificial Intelligence, 101(1):99-134, 1998. ISSN 0004-3702. doi: http:://bbboooo/loo///oo00000000 sciencedi rect com.com///aooo... httpe//www../eeeeeeea/aaaaaaa/a/
URL hetps: /ttp:://wwe /www
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.0400: 2021.
John Lambert, Ozan Sener, and Silvio Savarese. Deep learning under privileged information using heteroscedastic dropout. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8886-8895, 2018.
Jonathan Lee, Alekh Agarwal, Christoph Dann, and Tong Zhang. Learning in pomdps is sample- efficient with hindsight observability. In International Conference on Machine Learning, pp. 18733-18773. PMLR, 2023.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5(47) 2020a. doi:
10.11229sctrobbtccsabb66666 URL https: htnp:://wwer science org http://eeerrr/////////////////// http://oo//a///1.11//0 scirobotics.abc53
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning
quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc596,, 2020b.
Sergey Levine and Vladlen Koltun. Guided policy search. In International conference on machine learning, pp. 1-9. PMLR, 2013.
Michael L. Littman, Anthony R. Cassandra, and Leslie Pack Kaelbling, Learning policies for partially observable environments: scaling up. In Proceedings of the Twelfth International Conference on International Conference on Machine Learning, ICML'95, pp. 362-370, San Francisco, CA, USA, 1995. Morgan Kaufmann Publishers Inc. ISBN 1558603778.
Qinghua Liu, Alan Chung, Csaba Szepesvári, and Chi Jin. When partially observable reinforcement learning not scary? In Conference on Learning Theory, pp. 5175-5220. PMLR, 2022.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed processing systems, 30, 2017.
environments. Advances in neural information
Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning. arXiv preprint arXiv:2303.03982, 2023.
Omid Madani, Steve Hanks, and Anne Condon. On the undecidability of probabilistic planning and Moanite-horizon partially observable markov decision problems, Aaai/iaai, 10(315149.315395, 1999.
Lingheng Meng. Rob Gorbet, and Dana Kulić. Memory-based deep reinforcement learning for pomdps. In 202/ IEEE/RSJ international conference on intelligent robots and systems (IROS), pp- 5619-5626. IEEE, 2021.
William H Montgomery and Sergey Levine. Guided policy search via approximate mir- ror descent. D. Lee, Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.. 2016. URL httpp::/ppooeeeennnn...ouoe wwwp::/iii...eeienc.c.......ccce httpc//pppprffelleeeeeeeeeeeeeee f1se///pp///p////21//////// file 11l1:000050011222000322223633333
Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. POPGym: Benchmarking partially observable reinforcement learning. In The Eleventh International Confer-
ence Learning Representations, 2023. URL httpt::t/////.... https::/oppeeeee//////////
net. forum?id-
ChDrutUTsOK
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations:
Theory and application reward shaping. In Ieml, volume 99, pp. 278-287, 1999.
12

---

Published as a conference paper at ICLR 2026
Hai Nguyen, Andrea Baisero, Dian Wang, Christopher Amato, and Robert Platt. Leveraging fully observable policies for learning under partial observability. arXiv preprint arXiv:2211.01911, 2022.
Hai Huu Nguyen, Andrea Baisero, Dian Wang, Christopher Amato, and Robert Platt. Leveraging fully observable policies for learning under partial observability. In Karen Liu, Dana Kulic, and Jeff Ichnowski (eds.), Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Rexearch, pp. 1673-1683, PMLR, 14-18 Dec 2023. URL www.ppees//vvse. htttps:/prroceedinge wtt.:/aiinnnn.....p....///////// / html
Lerrel Pinto, Marcin Andrychowiez, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning. RSS, 2018.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning, pp. 1722-1732. PMLR, 2023.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, JMLR Workshop and Conference Proceedings, 2011.
Sasha Salter, Dushyant Rao, Markus Wulfmeier, Raia Hadsell, and Ingmar Posner. Attention- privileged reinforcement learning. In Conference on Robot Learning, pp. 394-408, PMLR, 2021.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/ 100.00572, 2015a.
05477
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. CoRR, 506.02438, 2015b. URL
https://api www.ntiisccc.ooooo
3075444
3075448
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/ abs/
1707.06347
Younggyo Seo, Junsu Kim, Stephen James, Kimin Lee, Jinwoo Shin, and Picter Abbeel. Multi-view masked world models for visual robotic mani manipulation. In International Conference Machine Learning, pp. 30613-3061 PMLR, 2023.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar. Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Iaan Shenfeld, Zhang- Wei Hong, Aviv Tamar, and Pulkit Agrawal. TGRL:
An algorithm for teacher guided reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31077-31093, PMLR, 23-29 Jul 2023a. URL https:://proceed.gga. ml press/ 2202/shenfeld23:5 html
Idan Shenfeld, Zhang- Wei Hong, Aviy Tamar, and Pulkit Agrawal. Tgrl: An algorithm for teacher guided reinforcement learning. In International Conference Machine Learning, p 31077- 31093. PMLR, 2023b.
Jialin Song, Ravi Lanka, Albert Zhao. Yisong Yue, and Masahiro Ono. Learning to search via self- imitation. CoRR, abs/1804 804.00846, 2018. httl:/ttpp.//axxi.ccrr///aa
00846
13

---

Published as a conference paper at ICLR 2026
Jialin Song, Ravi Lanka, Yisong Yue, and Masahiro Ono. Co-training for policy learning. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp. 1191-1201. PMLR, 22-25 Jul 2020. URL httt:tts::///tt httttpps:/proccceiingneoooooeeee ml wtt.ppree/sss/sssssepeeee httppp/ceeeooooo......eeeea/aaa/
html
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http:://incommpeeeeiiiee hwtps/eeeeeeooeoeeeeeeeeeeoeeeee book- 2nd
html
Voot Tangkaratt, Nontawat Charoenphakdee, and Masashi Sugiyama. Robust imitation learn- ing from noisy demonstrations. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceed- ings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 298-306 PMLR, 13-15 Apr 2021, URL a. html
httpp:://rrocees ngs m wtt.rpeeees/v////aaaeeeeeee htgp:///eeoooeeeeoooo...nooenean
Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzzdehe Mirror descent policy op- timization. CoRR, abs/2005.09814, 2020. /t/hat/pss//axxi...ooo////////// 09881000.0090..
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint frXiv:1805.015 2018.
Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged informa- tion, Neural networks, 22(5-6))544 557, 2009.
Aaron Walsman, Muru Zhang, Sanjihan Choudhury, Ali Farhadi, and Dieter Fox. Impossibly good experts and how to follow them. In International Conference on Learning Representations, 2023.
URL http:://ggg...aaaacccc.aaaaaaarp
Andrew Warrington, Jonathan Wilder Lavington, A. Scibior, Mark W. Schmidt, and Frank D. Wood. Robust asymmetric learning in pomdps. In Internatioona Conference on Machine Learning, 2020.
URL https:t//pp....... semant Leschhllllssc....c
Luca Weihs, Unnat Jain, lou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander Schwing. Bridging the imitation gap by adaptive insubordination. In Proceedings of the 35th International Conference Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 97817 778171384599..
Feiyang Wu, Zhaoyuan Gu, Ye Zhao, and Angi Wu. Learn to teach: Improve sample efficiency in teacher-student learning for sim-to-reat transfer, 2024. URL http:://axxv..r////// 2402. 06733
Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1-36, 2022. URL RRLLhttp:/// http::////.o...oorr/ppssss//////
Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. Embodied modti-modal agent trained by 1lm from parallel textworld, 2024. URL https: htap::://axxiix org/ abs/2311.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2022.
THE USE OF LARGE LANGUAGE MODELS (LLMs)
L.Mss used to polish the paper writing.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Although leveraging historical information has proven effective for solving POMDPs (Ig) al. 2018
Meng et al. 202T: Liu et al. 2022), additional information-ofier available in simulators during training- can be exploited to further aid learning. Leveraging additional information to accelerate learning in POMDPs has been explored across various frameworks and application domains (Vapnik) & Vashist 2009 Lambert et al. 2018 Lee et al., (023). A prominent line of research focuses on Imitation Learning (IL), where expert knowledge, often equipped with extra information, significantly enhances performance in practical domains like autonomous driving (Bansal ct al., 20118 De Haan al., 2019) and robot navigation and planning (Choudhury al.] 2017: Bhardwaj et al., 201 7). However, traditional IL methods such as Behavioral Cloning (BC) (Pomerleau) 1991 Torabi ct al.
2018) and DAgger (Ross et al.) T often lead to sub-optimal solutions in scenarios requiring active information gathering by the agent (Pinto et al. 2018 Warrington et al.) 2020).
To overcome these limitations, recent research has focused on hybrid approaches that integrate RL with II, often in the context of policy distillation (Czarnecki et al. 2019). For instance, (Nguyen ct al. 2022) modifies Soft Actor Critic (SAC) (Haarnoja et al. (018)) by replacing the entropy term with divergence mcasure between agent and expert policies each visited state. Similarly, (Weihs) ctt al. 2024) introduces a balancing mechanism between BC and RL training, adjusting based on the agent's ability mimic the expert. Additionally, (Walsman et al., (023) applies potential-baa -base reward shaping (Ng et al.] 1999) using the expert's ' value function to guide the agent's policy gradient, while (Shenfeld et al.) 2023b) augments entropy in SAC to blend task reward with expert guidance. where the balance is based on the agent's performance relative to reward-only learner.
Despite these advances, expert-driven approaches often assume access reliable expert, which may not be feasible when only supplementary information is available. This has led to a growing body of work on co-training approaches where the expert and agent are learned jointly, with the expert conditioned on additional information. For example, (Salter et al. (202T) proposes training separate policies for the agent and expert using spatial attention for image-based RI., aligning attention mechanisms through shared experiences. (Song et al.) (020) co-trains two policies, each conditioned on different information, and selects the most successful rollouts from both policies to guide subsequent learning via RL or IL. (Warrington et al. 2020) further develops this idea in adaptive asymmetric DAgger (A2D), where the expert is continuously refined through RL while supervising the agent. (Wu et al. 2024) also co-trains teacher and student, using the experience collected methods, a complementary approach involves embedding supplementary information directly into the value function within the actor-critic framework (Pinto et af. 20188] Andrychowicz et al. 2020 Baisero & Amato [202)) which also called asymmetric learning. This approach particularly useful in multi-agent settings where global information is naturally accessible (Foerster et al. 2018: Lowe et al., 2017 Yu et al. 2022).
by teacher to apply RL and BC for the student. Beyond expert-basee
Additional strategies include learning from noisy demonstrations (Tangkaratt et al. 2021). improving
via self self-correction from past trajectories (Song et al. (2018), and surpassing imperfect experts through regret-mininizzation frameworks (Chang et al.] 2015). Recent work also explores leveraging LLMs
as privileged experts, such as in embodied agents trained with reflective text-based guidance (Yang)
ct al. 2024).
Besides, there are also representation learning techniques provided in order to reconstruct the privileged information (or its latent representation) via partial observation. For example Sermanet et al. (2018): Seo et al. (2023) use multi-view setups (e.g.. image-based manipulation with additional camera views) to learn more informative embeddings. Others (Lee et al., 2020b; Salter et al.) 2021
Kumar al. 202 Qi et al. 2023) leverage privileged simulator states during training and design policies that operate on both observed and inferred states.
In our experiments, benchmark against several algorithms inspired by these lines of work, with detailed deseriptions of the baselines provided in Appendix E.
B OMITTED PROOFS
Proposition 1. If the guider's policy is updated using policy mirror descent in each GPO iteration:
fa arg min (minn-mm.0 (VV (v)((( (p(()),
+ 1 1-7 D Dusss (1, (())))
1
(13)
15

---

Published as a conference paper at ICLR 2026
then the learner's 's policy update follows constrained policy mirror descent:
((+1)) arg min =
1 (VV + D mn (1))
(14)
Proof. First, since D is a weighted sum of KI. divergence, it satisfies the definition of a Bregman divergence. Therefore, for any distributions p. 9 E ((4) we have
(p(p, 9) hy(p) (g) (Vh,(g).p-9) g)
(15)
where hp(p)= E d, Ps log Ps is the negative entropy weighted by the state distribution.
Next, by backtracking (A) to (8k) from the last time step, we get:
Thk (VV (N (1())) + 1ー D uaso (1, / (1)))
e = arg min
1-7
arg min
MJ (VV (TT (k)),
) 12)
+
1 2a (M, x1())
(16)
arg min
- (1-1))) - (VV))()))V(((())))))) σ) + ky (T) - (V ( (T( σ)
The optimality condition for requires:
-(1 7 ()(
(1) + Vhy
Why (ull)) =
(17)
where we use the fact that:
vpDx(p.g) = Vp)((p) -Vhh(()-Vpphh(000)
(18)
Now, consider the update of the learner's policy, which involves Bregman projection Pi:
((++1)_ (a) arg minD (K()) (TT, (.)
HEll
arg min min ( (ns (T) - (Vh ( (p), x²)
#Ell
arg min((hyu (TT) (V (Tr( +11 (1--7) 7) (1=1)))VV(x((*)) x3)
(19)
arg min - - 7m (VV (z)())) ), + h (T) - (Vi 10
well
arg min ( (VV (VV( x) + 1-7 ( x(4))}
#ell
This completes the proof.
Proposition For policy n, ps 3 and all state s, suppose Drv((u(1ss) S c/2, then we have
Ex- [[] p[]l-ps(sssa S 22harg-
(20)
Proof. First, let's examine the assumption (rv(u(-)s), B(-1s) S e/2 to check its validity.
Notice that at the start of each PPO policy update, the importance sampling ratio (* a) equals because the behavioral policy is equal to the policy being updated, i.c., {(ass) = u(als).
As PPO proceeds, (s, a) updated multiple times using the clipping function applied to p" a), i.c.. clip(p"(s,, a). 1-e, 1 ++ +c), only acttincttion pairs for which p"(s, a) E (1 (1-6. e, + e) get updated. Hence, in the early epochs of PPO, with properly tuned step size, we expect:
batch of samples. Due the
--, a)l S.
(21)
Now, recalling the definition of total variation (TV) distance:
1 Σ a
[u(als)- - 3(a[s) =
11/ e/2.
(22)
16

---

Published as a conference paper at ICLR 2026
This confirms that the assumption (iv(u(-)s), & e/2 is reasonable, especially for the first few policy updates.
By the triangle inequality for total variation distance:
Drv(a((lo),8((1b) - Drv(T((0)) - (rv(a(())),((11s))
(23)
we have
Drv(a(10),8((1s)) - V DRL + D
- dung + c/2,
where we use Pinsker's inequality to bound the total variation distance between я and y in terms of their KL divergence.
Finally, since total variation linked to the expected difference between probabilities under different policies, we have:
Ea-0 [11 [11- (s a) 2DTv E v 22harg
(24)
This result implies that, under the assumption, the majority of samples are valid for updating the learner's s policy during the early PPO epochs.
C
PSEUDO CODE
In this section, present the pseudo code of our algorithm (see Algorithm T. The algorithm is based on PPO, with an additional objective to leverage the extra information available during training.
Algorithm I: Guided Policy Optimization
Input: Initial policy parameters fo. value function parameters do
for & = 0, 1, .,..... do
Collect trajectory set D {Ti} by running guider policy μn uu-1opp (x);;));; Compute rewards-to-go Re;
Compute advantage estimates Å, using GAE w.r.t. current value function V
Update policy parameters Og 0c+1 by maximizing GPO objective 11] or 12
Fit value function parameters Φx+1 by minimizing mean squared error!!
T
Val(og))) - Ri.)
Φb+1 = arg min
D
TEDx 1=0
D GPO ON TIGERDOOR ALT PROBLEM
Table 4: TigerDoor-alt problem
action
a,
0.
state
31
2
0
SR
0
1
Here we provide an intuitive example to illustrate how GPO can achieve the optimal policy in the TogerDoo problem. At time step I, suppose the guider's and learner's policies
p+( (z)) (4(+))))) me (Iz, vt);
17

---

Published as a conference paper at ICLR 2026
where the two policies equal due to the backtracking from time step t 1. After update step, the guider's policy becomes:
(e(-1sL)= - (I) + Pt. Ve - Pt), Pe((1sR) = (2, + qe Vt - qc)
The key insight is that the higher reward for (SL, az) compared to (s aR) leads to larger gradient
step, implying Pt qr- The learner then imitates the guider, resulting in the updated policy:
P:
Pr 0t
T++1
(x+
2
Hence, (14) Te (a()),, meaning the leamner' 's policy moves closer the optimal policy (1, 0) in a monotonic fashion.
The critical mechanism is that actions yielding higher rewards induce larger updates in the guider's policy, which the learner then captures via imitation. Meanwhile, the backtracking step keeps the guider aligned with the learner, enabling stcady and consistent policy improvement.
E EXPERIMENTAL SETTINGS
E. BASELINES
In this section, we briefly introduce the baselines used in our experiments.
PPO. This is the standard algorithm used to train the agent without any additional information. The objective function given by:
((x)
Pas - E min (00, ()) (40
(25)
where the behavioral policy is 3 = Told-
GPO-naive. This variant of GPO uses the GPO-penalty without the auxiliary RL loss term. The objective function is:
LGPO-naive (0) -E min (0"e (o Pai A (o,, at) aDxL (Ho(+)os,,
(26)
DKL (Ma((1())))) To To no(-log))
GPO-ablation. This is another variant of GPO-penalty, but without the BC loss term. The objective is:
LcPo-ablation (0) -E min (p"o A ( u), del At (0g, a)) aDxi (noo
(27)
min (pra (o, a), Poli A A (o).
PPO-asym. This method trains the student using PPO, but with asymmetric value function taking og as input. The objective is:
((T) = E min (0"(o,, a)A (o,, a), Pasp e,)
a)) (o)
(28)
ADVISOR.
Given teacher's policy μ. ADVISOR (Weihs et al. 2024 uses balancing coefficient between BC and RL training, based on the distance between the teacher's policy μ and an auxiliary imitation policy f:
((T) -E u (cEE(((-og)))))))))))))))))))))) ((1o1)) (1 w) min (0
where w carpp-cle DaolDaDaaaaae (u(- og), # (-(o:))) and CE means cross-entropy.
ADVISOR-co. This is a modified version of the ADVISOR algorithm for co-training setting, as the original does not involve teacher training. The teacher's objective is:
((r) = -E min ( a) A (0,, a) P' ( a
(29)
18

---

Published as a conference paper at ICLR 2026
ADVISOR-Co can be viewed GPO-penalty without the backtrack term and with different update schedule. However, in the absence of backtracking, the coefficient w quickly diminishes, as the auxiliary policy cannot follow the teacher effectively, reducing this approach to pure PPO training for the student.
PPO+BC. In this approach, the teacher is trained using PPO:
((p) min (0 a) a) (og. a), (O ()) at
(30)
while the student is trained using BC with the teacher:
[(x)= = E DKi. (u(-)og); - (-1o))]]
(31)
PPO+BC-t.
Given teacher's policy ,a, the student is trained using combined loss of PPO and BC:
((T) = --[mmin min (o'(og, (0g a), Plaipplss ( a)
DRL
(32)
A2D. Adaptive Asymmetric DAgger (A2D) WWarrington et al. 2020) is closely related to GPO, as also involves co-training both the teacher and the student. A2D uses mixture policy 3(a og. 0r) Ap(alog) + (1 - (a)(a)or) to collect trajectories and train the expert p with a mixed value function Vog. 0:) (V,) (1 A)e (o:). The objective is:
C(u) -E min (o, a) (og, o1, a1,a), Pa, or, a, () ..)
(33)
while the student is updated through BC:
(TT) E (1(1))
(34)
In practice, A2D sets À = 0 or anneals it quickly for better performance. When = 0. A2D is equivalent to GPO-naive without the backtrack step, and it uses the student's behavioral policy a instead of the teacher's policy μr. While A2D implicitly constrains the teacher's policy through the PPO clipping mechanism (which prevents the teacher from deviating too far from the student's s behavioral policy), this is insufficient replace the explicit backtrack step. The gap between μ and can accumulate if the student fails to follow the teacher. Consequently, most samples will be clipped training progresses, leading A2D struggle in training strong teacher.
ELF. Given teacher's policy μ. ELF Distillation (Walsman et al. 2023 trains two policies jointly: a follower y mimic the teacher through BC:
[( = EE +
(35)
and a explorer nh trained through PPO:
(a, (oL, a)o(a))(oooo)) a), Peip (or, c)A(lo.a)) (...
((xe) = -E E min pT
(36)
To utilize teacher supervision, ELF applies potential-based reward shaping (Ng et al. 1999) r 7V" (df) - /* (or) to the explorer, where V*A the value function of follower. However, ELF needs to divide the interaction equally to train the follower and the explorer, which leads to inefficiencies.
ELF-asym. Since the follower is not required during execution, asymmetric value function v (og) is used instead of the original one. Although there are some performance improvement, ELF-asym still performs worse than PPO-asym due to inefficient experience usage.
L2T-PPO. Similar to PPO+BC, the teacher is trained using PPO:
C(μ) = E min (0 a) A (og, a), Patip (og a, <) e)
)
(37)
while the student is trained using combined loss of PPO and BC with the teacher:
((T) = - min (p'(og, a)A (o.. a), (og, Petp (og, c) ()) (0) (M())g))) (38)
where the behavioral policy 3 p.
19

---

Published as a conference paper at ICLR 2026
E.2 HYPERPARAMETERS
The experiments in Sections and 4.3 use the same codebase from (Lu et al. 2023). The hyperpa-
rameters for these experiments are listed Table 5 For GPO-clip. due to the asymmetry with large replace the clip(# 6, 1 5) with ( p) p) in the POPGym tasks.
For the experiments in Section 4.2 hyperparameter search for the original versions of the tasks and then fix the same hyperparameters for the partially observable and noisy variants. The hyperparameter search is detailed in Table and the selected hyperparameters for the experiments are provided in Table Other fixed hyperparameters are listed in Table H
the codebase from (Freeman al. 2021 We perform
All algorithms in Brax are run with 10 random seeds, whereas those in POPGym use 3 seeds, as the latter exhibits lower variance. Reward curves in this paper report the mean and standard deviation across these runs.
Table 5: Hyperparameters used in TigerDoor and POPGym.
Parameter
Value (TigerDoor)
Value (POPGym)
Adam Learning Rate
5e-5
5e-5
Number Environments
64
64
Unroll Length
1024
1024
Number of Timesteps
2e6
15e6
Number of Epochs
30
30
Number Minibatches
8
8
Discount
0.99
0.99
GAE X
1.0
1.0
Clipping Coefficient
0.2
0.2
Entropy Coefficient
0.0
0.0
Value Function Weight
1.0
1.0
Maximum Gradient Norm
0.5
0.5
Activation Function
LeakyReLU
LeakyReLU
Encoder Layer Sizes
128
[128,256]
Recurrent Layer Hidden Size
256
Action Decoder Layer Sizes
128
[128,128]
Value Decoder Layer Sizes
128
[128,122
KL Threshold d
0,001
0.1 (0.001 for CartPole)
Clip RL Coefficient
1.1
10 (1.2 for CartPole)
1
(1 for CartPole)
Table Sweeping procedure in the Brax domain.
Parameter
Value
Reward Scaling
[0.1, 1]
Discount y
[0.97, 0.99, 0.997]
Unroll Length
[5, 10, 20]
Batchsize b
[256, 512, 1024]
Number of Minibatches
[4, 8, 16, 32]
Number of Epochs Entropy Coefficient
[2, 4, 8]
[0.01, 0.001]
KL Threshold d
[0.01, 0.001]
Clip 5
[0.1, 0.3]
RL Coefficient or
[0, 2,
E.3 ENVIRONMENT DESCRIPTIONS
We provide brief overview of the environments used and the guider's observation settings.
20

---

Published as a conference paper at ICLR 2026
Table Adopted hyperparameters in the Brax domain.
Notations correspond to Table
Task
Y
b
C
d
Ant
0.1
0.97
5
1024
32
4
0.01
0.001
0.3
2
Halfcheetah
0.99
5
512
0,001
0.001
0.1
Humanoid
0.1
0.99
5
512
32
4
0.01
0.001
0.1
2
HumanoidStandup
0.1
0.99
5
256
32
0.01
0.001
InvertedDoubleeennbullm
0.997
20
256
8
4
0,01
0.001
0.1
Swimmer
0.997
5
256
32
0.01
0.001
0.3
Walker2d
0.99
5
512
32
4
0.01
0.001
0.1
Table Common hyperparameters used Brax domain.
Parameter
Value
Adam Learning Rate
3e-4
Number of Environments
2048
Episode Length
1024
Number of Timesteps GAE
3e7
0.95
Clipping Coefficient
0.3
Activation Function
SiLU
Value Layer Sizes
[128. 128]
Policy Layer Sizes
[128, 128]
Brax tasks and CartPole in POPGym: For these tasks, velocities and angular velocities removed
from the learner's observation. Gaussian noise with standard deviations of 0.1, 0.2, and 0.3 is added the observations, corresponding the difficulty levels Easy, Medium, and Hard, respectively. The guider, however, has access to the noiseless observations and the removed velocities.
During the WATCH phase, deck of cards shuffled and played in sequence the agent with the watch indicator set. The watch indicator is unset at the last card in the sequence, where the agent must then output the sequence of cards in order. The guider directly observes the correct card to be output at each timestep.
Autoencode:
Battleship: partially observable version of Battleship game, where the agent has no access the board and must derive its own internal representation. Observations contain either HIT or MISS and the position of the last salvo fired. The player receives positive reward for striking a ship, zero reward for hitting water, and negative reward for firing specific tile more than once. The guider has access to recorder that tracks all previous actions taken by the agent.
Count Recall: Each turn, the agent receives next value and query value. The agent must answer the query with the number of occurrences of a specific value. In other words, the agent must store running counts of each unique observed value, and report specific count back, based on the query value. The guider directly observes the running counts at each timestep.
At the first timestep, the agent receives of four values and remember indicator. Then it randomly receives one of the four values at each successive timestep without the remember indicator. The agent is rewarded for outputting the observation from constant k timesteps ago, ie. observation 0 at time t. The guider has direct access to the value ot-A at time t.
Repeat Previous:
E.4 ADDITIONAL COMPARATIVE EXPERIMENTS
Here, we present additional baselines that were not included in the main paper. First, I.2T-RL tackles problem similar to ours by employing fully observable teacher to supervise partially observable student. However, L2T-RL resembles PPO+BC, as it uses the teacher purely as behavioral policy without aligning it with the student's policy, which results in ineffective supervision. Moreover, L2T- RL relies on the teacher's experience to train the student through RL without any offline adaptation, further limiting the effectiveness of the RL training. Fig. 8 iiillustrtts the performance of the PPO- based L2TT-L (details Appendix E.T). where L2T-PPO performs similarly to PPO+BC and falls short of the other methods proposed in our work.
21

---

Published as a conference paper at ICLR 2026
GPO clp
GPO penalty
PPO-V
GPO nalve
PPO
co
A20
PPO-BC
LET PFO
A16
hallchettah
Humanood
Humemoebtardap
5200
2000
5050
34600
Episode Reward
1800
3903
59200
-200
-800
8020
Step (1e7)
Figure 8: Performance comparison on selected Brax tasks, including L2T-PPO.
Second, we provide a more detailed comparison with methods that train a teacher first and then apply TSL techniques address the challenge of an inimitable teacher. Two state-of-the-art approaches in this category are ADVISOR and TGRL, which are based on PPO and SAC, respectively. We evaluate these methods using PPO-trained teacher with full observability on the Ant task. The results, shown in Fig. [9] indicate that while both methods perform well and demonstrate improved efficiency over their respective base algorithms under full observability, their performance degrades in the partially observable Ant task. In this case where the teacher's policy is effectively inimitable- the TSL. methods perform comparably to their base RI. algorithms.
Fig. shows the KL divergence between the agent policies of these TSL methods and the teacher. Under full observability, where the teacher was trained, the agents can successfully mimic the teacher's policy. However, under partial observability, the agents struggle to imitate the teacher's s behavior, leading to a substantial KL divergence. Since both ADVISOR and TGRL revert to standard RL when the teacher becomes inimitable, this explains their performance similarity to the base algorithms in such scenarios.
Additionally, we report TGRL's performance across the 28 Brax tasks used in our experiments (see Fig. [Tp. Note that TGRL follows an off-policy training paradigm, in contrast to all other methods presented in the main paper, which makes it significantly slower (refer to Table(9). Therefore, TGRL for only IM steps, which is sufficient for convergence as shown by the learning curves.
Finally, we include the representation learning method RMA (Kumar et al. 2021), which aims to reconstruct the latent privileged information used by the teacher during the student training phase. Such representation learning approaches are promising when privileged information can be reliably approximated from partial observations. However, their effectiveness is limited in Brax, where observations are noisy (Figure TII. Since the noise cannot be removed by a deterministic mapping without additional structure or assumptions, regression-bastt identity mappings and fails to recover meaningful latent representations. Moreover, current encoder- based pipelines often lack theoretical guarantees for convergence or generalization across diverse tasks, particularly under partial observability.
reconstruction tends to collapse to
163
Ant (full observable)
Ant (partial observable)
(full observable)
(partial observable)
Episode Rewardd
Episade
ADVISOR
PRO+BC
TGRL SAC
PPO
teacher
3
Step
Stop
Stop
Stop
Figure 9: ADVISOR, PPO+BC and TGRL with pre-trained teacher.
E.5
ADDITIONAL FIGURES
Fig. 12 shows the reward curves of the experiments presented in Section Fig. 13 illustrates the performance influenced by the parameter sharing. We can observe that parameter sharing can sometimes impair performance, particularly when the observation dimension is large. For instance,
22

---

Published as a conference paper at ICLR 2026
Ant observable)
(partial obsenvable)
(full observablel
(portial observable!
PPO+8C
1.3
pivergence
0.5
0.0
Step
346
Figure 10: The KL divergence of ADVISOR, PPO+BC and TGRL with pre- pre-trained teacher.
the HumunoidStandup task, the observation dimension 400, which challenges the expressive capacity of the network. Thus, the decision to share the policy network represents trade-off between memory efficiency and performance.
E.6 COMPUTATIONAL COST
In this section, compare the computational cost of GPO (both GPO-penalty and GPO-clip share the same cost), PPO-asym, TGRL and the environmental step time across several environments. The results, presented in Table 9 show that GPO is approximately 10% 20% slower than PPO-asym. Importantly, GPO achieves this with no additional networks, underscoring its efficiency despite the modest increase in computational overhead.
Table 9: Frames Per Second (FPS) of GPO, PPO-asym and TGRL across several environments, computed on the NVIDIA GeForce RTX 4090.
Environment
GPO
PPO-asym
TGRL
Environmental Step
Ant Halfcheetah
1.19 x 10"
1.36 x 10
1.13 x 102
4.23
10
6.27 10a
7.21
10a
9.58 10a
2.55 10
Humanoid
6.29
10a
7.18
10a
9.92 x 10a
2.50
10
Swimmer
3.33 x 10
3.83 x 10*
1.04 x 10
1.50 x 10
23

---

Published as a conference paper at ICLR 2026
TGRL
RMA
Ardlom0)
Antio- 0.11
Arkion 0.21
Antip - 0.31
4000
4000
4000
4000
2000
2000
0-
0
0
halthheetnioomoo 0.1)
Halchwetab(o m02)
Halcchetannom 0.31
2000
2000
2000
2000
1000
1000
1000
1000
0
Humansid(a- 0t
Humaroidio 0.1)
Hamaroisfe 0.21
Humarcidlo - 0.31
3000
3000
3000
3000
2000
2000
2000
2000
1000
1000
1000
1000
HumareidSSaa-auu - 0)
HumanaldStandupien 0.11
Humancid Statandpppo 0.2)
Slddann upto 0.31
Episode Reward
20000
20000
20000
20000
15000
15000
15000
15000
10000
10000
10000
10000
Inver Invertedi deuubl Pendulmmio 0.1)
Tmwntedooualeerrduuaannn 0.20
DewatteDeudiieenrduuuu 0.3)
2000
2000
2000
2000
1000
1000
1000
1000
0
0
Swimmario 0)
Swimmeria 0.11
Swimmerta 02)
Swimmeria 0.3)
50
50
50
50
25
25
25
Walker2d(o
Waker2did 0.11
Walker2d 0.2)
Walker 20jo 0.3)
4000
4000
4000
4000
2000
2000
2000
0
0
0.5
0.5
Step (1e6)
Figure 11: TGRL and RMA with pre-trained teacher on 28 Brax tasks.
24

---

Published as a conference paper at ICLR 2026
penally
PPO-asy asym
naive
AndLo
Andio
0.31
3302
2000
2350.
3400
2100
1750
1450
-200
-300
-350
-350
tchhetahlp 0)
Ha tchetthlia
Hatrcheetarkom
Hallcheetahig- 0.31
1450
1302
1020
700-
400
250
100
550
Humanaistizso
Humanoister 0.11
Humonoidlo
Humanoosia
3850
4000
4520
3600
1950
1650
1520
1200-
Hamanoid
Humanoid 0.1)
HumaroidSanndullo- 0.2)
Reward
39250
32800
20800
23000
23600
19400
18400
17500
8000
7000
7000
WNettcdDoubbPeenuuuutt
certodDoubbFeencuuutt 0.11
weettssDouBkePnnd. umio 02)
InvottsDDuDeeeraaluliie 0.3)
4230
1980
1350
10m0-
1410
660
450
300
Swimmer(a 0)
Swimmer(a 0.1)
Swimmerlg
Swimmerfo
60
62-
30-
24-
WalterZakam 0)
Wal 0.21
WalcerZeto 0.3)
1980
1980
1530
1350.
660
662-
530
450
Step (1e7)
Figure 12: Comparing betweetn GPO and other baselines on 28 Brax tasks.
25

---

Published as a conference paper at ICLR 2026
shared actor
separated
Antlo
Antlo 0.31
Artio = 0.21
0.33
$150
3600
3000
2700.
1450
2402
2000
1000
1730
Ha Ichhestah(o-- 0)
Ha Ichhettahio-- 0.11
Hairchestako--022
Hallchoetario= 0.31
3000
2400
21.00
1500
1600
1400
1000
Episode Reward
1000
802-
500-
Jumanoi:Sannduploo
HumanodSannappoom 0.1)
HumaroifSinnduppo 0.21
56000
47000
44000
41000
40000
34009
24000
21000
20000
15000
8020
5000
8000
DovertddDobb PPnnnuuuoo 0)
IvartedDoub Pennouuuooo 0.11
WetterDDuakkPend uumo 02)
invortsDDoueePnnaallld =
3120
2100
15.0
1200
3750
700-
Step (1e7)
Figure 13: Comparing shared and separated policy networks of GPO-penalty.
GPO-1p
Halfcheatah
HumanokcStandup
Humanald
Swimmer
Episode Reward
4.8
2.6
3.5
1.4
0.0
-1.0
Step (1e7)
Figure 14: Comparing joint and separated update of GPO-clip.
guidar
HumanoicSSanndup
Humanaid
Swinmer
Episode Reward
1.a
2.6
0.0
1.0
1.0
Step (1e7)
Figure 15: Comparing the performance of guider and learner of GPO-clip.
26


Reviews:
[{"review_id": "WWGyXwGz9U", "reviewer": "Reviewer_wc4T", "summary": "This paper introduces Guided Policy Optimization (GPO),  a novel framework for Reinforcement Learning in partially observable environments. GPO co-trains a guider (using privileged information) and a learner (using partial observations and imitation learning) while employing an alignment step that constrains the guider to remain imitable by the learner. This mechanism ensures the guider acts as a \"possibly good\" teacher, significantly outperforming existing methods on continuous control and memory tasks.", "questions": "1. In Equations 11 and 12, are the advantage functions of the guider and learner the same since they both use the same trajectories and value functions? \n2. In Section 4.3, the input of the learner is the whole memory, while the input of the guider only keeps the important information, and is it correct? \n3. The paper aims to learn a \"possibly good\" teacher. Can the same be achieved by carefully tuning the update rate of the PPO+BC method? For example, restricting the teacher update size and increasing update steps for the student. \n4. Line 276, what does the exact backtracking mean?\n5. For Equation L1-L4, L1 and L3 are maximisation, and L2 and L4 are minimisation? It would be clearer if they represent the same.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The idea of the guider's backtracking step is novel and interesting. It interprets the principled solution to the \"impossibly good\" teacher problem. By actively keeping the privileged guider's policy within the learner's reachable policy region, GPO ensures that the supervision remains beneficial. \n- The implementation of the GPO method is quite clever, with additional state and observation and incorporated into PPO policy improvement. \n- The method is rigorously tested across a diverse set of tasks, including challenging continuous control (Brax) with partial observability and noise, and various memory-intensive tasks (POPGym), showing consistent and significant improvements over strong baselines.", "weaknesses": "- The main concern of the paper is the mismatch between the theory part and the practical implementation. For example, in Section 3.1, the GPO iteration is performed in four steps in sequence, but the actual loss of the 4 steps is combined in one policy improvement step in PPO, which might result in an optimisation issue. Besides, the use of L4 loss is not quite obvious, since the GPO iteration should work even if the learner doesn't have a value function. \n- There are some unclear design choices. For example, the paper focuses on the partially observed setup. In Section 2, the learner's policy depends on the trajectory, while in the following sections, the input is only a one-step observation o. In theory, it doesn't include enough information to solve the POMDP. Besides, the use of the memory-based experiment seems a little unmotivated. The idea should be using memory-based models like RNN to solve PODMP instead of on memory-based tasks, if I understand it correctly? It would be helpful to provide more explanations on these choices. \n- Some additional visualisations might help the understanding of the GPO mechanism. For example, the tunable parameters $\\alpha$ during training, and the performance gap between the guider and learner during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z4ORCcrI9n", "reviewer": "Reviewer_5vZi", "summary": "The paper discusses the learning of RL agents in POMDPs environment with access to a teacher's partial policy that might have used privileged information (modeles as full state observability). The method consists in iterations of teacher learning and guided student policy learning, where the learned student policy is used as guidance for the next step. Experimental results with a toy problem as well as with simulated and challenging environments show the superior performance of the proposed method (often achieving higher cumulative rewards and faster convergence) compared to SoTA approaches.", "questions": "How does the proposed model compare to the alternatives in terms of computational cost? Having two separate and intertwined learning seems to add significant computational overhead.\n\nThe high-level description of the algorithm suggest that the learned policy depends only on the current observation, which would be very suboptimal for a POMDP. Would learning from a sequence of observations and actions work just the same? What policy class was used in the experiments (in terms of input)?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Well-written, clear writing\n- Relevant problem\n- Good empirical support\n\n\nI can't vouch for its originality though as I'm not well familiar with the literature", "weaknesses": "- Despite Theorem 1, the proposed method is mostly heuristic, and good empirical performance might not replicate to other", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vjCgnb0eaH", "reviewer": "Reviewer_h6Kv", "summary": "This paper proposes a Guided Policy Optimization (GPO) algorithm that jointly trains a teacher and a student policy to mitigate the traditional imitation gap in reinforcement learning (RL). The approach draws an analogy to active learning. Experiments conducted on both didactic and locomotion benchmark tasks demonstrate empirical improvements over standard on-policy RL baselines, particularly in environments with partial or noisy observations for the student.", "questions": "**Q1.** How robust is the proposed algorithm to different configurations of the predefined thresholds used during training?\n\n**Q2.** In most IL or TSL settings, the teacher is a fixed pretrained model (e.g., an expert policy or human demonstrator). What realistic scenarios would allow both teacher and student to be learnable models, with the teacher having privileged information?\n\n**Q3**: How applicable would this method be given different backbone RL algorithms that are on-policy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\\+ The idea of co-training a teacher and student policy is intriguing. \n\n\\+ The proposed method is technically solid. \n\n\\+ Experiments seem comprehensive on SOTA locomotion tasks.", "weaknesses": "- The training process involves hyperparameters and seems intricate to tune. \n\n- Frequent heuristic descriptions: Terms such as *“possibly good region,” “impossibly good,”* and *“inimitable”* require clearer definitions. I would suggest that authors provide a more formal characterization of these concepts.\n\n- A few claims may not be factual: \n\n  * Authors used TSL to refer to IL and policy distillation \"as there is no fundamental distinction between them.\" (Line 82). However, IL usually refers to the case where no environmental rewards are available during learning. \n\n  * Crucially, none of the existing methods provides a theoretical guarantee that teacher supervision will actually be beneficial. (Line 119 - 120)\n\n- Fasibility of the core assumptions: (see Q2) The assumption that a learnable teacher model with privileged information can co-train effectively with the student (see Q2) may be restrictive in practice.\n\n- The method is evaluated only with PPO as the underlying RL algorithm, primarily due to its trust-region design (see Q3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uZe31oJhUV", "reviewer": "Reviewer_uwMy", "summary": "The paper proposes Guided Policy Optimization (GPO) for POMDPs with privileged information available only during training. GPO co-trains a guider and a learner, coupled by two mechanisms: (i) backtracking/alignment that constrains the guider to remain within a “possibly good” region trackable by the learner, and (ii) optionally adding RL updates to the learner to compensate when imitation lags. Empirically, the paper reports consistent gains over PPO, PPO+BC, A2D/ADVISOR-style teacher–student methods, and other variants across didactic settings, Brax continuous control with partial observability and added noise, and POPGym memory tasks, with ablations for thresholds/clipping and alignment strategies.", "questions": "See Weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper excels at motivating the work. The \"impossibly good\" teacher is a memorable concept, and the TigerDoor example  perfectly illustrates the concrete failure mode of naive TSL.\n\n2.Proposition 1  provides a strong theoretical foundation for the framework, giving confidence that the learner's imitation objective is not just a heuristic but is equivalent to a constrained RL update.", "weaknesses": "1.The strongest results rely on GPO-penalty/clip, where the learner uses a hybrid RL+IL objective. Current evidence does not isolate how much of the gain derives from constraining the guider versus adding RL to the learner. Controlled experiments (fix guider, swap learner objective; and vice versa) are needed to separate effects.\n\n2.While the mechanism is novel, the high-level concept of co-training a privileged teacher and a partially observable student is not. The paper does a good job arguing against prior work like A2D in the appendix , but the conceptual leap is more of a significant and highly effective refinement than a completely new paradigm.\n\n3.The failure mode analysis in Section 4.4 is honest but reveals a key weakness. GPO's performance is sensitive to the KL threshold or clip parameter. In difficult memory tasks (e.g., CountRecallHard), a poorly tuned threshold causes GPO to underperform PPO. The authors explain this is because the learner (GRU) fundamentally cannot store the information, making the guider truly inimitable. This is a valid explanation, but it implies GPO does not fully solve the imitation gap so much as manage it via a new, sensitive hyperparameter that may require careful task-specific tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)