Uncertainty-Sensitive Privileged Learning
Authors: Fan-Ming Luo
Lei Yuan
Yang Yu
Venue: NeurIPS 2025 poster ()
Keywords: Imitation Gap
Reinforcement Learning
Privileged Learning
Teacher-Student Learning
Primary area: reinforcement_learning

Abstract: Privileged learning efficiently tackles high-dimensional, partially observable decision-making problems by first training a privileged policy (PP) on low-dimensional privileged observations, and then deriving a deployment policy (DP) either by imitating the PP or coupling it with an observation encoder. However, since the DP relies on local and partial observations, a behavioral divergence (BD) often emerges between the DP and the PP, ultimately degrading deployment performance.  A promising strategy is to train a PP to learn the optimal behaviors attainable under the DP’s observation space by applying reward penalties in regions with large BD. However, producing these behaviors is challenging for the PP because they rely on the DP’s information-gathering progress, which is invisible to the PP. In this paper, we quantify the DP’s information-gathering progress by estimating the prediction uncertainty of privileged observations reconstructed from partial observations, and accordingly propose the framework of Uncertainty-Sensitive Privileged Learning (USPL). USPL feeds this uncertainty estimation to the PP and combines reward transformation with privileged-observation blurring, driving the PP to choose actions that actively reduce uncertainty and thus gather the necessary information. Experiments across nine tasks demonstrate that USPL significantly reduces the behavioral discrepancies, achieving superior deployment performance compared to baselines. Additional visualization results show that the DP accurately quantifies its uncertainty, and the PP effectively adapts to uncertainty variations. Code is available at https://github.com/FanmingL/USPL.

Uncertainty-Sensitiv Privileged Learning
Fan-Ming Luo Lei Yuan Yang Yu"
National Key Laboratory for Novel Software Technology, Nanjing University, China School of Artificial Intelligence, Nanjing University, China
Polixir.ai
luofm@lamde nju. edu, en, yuan1 nju. edu. en, yuyenju edu
Abstract
Privileged learning efficiently tackles high-dimensional. partially observable decision-making problems by first training a privileged policy (PP) on low- dimensional privileged observations, and then deriving deployment policy (DP) either by imitating the PP or coupling it with an observation encoder. However, since the DP relies local and partial observations, behavioral divergence (BD) often emerges between the DP and the PP, ultimately degrading deployment per- formance. A promising strategy is train PP to learn the optimal behaviors attainable under the DP's observation space by applying reward penalties regions with large BD. However, producing these behaviors challenging for the PP be- cause they rely on the DP's iirformmttiongaaterinnn progress, which is invisible to the PP. In this paper, quantify the DP's anformation-gatheriii progress by estimat- ing the prediction uncertainty of privileged observations reconstructed from partial observations, and accordingly propose the framework of Uncertainty-Sent -Sttiivvve Privileged Learning (USPL). USPL feeds this uncertainty estimation to the PP and combines reward transformation with priviegedd-observing PP to choose actions that actively reduce uncertainty and thus gather the necessary information. Experiments across nine tasks demonstrate that USPL significantly reduces the behavioral discrepancies, achieving superior deployment performance compared baselines. Additional visualization results show that the DP accurately quantifies its uncertainty, and the PP effectively adapts to uncertainty variations. Code available at httph:t///// /gggttb....u..g. com cwm/ htt.//aaamnnnUUSP...
blurring, driving the
(a) Target Reaching Task
(b) Vanilla
(c) RewReg PL
(d) Uncertainty- Sensitive PL (ours)
S
optimal policy
target
High uncertainty
Low uncertainty
Uncertainty reduce
Inconsistent behavior
Consistent behavior
Action
Figure 1: Policy comparison among different privileged learning (PL) methods on the Target Reaching task. The privileged policy (PP) observes both agent and target positions, whereas the deployment policy (DP) sees only the agent's position. (a) Task setup: the target randomly appears one of two locations, with a slight leftward bias. (b-d) PP and DP behaviors under Vanilla PL (b), Reward-Reguland PL. (c), and
Snvertainty-Sensiitie PL (d); in (d), uncertainty falls as the agent approaches the boundary, and the policies adapt accordingly.
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Real-worid decision prcciion-mmaiinge is typically partially observable and often relies on high-dimensional
visual observations, such as in autonomous driving [1. quadruped locomotion [2 3 and household
robotics [5], all of which pose significant challenges to reinforcement learning (RL) 6 introducing
substantial instability and inefficiency 87,899 9 The privileged learning framework 10) mitigates
these challenges by leveraging -in-demnsiooall privileged observations either complement missing
state information [1 12 13 14] or replace high-dimensional observations 15). thereby enabling
efficient training of privileged policy (PP). The resulting PP is then leveraged to construct
deployment policy (DP), either by training the DP to imitate the PP [16] or by combining the PP with an observation encoder that predicts privileged observations [3] Since the DP does not have access to privileged observations, the asymmetric observation spaces between PP and DP often result in behavioral discrepancies that degrade deployment performance [T). As illustrated in Fig. T in a Target Reaching task, the agent is required to navigate to a hidden target location based solely on partial observations. The PP has access to the full target position, while the DP only observes the agent's ss location. In Fig. [b), with the vanilla method, the PP can learn to successfully reach the target under both target position settings based on privileged observations. However, the behavior of the PP cannot be replicated by the DP, as the DP lacks information about the target's 's location. Simply imitating the PP's behavior results in suboptimal performance.
To reduce the behavioral divergence (BD) between PP and DP, two main approaches have been pro- posed. The first improves DP using either via RL-based fine-tuning [1]] or joint optimization of imitation and RL objectives [7 T6 19). While effective in low-dimensional or mildly asymmetric cases, these methods struggle in high-dimensional tasks with large BD due to RL instability, leading to inefficient training. The second approach targets the PP, making it easier to imitate 120 2, 21].
Directly training a PP to produce the best behavior achievable within the DP's observation space (the optimal behavior that can be replicated by the DP) via reward regularization 220 is promising and scalable direction, as it avoids exploration and RL on the DP. However, the optimal behavior of the DP may depend on its progress in gathering privileged information. For example, the optimal policy in Fig. [Ta initially moves to the left until it reaches the left boundary, and if the episode has not ended, indicating that the target is the right, it then turns back. Such behavior is actually reliant on its progress in gathering privileged information. When is impossible to infer privileged observations based on its historical observations, the DP executes fixed exploration behavior (moving left). Once it reaches the left boundary, the DP can infer that the target must be on the right, and it will then move towards the target. However, the PP cannot know the progress of gathering privileged information through its instantaneous observations, making it challenging for the PP to learn the
optimal behavior of the DP. As result, although Reward Regularization PL (Fig. [c)) maintains behavioral consistency between the DP and the PP, it still fails to achieve both objectives. Using
memory-baset architectures such RNNs 22 2 or Transformers
(24) may help, but these methods
suffer from inefficient training and require careful hyperparameter tuning 25 26).
In this paper, propose the framework of Snccrtainty-Senssidee Privileged Learning (USPL). employs an observation encoder to reconstruct privileged observations from the DP's partial observations and then estimates the uncertainty of these reconstructions, which quantifies how much information still needs to be gathered. USPL augments PP's observation with the estimated uncertainty, and further leverages reward transformation and privileged blurring incentivize the PP to produce behaviors that actively gather information by minimizing uncertainty. These techniques encourage the PP to learn DP's optimal behavior based on the uncertainty estimation. We empirically validate USPL across nine diverse tasks, including quadruped locomotion, quadrotor hovering, and robotic manipulation with both state-based and visual inputs. Our results show that USPL significantly reduces behavioral discrepancies between the PP and DP (as in Fig. [d), yielding higher deployment success rates than baseline methods. Furthermore, we observe that the DP accurately estimates its uncertainty, and the PP rapidly adapts its behavior in response to changes in uncertainty.
2 Background and Related Work
2.1 Partially Observable Markov Decision Processes
Partially Observable Markov Decision Processes (POMDPs) 27
provide a principled frame- 28 29
work for decision-malee making under uncertainty and incomplete state information. A POMDP is defined
2

---

by the tuple (S, Pu), where S, A, and O denote the state, action, and observation spaces, P and O are the transition and observation functions, r is the reward function, y is the discount factor, and Po is the initial-state distribution. Because the agent observes only o O, it must reason over a belief state-a distribution over to maximize expected cumulative return.
Mainstream RL algorithms for POMDP approximate the belief state using histories of observations and actions, typically with memoryy-augmented neural architectures such as RNNs 30 22 and Transformers [24]. These methods work well on low-dimensional tasks with modest memory
34 351; however, training becomes increasingly inefficient and unstable as task complexity grows [25] 32]. Luo et al. [25] showed that RNN-based RL demands finely tuned learning rates for stability, while Parisotto al. [26] reported that Transformer-
requirements 27
RL
requires specialized architectural modifications. Real-world applications further complicate matters by coupling ong-horizon memories with high-dimensional sensory inputs such as images 36 15
2.2 Privileged Learning
Given the inefficiencies of memory-augmented RL complex, high-dimensional POMDPs, priv- ileged learning [10] has emerged as a promising alternative. By introducing privileged observa- tions-compact, globally informative signals available only during training-priviie mitigates the sample inefficiency and instability of standard RL and enables the efficient training of a privileged policy (PP). During deployment, these privileged signals are no longer available, so a separate deployment policy (DP) must act solely on partial observations. In practice, the DP is typically obtained either by (i) directly imitating the PP through supervised learning (16), or encoder that reconstructs privileged observations from partial inputs and feeding them to the PP
learning
(ii) learning
[13. Both strategies retain the ppcision-making power of RI. while leveraging the stability of supervised learning, and have thus been widely adopted in real-world robotic control tasks 133 36 3 2. Nonetheless, a persistent challenge is the behavioral discrepancy between the PP and DP, which stems from the asymmetry in their observation spaces and can significantly impair deployment decision quality [66. Two main research lines have been proposed to address this issue.
One research line fine-tunes the DP with reinforeement-eerminng losses 15 or blends RL and imitation objectives during training [17 16 18 19). Balancing these two losses [6 18 and designing effective reward shaping [19] are persistent challenges. When the PP-DP gap is large, however, these methods eventually fall back on RL, inheriting its instability and exploration burden. A complementary line of work focuses optimizing the PP so that it is casier for the DP to imitate. One effective strategy reduces the PP's reliance on privileged observations by mapping them into a latent space with privileged encoder and then compressing that space through latent alignment 7 or an information bottleneck (21). A Alternatively. reward-penalyy methods (20) discourage behaviors that the DP struggles replicate by penalizing the PP in regions where DP errors are high, effectively searching-vis Ro-for behaviors that minimize behavior divergence. Because they avoid applying RL directly to the DP, these approaches are highly scalable. Nevertheless, without explicit access to the DP's progress in gathering privileged information, aligning the PP with the DP's optimal behavior remains challenging, which is the core problem USPL would like solve.
3 Method
In this section, we provide a detailed introduction to the design principles and details of USPL. Specifically, we describe the structure of the privileged and deployment policies in Sec. [3.1] explain the sampling and training processes in Sec. 3.2, and conclude the overall training process in Sec. 3.3
3.1 Framework
The overall framework of USPL is shown in Fig. USPL divides the observation information into three categories: common observation O₂ E target observation Ot O₁, and privileged observation Op E Opp where Oc. O,, Op denote the observation spaces. Privileged observation is only available during training and includes data that cannot be directly accessed during deployment, such as the coordinates of the goal points and the parameters of the environment dynamics. Common observation refers to the observation information that available both during training and deployment, such the joint angles, velocities, and positions of a quadruped robot. Target observation represents
3

---

Observations
Privileged and Deployment Policies
Sampling Process (Training)
Velocities,
Joint angles,
Deployment Policy
Op
Blurring
Op
agent
Positions,
Uncertainty dp
C1
t
O
Di
Depth images
Hp Observation Encader
P Train Switch
Transformation
Goal positions, Dynamics params,
Prodleged Polcy
simulator
Common
Target abs
Privllged obs
a
Figure 2: Overview of USPL.
high-dimensional observations that can be used to infer privileged observations during deployment, such as depth images captured by depth camera. This type of observation is optional, not necessary in all environments, depending on the task requirements. During training, we can access all types of observations, but during deployment, only common and target observations are accessible. Therefore, our goal to efficiently obtain deployment policy being able solve the task based solely on common and target observations.
USPL consists of two models: the privileged policy represented by an MLP and observation encoder that predicts the privileged observation. The observation encoder predicts the current privileged observation based on the common and target observations. Since target observations not globally available and may be images, the observation encoder is typically represented by a CNN- Transformer architecture handle both image and historical memory, process purely historical memory. The observation encoder outputs a mean Hip and standard deviation pp forming a distribution of the privileged observation, N(- Hp. of) This distribution represents the confidence interval feasible range for the current privileged observation based past target observations. The standard deviation Op provides an estimate of the uncertainty in the predicted privileged observation: the larger the op. the less reliable the output /4p- also indicating that the DP has yet to gather sufficient privileged information.
Transformer structure
The privileged policy is represented by simpler MLP network, which takes input the common observation, privileged observation, and the uncertainty output by the observation encoder, and then outputs action. In the deployment scenario, we treat the combination of the observation encoder and the privileged policy as the deployment policy, where the privileged observation is replaced by the Mp predicted by the observation encoder and fed into the privileged policy. The key difference from previous works is that we input the uncertainty into the privileged policy, allowing it to directly and dynamically sense the uncertainty of the observation encoder in the current state. This enables the privileged policy to intelligently identify which dimensions are inaccurate based on this uncertainty and take corresponding information-gathemee actions minimize the uncertainty, ensuring that the privileged observation can be accurately restored during deployment.
3.2 Training Process
The observation encoder is optimized and updated through a simple maximum likelihood estimation (MLE). Let the parameters of the observation encoder Es (oc, or; Φ) be denoted as øi, and for conve- nience, let El' oc: Φ) and Eg (Oez 0g 05) represent the mean and standard deviation vectors output by the observation encoder, respectively. Then, given a dataset constructed by D = {(o., or, op)}. the loss function for & is given by the following MLE loss:
(MLE(O, D) -B [log (o,, oz Φ), (E, (o., o6; o))
(1)
The training data D consists of two parts: one half comes from on-policy data collected by ne, while the other half comes from past training data stored in a replay buffer. The inclusion of off-policy data is intended to enhance the diversity of the observation encoder's training dataset, helping to prevent the observation encoder from overfitting to the behavior of the current privileged policy.
The privileged policy is trained using PPO. To ensure that the deployment policy can also complete the task, we want the privileged policy to learn such a behavior pattern: when uncertainty is high, should avoid executing behaviors that rely on the privileged observation to reduce uncertainty, thus allowing the observation encoder's output to converge to the actual privileged observation. When uncertainty is low, the privileged policy should efficiently complete the task based the privileged observation.

---

This way, the deployment policy can consistently align with the behavior of the privileged policy. To achieve this, as shown in the right part of Fig. 2] we introduce two modifications during PPO sampling: reward modification and privileged blurring. The reward modification shares similarity with previous works (20), where the reward of the privileged policy adjusted hased on the difference between the privileged policy and the deployment policy. Unlike earlier methods [20] which applied an additive penalty to the original reward, we propose the following reward transformation function:
fr, op) [clip(Cmin - log( O, Cmax) + -r.rr.
(2)
Here, Cmin and CIax are two ayper-parameters, such as Ciin log(0.2) and CiaA Cmin (0.(0.. o represents the average over each dimension of op. and o is a small value that prevents the reward from being zero. This function scales the original reward, giving lower scale coefficient for high uncertainty rewards and higher scale coefficient for low uncertainty rewards. When uncertainty is high, we want to encourage the privileged policy to explore the environment, which may conflict with the task reward's 's objective, potentially leading to lower or even negative rewards. In this case, we handle the reward with a lower scale to reduce the penalty for exploration. In contrast, when uncertainty is low, we encourage the privileged policy to focus on completing the task, thus giving a higher scale to focus on task completion.
In addition to reward modification, we inject noise into the privileged observation based on uncertainty. For conlindous-valued privileged observations, we process them as follows:
0p (op op) Op Kapte;
where K is a hyperparameter that controls the magnitude of the noise perturbation, and e, is random noise. The choice of distribution for < primarily depends on the properties of 0p: for bounded Op. we adopt a uniform distribution (1-1,1) as the distribution for c.- After the distribution is scaled by K, we clip its upper and lower bounds to ensure that the range of Op remains within the valid bounds of Op. For unbounded Op, a Gaussian distribution (e. 1) can also be used as the noise distribution. For discrete-valued privileged observations, the perturbation is handled as follows:
Öp- (o (ep:o)) op) TTea < J, + I(ea 2 op)
where E2 ~ (0, 1), If-} is an indicator function, and o, ~ ((O,,) is selecting uniformly at random from Op. Blurring the privileged observations has several advantages. Firstly, allows the privileged policy to behave without relying on privileged observations. Secondly, during high uncertainty, the noise added to the privileged observation encourages the privileged policy to explore the environment more. Lastly, this approach helps make the privileged policy more robust noise in the privileged observation, enabling it to tolerate errors from the observation encoder during deployment.
3.3 Practical Algorithm
The entire algorithmic process is summarized in Alg. 1 in App. A. In each iteration, we first collect the common, target, and privileged observations, denoted as Oc. Ot, and Op. respectively, from the environment. Next, the observation encoder is used to compute the current uncertainty Tp- The privileged observation Op is then perturbed using Eq. 3 or Eq. 4to obtain op. The privileged policy is invoked to generate the action output a based on (op. oe, Tp). After executing the action, the next of observations and the reward are received from the environment. The reward is scaled using Eq. to obtain f. Finally, the state transition and i are inserted into both the on- policy and off- off-po bufters. After sample collection is complete, we train the policy using PPO [37] with the (ōp, Oc, f, a) samples from the on-policy buffer. Additionally, the observation encoder is optimized using the Op, Oc, Ot samples from both the on-policy and off-policy buffers in conjunction with Eq, I More implementation details can be found in App. C.3
3.4 Theoretical Analysis
In this subsection, we analyze the distribution divergence between PP and DP of USPL. Let the USPL policy be denoted as T(- Oz. Op, Op. dp), where o,, op, and 0, represent the common observation, privileged observation, and uncertainty estimation, respectively. At any timestep h. PP takes the true privileged observation of: as its input, while the DP uses a predicted observation ob as the input.
Theorem 1. Assume for each timestep h E (1,...,H, there exists ch > 0 such that the prediction error is bounded:
||o ob Se
(5)
5

---

Further assume the policy is Lipschitz continuous with respect to Og. such that for any og. 0%.
Drv (T(. oc, oy op), T(. Oc. op)) I L(ee, 0p) llob 02.12.
(6)
where Loe, pp) is the Lipschitz constant. Let Th = x(- oe, obee0) and DP = *(. oe, eh)
Then, the TV distance between the state visitation distributions dī
pp (oo) and dz (0o) is bounded:
H E Drv
1
dh
dh
E R (T)
h=1
Proof. Please refer to App.
Theorem indicates that the divergence is governed by two factors: the prediction error ex and the Lipschitz constant L(oe, c²), which quantifies the policy's 's reliance on the privileged observation Op. As the true error bound e is inaccessible in practice, we utilize the uncertainty estimation dp as its approximation. The bound suggests that the divergence can be minimized by reducing either e" or L(o,, e*). In USPL, the reward transformation (Eq. 2 is designed to minimize ex, while the privileged observation perturbation (Eqs. and 4 aims to minimize L(o,, e
In practice, neither term can be guaranteed to remain minimal throughout an episode. Insufficient information, particularly early an episode, often leads high prediction error (large e Similarly, ((o,<*)) cannot be persistently minimized, the DP must utilize op at certain timesteps to maximize rewards; otherwise, the training DP would be very difficult. large divergence arises when both e" and L(oe, e²) are simultaneously high. A core contribution of USPL is to make the policy by providing the uncertainty estimate op (as an approximation of c²) as an explicit input. This enables the policy to learn dynamic strategy: it learns to minimize its reliance on Op (lowering L(,,, e²) )) when uncertainty is high, and conversely, to leverage Op to complete the task when uncertainty low.
4 Experiments
In this section, we answer the following questions through experiments:
What is the behavioral difference between the USPL privileged policy and the deployment policy? (Fig. 4
How well does USPL perform in various tasks? (Tab. and Fig. 5
Can the USPL observation encoder accurately predict uncertainty, and is the privileged policy sensitive to uncertainty? (Figs. 6 7
How the different components of USPL impact the final performance? (Tab.
4.1 Experiment Setup
(a)
Stainway Search
b) Lateral Choice
(c) Blind Mass Stack
Sumounded
possible goals training phase
current goal
Mi Midpoint Choice
Signpost
(e) Square Maze
highasz
heggt
lpsreesto to Geal
uniformly randomly distributed
Figure 3: Illustrations of the environments used in the experiments.
We consider three common types of robots in Isaacgym 38 quadruped robot, a quadrotor UAV, and robotic arm. As shown in Fig. 3 constructed environments based on these three types of

---

robots, with two of them supporting both image-based and con-image-baseee inputs, resulting in total of tasks. These tasks all require the agent to possess strong information unformation-gathering abilities and the capacity to handle long-term memory and images. For example, in the Midpoint Choice task, the quadruped robot needs to identify the second tallest of three pillars of different heights. However, it can only perceive small area in front of it, so it must approach each pillar, remember its height, and finally compare to determine the position of the second tallest pillar. The agent not only needs to actively explore the environment but also remember what it saw during its movement. In the Signpost Navigation task with depth image observation, the quadruped robot should first approach signpost, use the depth camera to determine the position of the target point, and then remember this position while gradually moving toward the target. This requires processing images while maintaining long-term memory. For more detailed environment descriptions, refer to App. C.1
We consider four baseline methods: (1) RMA 13 a privileged learning method, which differs from USPL in that it does not apply any regularization to the training of the privileged policy; (2) RESeL- PPO, which applies the state-of-the-art POMDP RL. method RESeL. [25] to the PPO algorithm [77 training the deployment policy purely via RL; (3) PrivRecons, hybrid method combining RL and privileged learning, which samples based on the deployment policy but only uses RL to update the privileged policy part, while updating the observation encoder via supervised learning [39 33 (4) RewReg, a privileged learning variant that penalizes the PP's rewards in states where the DP exhibits large behavior divergence, done in [20]. Refer to App. C.2 for more baseline details.
4.2 Trajectory Divergence between Privileged and Deployment Policies
The central motivation of USPL. is to reduce the behavioral discrepancy between the Deployment Policy (DP) and the Privileged Policy (PP). To evaluate whether this objective has been achieved, we conduct experiments on four quadruped navigation tasks, measuring the divergence in trajectories between the DP and PP. As shown in Fig. 4 USPL exhibits sigmificantly lower trajectory discrepancies compared to RMA. Notably, in environments such as Signpost Nav, Square Maze, and Lateral Choice, the discrepancy remains consistently close to zero over time, indicating that USPL effectively minimizes behavioral differences. This suggests that the DP and PP maintain highly consistent trajectories even after long-term interaction with the environment.
Signpost Nav
Squara Maze
Lateral Choice
Midpoint Choice
Trajectory Dive ggance
8-
4-
2-
-0.44
6-
2-
0.24
-0.18
2-
0.10
0.16
-0.06
2-
0.05
-0.08
0+
0.00
600
1000
200
400
200
500
750
200
400
limeolag
Smeslep
lmeslep
lineslep
USPI (Tajeetoyy Diveegencc)
RB/A (Tajectry Divergence)
USP4 (Prv Prediction Firor) -
RMA (Prt Predidion Emror)
Figure 4: Trajectory divergence between PP and PD and privileged observation prediction error over time for different algorithms. The discrepancies in the lengths of the curves come from the different trajectory lengths for different algorithms.
The dashed lines in Fig. 4 represent the real-time prediction error of the observation encoder, which attempts to reconstruct the privileged observation from target observations during DP execution. Initially, when informative observations have not yet been revealed, both RMA and USPL exhibit similar levels of prediction error. However, critical information is acquired, USPL's prediction error steadily decreases and eventually converges near zero. In contrast, RMA shows a noticeable reduction in prediction error only in the Lateral Choice environment.
From the trends in prediction error, two key conclusions emerge. First, USPL learns effective information-gathenn
behaviors that actively reduce the prediction error of the observation encoder during interaction. Second, even when the prediction error is high, USPL is able to maintain low behavioral discrepancies between DP and PP. This indicates that the privileged policy learns to selectively utilize privileged observations in a way that reduces the behavioral gap between the privileged and deployment policies.
7

---

4.3
Policy Performance Comparisons
Table 1: Success-rate comparisons over 5 seeds. "-p" denotes the privileged policies.
Task
PrivRecons
RESeI.-PPO
RewReg
RMA
USPL
RMA-p
USPL.-
Signpost Nav
18.1
3.9+0.3
14.2
9.5+0.66
97.
100.0
100.0
Square Maze
26.7+10.3
5.2+-4.3
44.9=1.1
24.5+10
98.5=0.1
98.1
99.8
I Lateral Choice
96.0++055
46.7+-21.8
56.4+2.0
89.6+5.2
98.8+0.4
100.0
100.0
Midpoinl Chuice
80.9.13.88
0.1::0.1
50.1
38.81004
97.710.7
100.0
98.9
Biased Quadrotor
2.210.0
0.0:00.0
0.6110.2
0.0:1:.00
96.7100.1
100.0
100.0
Blind-Maa Mass Stack
55.1122.5
50.2+1.2
55.828.8
50.12006
97.5=10
100.0
100.0
Average
46.8
17.7
40.3
35.4
97.8
99.8
Success Rate (%)
100
PrivRacons
RMA
USPL
RMA-p
USPL-p
Signpost Nav (image) Midpoint Choice (image)
Slairway Search
Average
Figure 5: Success rate comparisons in image-based tasks. '-p' denotes the privileged policies.
To further quantify the performance of different algorithms during deployment, compare the task success rates of the deployment policies across six non-visual tasks, as shown in Tab. To better highlight the performance gap between privileged and deployment policies, the last two columns of the table report the task success rates of the privileged policies for both USPL and RMA. Comparing USPL with other methods, higher deployment success rates. The pure RL method (RESeL-PPO) and privileged learning methods with or without regularization on the privileged policy (RewReg and RMA) underperform the hybrid approach PrivRecons. This is primarily because the tasks heavily rely on long-term memory, rendering RL inefficient. Moreover, there exists a substantial behavioral discrepancy between the optimal privileged and deployment policies. As a result, even though RMA'S privileged policy achieves over 99% average success rate, its deployment policy still performs poorly. With reward penalty regularizing the PP, RewReg outperforms RMA in 5/6 tasks but still inferior to PrivRecons, which combines RL with supervised learning. PrivRecon provides some performance improvement over other methods but lags behind USPL by up to 50%. In contrast, USPL exhibits minimal performance gap between the privileged and deployment policies, achieving over 95% success rate across all tasks. These results demonstrate the superior deployment performance of USPL in memory-intensive tasks. The learning curves of USPL and other methods can be found in App.D.I. We also compare USPL with the basclines in non-robotic environment in App. D.2
observe that USPL consistently achieves significantly
To further evaluate the effectiveness USPL. in high-dimensional settings, we conducted experiments on three image-based tasks. As shown in Fig. 5 USPI. achieves over 95% success rates, significantly outperforming baselines such as RMA and PrivRecons. Notably, the performance gap between USPL's deployment policy and its privileged policy remains minimal, with much smaller drop compared to RMA.
4.4 Uncertainty Estimation Analysis
To further understand how uncertainty estimation helps reduce the behavioral divergence between the privileged policy and the deployment policy, we conducted additional experiments. In Tig. 6 we visu- alize the temporal evolution of the deployment policy's 's uncertainty estimation in the Stairway Search task, along with the corresponding policy behaviors and the distribution of privileged observations predicted by the observation encoder at key time steps. Initially, the uncertainty estimation is around 2.0, and the predicted privileged observation distribution forms Gaussian centered in the middle of the map with wide spatial spread. As the robot approaches the edge of the platform and observes no stairs on that side, the uncertainty estimation drops for the first time, and the distribution N/ (- Mp: og) quickly shifts toward the lower part of the map. Upon reaching the lower edge and looking to the right, the robot again finds no stairs, prompting a second drop uncertainty- now suspecting that the stairs might be in the bottom-left corner. At this point, (- Mp, o3) centers around the bottom-left.
8

---

Uncertainty estimation 6
2.0
platform
N (Poe)))
Quadraped
1.5
1.0
0.0
100
400
500
timestep
Figure 6: The uncertainty estimation of the deployment policy over timesteps, with visualizations of the corresponding behaviors and the distribution N Ppp op) key timesteps.
Step Control Op
Periodic Switching Og
Observation Encoder dp
0.50
value
0.25
Landed at this height
0.00
200
400
200
400
200
400
timestep
timestep
timestep
target height
current height
uncertainty eslimation
Figure 7: Altitude trajectories under different uncertainty inputs the Biased Quadrotor task. The quadrotor aims to maintain level flight but receives biased attitude and altitude observations. During deployment, it must first land on plat ground to calibrate its sensors. The privileged observation provides the ground-truth bias.
though op does not collapse zero since the agent has yet directly observed the stairs. Once the robot turns to the bottom-left and detects the stairs, the uncertainty estimation immediately collapses zero, and (- Mp: og) converges the target location. This result indicates that our observation encoder provides highly sensitive estimation of uncertainty, which accurately reflects the amount of information still be perffctly aligning with our expectations.
To better examine how the privileged policy responds to uncertainty estimation, we further conduct experiments the Biased Quadrotor task. We directly feed the ground-truth privileged observation into the privileged policy and vary the uncertainty estimation across three settings. The resulting altitude trajectories are shown in Fig. 7 Under the step control setting, apply high uncertainty estimation for the first 250 steps, then switch to a low value. Remarkably. although the policy receives the ground-trute truth privileged observation, it remains grounded for the first 250 steps, refusing to initiate altitude control. This because the policy considers the calibration incomplete, and any premature action might diverge from the deployment policy. Once the uncertainty drops after step 250, the quadrotor immediately takes off and stabilizes at the desired altitude. Notably, the reward remains negative while grounded, yet the policy prioritizes reducing behavioral divergence by waiting for uncertainty to decrease. Similar behavior is observed under periodic switching and when using encoder- predicted op: the policy remains grounded during high uncertainty for calibration, and takes off only when uncertainty becomes low. These results show that the USPL privileged policy is highly sensitive uncertainty estimation and adapts its behavior accordingly.
encoder-predicted
Taken together, Figs. and illlustrate how USPL works. The observation encoder provides rapid and accurate uncertainty predictions, while the privileged policy reacts sensitively these sig- nals-avoiding reliance on inaccurate privileged observations under high uncertainty and leveraging them effectively under low uncertainty accomplish the task. Additional results similar to those in Figs. and covering more environments, are presented in App. D.3
4.5
Ablation Studies
USPI. consists of three core components: using uncertainty estimation as an input to the privileged policy (UI), applying blurring to the privileged observations (blur), and modifying the reward

---

Table 2: Ablation studies on uncertainty estimation input (UT), privileged blurring (blur), and reward transformation (rew) over 3 seeds.
USPL
w/o blur
w/o rew
w/o UI
only UI
RMA
Signpost Nav
97.8 + 0.2
96.5
+ 0.8
35.2
+ 1.0
97.0 + 0.1
9.9 + 0.4
9.5 + 0,6
Square Maze
98.5 + 0.1
38.9
+ 6.1
88.6
+ 8.3
94.7 + 2.2
24.9 + 0.4
24.5 + 1.0
Lateral Choice
98.8 + 0.4
88.1
+ 4.5
91.1
+ 3.5
92.0 0.8
68.9 + 2.1
89.6
+ 5.2
Midpoint Choice
97.7 + 0.7
50.9 - 5.8
91.410
78.9 0.6
0.0 = 0.0
38.8 = 0.4
Avg. Success Rate
98.2
68.6
77.3
90.7
25.9
40.6
Avg. Failure Rate
1.8
31.4
22.7
9.3
74.1
59.4
function (rew). To investigate the contribution of each component, we conducted ablation and sensitivity studies, with the ablation results shown in Tab. [ and the sensitivity results in App. D.4 We found that both blur and rew significantly impact policy exploration. When both are removed (i.e., only UI), performance drops sharply across all four environments,
the PP lacks proper guidance and fails to minimize behavioral divergence from the DP. The effects of blur and rew vary by task: in the Signpost Nav task, rew is crucial-its removal leads to major performance drop-while blur has little effect. In contrast, for the other three tasks, removing blur causes substantial degradation, whereas removing rew has smaller impact. Combining both consistently yields strong performance across all tasks. The results show that removing UI still results in reasonable performance under the strong guidance of rew and blur, but it falls short of optimal. We found that introducing UI significantly reduces the policy's 's failure rate. UI gains an 81% relative reduction in the failure rate (comparing USPL w/o UI), substantially improving policy robustness.
Tab. 2 demonstrates how different components play distinct yet complementary roles in a multi-stage process. Note that optimal policy must adapt its behavior online based on its uncertainty estimation, pp. USPL achieves this in two stages. First, the policy must Iearn a complete spectrum of optimal behaviors corresponding to different uncertainty levels. This is achieved via two key components: Privileged blurring simulates varying error levels in the privileged observation, thereby training the agent to master behaviors suited for different reliability levels of this input. Simultaneously, the reward transformation incentivizes the agent to learn exploratory behaviors that actively minimize uncertainty. Tab. confirms that both components are essential for discovering this full behavioral spectrum. Second, the policy must dynamically select the appropriate behavior online. Providing op as direct input acts as a real-time signal, enabling the policy to seamlessly switch between exploratory behaviors (high op) and exploitative behaviors that utilize the privileged observation for task execution (low σp). This online adaptation fails if the behavioral spectrum was learned in the first stage, which explains the poor performance observed when only providing the uncertainty input.
5 Conclusions
In this paper, we propose Uncertainty- -Sensiivv Privileged Learning (USPL) to address the limitation that the privileged policy (PP) in prior methods lacks real-time awareness of the progress of privileged information gathering. USPL constructs the deployment policy (DP) by combining the privileged policy (PP) with an observation encoder that predicts privileged observations and outputs an associated estimate of uncertainty. The uncertainty, which reflects how much information still needs to be gathered, is fed to the PP so it can adapt its behavior as information gathering progresses. Additionally, USPL encourages the PP to produce DP-replicable behaviors via reward transformation and privileged observation blurring. Experiments show that the DP can accurately estimate uncertainty, the PP learns to actively explore and calibrate, and the behavior divergence between PP and DP is largely reduced, achieving over 95% deployment success rate all 9 tasks.
Limitations. Despite the promising results, USPL still has two known limitations: (1) It currently handles only cow-dimeniional privileged observations. Extending USPL to high-dimensional, re- dundant, or time- time-varying privileged observations remains an open challenge. It may be addressed by introducing a privileged encoder that maps privileged observations into a latent space [3] (2) Another potential direction is to eliminate the use of the DP's encoder during PP training, thereby avoiding the need to process high-dimensional target observation information and thus improving both training and simulation speed. One possible solution to train a proxy encoder that directly predicts uncertainty from historical privileged observations (20).
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Science Foundation of China (62495093,625061.55 U24A20324), the Natural Science Foundation of Jiangsu (BK20241199, BK20243039), and the AI & AI for Science Project of Nanjing University. The authors thank Dr. Tian Xu for his help with the theoretical derivations. The authors thank the anonymous reviewers for their support and helpful discussions on improving the paper.
References
[1] B. Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Ku- mar Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions on Intelligent Transportation Systems, 23(6):4909-49222 2022.
[2] Zipeng Fu, Xuxin Cheng, and Deepak Pathak. Deep whole-body control: Learning unified policy for manipulation and locomotion. In Proceedings of the Conference on Robot Learning. Auckland, New Zealand, 2022.
[3] Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in challenging terrains using egocentric vision. In Proceedings of the Conference on Robot Learning, Auckland, New Zealand, 2022.
[4] Zhaohan Feng, Ruiqi Xue, Lei Yuan, Yang Yu, Ning Ding, Meiqin Liu, Bingzhao Gao, Jian Sun, and Gang Wang. Multi-agent embodied Al: advances and future directions. CoRR, abs/2505.05100 2025.
[5] Yunfan Jiang, Ruohan Zhang, Josiah Wong, Chen Wang, Yanjie Ze, Hang Yin, Cem Gokmen, Shuran Song, Jiajun Wu, and Li Fei-Fei. BEHAVIOR robot suite: Streamlining real-world whole-body manipulation for everyday household activities. CoRR, abs/2503.05655. 2025.
[6] Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction, 2nd Edition. MIT Press, 2018.
[7] Hang Lai, Jiahang Cao, Jiafeng Xu, Hongtao Wu, Yunfeng Lin, Tao Kong, Yong Yu, and Weinan Zhang. World model-based perception for visual legged locomotion. CoRR, (ttp//409.167884 2024.
[8] Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R. Devon Hjelm. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems, Vancouver, Canada, 2019.
[9] Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Lue Bacon. When do transformers shine in RL? decoupling memory from credit assignment. In Advances in Neural Information Processing Systems, Orleasn, LA, 2023.
[10] Joonho Lee, Jemin Hwangbo. Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5(47):5986, 2020.
[11] Yang Cai, Xiangyu Liu, Argyris Oikonomou, and Kaiqing Zhang. Provable partially observ- able reinforcement learning with privileged information. In Advances in Neural Information Processing Systems, Vancouver, Canada, 2024.
[12] Edward S. Hu, James Springer, Olch Rybkin, and Dinesh Jayaraman. Privileged sensing scaffolds reinforcement learning. Proceedings of the International Conference on Learning Representations, Vienna, Austria, 2024.
[13] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: rapid motor adaptation for legged robots. In Robotics: Science and Systems, virtual, 2021.
[14] Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62), 2022.
11

---

[15] Xuxin Cheng, Kexin Shi, Ananye Agarwal, and Deepak Pathak. Extreme parkour with legged robots. In Proceedings of the IEEE International Conference on Robotics and Automation. Yokohama, Japan, 2024.
[16] Luca Weihs, Unnat Jain, Iou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander G. Schwing. Bridging the imitation gap by adaptive insubordination. In Advances in Neural Information Processing Systems, virtual, 2021.
[17] Hai Nguyen, Andrea Baisero, Dian Wang, Christopher Amato, and Robert Platt. Leveraging fully observable policies for learning under partial observability. In Proceedings of the Conference on Robot Learning. Auckland, New Zealand, 2022.
[18] Idan Shenfeld, Zhang-Wa Wei Hong, Aviv Tamar, and Pulkit Agrawal. TGRL: algorithm for teacher guided reinforcement learning. In Proceedings of the International Conference on Machine Learning, Huonlulu, HI, 2023.
[19] Aaron Walsman, Muru Zhang, Sanjiban Choudhury, Dieter Fox, and Ali Farhadi. Impossibly good experts and how to follow them. In Proceedings of the International Conference on Learning Representations, Kigali, Rwanda, 2023.
[20] Nico Messikommer, Jiaxu Xing, Elie Aljalbout, and Davide Scaramuzza. Student-informed teacher training. In Proceedings of the International Conference Learning Representations, Singapore, 2025.
[21] Haoran He, Peilin Wu, Chenjia Bai, Hang Lai, Lingxiao Wang, Ling Pan, Xiaolin Hu, and Weinan Zhang. Bridging the sim-to-real gap from the information bottleneck perspective. In Proceedings of the Conference on Robot Learning, 2024.
[22] Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014.
[23] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.0075 2023.
[24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, In Advances in Neural Informa- tion Processing Systems, Long Beach, CA, 2017.
[25] Fan-Ming Luo, Zuolin Tu, Zefang Huang, and Yang Yu. Efficient recurrent off-policy RL requires a context-encodee--pecfffe learning rate. In Advances in Neural Information Processing Systems, Vancouver, Canada, 2024.
[26] Emilio Parisotto, H. Francis Song. Jack W. Rae, Razvan Pascanu, Çaglar Gülçehre, Siddhant M. Jayakumar, Max Jaderberg, Raphaäl Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Proceedings of the International Conference on Machine Learning, virtual, 2020.
[27] Siegmund Duell, Steffen Udluft, and Volkmar Sterzing. Solving partially observable reinforce- ment learning problems with recurrent neural networks. In Neural Networks: Tricks of the Trade Second Edition, volume 7700 of Lecture Notes in Computer Science, pages 709-733. Springer, 2012.
[28] Matthijs J. Spaan. Partially observable markov decision processes. In Reinforcement Learning, volume 12 of Adaptation, Learning, and Optimization, pages 387-414. Springer, 2012.
[29] Hanna Kurniawati. Partially observable markov decision processes and robotics. Annual Review of Control, Robotics, and Autonomous Systems, 5:253-277, 2022.
[30] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 98)): 17800 1997.
[31] Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many POMDPs. Proceedings of the International Conference on Machine Learning, Baltimore, MD, 2022.
12

---

[32] Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon. When do transformers shine in RL? decoupling memory from credit assignment. Advances in Neural Information Processing Systems, New Orleans, LA, 2023.
[33] Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A. Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. CoRR, abs//905 066444 2019.
[34] Fan-Ming Luo, Shengyi Jiang, Yang Yu, Zongzhang Zhang, and Yi-Feng Zhang. Adapt to environment sudden changes by learning context sensitive policy. In Proceedings of the AAAI Conference on Artificial Intelligence, virtual, 2022.
[35] Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. A survey of progress on cooperative multi-agent reinforcement learning open environment. CoRR, abs/2312.01058 2023.
[36] Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher G. Atkeson, Sören Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. In Proceedings of the Conference on Robot Learning, Atlanta, GA, 2023.
[37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06344 2017.
[38] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance GPU based physics simulation for robot learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, virtual, 2021.
[39] Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. In Robotics: Science and Systems XIII, Cambridge, MA, 2017.
[40] Tian Xu, Ziniu Li, Yang Yu, and Zhi-Quan Luo. Understanding adversarial imitation learning small sample regime: stage-coupled analysis. CoRR, abs/2208.0 01999,, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We state in the Abstract and Introduction that our main contribution is enabling the privileged policy to be aware of the state of the deployment policy.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
We do not provide any theoretical proof in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Sec.
App. and App.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Code is available at https://github.. com www./anm//aaaaaaccccaaa htt.://...ooooaaiigg///////
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com http///bbb///piiidedoeee/eaaaaaa cassiicyncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer to Sec.
App.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We represent error bars using + standard error in tables and shaded areas in
curves.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to App. C.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pwwppaapp....p.o//po/eete
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field of privileged learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any models or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

A Algorithm Details
A.1 Algorithm
Algorithm 1 Un Sensitive Privileged Learning (USPL)
Initialize Ti, E.. on-policy buffer oe and off-policy buffer Bott
2: for each training iteration do
3: Reset simulator, obtain common, target, and privileged observations Oc: Ot, Op. Bon 0:
4: for each step do
5:
Pp, Sp <- (o, 0z; Ø):
6:
Op + folop, Tp) following Eq. a 4 (Opp Oe: Tp);
3 or Eq. based on the continuity of op:
7:
8:
Apply a to the simulator, obtain reward and next observations de 0/2 0/2
9:
f + f-(r, following Eq. 2
Store (Øe; Op. Op, Oc, a, r, oe 02 o/)) in both Bon and Ban;
11:
end for
12: Update ne using PPO with Bon:
13: Update E, using Eq. 1 with Bur and Boni
14: end for
B Proof
B.1
Proof Preliminaries
Lemma (Lemma in (40). For any tabular and episodic MDP, considering two policies a and n², let dz. ()) denote the state distribution induced by in time step h. When h 2 2, we have that
11 d.h (-11a S
of of ()llme( (s) - s)]]].
(8)
Proof. Please refer to 40
Corollary For any tabular and episodic MDP, considering two policies and n², let d),()) denote the state distribution induced by in time step h. If there exists E R such that
H
E
dpd (ev(Te(-1&),,5e s))] s E,
(9)
H
1=1
then
( $ He.
(10)
Proof. We adopt the standard definition of the TV distance:
Drv = lp 4/11-
Thus, Eq. 8 can be written as follows:
h-1
Drv(d d
Σ B Drv(Te( ( () s))]
Summing over h = =1,.... I,.... H and exchanging the order of summations,
Drv(dh dh) <
(H E) E Ewayy' (Div(Tel: 18), al( [s)]]
Fm1
< Ba [Drv(Te(- s), 7f6 ((((1s))]]
Divide both sides by H and apply Eq. 9 to obtain the corollary.
21

---

B.2 Proof of Theorem 1
Proof. We first bound the single-step policy divergence at timestep for a given common observation
Oe- By the definitions of Th
PP and Th DP the Lipschitz continuity assumption in Eq. and the
prediction error bound in Eq.
[] we have:
Drv(Th
PP (. oc), Fh p (o)) Drv(f(-)) oc: o")e"), x(. ocs los ob)
( L(oes en) 1lo8 0/ - L(o cn) eh
(11)
Taking the expectation of this inequality with respect to 0c dhi²e gives:
a Pr Drv(mm 0c), ( oz))] I E o ((o
(12)
From Corollary D we have the following bound on the average state visitation divergence, which relates the sum of expected single-step policy divergences:
H
H
EDV DTv dh
d2
E
o
PP
[DTv ( oo), R [o)))))
(13)
A=1
hum
Finally, substituting the bound from Eq. inequality:
12
into the right-hand side of Eq. 13 yields the claimed
Σ
Drv(dh
dh "h
S
oc a [L(Oc,E")e
(14)
H A=1
C Experiment Settings
C.1
Tasks Descriptions
C.1.1
Stairway Search
Observation space: depth images from an onboard depth camera, current robot position, and orientation.
Action space: desired change in heading (yaw) and desired body pitch.
Privileged observation: coordinates of the target platform.
Reward: a shaped reward for approaching the target platform and a terminal reward for reaching it.
Task description. The robot starts on a large platform and must step onto smaller platform equipped with stairways on both sides so that it can later descend to the ground. To focus the policy on high- level decision making, the agent does train a low-level controller- based on the open-source implementation of (I5) that receives three commands: target speed magnitude, desired body pitch, and desired yaw. The controller's inputs include motor positions and velocities, the base linear and angular velocities, pitch and roll angles, the robot's 's inertial parameters, and the three command signals. Because precise speed control is critical here, the commanded speed is fixed at 0.5 m/s.
command the torques of all 12 joints directly. Instead,
Optimal behavior. The robot should peer the platform's 's edge to identify the smaller platform that has stairs on both sides, then walk onto that platform.
C.1.2 Lateral Choice
Observation space: current robot position and orientation.
Action space: desired change in heading (yaw).
Privileged observation: coordinates of the goal point.
Reward: shaped reward for approaching the goal and terminal reward for reaching it.
22

---

Task description. The goal may appear either on the left or on the right, but the robot can perceive only its own pose-it has no information about the terrain. Under this blind setting the robot must navigate to the goal; the episode terminates once the goal is reached.
Optimal behavior. As illustrated in Fig. [1] the robot first walks toward side. If the episode does not terminate, that side is not the goal, so the robot turns around and heads to the opposite side.
C.1.3 Blind-Mass Stack
Observation space: positions of the three cubes; end-effector position; index of the green cube; boolean flag indicating whether a cube is grasped; index of the grasped cube (if any); measured mass of the grasped cube.
Action space (three discrete dimensions):
end-effector æ, y sarget-discretie into three values, each corresponding to the æ, y position one cube (the end-effector moves toward the selected cube);
end-effector target- target-discrrtized discretized into three height levels;
gripper command-ove open close.
Privileged observation: index of the red cube and the bias applied to the weight sensor.
Reward. the reward for the end-effector approaching the red block, the reward for the end-effector lifting the red block, the reward for approaching the green block while the red block is still lifted, and the reward for task completion.
Task description. Three rebe green, and blue-are placed from left to right and numbered 2, The green cube (index known) weighs kg; the red cube weighs 0.75 kg, but its index is unknown; the blue cube's weight and index are both unknown. The arm's weight sensor is biased by fixed multiplicative factor. The objective to identify the red cube via weight sensing and stack on top of the green cube, implicitly learning to self-calibrate the sensor.
Optimal behavior. Because the green cube's index and true mass known, the arm first grasps the green cube to estimate the bias factor and calibrate the sensor. then picks either remaining cube, checks whether its (calibrated) mass is 0.75 kg; if so, that cube is red and should be stacked on the green cube, otherwise the other cube stacked.
C.1.4 Signpost Nav
Observation space: current robot position and orientation, plus either (i) terrain heights in a small frontal region obtained via scandot (ii) depth images from depth camera.
Action space: desired change in heading (yaw).
Privileged observation: coordinates of the goal point.
Reward: shaped reward for moving toward the goal and terminal reward for reaching it. Task deseription. The robot must reach hidden goal point whose location directly observable. Instead, a signpost encodes the goal: its orientation indicates direction, and its length encodes the distance from the signpost to the goal.
Optimal behavior. The robot first walks to the signpost, uses scandot or depth perception to infer the signpost's geometry, predicts the goal location accordingly, and finally travels the goal.
C.1.5 Square Maze
Observation space: current robot position and orientation.
Action space: desired change in heading (yaw).
Privileged observation: coordinates of the goal point.
Reward: shaped reward for approaching the goal and terminal reward for reaching it.
23

---

Figure Visualization of the four Square Maze layouts and their corresponding goal locations.
Task description. The goal can be located at any of the four corners. The maze layout varies slightly with the goal location, as shown in Fig. [8] At the first junction, if the junction "T" shape the goal to the right; otherwise it is to the left. A similar relationship holds at the second junction.
Optimal behavior. Because the robot blind the maze geometry, it probes each junction by walking straight ahead. it collides with wall, it turns right; otherwise it turns left, thereby inferring the junction type and, recursively, the goal corner.
C.1.6 Midpoint Choice
current robot position and orientation, plus either (i) terrain heights in a small frontal region (scandot) (ii) depth images from depth camera in front of the robot head.
Observation space:
Action space: desired change in heading (yaw).
Privileged observation: coordinates of the goal point.
Reward: shaped reward for approaching the goal and terminal reward for reaching it.
Task description. The goal resides on one of four corner platforms. Each platform is preceded by two columns, but only the goal platform has the second-tallest pair. With image observations, the robot can measure column heights from the depth camera; with scandot, obtains local height data directly.
The robot circles the platforms, records the heights of the three column pairs encounters, identifies the second tallest pair, and moves to the platform behind it.
Optimal behavior.
C.1.7 Biased Quadrotor
Observation space: perceived altitude, roll, pitch, angular velocity, vertical velocity, and flag indicating whether the quadrotor has landed.
Action space: target pitch, target roll, and target altitude.
Privileged observation: biases in the altitude, roll, and pitch sensors.
Reward: a shaped reward for maintaining the target altitude and level attitude, a terminal reward for achieving them, and penalty for flying low (touching the ground).
Task description. The quadrotor must perform level hover at target altitude, but state estimates corrupted by unknown sensor biases caused by in in-flight malfunction.

---

Optimal behavior. The quadrotor first lands, because ground contact provides a reference of zero altitude, roll, and pitch. Using these reference readings it calibrates sensors, then takes off again and performs steady hover with the calibrated measurements.
Table 3: Summary of environments, observation modalities, and desired behaviors.
Env. name
Observations
Task description
Optimal behavior
Stairway Search
common: robot pose privileged: target coords target: depth image
The robot must leave a large platform and step onto a smaller one with stairs both sides.
Navigate to platform edge, use depth perception to identify the staircase, and ascend onto the target platform.
Lateral Choice
common: robot pose privileged: goal coords target: N/A
A goal spawns either left or right, but only the robot's 's own pose is observable.
Walk toward one side; if the goal is not reached, turn around and head the opposite side.
Blind-Mass Stack
common: cube EE states, grasp info
Identify the red cube via weight sensing and stack it on the green cube while self-calibrating the arm's 's scale.
Calibrate on the green cube, test the others, then stack the red cube on the green one.
privileged: red-cube index, sensor bias target: N/A
Signpost Nav
common: robot pose
signpost encodes the hidden goal:
Approach the signpost, infer the goal location from its geometry, then travel to the goal.
privileged: goal coords target: depth / scandot
orientation gives direction, length gives distance.
Square Maze
common: robot pose privileged: goal coords target: N/A
The goal lies at of four corners; junction geometry hints the correct path.
Walk straight to probe each junction; collide/clear determine turns that guide the robot the goal corner.
Midpoint Choice
common: robot pose privileged: goal coords target: depth / scandot
Four corner platforms, three of which have two columns; the goal platform has the second-tallest pair.
Circle the platforms, measure column heights, then move the platform behind the second-tallest pair.
Biased Quadrotor common:
altitude, attitude, velocities, landed flag privileged: sensor biases target: N/A
Hover level at a target altitude despite unknown biases in altitude and attitude sensors.
Land to obtain zero references, estimate biases, then take off and hover stably.
We summarize the description the environments Tab.
C.2 Baselines
We mainly evaluate four categories of baselines:
1. PrivRecons. PrivRecons adopts exactly the same network architecture USPL: the observation encoder is trained with an MLE loss, while the privileged policy optimized with PPO. Unlike USPL, however, PrivRecons collects trajectories using the deployment
25

---

policy so that the privileged policy experiences no train-test gap. During learning, the method does not exploit privileged information as auxiliary imput and is not targeted to actively gather it. It performs well only when privileged information is easy to obtain-i.e.. when random exploration quickly uncovers observations correlated with that information. For tasks in which privileged information is hard to gather (or costly in reward terms) such as Stairway Search and Biased Quadrotor, the efficiency of PrivRecons drops sharply.
2. RMA. [3] RMA removes all innovations introduced by USPL: it neither feeds uncertainty into the privileged policy nor applies reward transformation or privileged blurring. Because the privileged policy is trained without considering the deployment setting, its performance is generally poor.
3. RewReg. (20) On top of RMA, RewReg adds reward guidance that encourages the privi- leged policy to minimize its behavioral divergence from the deployment policy while still completing the task. Such methods implicitly assume that the privileged policy can gener- ate optimal behavior that is replicable during deployment. However, as discussed earlier, achieving these behaviors is challenging for the privileged policy because its input lacks information--atterrin progress. RewReg outperforms RMA in most environments, but its overall performance remains far below that of USPI..
4. RESeL_ PPO. We apply RESeL's stabilization techniques 25 to PPO training. Concretely, the deployment policy is a Transformer trained directly with PPO, and we use reduced learning rate to stabilize Transformer optimization. Nevertheless, the tasks considered here involve long-term credit-assignment challenges. Ni et al. suggest that Transformer- -basededdde RL struggles to solve such problems efficiently [32]. As a result, RESeL_PPO still shows training inefficiency our benchmark.
C.3 Implementation Details of USPI
C.3.1 More Implementation Tricks
Learning Schedule. our implementation, we employ a cosine schedule to anneal both the standard deviation of the PPO policy and its learning rate. In addition, separate cosine schedule is applied to gradually increase the degree of privileged blurring and reward transformation. Training begins with a 100-epoch warm-up phase, during which neither privileged blurring nor reward transformation is applied. This results in a policy that converges to an The goal of this phase is to enable the privileged policy (PP) to acquire basic task-solving skills. After the warm-up, the influence of privileged blurring and reward transformation is gradually introduced following cosine schedule. Let the progress coefficient output by this schedule be l E [0, 1]. which increases from 0 to throughout training. The reward transformation is defined as:
RMA.
d = (elip(C - log((,,)) 0, mmxx) ) + 0.05],
f = (r, = [(1 - d) * (1 - E) + d]-r -r.
For vomtinuous-valued privileged observations, privileee blurring is formulated as:
0p- [o op) = op + (Kopeer e, 1(1,1).
For discrete-valued privileged observations, the formulation becomes:
op = (o op) = Ilea < fa + HEa 2 to
This gradual process encourages the PP to progressively reduce its reliance on privileged observations.
Off-Policy Replay Buffer. We use an off off-policy replay buffer train the observation encoder.
To further enhance the diversity of the data stored in the buffer, we additionally collect a batch of trajectories with mismatched privileged observations during each sampling iteration. Specifically, we randomly swap privileged observations across different environments, forcing the PP to act based on incorrect privileged inputs. This strategy prevents the observation encoder from trivially inferring privileged information solely from the behavior of the PP. For example, in the Lateral Choice task, after the warm-up phase, the PP quickly converges to directly reaching the goal without exploration. As result, the observation encoder could deduce the goal location simply by observing whether the PP moves left or right. By incorporating mismatched data, we eliminate this shortcut and promote more robust encoder training.
26

---

Uncertainty Smoothing, During sampling, we apply a temporal smoothing operation to the uncertainty values before feeding them into the PP. This necessary because the uncertainty signal can be highly volatile in the early stages of training. If the uncertainty drops abruptly to small value, the PP can immediately get the actual privileged observation and exploit without performing meaningful exploration-thhn thits behavior is referred to an "uncertainty hack." By smoothing the uncertainty over time, we mitigate such exploitation and ensure that the PP continues to explore the environment effectively until sufficient information is truly gathered.
solve the task
C.3.2 Hyperparamters
Tabs. 4 and 5 list the hyperparameters used for the quadruped robot and for the other two environments, respectively. Because the trajectory lengths, observation spaces, and task complexities differ between the quadruped robot and the two additional tasks, employ two distinct sets of hyperparameters.
Table 4: Key hyperparameters for the Quadruped Robot
Attribute
Value
Learning rate for F.,
5 x 10-5
Weight decay for E.
0.0
Minimal reward multiplication factor (6 in Eq 2
0.05
Distribution of the privileged observation noise (c, in Eq.
uniform distribution
Initial sampling standard deviation
0.35
Initial learning rate for the MLP policy of PP Initial learning rate for the MIP value function Initial learning rate for the context encoder of PP
1 x 10
5 x 10
2 x 10-5
Initial learning rate for the context encoder of the value function
5 x 10-4
Discount factor y
0.998
Number of parallel agents
900
Batch size for observation eeneeeetioneenooder training
8000
Training gradient steps of the observation encoder per epoch
450
Training gradient steps of the privileged policy per epoch
150
Cosine schedule period of the privileged blurring and reward transformation
300
Table 5: Key hyperparameters for the Biased Quadrotor and Blind-Mass Stack Tasks
Attribute
Value
Learning rate for E. Weight decay for E,
2 x 10-5
2 x 10
Minimal reward multiplication factor (6 in Eq, 2
0.05
Distribution of the privileged observation noise (c. in Eq.
uniform distribution
Initial sampling standard deviation
0.65
Initial learning rate for the MLP policy of PP Initial learning rate for the MLP value function Initial learning rate for the context encoder of PP
1 x 10-1
5 10
2 x 10-5
Initial learning rate for the context encoder of the value function Discount factor y
5 x 10-4
0.995
Number of parallel agents
4000
Batch size for observation-enncodr training
16000
Training gradient steps of the observation encoder per epoch
170
Training gradient steps of the privileged policy per epoch
150
Cosine schedule period of the privileged blurring and reward transformation
150
27

---

C.3.3 Network Architectures
In this part, we describe the architectures of the two key modules involved in USPL.
Privileged Policy The architecture of the Privileged Policy is consistent with the policy design in RESeL [25], as our training framework is built upon the RESeL codebase. A context encoder, implemented as a two-layer MLP with hidden dimensions [256, 256], first encodes the environment information based on the last observation, last action, and current observation. Here, the observation includes both the common observation and the privileged observation. The context encoder outputs 28-dimensional representation, which is then concatenated with the current observation and fed into another MLP-based policy network with hidden layers of size [256, 256]. This policy network finally outputs the action. ELU is used as the activation function for all intermediate layers.
Observation Encoder (non-image) For non-image-ber based tasks, the observation encoder first processes the common observation through a three-layer MLP with hidden sizes [512, 256, 128]. The output is then passed to temporal sub-module. This sub-module first applies three-layer MLP with hidden sizes [1024, 512, 256] to extract features, followed by a Transformer block. The Transformer block is configured with 16 attention heads, each with 64 dimensions. The output of the Transformer is then processed by 256-unit MIP to produce a 128-dimensional embedding. This embedding is further mapped the mean and log standard deviation of the current privileged observation distribution by two MLPs with hidden sizes [512, 256]. ELU is used as the activation function in all intermediate layers, while the final embedding is activated using the Tanh function.
Observation Encoder (image) In image-based tasks, the image input is first processed by CNN encoder. The encoded image features are then concatenated with the common observation features, which are separately processed by MLP with hidden layers [512, 256]. The combined representation is subsequently fed into the same temporal processing module as in the non-image encoder.
The CNN encoder is adapted from the implementation in [15] and consists of a convolutional layer with kernel size 5 and 32 output channels, followed by max pooling layer with kernel size 2 and stride 2. This followed by another convolutional layer with kernel size and 64 output channels. The resulting feature map is flattened and passed through two fully connected layers, each with 128 units. ELU activation is applied to all intermediate layers, and the final embedding is activated using the Tanh function.
C.4 Computational Infrastructure and Computational Overhead
Our experiments were conducted on GPUs with 82.6 TFLOPS of compute and 24 GB of memory. For non-image-base tasks, training was performed using a single GPU, and the training times are summarized in Tab. 6
Table 6: Training time for non-image tasks
Task
Time per iteration (s) Total training time (h)
Biased Quadrotor
37.8
10.5
Blind-Mass Stack
33.0
18.3
Lateral Choice
47.8
6.6
Midpoint Choice
57.2
31.8
Signpost Nav
53.9
11.2
Square Maze
50.1
13.9
For image-based tasks, we utilized five GPUs in parallel to accelerate training. The training durations for these tasks shown in Tab.
We find that the training speed for non-image tasks is generally acceptable. However, training time for image-based tasks is significantly longer. This inefficiency is primarily caused by three factors: (1) synchronization overhead across multiple GPUs, (2) memory and GPU memory conversions when sampling batches from the replay buffer, and (3) image rendering in simulation, where environments
28

---

Table Training time for image-based based tasks
Task
Time per iteration (s)
Total training time (h)
Signpost Nav (image)
166.9
34.8
Midpoint Choice (image)
177.0
49.0
Stairway Search
165.0
59.9
processed sequentially, leading considerable delays. Improving the training efficiency of image-hased tasks represents a valuable and meaningful direction for future work.
Table 8: Training time of different methods on the Sigmpost Nav task.
Method
Average Training Time per Epoch (s)
USPL
53.9
RMA
53.1
PrivRecons
53.2
RewReg
53.4
RESeL-PPO
46.8
We also compared the training time of different methods. We compared the average training time per epoch on the Signpost Nav task. The results are listed in Tab. 8 We find RESeL-PPO, which trains the entire DP network end-to-end with a PPO loss, is approximately 13% faster than the other methods that require additional loss computations. The training times for USPL and the other three baselines show no significant differences, indicating that USPL does introduce significant computational overhead.
D
More Experiment Results
D.1
Learning Curves
biased quadrotor
blind-mass stack
lateral choice
1.00
1.00
1.00
Success Rate
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.0
0.5
1.0
1
2
2
4
iteration midpoint choice
1e3
iteration signpost nav
1e3
iteration
1e2
square maze
1.00
1.00
1.00
Success Rate
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
2
4
0.0
1.0
Iteration
1e3
iteration
1e2
Iteration
1e3
USPL
PrivRecons
RMA
RewReg
RESeL_PPO -PO
Figure 9: Success-rate curves of different algorithms over the course of training.
presents the training curves of the different algorithms. The USPL curve is notably smooth with narrow confidence hand, indicating both robustness to random initialization and high training stability. By contrast, the pure RL baseline (RESeL PPO) exhibits
As supplement to Tab. Fig. 9
29

---

pronounced oscillations and much wider confidence region, suggesting that pure RL suffers from instability on these tasks and yields highly variable performance across different random seeds,
D.2 Experiments on Non-Robotic Environments
In addition to the robotic control tasks from previous experiments, evaluated USPL on a set of non-robotic environments. We consider the challenging credit assignment task from [32 known as Active T-Maze, setting the credit assignment length 500. Active T-Maze grid-world task where the agent's s navigable area is shaped like a "T" rotated 90 degrees clockwise (as in Fig. of [32]. The goal is located either the top-right or bottom-right corner. The optimal policy requires the agent to first move one step left to an "oracle" position, which reveals the goal's 's location. The agent must then move right to a junction and decide whether to proceed up or down based on this privileged information. 32 found that traditional Transformer-be based and RNN- based RL methods struggle to solve such credit assignment tasks efficiently. We approach the Active T-Maze task from a privileged learning perspective. For this experiment, USPL was trained using 5 different random seeds, while all other baselines were trained with random seeds due to limited computational resources.
Table 9: Success-rate comparisons in the Active T-Maze task. "-p" denotes the privileged policies.
PrivRecons
RESeL-PPO
RewReg
RMA
USPL
RMA-P
USPL-p
Success Rate
50.7+0.1
43.4+2.1
49.5+0.6
51.4+0.2
100.0+0.0
100.0
100.0
As shown in Tab. 9 USPL. achieves perfect success rate on this highly challenging task, whereas the other baselines only reach approximately 50% success rate. We hypothesize that USPL's explicit uncertainty input and reward transformation mechanisms significantly alleviate the difficulty of this long-horizon credit assignment. These results demonstrate USPL's effectiveness in solving challenging credit assignment problems.
D.3
More Uncertainty Analysis Results
D.3.1 More Uncertainty Visualizations
As an extension to Fig. 6 we visualize in Figs. 10 to 14 the evolution of agent behavior and the predicted privilegee-oobeervation distribution over time in the Lateral Choice, Square Maze, Midpoint Choice (image), Blind-Mass Stack, and Biased Quadrotor tasks, respectively. These figures reinforce the finding of Fig. 6 the uncertainty produced by the observation encoder provides sensitive indicator of information -attorringgtttggggnoo progress. Once the key information has been collected, this uncertainty contracts rapidly, allowing the policy to adjust its behavior promptly in response the current uncertainty level.
D.3.2 Manually Setting Uncertainties
Complementing Fig. 7 Figs. 15 to 18 illustrate how the policy behavior and the privileged-observation distribution change when manually modify the uncertainty fed into the privileged policy (PP). Consistent with Fig. 7 the PP proves highly sensitive to the uncertainty input: it strictly follows the paradigm of gathering information when the uncertainty is high and exploiting the current privileged observation when the uncertainty is low. Because of this property, PP greatly reduces the behavioral divergence between DP and PP.
D.4 Sensitivity Studies
To analyze USPL's sensitivity its hyperparameters, we conduct studies the newly introduced parameters: those affecting the reward transformation (Cmins Cmax) and the hyperparameter K, which controls the magnitude of the perturbation noise applied to Op- We present our sensitivity analysis for these two types of hyperparameters.
First, we analyze the sensitivity to Cmin and Cinax. which jointly define the range of the scaling coefficient applied to the original reward. This coefficient determined by ap- Specifically, log(o,) Cmin- the coefficient reaches its minimum value. As op decreases, the coefficient
30

---

(1) Ims: C13
[2] time 1.16
()) bms: 4.44
(4) Imer
biee b.J3
()) bme:: 744
(g) Time 12.27
[A] tim 18.19
Figure 10: Deployment-policy (DP) behavior and the predicted privileged-observaatie distribution in Lateral Choice. The current time shown in the top-right corner, and the red ellipse in the bottom-right visualizes the Gaussian distribution (mean and variance) produced by the observation encoder. The star marks the goal, while the triangle marks the robot's position. At the start of the episode (1-2) the predicted distribution collapses into horizontal line slightly left of center, prompting the robot to explore leftwards (3). Upon reaching the left boundary realizes the episode has not ended, implying the goal lies to the right; the distribution therefore narrows sharply (4). The robot then moves right and the predicted distribution converges on the goal position (5-8).
(2)
Ims:
(5)
Figure 11: Deployment-polic (DP) behavior and the predicted privileged-obserratii distribution in Square Maze. The layout is shown in Fig. TO. At the beginning (1-2) the predicted distribution spans large area. After leaving the start, the robot overshoots the first junction; finding no wall ahead, it deduces that the goal is to its left, and the distribution shifts leftwards (3). The robot turns left (4); at the second junction again moves forward, discovers no wall (5), and infers the goal is further left. The distribution then collapses onto the goal, and the robot heads straight towards (5-7).
31

---

Figure 12: Deployment-policy (DP) behavior and the predicted privileged-observation distribution in Midpoint Choice. The layout is shown in Fig. [0] At the episode start (1-2) the predicted distribution covers nearly the whole arena. The robot circles the central platform (3). Observing that one upper platform lacks pillar and the other has very short pillar, infers that the goal must lie in the lower half (4) and begins exploring downward. After comparing the pillar heights of the two lower platforms, identifies the lower-right pillar as the second tallest (5) and proceeds directly to the goal (6-7).
Iie:
( time: 0.8
Uime: 1.95
(4) Ims: 2.3
(5) 243
( Img:: 617
10 lee 501
Ims: 5.54
Figure 13: Deployment-police (DP) behavior and the predicted siivileged-observate distribution in Blind-Mass Stack. Red and blue bars depict the two-dimensional Gaussian output from the observation encoder; the dashed line between the two triangles indicates the true divileged-obbervation value. Initially, uncertain which cube red, the manipulator grasps a red cube (without knowing it is red), records its mass, and puts it down (1-3). To verify its color, next grasps green cube with known ID and mass (4-5), uses it to calibrate the force sensor, and, combining this with the earlier measurement, infers that the first cube was red. At this point the red and blue bars converge the true privileged value (6), and the manipulator successfully stacks the red cube on the green cube.
32

---

(1) Intr 6.01
G line 0.16
Ims: 0.35
(6) Ims: 6.57
31 line: 0.35
c bing: 1.01
(1) line: 1:31
Figure 14: Deployment-policy (DP) behavior and the predicted privileged-observation distribution in Biased Quadrotor. Red, blue and green bars represent the observation encoder's predictions of altitude, pitch and roll, respectively; dashed lines between triangles show their true values. At the beginning (1-2) the estimates deviate from the state, but the quadrotor descends (3-4) the sensors become calibrated and the predictions align with reality, The quadrotor then takes off again and completes the task (5-7).
line
(9)
(1)
Figure 15: Deployment-pole policy (DP) behavior and the predicted privileged observation distribution in Lateral Choice with manually adjusted uncertainty. The current time is shown in the top-right corner, while the red ellipse in the bottom-right visualizes the Gaussian output (mean and variance) from the observation encoder. The star marks the goal and the triangle the robot's 's position; the top-left corner displays the externally supplied uncertainty value. the start the episode (1-2) the robot moves left to test whether the goal is there. After reaching the left boundary, realizes the episode has not ended, so the predicted distribution shifts rightwards (3-4). Because the strategy still perceives high uncertainty, it remains in place and continues exploring (5-6). Only after we set the uncertainty to lower value does it turn and head directly for the goal (7-8).
33

---

Figure 16: DP behavior and the predicted privileged observvation distribution in Signpost Nav under manual uncertainty manipulation. The top-right corner shows time; the bottom-right red ellipse depicts the Gaussian output the observation encoder. The star indicates the goal, the triangle the robot. The top-left corner shows the current uncertainty input, and the upper-right inset presents the depth-camera view. Early in the episode (1-2) the robot moves toward the signpost infer the goal's location. Once the signpost appears in the depth image (3) the goal position is immediately inferred, yet the robot continues to approach and then waits near the signpost because the uncertainty still high (4-6), After manually lower the uncertainty, it turns back and moves straight toward the goal (7-8).
increases, until Cmin log(op)) Cmaxx which point the coefficient reaches its maximum value. We tested various combinations of Cmin and Cmax in both the Signpost Nav and Biased Quadrotor tasks. The results presented in Tab. [T)
For the Signpost Nav task, the results (first three rows of Tab. TOP show that a narrow range for the scaling coefficient limits the effectiveness of the reward transformation, yiclding success rate below 90%. As the range expands, the success rate consistently remains above 97%. The Biased Quadrotor task exhibits similar trend: the narrowest range results in very low success rate (12.5%). However, this task also reveals that an excessively large range can be detrimental. As shown in the last two rows, when the range becomes too wide, performance drops significantly to 14.0% and 53. 1 1%, respectively.
Overall, these results suggest that as long as the range of the scaling coefficient not excessively large or small, the policy's success rate is largely insensitive to the specific values of Cmin and Cmax- Within an appropriate range, different combinations of these parameters can achieve near-optimal performance.
Table 10: Sensitivity Studies Cuin and Cmax-
Cmin
Cuax
Signpost Nav
Biased Quadrotor
log(0.058)
Cmin log(0.053)
38.6 + 2.5
12.5 + 2.1
log(0.06)
Gmin log(0.05)
42.7 + 9.3
97.7 + 0.2
log(0.075)
Cmin log(0.04)
87.7 + 7.1
97.8 + 0.6
log(0.1)
Cmin log (..0)
97.6 0.5
96.0 0.3
log(0.2)
Cuin log(0.01)
9.2
97.0
log(0.4)
Cmin log(0.005)
98.2
14.0 0.7
log(0.6)
Cmin log(0.003)
98.1 0.2
31.7
34

---

Figure 17: DP behavior and the predicted crivilgedd-oseevaan distribution in Stairway Search while manually varying uncertainty. The current time is shown in the top-right, the Gaussian prediction appears red ellipse in the bottom-right, the star marks the goal and the triangle the robot. The top-left corner shows the supplied uncertainty value, and the upper-right inset displays the depth- camera image. At the beginning (1-3) the robot approaches the upper platform edge and looks down for stairs. Finding none, the predicted distribution collapses to the lower half. Because the supplied uncertainty remains high, the policy keeps inspecting the lower edge repeatedly (4-9). After we reduce uncertainty at step 10, the robot descends, locates the stairs, and walks to the goal.
Next, we perform a sensitivity analysis on K, which controls the magnitude of the perturbation noise applied Op- The results are presented in Tab. [I the Signpost Nav task, the value of K has almost no impact on the policy's success rate. For the Biased Quadrotor task, however, smaller values of K (e.g., K < 10) lead a significant drop in the success rate. For K > 10, the policy consistently achieves a high success rate. This indicates that USPL performs well across a wide range of K values, and its performance becomes largely insensitive K once it sufficiently large (K 2 10).
Table 11: Sensitivity Studies on K.
K
Signpost Nav
Biased Quadrotor
95.7 1.6
40.7 + 20.8
2.5
98.4 + 0.4
9.6 1.2
97.7
8.2
10.0
97.8
9.67
20.0
98.0
9..
40.0
97.8 0.3
9.3
35

---

2.55 Meow
line: 3.1
4.0 Wwoo
7.61
12.03
Figure 18: DP behavior and the predicted privileged-obsee with manual uncertainty control. The top-right corner shows time. Red and blue bars represent the two-dimensional Gaussian output of the observation encoder; the dashed line between two triangles indicates the true privileged-observattee
distribution in Blind-Mas Stack
value. The top-left corner lists the current uncertainty input. To determine which cube is the 0.75 kg red cube, the manipulator sequentially grasps red and then a green cube (1-5). Although the predicted privileged observation converges to the true value, the strategy still detects high uncertainty and keeps repeating the calibration behavior pehavior-grasping red and green cubes alternately (6-10). Only after we lower the uncertainty does it finally pick up the red cube and stack it the green one.
36


Reviews:
[{"review_id": "JqMAiznpox", "reviewer": "Reviewer_ru4E", "summary": "This paper considers the problem of learning an effective deployment policy from partial observations that exhibit less behavioral divergence compared to the privileged policy, which has access to privileged observations during training. This work proposes to incorporate the uncertainty of the (partial) observation encoder (that predicts the privileged observation) into the privileged policy learning. The idea is that high uncertainty will drive the policy to collect more information that would reduce the behavioral discrepancy. Experimental results on different environments show the effectiveness of the method.", "questions": "The limitation section talks about the elimination of the DP encoder during PP training. Even if you opt for a proxy encoder, does it not need to be trained alongside PP?\n\nThere is much room for improvement in the writing part. Sections 1 and 2 often sound repetitive.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n1. The idea of injecting uncertainty into the privileged learning is interesting. This enables to capture the discrepancy of the deployment scenarios and facilitates generalization. Also, the controlled blurring of privileged information helps to imitate a deployment-level scenario. \n\n2. The additional loss functions to learn the observational encoder are simple and easy to implement.\n\n3. The experiments cover image and non-image observations, and the evaluations show that the deployment policy can achieve performance similar to the privileged policy. Also, the uncertainty estimation analysis and ablation study sections provide meaningful and rational insights. \n\n\n### Weaknesses:\n\n1. Eqs 3 and 4 introduce new hyperparameters, thus making it more critical to optimize. I do not see any sensitivity analysis to those hyperparameters.\n\n2. The experiments are limited to robotic environments. It would be nice to see some experiments on other applications of privileged learning.", "was_revised": true, "final_justification": "This paper incirporates uncertainty estimation within the privileged learning farmework in a simple and effective way. The addition of non-robotic experiments and hyperparameter sensitivity analysis added during the rebuttal provide more insights. I believe adopting a more theoretical discussion along with additional experiments such as other means of corrupting privileged information would strengthen the paper.", "rebuttal": "### W1\n\n> Lack of sensitivity analysis to hyperparameters.\n\nThank you for raising this important point. Our main hyperparameters include those that affect the reward transformation ($C_{\\min}$, $C_{\\max}$) and the hyperparameter $K$, which controls the magnitude of the perturbation noise to $o_p$. We present the results of our sensitivity analysis for these two types of hyperparameters.\n\nFirst, we conduct an sensitivity study on $C_{\\min}$ and $C_{\\max}$, which affect the scale coefficient for the original reward. If $\\log(\\sigma_p) > C_{\\min}$, the scale coefficient reaches its minimum. Conversely, the scale coefficient increases as $\\sigma_p$ decreases, until $C_{\\min} - \\log(\\sigma_p) > C_{\\max}$, at which point the coefficient reaches its maximum and stops increasing. Therefore, $C_{\\min}$ and $C_{\\max}$ jointly determine the range of the scale coefficient. We tested different combinations of $C_{\\min}$ and $C_{\\max}$ in both the Signpost Nav and Biased Quadrotor tasks, with the results shown in the table below. \n\nFor the Signpost Nav task, the first three rows show that a relatively small range for the scale coefficient limits the impact of the reward transformation, resulting in a task success rate below 90%. As the range expands, the success rate consistently stays above 97%. The Biased Quadrotor task exhibits a similar trend at the lower end of the range, with the narrowest range resulting in a very low success rate ($12.5\\%$). However, it also reveals that an overly large range can be detrimental. As shown in the last two rows, when the range becomes too wide, the performance drops significantly to $14.0\\%$ and $53.1\\%$, respectively.\n\nOverall, the results suggest that as long as the range of the scale coefficient is not excessively large or small, the policy's success rate is largely insensitive to the specific values of $C_{\\min}$ and $C_{\\max}$. Within an appropriate range, different combinations of these parameters are capable of achieving near-optimal performance.\n\n| $C_{\\min}$    | $C_{\\max}$             | Signpost Nav   | Biased Quadrotor |\n| ------------- | ---------------------- | -------------- | ---------------- |\n| $\\log⁡(0.058)$ | $C_{\\min}-\\log(0.053)$ | $38.6 \\pm 2.5$ | $12.5\\pm 2.1$    |\n| $\\log(0.06)$  | $C_{\\min}-\\log(0.05)$  | $42.7\\pm 9.3$  | $97.7\\pm 0.2$    |\n| $\\log(0.075)$ | $C_{\\min}-\\log(0.04)$  | $87.7\\pm 7.1$  | $97.8\\pm 0.6$    |\n| $\\log(0.1)$   | $C_{\\min}-\\log(0.02)$  | $97.6\\pm 0.5$  | $96.0\\pm 0.3$    |\n| $\\log(0.2)$   | $C_{\\min}-\\log(0.01)$  | $97.8 \\pm 0.2$ | $97.0\\pm 0.4$    |\n| $\\log(0.4)$   | $C_{\\min}-\\log(0.005)$ | $98.2\\pm 0.2$  | $14.0\\pm 0.7$    |\n| $\\log(0.6)$   | $C_{\\min}-\\log(0.003)$ | $98.1\\pm 0.2$  | $53.1\\pm 31.7$   |\n\nNext, we perform a sensitivity analysis on $K$, which affects the magnitude of the perturbation noise to $o_p$. The results are in the table below. The results in Signpost Nav show that the value of $K$ had almost no impact on the policy's success rate. As for the Biased Quadrotor task, the results show that a smaller value of $K$ leads to a significant drop in the success rate, whereas for $K \\geq 10$, the policy consistently achieves a high success rate. USPL performs well across a wide range of values for $K$. After $K \\geq 10$, the policy's success rate is largely insensitive to the value of $K$.\n\n| $K$  | Signpost Nav   | Biased Quadrotor |\n| ---- | -------------- | ---------------- |\n| 1    | $95.7 \\pm 1.6$ | $40.7\\pm 20.8$   |\n| 2.5  | $98.4 \\pm 0.4$ | $9.6\\pm 1.2$     |\n| 5.0  | $97.7 \\pm 0.5$ | $8.2\\pm 0.5$     |\n| 10.0 | $97.8 \\pm 0.2$ | $96.7 \\pm 0.6$   |\n| 20.0 | $98.0 \\pm 0.2$ | $97.7\\pm 0.3$    |\n| 40.0 | $97.8 \\pm 0.3$ | $97.5\\pm 0.3$    |\n\nWe will include these results and the corresponding discussion in the revised version of our paper.\n\n### W2\n\n> It would be nice to see some experiments on other applications of privileged learning.\n\nThank you for this constructive suggestion. We agree that demonstrating the applicability of our method beyond robotics environments would significantly strengthen the paper. As a result, we test our method on the challenging credit assignment task discussed in [1], Active T-Maze (with the credit assignment length set to 500). Active T-Maze is a grid-world task where the agent's navigable area is shaped like a \"T\" rotated 90 degrees clockwise (as in Fig. 1 of [1]). The goal is located at either the top-right or bottom-right corner. The optimal policy for the agent is to first move one step left to an \"oracle\" position to reveal the goal's location, then move right to a junction, and finally decide whether to move up or down based on the privileged information it received. [1] found that traditional Transformer-based and RNN-based RL methods struggle to solve such credit assignment tasks efficiently. We approach the Active T-Maze task from a privileged learning perspective, with the results shown below. In this experiment, USPL was trained with 5 different random seeds, while other baselines were trained with 3 random seeds due to limited computational resources.\n\n| Method       | PrivRecons    | RESeL-PPO     | RewReg        | RMA           | USPL (ours)    | RMA-p           | USPL-p         |\n| ------------ | ------------- | ------------- | ------------- | ------------- | -------------- | --------------- | -------------- |\n| Success Rate | $50.7\\pm 0.1$ | $43.4\\pm 2.1$ | $49.5\\pm 0.6$ | $51.4\\pm 0.2$ | $100.0\\pm 0.0$ | $100.0 \\pm 0.0$ | $100.0\\pm 0.0$ |\n\nAs the results show, USPL achieves a perfect success rate on this highly challenging task, while the other baselines only achieve around a 50% success rate. We hypothesize that USPL's explicit uncertainty input and reward transformation significantly alleviate the difficulty of credit assignment. These results demonstrate the effectiveness of USPL on credit assignment problems. We will add these findings and discussion to the revised version of our paper.\n\n[1] When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment. NeurIPS 2023.\n\n### Q1\n\n> Even if you opt for a proxy encoder, does it not need to be trained alongside PP?\n\nYou are right, the proxy encoder does need to be trained alongside the PP. However, this proxy encoder would be more lightweight than the observation encoder we currently use, leading to faster training. This is especially true for image-based inputs. Currently, training the PP requires processing images with the observation encoder, which is slow. By contrast, using a proxy encoder to process the generally lower-dimensional historical privileged observations would be much more efficient.\n\n### Q2\n\n> There is much room for improvement in the writing part. Sections 1 and 2 often sound repetitive.\n\nThank you for this valuable feedback. We will thoroughly revise the writing of the entire paper in the next version, with a particular focus on Sections 1 and 2, to improve clarity and readability.\n\n---\n\nThank you once again for your detailed review and constructive suggestions. Your feedback has been crucial in helping us enhance the quality of our paper. In light of these clarifications and additional experiments, we would be grateful if you would consider a higher rating. We would be more than happy to provide any further clarifications if needed."}, {"review_id": "hMvr8UZGOb", "reviewer": "Reviewer_mnJ9", "summary": "The authors propose an approach for operating in POMDPs where the belief state (or a privileged information) is inferred via a special encoder. This encoder is trained by predicting belief state (or a privileged information) during training.", "questions": "Could the authors more clearly explain how does their work differ from [1] and why it is novel?", "limitations": "Explained above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The proposed approach bears strong similarity to a method described in [1]. In [1], similarly to the current work, the authors have privileged information during training (belief state) and they train a special network (belief network) to predict the belief state (in the terminology of this work, it is called privileged information). In [1], there is also a partial information which can be available during sampling (called a \"cue\"), which is called \"target information\" in the current work. The difference between [1] and current work is that in [1], the authors use RNNs, while here the setting is not limited to RNNs and there is an additional blurring of the privileged information.\n\nIn light of [1], the novelty of the current work is quite limited, and the authors should discuss the novelty more explicitly in their paper.\n\nThe experiments of the current work are promising but the results are still limited to low-dimensional privileged observations (but [1] would also suffer from the same limitations).\n\nReferences:\n\n[1] Meta reinforcement learning as task inference, Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A. Ortega, Yee Whye Teh, Nicolas Heess, 2019", "was_revised": true, "final_justification": "The authors addressed some of my comments and have provided an extensive sensitivity analysis of their method.\n\nI think the approach is sound, however, it should be more clearly presented how it differs from the related work as well as provide evidence of its scalability to high-dimensional situations.", "rebuttal": "### W1\n\n> Could the authors more clearly explain how does their work differ from [1] and why it is novel?\n>\n> [1] Meta reinforcement learning as task inference, Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A. Ortega, Yee Whye Teh, Nicolas Heess, 2019\n\n\nWe thank the reviewer for pointing out this relevant work. USPL is fundamentally distinct from [1] as USPL is a privileged learning (PL) framework, while [1] is not. This core difference leads to distinct training methodologies, primary challenges, and novel contributions. We highlight three key distinctions:\n\n1. Usage of privileged observations \n\n- **USPL:** During training, the Privileged Policy (PP) has **direct, real-time access** to privileged observations. This allows the PP to focus on completing the task without needing to learn active information-gathering behaviors, leading to a more efficient RL training process. Our primary focus is then on bridging the \"behavioral divergence\" between this PP and a deployment policy that operates on target observations.\n- **Method in [1]:** The policy **never** directly uses privileged information as an input. Instead, privileged data is used solely to supervise an auxiliary loss for a belief encoder. Consequently, their single policy must learn complex, active information-gathering behaviors to infer the missing information, making the policy training significantly more challenging.\n\n2. Core technical contribution\n\n- **USPL:** A central contribution of our work is the **prediction and utilization of uncertainty**. We explicitly model the uncertainty of the privileged observation prediction and leverage this uncertainty to mitigate behavioral divergence.\n- **Method in [1]:** The concept of prediction uncertainty is **entirely absent**. Their work focuses on task inference through belief state representation without considering the reliability of that belief.\n\n3. Empirical comparison\n\n- Our baseline \"PrivRecons\" in Table 1 is conceptually analogous to the approach in [1]. PrivRecons uses a observation encoder trained via supervised learning to predict privileged observations, and a single policy uses this encoder's output for both training and deployment.\n\n- The experimental results clearly show that USPL substantially outperforms PrivRecons. This result empirically validates the benefits of our privileged learning framework and our novel use of uncertainty, highlighting the advantages over the methodology presented in [1].\n\nWe hope this comparison clarifies the distinctions of our approach and its contributions relative to [1]. We will include the comparisons with [1] in the revised version of our paper.\n\n### W2\n\n> The experiments are still limited to low-dimensional privileged observations.\n\nThis is indeed a known limitation of the current work. Extending USPL to scenarios with high-dimensional, redundant, and time-varying privileged observations is indeed a challenging and important direction for future research. Without additional techniques, the performance of USPL in such settings would heavily depend on the capacity and learning efficiency of the observation encoder.\n\nA promising approach to address this would be to integrate dimensionality reduction techniques. For instance, [2] and [3] introduced a privileged encoder that maps high-dimensional privileged observations into a low-dimensional latent space, and they have demonstrated excellent results. Similarly, USPL could adopt such a method to first project the privileged observations into a compact latent space and then train the observation encoder to predict this lower-dimensional representation.\n\nWe will continue to conduct more in-depth research on this limitation in our future work. We will also extend the discussion of this limitation in the revised version of our paper.\n\n\n\n[2] RMA: Rapid motor adaptation for legged robots. RSS 2021.\n\n[3] Deep whole-body control: Learning a unified policy for manipulation and locomotion. CoRL 2022.\n\n---\n\nWe would like to express our sincere gratitude once again for your insightful and constructive suggestions. We hope that our response and the proposed revisions have thoroughly addressed your concerns. We believe the paper is significantly stronger as a result of your guidance, and we would be grateful if you would consider these clarifications in your final assessment. Please do not hesitate to let us know if any further questions arise."}, {"review_id": "f9jEwylIMd", "reviewer": "Reviewer_GVJr", "summary": "The paper proposed USPL, using the uncertainty estimation to guide DP to minimize the divergence between DP and PP. Experiments show that USPL can achieve superior performance than baseline methods.", "questions": "1. Have you compared the compuation cost across all methods tested, including the baseline methods?\n2. If the removing uncertainty estimation does not have the most significant improvement compared with other components as shown in 4.5, is the paper's argument about the importance of uncertainty estimation still valid? Is it possible to apply these components to the baseline methods and also significantly improves the performance?", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The motivation is clear and using the uncertainty estimation to contrain the behavior of DP sounds reasonable.\n2. The empirical performance of the proposed method is signficantly better than the baseline methods in the environment provided in the paper.\n3. Extensive empirical analysis and ablation studies of the proposed method. I am no expert in privileged learning so not sure whether the task range is limited though.\n\nWeaknesses:\n1. There seems to be a lack of comparison in terms of the computation cost with the baseline methods, since it includes training and inference of the encoder model.  See Q1.\n2. In the ablation studies, the performance improvement seems to largely come from other compoenents like blurring and modifying the reward, which make it questionable that what the main contribution of the propose method should provide. See Q2.", "was_revised": true, "final_justification": "Thank you for the rebuttal. It solved most concerns and I will keep my score.", "rebuttal": "### W1\n\n> Lack of comparison in terms of the computation cost with the baseline methods.\n\nThank you for your valuable feedback. We agree that a discussion on computational cost is crucial for a complete comparison. In our experiment, all baseline methods were implemented with identical network architectures to ensure a fair comparison. The differences between the baselines lie in the loss functions used for training various modules. We observed that the primary computational overhead stems from environment interaction and network updates, rather than the specific loss calculations. Consequently, the overall computational costs for different methods are quite similar.\n\nWe show the average training time per epoch for each method on the Signpost Nav task in the following:\n\n| Method     | Average Training Time per Epoch (s) |\n| :--------- | :---------------------------------- |\n| USPL       | 53.7                                |\n| RMA        | 53.1                                |\n| PrivRecons | 53.2                                |\n| RewReg     | 53.4                                |\n| RESeL-PPO  | 46.8                                |\n\nAs shown, RESeL-PPO, which trains the entire DP network end-to-end with a PPO loss, is approximately 13% faster than the other methods that require additional loss computations. The training times for USPL and the other three baselines show no significant differences. We will incorporate this comparison into the revised version of our paper.\n\n### W2\n\n> In the ablation studies, the performance improvement seems to largely come from other compoenents like blurring and modifying the reward, which make it questionable that what the main contribution of the propose method should provide. Is it possible to apply these components to the baseline methods and also significantly improves the performance?\n\nWe sincerely thank the reviewer for this insightful question. You have pinpointed a crucial aspect of our ablation study that merits a more detailed explanation:\n\n1. We first want to address the question of whether privileged blurring and reward modification could be applied to baseline methods. These components are not standalone techniques, and are **fundamentally reliant on the uncertainty estimation $\\sigma_p$**, which is the central contribution of USPL. They cannot be applied to baseline methods that do not estimate this uncertainty. We also recognize, with thanks to your comment, that our original table caption was a source of confusion. The \"w/o uncert\" variant does, in fact, still compute and use uncertainty for blurring and reward modification. It only omits uncertainty as a direct input to the PP. We will rename this variant to prevent future ambiguity.\n\n2. Regarding the performance gain comparison of different components, we would like to clarify that our ablation results should be viewed not as a competition between components, but as a demonstration of how they play distinct and complementary roles in a multi-stage process.\n\n   An optimal DP would ideally adapt its behavior online based on its current **Progress of Collecting Privileged Information (PCPI)**, which is inversely correlated with our uncertainty estimation $\\sigma_p$. Our framework aims to build such a policy based on PP, and this requires satisfying two essential conditions:\n\n   - First, the PP must **explore and learn the optimal behaviors for different levels of PCPI**. During training, our privileged blurring and reward transformation components work in concert to achieve this foundational goal. **Privileged blurring** simulates states of high uncertainty, forcing the policy to learn how to act when privileged information is unreliable. Concurrently, **reward transformation** incentivizes the policy to actively explore and discover actions that reduce uncertainty (i.e., increase PCPI). The necessity of this exploration stage is evident in Table 2: removing either privileged blurring or reward transformation leads to a collapse in the success rate, proving they are essential for learning the necessary behaviors.\n\n   - Second, the PP must be able to **dynamically select the appropriate behavior online** based on its current PCPI. This is accomplished by providing the uncertainty $\\sigma_p$ as a direct input to the policy. The uncertainty serves as a real-time signal, allowing the policy to intelligently switch between information gathering (when uncertainty is high) and task execution (when uncertainty is low). However, this online adaptation is only effective if the optimal behaviors have already been discovered via privileged blurring and reward transformation. This explains the poor performance of `only uncert` in Table 2. \n\n   Incorporating uncertainty as a policy input is crucial for achieving optimality and reliability. To quantify its importance, we analyzed the task failure rate, a key metric for evaluating policy reliability. The results are presented in the table below. As the table shows, even with the powerful exploration capabilities provided by blurring and reward transformation, the policy still has a notable average failure rate of 9.4%. Providing uncertainty as a direct policy input **reduces this failure rate by 80.9%**, bringing it down to just 1.8%.\n\n   This result powerfully illustrates that while blurring and reward shaping are essential for teaching the policy *what to do*, the uncertainty input is the crucial final piece that tells the policy *when to do it*. It is this ability to dynamically adapt based on a direct measure of its own uncertainty that makes the policy reliable, and ultimately more successful.\n\n   | Task                | Failure Rate (w/o Uncert. Input) | Failure Rate (w/ Uncert. Input) | Failure Rate Reduction |\n   | ------------------- | -------------------------------- | ------------------------------- | ---------------------- |\n   | **Signpost Nav**    | 3.0%                             | 2.2%                            | 26.7%                  |\n   | **Square Maze**     | 5.3%                             | 1.5%                            | 71.7%                  |\n   | **Lateral Choice**  | 8.0%                             | 1.2%                            | 85.0%                  |\n   | **Midpoint Choice** | 21.1%                            | 2.3%                            | 89.1%                  |\n   | **Average**         | 9.4%                             | 1.8%                            | 80.9%                  |\n\n   ---\n\n   We are truly grateful for your perceptive feedback. It prompted a deeper analysis that has, we believe, significantly improved the paper. We hope our explanation clarifies the unique role of each component within our framework. We sincerely look forward to any further discussion and hope that these clarifications and new results will lead you to view our contributions even more favorably."}, {"review_id": "NvC7fVmFej", "reviewer": "Reviewer_dzCL", "summary": "The paper addresses the challenge of behavioural discrepancies between the privileged policy (PP) and the deployment policy (DP) in the context of privileged learning. The authors propose a novel framework, Uncertainty-Sensitive Privileged Learning (USPL), which estimates the prediction uncertainty of privileged observations and then uses prediction uncertainty to guide the PP with reward transformation and privileged observation blurring. Empirical evaluations on nine diverse tasks demonstrate that USPL achieves superior deployment performance, particularly in reducing behavioural discrepancies, when compared to several strong baselines.", "questions": "Along with the points mentioned in the section on weakness, I have two further questions. \n\n1. I do not follow the choice of a uniform distribution for the noise in Equation (3). The authors claim that the PP is robust to Gaussian noise, but they do not clarify why uniform noise was chosen instead. If the noise is used to simulate uncertainty or model robustness, Gaussian noise would be the more canonical choice, especially if the observation encoder models uncertainty via Gaussian distributions. Is there any requirement that the noise must be bounded and symmetric for the analysis or experiments? If the uniform noise is meant to be bounded, that rationale should be made explicit. Without such clarification, the design choice feels somewhat arbitrary.\n\n2. In Table 1, the success rate for the Biased Quadrotor task under USPL is reported as nearly perfect with a variance of 0.00, which is suspicious at first glance. Could the authors clarify this point? If the performance is indeed identical across all seeds?", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The motivation to reduce behavioural discrepancies between the PP and the DP due to asymmetric observation spaces is clearly stated and, to the best of my knowledge, remains relatively underexplored. The author discusses the limitations of the existing approach (e.g., RL-based fine-tuning DP, regularization) and proposes an original direction of using the prediction uncertainty of privileged observations to guide the PP. I find the idea intuitive and sound. The framework is evaluated across a diverse set of challenging tasks, and the results consistently demonstrate superior performance over the baselines, particularly in reducing behavioural discrepancies. \n\nMy major concern lies in the lack of theoretical justification for the proposed framework. It remains unclear to me how the estimated prediction uncertainty directly influences the reduction in behavioural discrepancies between the PP and DP. While the authors provide explanations for the role of uncertainty in guiding the privileged policy, these arguments are largely intuitive. A formal analysis, such as an error bound between uncertainty and discrepancies. For instance, I suspect that the BD could be upper bounded by a function of the uncertainty in the privileged observation prediction, along with other common factors such as the discount factor, the observation noise level, or the dimensionality of the state space. While I agree it could be hard, a further discussion of challenges could be helpful. It would still be valuable for the authors to include a discussion of the key theoretical challenges and possibly sketch out the conditions under which such a bound might hold. This would help clarify the underlying assumptions of the framework and restrictions.", "was_revised": true, "final_justification": "I have no further questions and appreciate the clarifications provided by the authors. My only remaining concern is that the performance of the proposed approach appears to rely on the choice of several hyperparameters. While the authors have included a sensitivity analysis, I still have some reservations about the generalizability of the method. I have increased my score, but I leave the final decision to the AC and other reviewers.", "rebuttal": "### W1\n\n> Theoretical justification.\n\nWe are grateful to the reviewer for this insightful comment regarding the theoretical justification for our proposed framework. Heeding this suggestion, we have developed a formal analysis to address this concern. Specifically, by combining a Lipschitz assumption on the policy with a result from Lemma 4 in [1], we derive the following cumulative error bound that formally connects prediction uncertainty to the behavioral discrepancy between the PP and DP.\n\nLet the policy in USPL be denoted as $\\pi(\\cdot\\mid o_ c, o_ p, \\sigma_ p)$, where $o_ c, o_ p,$ and $\\sigma_ p$ represent the common observation, privileged observation, and uncertainty estimation, respectively. The key difference between the PP and the DP lies in the source of $o_ p$. Let the privileged observation input to the PP at timestep $h$ be $o_ p^h$ (the ground truth) and the one input to the DP be $\\hat{o}_ p^h$ (the prediction). We assume there exists an error bound $\\epsilon^h$ such that:\n$$\n\\Vert o_ p^h - \\hat{o}_ p^h \\Vert_ 2 \\leq \\epsilon^h.\n$$\n\nHere, $\\epsilon^h$ represents the upper bound on the prediction error of the privileged observation. We assume that the policy $\\pi(\\cdot\\mid o_ c, o_ p, \\sigma_ p)$ is Lipschitz continuous with respect to the privileged observation $o_p$:\n\n$$\nD_ {TV}(\\pi(\\cdot\\mid o_ c, o_ p^h, \\sigma_ p), \\pi(\\cdot\\mid o_ c, \\hat{o}_ p^h, \\sigma_ p)) \\leq L(o_ c, \\sigma_ p) \\Vert o_ p^h - \\hat{o}_ p^h \\Vert_ 2,\n$$\n\nwhere $L(o_ c, \\sigma_ p)$ is the Lipschitz constant of $\\pi$ with respect to $o_ p$ for a given $o_ c$ and $\\sigma_ p$. This constant will be large if the policy is highly sensitive to privileged observations. We assume $\\epsilon^h$ is known and is used as the uncertainty input to the policy. Let $\\pi_ h^{PP} = \\pi(\\cdot\\mid o_c, o_ p^h, \\epsilon^h)$ and $\\pi_ h^{DP} = \\pi(\\cdot\\mid o_ c, \\hat{o}_ p^h, \\epsilon^h)$. Under these assumptions, the following inequality holds for the cumulative discrepancy between the state visitation distributions:\n\n$$\n\\frac{1}{H}\\sum_ {h=1}^H D_ {TV}(d_ h^{\\pi^{PP}}, d_ h^{\\pi^{DP}}) \\leq \\sum_ {h=1}^H \\mathbb{E}_ {o_ c \\sim d_ h^{\\pi^{PP}}}\\left[L(o_ c, \\epsilon^h)\\epsilon^h\\right],\n$$\n\nwhere $d_ h^{\\pi^{PP}}$ and $d_ h^{\\pi^{DP}}$ are the visitation distributions of $o_ c$ at step $h$ for the PP and DP, respectively, and $H$ is the horizon. A detailed derivation of this bound will be provided in the revised paper.\n\n**Relationship between the Error Bound and our USPL Framework:**\n\nThis bound formally demonstrates that the cumulative behavioral discrepancy is upper-bounded by a function of the privileged observation prediction error ($\\epsilon^h$) and the policy's sensitivity to that information ($L(o_c, \\epsilon^h)$). To minimize this compounding error, one can pursue two primary strategies:\n\n* **(S1) Minimize the prediction error $\\epsilon^h$**: This involves improving the prediction accuracy of the observation encoder.\n* **(S2) Reduce the policy's sensitivity to the privileged observation**: This involves decreasing the Lipschitz constant $L(o_c, \\epsilon^h)$.\n\nIn USPL, S1 is directly addressed by the optimization objective of our observation encoder and reward transformation, both of which aim to reduce $\\epsilon^h$. S2 is realized through privileged blurring, which adds noise to the privileged observation during training to reduce the policy's over-reliance on it. However, at the beginning of an episode, when the agent's perception of the environment is insufficient, $\\epsilon^h$ is typically large. At the same time, solving the task often requires leveraging the privileged observation, which means $L(o_ c, \\epsilon^h)$ also needs to be large. To prevent the compounding error from escalating due to both $\\epsilon^h$ and $L(o_ c, \\epsilon^h)$ being high simultaneously, USPL also achieves an adaptive dependence on privileged observations by taking uncertainty as policy input: \n\n* **When uncertainty ($\\approx \\epsilon^h$) is high,** the privileged observation is very noisy. Thus, the policy learns to distrust the privileged input (effectively reducing $L(o_ c, \\epsilon^h)$). Guided by the transformed reward, it prioritizes exploration to gather information, thereby reducing future uncertainty.\n* **As uncertainty ($\\approx \\epsilon^h$) decreases,** the privileged observation becomes more reliable. The reward for task completion is magnified by reward transformation. As a result, the policy learns to increase its reliance on the privileged observation (increasing $L(o_c, \\epsilon^h)$) to efficiently solve the task.\n\nThis coordinated control of both $\\epsilon^h$ and $L(o_c, \\epsilon^h)$ effectively tightens the upper bound on the cumulative error. This, in turn, minimizes the behavioral discrepancy between the PP and DP, mitigating performance degradation at deployment.\n\n**Empirical Evidence:** As demonstrated by our experiments with manually set uncertainty (Figure 7, Figs. 14-17 in the appendix, and videos in the supplementary material), USPL policy successfully learns this adaptive capability. When provided with high uncertainty, the policy indeed refrains from exploiting the privileged observation—even when the prediction is accurate—and instead prioritizes exploration. It only begins to leverage the privileged observation to complete the task once the uncertainty is reduced, validating our theoretical analysis.\n\nWe will add a new subsection in Sec. 3 of the revised paper to elaborate on this theoretical analysis, thereby strengthening the paper's rationale.\n\n[1] Understanding adversarial imitation learning in small sample regime: A stage-coupled analysis. arXiv preprint arXiv:2208.01899.\n\n### Q1\n\n> The choice of a uniform distribution to randomize the privilieged observations.\n\nWe apologize for the confusion. The form of noise perturbation in Eq. (3) is indeed a hyperparameter. We found that when using Gaussian noise ($\\mathcal{N}(\\cdot\\mid 0, \\sigma_p^2)$), even with a high fixed uncertainty $\\sigma_p$, the PP could sometimes still complete the task without performing information-gathering actions. This led to a significant behavioral discrepancy between the PP and the DP. On the other hand, increasing the noise magnitude using a distribution like $\\mathcal{N}(\\cdot\\mid 0, (K\\sigma_p)^2)$ with $K>1$ often resulted in perturbed observations $\\hat{o}_p$ that fell outside their practical range and the output range of the observation encoder's mean $\\mu_p$. These are values that would never be encountered during deployment, and their frequent occurrence could potentially reduce training efficiency. Moreover, in our experiments, the environment parameters corresponding to $o_p$ are sampled from a bounded uniform distribution. By using uniform noise for perturbation, we can adjust the distribution's bounds to ensure that the final $\\hat{o}_p$ remains within a valid range. The $\\hat{o}_p$ with the maximum perturbation follows the prior distribution of $o_p$. We will incorporate the aforementioned discussion into Sec. 3.2 of the revised version of the paper.\n\nThe table below compares the success rate using the perturbation method from Eq. (3) versus a Gaussian distribution $\\mathcal{N}(\\cdot\\mid 0, (K\\sigma_p)^2)$.\n\n| Task             | USPL           | USPL w/ Gaussian Noise |\n| ---------------- | -------------- | ---------------------- |\n| Signpost Nav     | $97.8 \\pm 0.2$ | $97.2 \\pm 0.3$         |\n| Biased Quadrotor | $96.7 \\pm 0.6$ | $17.8 \\pm 6.3$         |\n\nIn the Signpost Nav task, the performance difference is minor. However, in the Biased Quadrotor task, the performance with Gaussian noise is substantially worse.\n\n### Q2\n\n> The anomalous success rate standard error for the Biased Quadrotor in Table 1.\n\nThank you for pointing this issue out. This was indeed a calculation error on our part. The success rates for the 5 seeds were [0.983, 0.955, 0.956, 0.963, 0.977], with a mean of approximately 0.967 and a standard error of 0.0056. We mistakenly forgot to multiply the standard error by 100, which led to the anomalous value in the table. The correct success rate is $96.7 \\pm 0.6$. We will correct this in the revised version of our paper.\n\n---\n\nWe are truly grateful for your thoughtful and constructive feedback. Your suggestions were incredibly helpful and have guided us in making significant improvements to the paper. We believe the paper is much stronger now as a result. We hope our responses have thoroughly addressed all your concerns, and we look forward to your feedback. Please do not hesitate to contact us if any points require further discussion."}]

Author rebuttal:

Meta-review: In this paper, the authors quantify the deployement policy’s information-gathering progress by estimating the prediction uncertainty of privileged observations reconstructed from partial observations, and propose the framework of Uncertainty-Sensitive Privileged Learning (USPL). This framework feeds this uncertainty estimation to the privileged policy (PP) and combines reward transformation with privileged-observation blurring, driving the PP to choose actions that actively reduce uncertainty and thus gather the necessary information. Experiments across nine tasks demonstrate that USPL significantly reduces the behavioral discrepancies, achieving superior deployment performance compared to baselines. The reviewers have stated that the idea of injecting uncertainty into priviledge learning is interesting. The additional loss functions used to learn the observational encoder are simple and easy to implement. Moreover, the experiments cover image and non-image observations, and show that the deployment policy can achieve performances similar to those of the privileged policy. Also, the uncertainty estimation analysis and ablation study sections provide meaningful and rational insights. Furthermore, the motivation is clear and the empirical performance of the proposed method is signficantly better than the baseline methods in the environment provided in the paper. Specifically, the results consistently demonstrate superior performance over the baselines. Overall I believe that this is a nice paper that will get the attention of the community working on learning under privileged information and RL.
Decision: Accept (poster)