Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
Authors: Haoran He
Peilin Wu
Chenjia Bai
Hang Lai
Lingxiao Wang
Ling Pan
Xiaolin Hu
Weinan Zhang
Venue: CoRL 2024 (2024)
Keywords: Sim-to-Real
Information Bottleneck
Reinforcement Learning
Locomotion
TL;DR: We propose a novel privileged knowledge distillation method called the Historical Information Bottleneck (HIB) to bridge the sim-to-real gap.

Abstract: Reinforcement Learning (RL) has recently achieved remarkable success in robotic control. However, most works in RL operate in simulated environments where privileged knowledge (e.g., dynamics, surroundings, terrains) is readily available. Conversely, in real-world scenarios, robot agents usually rely solely on local states (e.g., proprioceptive feedback of robot joints) to select actions, leading to a significant sim-to-real gap. Existing methods address this gap by either gradually reducing the reliance on privileged knowledge or performing a two-stage policy imitation. However, we argue that these methods are limited in their ability to fully leverage the available privileged knowledge, resulting in suboptimal performance. In this paper, we formulate the sim-to-real gap as an information bottleneck problem and therefore propose a novel privileged knowledge distillation method called the Historical Information Bottleneck (HIB). In particular, HIB learns a privileged knowledge representation from historical trajectories by capturing the underlying changeable dynamic information. Theoretical analysis shows that the learned privileged knowledge representation helps reduce the value discrepancy between the oracle and learned policies. Empirical experiments on both simulated and real-world tasks demonstrate that HIB yields improved generalizability compared to previous methods.

Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
Haoran He Hel? Peilin Wu'
Chenjia Bai
Hang Lai'
Lingxiao Wang*
Ling Pan?
Xiaolin HuS
Weinan Zhang!t
Shanghaii Jiao Tong University "Hong Kong University of Science and Technology
Institute of Artificial Intelligence (TeleAI), China Telecom
WNrttwwestern University TTinnhua University
Reinforcement Learning (RL) has recently achieved remarkable success in robotic control. However, most works in RL operate in simulated environments where privileged knowledge (e.g.. dynamics, surroundings, terrains) is readily avail- able. Conversely, in real-world scenarios, robot agents usually rely solely on local states (e.g., proprioceptive feedback of robot joints) to select actions, leading to a significant sim-to-real gap. Existing methods address this gap by either gradually reducing the reliance on privileged knowledge or performing two-stage policy imitation. However, we argue that these methods are limited in their ability to fully leverage the available privileged knowledge, resulting in suboptimal performance. In this paper, we formulate the sim-to-real gap as an information bottleneck prob- lem and therefore propose a novel privileged knowledge distillation method called the Historical Information Bottleneck (HIIB. In particular, HIB learns privileged knowledge representation from historical trajectories by capturing the underlying changeable dynamic information. Theoretical analysis shows that the learned privileged knowledge representation helps reduce the value discrepancy between the oracle and learned policies. Empirical experiments on both simulated and real-world tasks demonstrate that HIB yields improved generalizability compared to previous methods.
Abstract:
Keywords: Sim-to-Real, Information Bottleneck, Reinforcement Learning
Introduction
Reinforcement Learning (RL) has made significant advancements across various simulated environ- ments (e.g., games [1]. financial trading [2]), but its applications in real-world scenarios still remain challenge. The primary obstacle is the sim-to-real gap, an inherent mismatch between simulated and real environments that cause policies learned in simulation to perform sub-optimally in the real world. Previous works tackle this problem through more realistic simulation [3, 4], adversarial training [5, 6]. and domain randomization [7, 8] to minimize the mismatch. However, building high-quality simulators difficult, and excessive introduced assumptions can lead to an overly conservative policy.
Recently, more effective branch of methods proposes to utilize privileged knowledge to address the sim-to-real problem, where privileged knowledge is the information available in simulation (e.g., dynamics, surroundings, terrains) but inaccessible in real environments. Previous methods solve this problem via a two-stage policy distillation process [9]. Specifically, a teacher policy is first trained the simulator with privileged states accessible, and then a student policy relying on local states (without privileged information) is trained by imitating the teacher policy. Nevertheless. such a two-stage paradigm computationally expensive and requires careful design for imitation. An alternative approach involves gradually dropping the privileged information the policy is trained [10] or conditioning the critic on privileged information [11]. However, these methods lack theoretical understanding of the sim-to-real problem and do not fully exploit the available privileged information during training.
1 "Coreeponndence to: Weinan Zhang (wnzhang @ sjtu.edu.cn).
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

In this paper, we present a representation- approach, instead of policy distillation, to better utilize the privileged knowledge from simulation with a single-stage learning paradigm. Inspired by the Information Bottleneck (IB) method [12, 13], which learns a minimal sufficient representation Z of a given input source with the target source Y, we propose a novel method called Historical Information Bottleneck (HIB). Similar to the goal of the IB method, HIB aims find a maximally compressed representation of the privileged knowledge while preserving sufficient information about the current environment for real-world decision-making historical information that contains previous local states and actions to learn a history representation, which is trained by maximizing the mutual information (MI) between the representation and the privileged knowledge. Theoretically, we show that maximizing such an MI term will minimize the privileged knowledge modeling error, reducing the discrepancy between the optimal value function and the learned value function. Furthermore, benefiting from the IB principle, we compress the decision-irrelevann information from the history and obtain a more robust representation. The IB objective is approximated by variational lower bounds to handle the high-dimensional state space.
In particular, HIB takes advantage of
In summary, our contributions are threefold: (i) We propose a novel policy generalization method called HIB that follows the IB principle to distill privileged knowledge from a fixed length of history. (ii) We provide a theoretical analysis of both the policy distillation methods and the proposed method, which shows that minimizing the privilege modeling error is crucial in learning a near-optimal policy, (iii) Empirically, we show that HIB learns robust representation in randomized RL environments and achieves better generalization performance in both simulated and real-world environments than state-of-the-art (SOTA) algorithms, including -utrof-distribution test environments.
2 Related Work
Sim-to-Real Transfer. Transferring RL policies from simulation to reality is challenging due to the domain mismatch. To this end, the previous study hinges on domain randomization, which trains the policy under a wide range of environmental parameters and sensor noises [14, 15, 16, 17, 18]. However, domain randomization typically sacrifices the optimality for robustness, leading to an over-conservative policy [19]. To address this problem, various works perform privilege distillation by teacher-student learning [20, 21, 22, 23, 24], teacher demonstration exploration [25, 26, 27, 28] and teacher policy progressive imitation [29, 30J. However, these methods are sample inefficient due to the requirement of training an additional teacher policy. Aside from privileged policy, recent works exploit the privileged information by introducing privileged critic [31,32, 11] or privileged world models [33, 34). An alternative method gradually drops privileged knowledge [10]. Nevertheless, these methods are limited to fully leveraging historical knowledge for better generalization. Different from the above approaches, our method learns a privileged representation via informative historical trajectories from the IB perspective, resulting in better utilization of historical information.
Information Bottleneck for RL. The IB principle [35, 13] was initially proposed to trade off the accuracy and complexity of the representation in supervised learning. Specifically, IB maximizes the MI between representation and targets to extract useful features, while also compressing the irrelevant information by limiting the MI between representation and raw inputs [36, 37]. Recently, IB has been employed in RL acquire compact and robust representation. For example, Fan and Li [38] takes advantage of IB to learn task-relevant representation via multi-view augmentation. Other methods [39, 40, 41] maximize the MI between representation and dynamics or value function, and restrict the information to encourage the encoder to extract only the task-relevant information. Unlike the previous works that neither tackle the policy generalization problem nor utilize historical information, HIB derives a novel objective based on IB, which aims to learn a robust representation of privileged information.
knowledge from history while simultaneously removing redundant
3 Preliminaries
In this section, we briefly introduce the problem definition and the corresponding notations used throughout this paper. We give the definition of privileged knowledge in robot learning follows.

---

Definition 1 (Privileged Knowledge). Privileged knowledge is the hidden state that is inaccessible in the real environment but can be obtained in the simulator, egg,, surrounding heights, terrain types, morphology parameters like length of legs, and dynamic parameters like friction and damping. An oracle (teucher) policy is defined as the optimal policy with privileged knowledge visible.
In this paper, extend the concept of the sim-to-real gap to general policy generalization problem with a knowledge gap. We define the MDP as M (S', A. where [st s s represents the oracle state s" that contains s E S (i.e., the local state space) and s" E S" (i.e., the privileged state space), where s contains privileged knowledge defined in Definition 1. A the action space. The transition function P [5%, az) and reward function r(s", a) follows the ground-truth dynamics based on the oracle states. Based on the MDP, we define two policies: (a|s', sP) and # ( (als'). for the simulation and real world, respectively. Specifically, [[ass, s²) is privileged policy that access the privileged knowledge, which is only accessible in the simulator. In contrast, a(lss)) is a local policy without accessing the privileged knowledge throughout the interaction process, which is common in the real world. A thorough discussion about our problem and Partially Observable MDP (POMDP) [42, 43] is provided in Appendix A.I.
Based on the above definition, our objective is to find the optimal local policy #* based on the local state st that maximizes the expected return, denoted
arg max E (111() = 7-r(s),
E ai)
Theoretical Analysis & Motivation
4.1 Value Discrepancy for Policy Generalization
In this section, give a theoretical analysis of traditional oracle policy imitation algorithms [44]. Note that previous works give similar analyses but in different formulations [22, 27, 30]. Specifically, the local policy is learned by imitating the optimal oracle policy x². We denote the optimal value function of the policy "t learned with oracle states 8 = [ sP] as (s' 8 a), and the value function of policy learned with local states as (s's a). The following theorem analyzes the relationship between the value discrepancy and the policy imitation error in a finite MDP setting.
Theorem 1 (Policy imitation discrepancy). The value discrepancy between the optimal value function with privileged knowledge and the value function with the local state is bounded as
(8"(8'88P000 ,,,aa
(**ss" (1 7)2 E3
2-TTmax,
sup
where
C = sup Drv (T" (T (-Is², sP) (-)()+
6',00
the policy divergence between πT" and ft, and Tmax is the maximum reward in each step.
The proof is given in Appendix A.2. Theorem shows that minimizing the total variation (TV) distance between π (-Is', sP) and #--[s²) reduces the value discrepancy. However, minimizing ea can be more difficult than ordinary imitation learning where '7" and क have the same state space. Specifically, if n² and have the same inputs, Dry (T² (+1=)][ (+1s) will approach zero with sufficient model capacity and large iteration steps, least in theory. In contrast, due to the lack of privileged knowledge in our problem, the policy error term in Eq. (3) can still be large after optimization as T (+(ss',, s²) and 4-((s)) have different inputs.
Previous works [21, 20) try to use historical trajectory
he {
(4)
with a fixed length to help infer the oracle policy, and the policy imitation error becomes (r (+) h)). However, without appropriately using the historical information, im- itating well-trained oracle policy can still be difficult for an agent with limited capacity, which results in suboptimal performance. Meanwhile, such two-stage policy imitation methods are also sample-inefficient.
3

---

4.2 Privilege Modeling Diserepancy
To address the above challenges, we raise an alternative theoretical motivation to relax the requirement of the oracle policy. Specifically, we quantify the discrepancy between the optimal value function and the learned value function based on the error bound in reconstructing the privileged state sy via historical information he, which eliminates the reliance on the oracle policy and makes our method single-stage distillation algorithm. Specifically,
define density model ((s][h,) to predict the privileged state based on history he in Eq. (4). Then the predicted privileged state can be sampled as Sp ~ P(-Iha). In policy learning, we concatenate the local state s' and the predicted @ as input. The following theorem gives the value discrepancy of Q" and the value function Q with the predicted sP.
Theorem (Privilege modeling discrepancy). Let the divergence between the distribution of privi- leged state model P( (/++1) and the true distribution of privileged state ((st+11b44) be bounded as
sup sup Drv (P(: (e(1)) || PP. hh-1)).
(5)
12to he+1
Then the performance discrepancy bound between the optimal value function with P and the value function with P holds, as
Ag (1 (1-7) + 27ymmx
sup sup (s) sf,)))))))))))) ar) Qc(s) 82 a.) 12 at ,7,a
(1- Ep.
(6)
where Ag = SuP -E--E--rein 10 + - 111oo is the difference in the same value function with sumpled oy and the expectation of oy conditioned on ht-
We defer the detailed proof in Appendix A.3. In Ag, we consider using sufficiently long (i.e.. by using history greater than some to) and informative (i.e., by extracting useful features) history to make Ag small in practice. We remark that Ag captures the inherent difficulty of learning without privileged information. The error is small if privileged information is near deterministic given the history, or if the privileged information is not useful given the history. Ep measures the model discrepancy in the worst case with an informative history. In the next section, we provide an instantiation method inspired by Theorem
Methodology
In this section, we propose a practical algorithm named HIB to perform privilege distillation via historical representation. HIB only acquires the oracle state without an oracle policy in training. In evaluation, HIB relies on the local state and the learned historical representation to choose actions.
5.1 Reducing the Discrepancy via MI
Theorem 2 indicates that minimizing ep yields a tighter performance discrepancy bound. We then start by analyzing the privilege modeling discrepancy ep in Eq. (5). We denote the parameter of Pa by o, then the optimal solution o" can be obtained by minimizing the TV divergence for Vt, as
Φ² = arg ming Drv (P(-)he) (-h)))) = arg mins DKL (PL()))))) ( P-())h)
arg max & Epcor A..) log Po hs] A arg max Fpred:
(8)
where the true distribution P(-Ih+)) is irrelevant Φ, and convert the TV distance to the KL distance in Eq. (7) by following Pinsker's inequality. Since h, is usually high-dimensional, which is of linear complexity with respect to time, it is necessary to project h, in a representation space and then predict s.. Thus, we split the parameter of Po as Φ = [2]. where Φi aims to learn a historical representation fer (hr) first, and d2 aims to predict the distribution Po.. (2) of privileged state (e.g.. Gaussian). In the following, we show that maximizing Tpred is closely related to maximizing the MI between the historical representation and the privileged state. In particular, we have
I Ipred Ep(ep. [log Pe. (sflif, (hx))]
= Eppotdee [oog P(s.lfo, (h+))] D -H (H.)) [KL[P[]P]
(9)
= =S(S (1+)) H(S?) (S)P:,,o

---

AutoEncoder fu
Targer projector Be
Privilege info s
Representation
projection y
Momentum update
Cosine similarity
TCN encoder
Online projector ge
Trajectory history
Latent variable R4
projection Yr
dipergence
objective
Overall training framework. HIB adopts the IB principle to recover privileged knowledge from fixed length of local history information. The RL objective also provides gradients to the history encoder fv. implying that the learned representation can be combined with any RL algorithm effectively.
Figure 1:
where we denote the random variables for sy and he by S and H₁, respectively. In Eq. (9), the upper bound is obtained by the non non-negativity of the Shannon entropy and KL divergence. The bound is tight since the entropy of the privileged state H(S,") is usually constant, and DRL(PI| can be small when we use variational Ps with an expressive network.
According to Eq. (9), maximizing the predictive objective 4prot is closely related to maximizing the MI between S and for (Hz). In HIB, to address the difficulty of reconstructing the raw privileged state that can be noisy and high-dimentn dimensional in pred objective, we adopt contrastive learning [45] as an alternative variational approximator [46] approximate MI in representation space. Moreover, HIB restricts the capacity of representation to remove decision-irrelevann information from the history, which resembles the IB principle [35] in information theory.
5.2 Historical Information Bottleneck
To optimize the MI in Eq. (9) via a contrastive objective [47], we introduce a historical representation 2, fo (ht) to extract useful features that contain privileged information from long historical vector his where J is temporal convolution network (TCN) [48] that captures long-term information along the time dimension. We another notation và distinguish it from the predictive encoder di in Eq. (9), since the contrastive objective and predictive objective Ipred learn distinct representations by optimizing different variational bounds. In our IB objective. the input variable is H,, and the corresponding target variable is Sy- Our objective is to maximize the MI term T()+; while minimizing the MI term 1 [HH 2,) with Z, = (H.), which takes the form of
min -1,, + a:l (H; Z.),
(10)
where a is a Lagrange multiplier. The I(Z,; S term quantifies the amount of information about the privileged knowledge preserved in Ze. and the [HH;; Z,) term is a regularizer that controls the complexity of representation learning. With a well-tuned a, we do not discard useful information that is relevant the privileged knowledge.
We minimize the MI term I [HH;; Z.) in Eq. (10) by minimizing the following tractable upper bound:
[HH;; Zp) = Ep(z.. log p(2albe) (hx) p(=))
loog p(2(lhe - D
= Ep(eah))))
((=1)
(11)
S DRI. DKII.p((s(lee)lll
where the inequality follows the non-negativity of the KI. divergence, and g(z;) is an approximation of the marginal distribution of Zz. We follow Alemi al. [36] and use a spherical Gaussian ((xx) (0, /) as approximation.
One can maximize the MI term 1 (2,;;) SP') in Eq. (10) based on the contrastive objective [47]. Specif- ically, for a given s/, the positive sample 2, ~ fo (h,) is the feature of corresponding history in timestep f. and the negative sample 2 can be extracted from randomly sampled historical vectors. However, considering unresolved trade-offs involved in negative sampling [49, 50], we simplify the contrastive objective by removing negative sampling. In HIB, the contrastive loss becomes cosine similarity with only positive pairs, and we empirically find that the performance does not decrease. Such simplification was also adopted by recent contrastive methods for RL [51, 52].
5

---

We adopt a two-stream architecture to learn 2,, including an online and a target network. Each network contains an encoder and a projector, as shown in Fig. 1. The online network is trained to use history to predict the corresponding privilege representation. Given a pair of a history sequence and privileged state 8.') we obtain & = [=(( with an encoder fw to get the representation of sy. Here, fw used project sy into the same dimensional space as zt. so fw can be a simple AutoEncoder [53] trained by reconstruction loss free = || Du- (f.)(( where Dwr is an additional trained decoder (see Appendix C.I for details of implementation). Then we use TCN as the history encoder J to learn the latent representation ž₁ f (-1,,)) As éz and žn have the same dimensions, the projectors share the same architecture. The online projector go outputs yt ga (2x) and the target projector ge- outputs V/ = 9o- (êe). We use the following cosine similarity loss between yt and Dt, and use stop gradient (sg[-) for the target value y, as
Ssim
Σ
yt
sg sg(üe] |sg[Bo]||2
(12)
To prevent collapsed solutions in the two-stream architecture, we follow previous architectures [54] by using a momentum update for the target network to avoid collapsed solutions. Specifically, the parameter of the target network 0 takes an exponential moving average of the online parameters 0
with a factor т E [0, 1]. as A + r + (1 (1-r)8.
At each training step, we perform stochastic optimization step to minimize Caim with respect to and sb. Meanwhile, we learn RL policy x (alss), 2x) based on the historical representation 2,, and the RL objective is also used to train the TCN encoder fa. The dynamics are summarized as
+ o0timizer(0, VeLimm), & c- optimizer(M) (v), (Ai Cai + A2CKI. + Csi z)) w + optimizer(w, V.Lre))
(13)
where LKI. (f and Cr. is the loss function for an arbitrary RL algorithm. We use the Adam optimizer [55]. We summarize our algorithm in Alg. in Appendix C.I.
1)) is the IB term in Eq. (11) that controls the latent complexity,
Experiments
6.1 Benchmarks and Compared Methods
To quantify the generalizability of the proposed HIB, we conduct experiments in simulated envi- ronments that include multiple domains for comprehensive evaluation, and also the legged robot locomotion task to evaluate the generalizability in sim-to-real transfer.
Privileged DMC Benchmark. We conduct experiments on DeepMind Control Suite (DMC) [56] with manually defined privileged information, which contains dynamic parameters such as friction and torque strength, and morphology parameters such as the lengths of specific legs. The privileged knowledge is only visible in the training process. Following Benjamins et al. [57], we randomize the privilege parameters at the beginning of each episode, and the randomization range can be different for training and testing. Specifically, we choose three randomization ranges for varied difficulty levels, ie., ordinary, o.o.d. and far o.o.d. The ordinary setting means that the test environment has the same randomization range as in training, while o.o.d. and far o.o.d. indicate that the randomization ranges are larger with different degrees, causing test environments being out-of-distribution compared with training environment. The detailed setup can be found in Appendix B. For dynamic parameters, we evaluate the algorithms in three different domains, namely pendulum, finger spin, and quadruped walk; For morphology parameters, we evaluate the algorithms quadruped walk with varying lengths of front legs and back legs. Both settings cover various difficulties ranging from ordinary to far o.o.d. This benchmark is denoted as the DMC benchmark in the following for simplicity.
Sim-to-Real in Blind Legged Robot. This experiment is conducted on quadrupedal robot. In this domain, privileged knowledge is defined as terrain information (c.g. heights of surroundings) of the environment and dynamic information such as friction, mass, and damping of the quadrupedal robot. We leverage the Isaac Gym simulator [58] for training, which also provides multi-terrain simulation (e.g., slopes, stairs, and discrete obstacles). Details can be found in Appendix B.2. For experiments in the real robot, utilize the Unitree Al robot [59] to facilitate real-world deployment.

---

Table 1: Evaluated episodic return achieved by HIB and baselines on DMC tasks. SAC is adopted as the basic RL algorithm. We report the mean and standard deviation for 100K steps. Variances are large because dynamic parameters are changed for each episode. We refer to Appendix B.1 for the details.
Domain
Testing Difficuity
Tencher
HIR (ours)
DR
Stadent (RMA))
DreamWaQ
Dropper
DS, 30.24
107.69
115.055
204.50
Penvision
oad.
-251, 164304.2
502
-315.39 315.339..s...
for o.dd.
677 188
500.622548
20w.000 as
-674 -674-43-690
ordinary
826. 19a
623 32 114 in
May 401
G41.08 a5
Finger Spin
o.dd.
7U8. 18 1w6
669.2
+:30.22
160.79 so
6419 58 or
618. 72saus 64
551. 6l
for 0.0.d.
663 73ka9m
645.
6-45.42.31..0
453 (SS.0eaati..
608 76
606, 01 sosw
51% 06
orcimary
22412299
Quadraped Wolk
945.60 42.28
927.91.55..0
$20.27 60.27
$20.45
far o.n.d.
022.400-a
904.722-0.76
164.79
703. =000o
801 10..088:6..0
286. 00:
Table 2: Evaluated episodic return achieved by HIB and baselines quadruped walk task with morphology shifts. The lengths of the front legs and back legs changed episodically. See Appendix B.1 for more details.
Changed Legs
Testing Difficulty
Teacher 3322,89134 o7
HIB (ours)
DR
Student (RMA)
Dream
Dropper
ordinary
912.92255.11
306.21 55
880.78 130.118
732.600 1ot
Frout
o.o.d.
114.26=54.0 a6
896.69=42.77
300.94 000.91427..88 a2
851. 351.799 144.78
728. 102..00
238. 81 in
lar o.o.d.
861 99990.1
867. 18.96.
294 00
814.52 178.50
721. 128.2
220. 11
onlinary
726.731-2.32
914.03 5.2002
66 26.333.m0 vn.4s
880. aa 135.38
733121 12
230.390 200.70
Back
o.o.d.
906.63
903.93
G6 30. oe
815. 98
60
728. 40 101 54
180 95 as
o.o.d.
831.72x11..7
812. 1.2.08881..
61 08:
616.99+17a.98
727.81=115.08
150.39-220.54
Baselines. We compare HIB to the following baselines. (i) Teacher policy is learned by oracle states with privileged knowledge. (ii) Student policy follows RMA [21] that mimics the teacher policy through supervised learning, with the same architecture as the history encoder HIB. We remark that student needs a two-stage training process obtain the policy. (iv) Dream WaQ follows the implementation in Nahrendra et al. [31], conditioning the critic on privileged information and utilizing history trajectories to learn a latent variable for action. (iv) Dropper is implemented according to Li et al. [10], which gradually drops the privileged information and finally converts to a normal agent that only takes local states as input. (v) DR agent utilizes domain randomization for generalization and is directly trained with standard RL algorithms with local states as input.
6.2 Simulation Comparison
For varying dynamic parameters, the results are shown in Table 1. Our method achieves the best performance in almost all test environments, closely matching the returns of the oracle (teacher) method. The results in Table 2 demonstrate that HIB can also generalize unseen environments with morphology shifts, highlighting its superior capability which benefits from our proposed IB- style training. While the student baseline can perform slightly better than HIB, it exhibits larger variance because its performance heavily relies on the teacher policy and its exploration ability is restricted. The results on the Legged Robot benchmark (Table 3) further verify the advantage of HIB. Our method outperforms the two strongest baselines, student and Dream WaQ, on the three most challenging terrains: stairs, discrete obstacles, and slopes. From the simulation results, HIB can be seamlessly combined with different RL algorithms and generalizes well across different domains and tasks, demonstrating the efficacy and high scalability of HIB.
Table 3: Success rates and average distances achieved by HIB and baselines of legged robot for 1000 steps evaluated in the Isaac-Gym simulator. Results are averaged over 1000 trajectories with different difficulties. Reward designs follow [3], and environmental details can be achieved in Appendix B.2.
Success Rate (%)r f
Average Distance (m) 1 Discrete Obstacles
Stairs
Discrete Obstacles
Slope
Stairs
Slope
Student (RMA)
94.2
82.1
96.1
7.96#1.21
7.67+2.14
8.69
Dream WaQ
94.1
81.3
97.1
8.09
7.5142.36
8.78
HIB (ours)
96.7
85.8
97.7
8.67.11.1
7.8412.07
9.04
Teacher
98.0
86.9
98.3
8,88+1.09
8.21+2.01
9.24+1.06
6.3
Visualization and Ablation Study
To investigate the ability of HIB in modeling the privileged knowledge, we visualize the latent representation learned by HIB and student via dimensional reduction with t-SNE [60]. We also visualize the true privileged information for comparison. The visualization is conducted in finger spin task, and the results are given in Fig. 3. We find that the student agent can only recover part of the privileged knowledge that covers the bottom left and upper right of the true privilege distribution. In
7

---

contrast, the learned representation of HIB has almost the same distribution as privileged information. This may help explain why our method outperforms other baselines and generalizes to o.o.d. scenarios without significant performance degradation.
We conduct an ablation study for each component of HIB to verify their effectiveness. Specifically, we design the following variants to compare with. (i) HIB-w/o-ib only uses RL loss update history encoder fu. which is similar to standard recurrent neural network policy. (ii) HIB-w/o-rl only uses HIB loss to update the history encoder without the RL objective. (iii) HIB- HIB-w/o-proj drops the projectors in HIB and directly compute cosine similarity loss between 2 and é (iv) HIB-contra employs contrastive loss [61] instead of cosine similarity, using score function that assigns high scores positive pairs and low scores negative pairs.
From the result in Fig. 4, we observe that HIB- IIB-w/ almost fails and HIB-w/o-rl can get relatively high scores, which signifies that both HIB loss and RL loss are important for the agent to learn well-generalized policy, especially the HIB loss. The HIB loss helps the agent learn historical representation that contains privileged knowledge for better generalization. Furthermore, projectors and the momentum update mechanism are also crucial in learning robust and effective representation. Moreover, HIIB-conte performs well at the beginning but fails later, which indicates that the con- trastive objective requests constructing valid negative pairs and learning good score function, which is challenging the general state-based RL setting.
Figure 3: t-SNE visualization for the privilege rep- resentation and the learned latent representation of Figure 2: A Unitree A travers- history encoder in finger spin. ing different terrains.
Figure 4: Comparison of different HIB variants in quadruped walk.
6.4 Real-world Application
To further evaluate the generalizability of HIB in the real world, deploy the HIB policy trained in the Legged Robot benchmark on real-world A1 robot without any fine-tuning. Detailed setup is referred to Appendix B.3. Note that the policy is directly run on the Al hardware, and the local state is read or estimated from the onboard sensors and IMU, making the real-world control noisy and challenging. Fig. shows the snapshots of our HIB agent traversing different terrains. The agent generalize different challenging terrains with stable control behavior and there is no failure in the whole experimental process. Fig. 5 further showcases the advantages of our proposed HIB, enabling the agent to navigate down high stairs with a 0.6m height, achieving a 100% success rate 10 trials, while the strongest baseline (e.g. Dream WaQ) completely fail. These real experiments demonstrate that HIB enhances the ability to bridge the sim-to-real gap without any additional tuning real environments.
(a) w/ HIB
(b) w/o HIB (Dreum Wa()
Figure 5: HIB successfully handles high stairs. See more examples in our supplementary video.
7 Conclusion and Limitation
We propose a novel privileged knowledge distillation method based on the Information Bottleneck to narrow the knowledge gap between local and oracle RL environments. In particular, the proposed two- stream model design and HIB loss help reduce the performance discrepancy given in our theoretical analysis. Our experimental results on both simulated and real-world environments show that (i) HIB learns robust representations to reconstruct privileged knowledge from local historical trajectories and boosts the RL agent's 's performance, and (ii) HIB can achieve improved generalizability out- of-distribution environments compared to previous methods. However, HIB is limited in recovering privileged knowledge (e.g., RGB images), which is more nigh-dimensional and complex.
8

---

Acknowledgments
This work is partially supported by Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102 and National Natural Science Foundation of China (62322603 & 62076161). We also thank the anonymous reviewers for their valuable suggestions.
References
[1] S. Kapturowski, V. Campos, R. Jiang, N. Rakićević, H. van Hasselt, C. Blundell, and A. P. Badia. Human-level atari 200x faster. arXiv preprint (0t):22299.05550, 2022.
[2] X.-Y. Liu, H. Yang. J. Gao, and C. D. Wang. Finrl: Deep reinforcement learning framework automate trading in quantitative finance. In Proceedings of the second ACM international conference Al in finance, pages 1-9, 2021.
[3] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, 2022.
[4] K. Kristinsson and G. A. Dumont. System identification and control using genetic algorithms. IEEE Transactions on Systems, Man, and Cybernetics, 2235):003--1045 1992.
[5] N. Akhtar and A. Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access, 2018. doi: 1109/ACCESSS200 2.8877887.
[6] Y. Jiang, T. Zhang, D. Ho, Y. Bai, C. K. Liu, S. Levine, and J. Tan. Simgan: Hybrid simulator identification for domain adaptation via adversarial reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021. doi: Noi:10.1109//0 1199/CCA448066 2021.9561731.
[7] X. Chen, J. Hu, C. Jin, L. Li, and L. Wang. Understanding domain randomization for sim-to-real transfer. In ICLR, 2021.
[8] Muratore, C. Bilers, M. Gienger, and J. Peters. Bayesian domain randomization for sim-to-real transfer. CoRR, 2020.
[9] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation, 2015. URL abs/1511 06295.
[10] Li, S. Koyamada, Q. Ye, Liu, C. Wang, R. Yang. L. Zhao, T. Qin, T. Liu, and H. Hon. Suphx: Mastering mahjong with deep reinforcement learning. CoRR, 2020.
[11] L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for masee-based robot learning, arXiv preprint arXiv: 1710.0652,, 2017.
[12] N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
[13] N. Tishby and N. Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (inw), 2015.
[14] W. Zhao, J. P. Queralta, and T. Westerlund. for robotics: a survey. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), 2020.
transfer in deep reinforcement learning
[15] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, Issac, N. Ratiff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference Robotics and Automation (ICRA), 2019.

---

[16] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), 2018.
[17] A. Handa, A. Allshire, V. Makoviychuk, Petrenko, Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 5977-5984. IEEE, 2023.
[18] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The Interna- tional Journal of Robotics Research, 39(1):3-20, 2020.
[19] Z. Xie, Da, M. van de Panne, Babich, and A. Garg. Dynamics randomization revisited: A case study for quadrupedal locomotion. In 202/ IEEE International Conference Robotics and Automation (ICRA), 2021.
[20] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 2020.
[21] A. Kumar, Z. Fu, D. Pathak, and J. Malik, Rma: Rapid motor adaptation for legged robots. In Robotics: Science and Systems, 2021.
[22] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. In 6th Annual Conference on Robot Learning, 2022.
[23] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl.. Learning by cheating. In Conference on Robot Learning, pages 66-75. PMLR, 2020.
[24] Wu, Xin, Qi, and Y. Xue. Learning robust and agile legged locomotion using adversarial motion priors. IEEE Robotics and Automation Letters, 2023.
[25] A. Walsman, M. Zhang, S. Choudhury, D. Fox, and A. Farhadi. Impossibly good experts and how to follow them. In The Eleventh International Conference on Learning Representations,
2023. URL https::///// / http//pppnneeeeeeeeeeee net htt////rrmm//// wer.miiidsciiA.xyoooo. sciA_
[26] H. H. Nguyen, A. Baisero, D. Wang, C. Amato, and R. Platt. Leveraging fully observable policies for learning under partial observability, In 6th Annual Conference on Robot Learning, 2022. URL https: htt::///ppeeeieeeeee wttp:/ewwwiei.eteerreeeeee ntt/forumm hi---ii-oppo--ooo
[27] L. Weihs, U. Jain, I.J. Liu, J. Salvador, S. Lazebnik, A. Kembhavi, and A. Schwing. Bridging the imitation gap by adaptive insubordination. Advances in Neural Information Processing
Systems, 34:19134-19 1114344--66,,,111 2021.
[28] I. Shenfeld, Z.-W. Hong, A. Tamar, and P. Agrawal. Tgrl:
An algorithm for teacher guided reinforcement learning. In International Conference on Machine Learning, 2023. URI. https: www.steeeanntteeeeee ieschooas.c.... wwwphoool..rrr//oooooooooooeeeeo http//ooppuIID::331106
[29] Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: Learning unified policy for manipulation and locomotion. In Conference on Robot Learning, pages 138-149. PMLR, 2023.
[30] A. Warrington, J. W. Lavington, A. Scibior, M. Schmidt, and F. Wood. Robust asymmetric learning in pomdps. In International Conference on Machine Learning, pages 11013-11023. PMLR, 2021.
[31] I. M. A. Nahrendra, B. Yu, and H. Myung. Dreamwaq: Learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 5078-5084. IEEE, 2023.
10

---

[32] J. Wu, Y. Xue, and C. Qi. Learning multiple gaits within latent space for quadruped robots. arXiv preprint arXiv:2308.030144 2023.
[33] G. Lambrechts, A. Bolland, and D. Ernst. Informed pomdp:
Leveraging additional
information in model-based rl. ArXiv, abs/2306. 1488, 2023. URL https://api.
semanticscholad org. wwtpCoopssID:25922222222
[34] E. S. Hu, J. Springer, O. Rybkin, and D. Jayaraman. Privileged sensing scaffolds reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https:
//pppreeeeeeeee net/ / forum? iddeuneeddeeeeee jAjdx.
[35] N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method, 2000. URL https:// awwpvvaxivvv//va///vv//... wwwp:/wwwwwi.ooooooooooo org/ abs http://////pssass//oo///////////
[36] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck.
In ICLR, 2017. URL https: /ttpp::://peeeeveee... net/ fett/oormmidddyyyoooeeee Hy
[37] A. M. Saxe, Y. Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019.
[38] J. Fan and W. Li. DRIBO: Robust deep reinforcement learning via multi-view information bottleneck. In ICML, 2022.
[39] Y. Kim, W. Nam, H. Kim, J.-H. Kim, and G. Kim. Curiosity-bottleneck: Exploration by distilling tpsk-specific novelty. In ICML, 2019.
[40] X. Lu, K. Lee, P. Abbeel, and S. Tiomkin. Dynamics generalization via information bottleneck in deep reinforcement learning. CoRR, 2020.
[41] C. Bai, Wang, L. Han, A. Garg, J. Hao, P. Liu, and Z. Wang. Dynamic bottleneck for robust self- superviedd exploration. In Advances in Neural Information Processing Systems, 2021.
[42] M. Igl. L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson. Deep variational reinforcement learning for pomdps. In International Conference on Machine Learning, pages 21 17-21226. PMLR, 2018.
[43] L. Meng, R. Gorbet, and D. Kulić. Memory-basee deep reinforcement learning for pomdps. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5619-5626. IREE, 2021.
[44] Y. Fang, K. Ren, W. Liu, D. Zhou, W. Zhang, J. Bian, Y. Yu, and T.-Y, Liu. Universal trading for order execution with oracle policy distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[45] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. CoRR, 2018.
[46] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker. On variational bounds of mutual information. In JCML., 2019.
[47] X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[48] S. Bai, J. Z. Kolter, and V. Koltun. An empirical evaluation generic convolutional and recurrent networks for sequence modeling. ArXiv, 2018
[49] F. Wang and H. Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
11

---

[50] Т. Huynh, S. Kornblith, M. R. Walter, M. Maire, and M. Khademi. Boosting contrastive self-supervised learning with false negative cancellation. In 2022 IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), 2022. doi:10.1 110.11199/AAAY3148882222222
[51] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. C. Courville, and P. Bachman. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, 2020.
[52] S. Qiu, Wang, C. Bai, Z. Yang, and Z. Wang. Contrastive ucb: Provably efficient contrastive self sulf-vuper learning in online reinforcement learning. In ICML, 2022.
[53] D. E. Rumelhart, G. Hinton, and R. Williams. Learning internal representations by error propagation. 1986. URL https: /api. semanticschos org/CorpusID: 62245742.
[54] J.-B. Grill, F. Strub, Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent new approach self-supervised learning. In NeurlPS, 2020.
[55] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, : 6628106.
2014. URL http:: //ppi//ppppppp/// www.smmmiiiii. www.antcccccclla. ar
[56] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap. N. Heess, and Y. Tassa. dm.control: Software and tasks for continuous control. Software Impacts, 2020.
[57] C. Benjamins, Eimer, F. Schubert, A. Biedenkapp. Rosenhahn, Hutter, and M. Lindauer. Carl: A benchmark for contextual and adaptive reinforcement learning. In NeurlPS 2021 Workshop Ecological Theory of Reinforcement Learning. 2021.
[58] V. Makoviychuk, Wawrzyniak, Guo, Lu, K. Storey, Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physies simulation for robot learning. CoRR, 2021.
[59] Unitree. Unitree robotics. https: /www. unitree. 2022.
[60] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008
[61] A. Srinivas, M. Laskin, and Abbeel. Curl: reinforcement learning. In International Conference on Machine Learning, 2020.
Contrastive unsupervised representations for
[62] A. Escontrela, X. B. Peng, W. Yu, T. Zhang, A. Iscen, K. Goldberg, and Abbeel. Adversarial motion priors make good substitutes for complex reward functions. 2022 IEEF/RSI International Conference on Intelligent Robots and Systems (IROS), pages 25-32, 2022. URL https: //api. www.nt insenntiiss cholar. org/ wtrg/Corpusiiii ID: 2477789244.
[63] J. Wu, Y. Xue, and C. Qi. Learning multiple gaits within latent space for quadruped robots. ArXiv, abs/2308.03014, 2023. URL https: : //api.ppp.////// semanticscholat orggCorrpssii 2600 916.
[64] Y. Wang, Z. Jiang, and J. Chen. Learning robust, agile, natural legged locomotion skills in the wild. 2023. URL https::://.......... : /tss:///ppi.. /api.. semanttttiii sommnntcccholas :ttpaaarr.rrroooopt::::22000
12

---

A Theorems and Proofs
A.1 POMDP and Sim-to-Real Problems
We summarize common points and differences between POMDP and our setting as follows.
The transition function and reward function for both problems follows the ground-truth dynamics of the environment.
2. In POMDP, the agent cannot access the oracle state in both training and evaluation, while in sim-to-real adaptation, the agent can access the oracle in training (in the simulation).
cp
S++1
Tt
Tr+1
sim
Tt
T++1
sf
at
s
A++1
a2
St+1
Ac+1
S++1
b
Dr+1
Tt
T++1
real
belief state
S
Qt
Sp+1
1++1
POMDP train/eval
Sim to real adaptation
Figure 6: The difference between POMDP setting and the sim-to-real problem.
A.2
Proof of Theorem
We begin by deriving the imitation discrepancy bound in ordinary MDP, shown in the following theorem.
Theorem A.1 (General oracle imitation discrepancy bound). Let the policy divergence between the oracle policy x² and the learned policy ft is 6g = sup, Drv(x" (1s))[((+1s)) Then the difference between the optimal action value function Q" of x and the action value function Q of bounded as
sup (s, -(s)s)))((ss $
(1-7)² Emi
27rmax
(14)
where y is the discount factor, and Tmax is the maximum reward in each step.
Proof. For any (s, a) pair that E S, E A, the difference between Q* a) and ((s, a) is
Q' a) (s,
= t (1 r(s,a) -2 12(s', d")]
=y Σ P(5\ Q(1
s'.0'
(15)
13

---

Then we introduce some intermediate terms and obtain
(s,
(()
-( a¹)l
=7
Σ P(s'ss (d'ls) ((
((s' +2 E P( ( +(a
aaa
s/.a¹
s <> @1lcc + + -- #|l=
> Hölder's inequality
s < @1 + +2y (1
s>|QQ' @1 + 1 -- Ex.
2yrmax
D Q(+, a) S -
Tmax V8, a
(16)
Thus,
|1-- - @11 ~>|1Q- - -ll1cc -
2n max
6x
1 1-7
(17)
(1 (1- -1=1 -Qll -
27/Tmax CT
Then we have
|1" - -ll 1 2aTmmax
(1 7)2 CT:
(18)
which completes our proof.
Next, we consider the value discrepancy bound in policy imitation with the knowledge gap. Specifi- cally, we denote the optimal value function with oracle states s" = [[s', s as Q'' sP, a), and the value function with local states Q(s', a). Then have the following discrepancy bound (restate of Theorems 1).
Theorem A.2 (Policy imitation discrepancy). The value discrepancy between the optimal value function with privileged knowledge and the value function with local state is bounded
sup (2' a) (s'(("(' ((s'a)]] ,
(1 7) ea.
27Tmmax
(19)
where
sup DrY (nT (-1s) (11s,,8")11 (+1(()
(20)
s
the policy divergence between π and & and Tmax is the maximum reward in each step.
Proof. We note that the transition functions for learning tta]s') and x (a]s", sP) are the same and follow the ground-truth dynamics. Although we cannot observe the privileged state s" in learning the local policy, the transition of the next state P(s i4+1 1/ 8/)) still depends on the oracle state of the previous step that contains the privileged state s. Since the agent is directly interacting with the environment, the privileged state does affect the transition functions. As a result, for policy imitation of the local policy, we have similar derivation as in (15) because the transition function in (15) is the same for n" (at+ (64+1)Q" Q("(((++==+)) ((+1) and (me+ 1 1e+1), which leads similar discrepancy bound as in ordinary MDP.
A.3 Proof of Theorem 2
We have defined the history encoder P& which takes the history as input and the privileged state as output:
SPP = (-
(21)
In the following, we give the performance discrepancy bound between the optimal value function with oracle state se = |s], s!'] and the value function with local state sf well as the predicted privileged state 3 (restate of Theorem 2).
14

---

Theorem A.3 (Privilege modeling discrepancy). Let the divergence between the privileged state model (( h(+1) and the true distribution of privileged state P be bounded as
Cp
sup sup Drv (P(: (141) || P(: hh-1)).
(22)
12t be+1
Then the performance discrepancy bound between the optimal value function with P and the value function with holds, as
DB (s), st, at) - Q,,,,, at) ) (1 (1 sup sup
+ 2aymmxx
(1 7)2 pr.
(23)
12 ,
where DR supi2to Q2 P(ophe) ( 1 E P 11 is the difference the same value function with sampled sy and the expectation of sy conditioned on ht-
Proof. For Q that estimates the value function of state (st, 8)), the value is stochastic since the hidden prediction 88 is single sample from the output of model P(-)/x). We take the expectation to the output of P(- and obtain the expected Q as
Eg P(Ph,) 12
Q $6 az)]
P((lh,, Pille aa max Q1+1(11+1 & > (21) E Peg s3) + -
a'
= Ee -E sf)l +
p a)) P(-lbr-)) max 1141(1/444)))) &f
(24)
where h441 h4 U ( We remark that the reward function r(s/ s(:) is returned by the real environment, thus sy follows the true P in the reward.
Remark (Explanation of the Bellman Equation). Our goal is to fit the Bellman equation in (24). Fitting the Bellman equation in (24) has several benefits. Firstly, it is easily implemented based the fitted privileged state model P. Specifically, solving (24) is almost the same solving ordinary MDP, with S generated by P based on the history in place of the true privileged state of s (which is unavailable). Secondly and most importantly, when the history is sufficiently strong in predicting s. solving (24) leads to approximately optimal solutions, as we show this theorem.
For Q" that represents the optimal value function with the true privileged state, we have
(s,,, = r(ss (())..8) st)) + (s Plls Imax
(25)
= =r(s/, st) + (st )~ P( P(llee [max Qi+ (8/ (4
where the second equation holds since he contains s/. Then we take a similar expectation to the true hidden transition function P as
E.p-((,))QQ" (Q" sp, at)]
E|r(s}, sf] + ME (s) --P
[max (5 a4)
= E|r(s|, s + y (s]
((),))))) mmax Qi (s) ( s41 d')l
according to Fig.
B[r(s}, sf) + > P(
Imax (5 8
E/r(s|, sf)l + TE P - ((-lee.))) mmax Qi+1 (5) (5 s4t,,
(26)
where the last equation follows h141 = h4 J [5/+1 at). and the second equation follows the causal graph shown in Fig. 7. Specifically, following the relationship in Fig. 7, we have
P(s hy sf, ax) P(h.) h. P(a, hy P(s sc).
(27)
15

---

h
of
Figure 7: The causal graph. The agent takes action di by only considering local states without accessing the privileged state of. The privileged state is based on history he- The next-state is conditional on h, sf, and ag-
Then we have
P( n4, si) P(s)]h.)
P(s he, sa)
P(h)) (ha
(28)
P( hes 0e, P(stt1;
P(an, hz)
where we denote 1/+1 [sh-1: 1642
Based on above analysis, we derive the value discrepancy bound as follows. For Vs!, sf the discrepancy between Q" (*( sy. az) and @e()), sp, az) can be decomposed as
Q'(1/ of at) Qr(s/s sp,
Q" st, a.) B (s), sf, at) (s), an) + M( &
of at))
(AA)
(i)
+ +Bg-r((h (5 st, at) 1n at)
(29)
(diz) value emvrr
+E, +B..-((in,)l(((4) s P Qu
(su) model
where we add three terms with positive sign and negative sign, including Q' with the expectation of P(-[he). Qu with the expectation of P(-[hz), and Qe with the expectation of P(-Ihe).
For (i) and (ii), we take the absolute value and get AØ Åz s |Ag + |Ågl = Ag, where
A Åg
A sup (1S + (ÂEl)
of.of
sup (* s a) E.g-r( QQ" (5/= 2e a)]l + (0, of,) - E.g a)]
of.
= ||Q" E,- Q21) 1- + - E 1011
(30)
which represents the difference in the same value function caused by the expectation of sf with respect to the history.
16

---

For (diz) that represents the value difference in the optimal value function Q" and the current value function Q with the same distribution of state-action, we have the following bound as
Exp-r(ha0 (s of az)] Exp A.) (2 ot ax)l Eir(s,, sf)] + y Pl P( max Q (5 (()
E[r(s) 3)] + ? ( ~PP
P(-la, 1) I Q (1641) & S41 d")]
D from (26)
=
max Q"
o d') max uex "422,
S max Bst ~ -(()))
( (411 d') Q041(5441:
S sup
1Q" (6/41) (141
41 Bitiaa
=> 11o0
(31)
For (iv) that represents the model difference hidden state with different distributions, we have the following bound by following (24), as
E Eeg- sf, at) - B P1lba (ch, St, at)
B|r(s), sf)] + DIE P(( p((bbaa)))) P( (b max Q441 (411
E|r Elr(s/ [1]1 1E
mmax Qu41 SP a")] (1111)
a"
=> p((baaa)) P( max Q 1 8 [4)] - E sp Ptylen mmax 1
Σ max Qu+1 a') P(( max - P(-bhc.)))))))
&4 a") ((sttilteet))
(32)
Define new function as P (r(() ar) ac))
k+4)) = max, ( x.)1,)), a), then we have
E Exppe(in) Qr(s) sh, ar)
=yE, ( p pellhe,ee)
P 1 (141 (tr P(5
P(st41lhe4))
(33)
$ { ((-1he++) - P-11h++1]l1
Hölder's inequality
27Tmax Dry (P(())++)))P((((ee))))
Frax &Taax/1 1-7
1-7
According to derivation of (i) (iv), we take the supreme of s sy a, and obtain the discrepancy between (* of ar) and (x SP, at)
2yTmaxx Drv Drr((((((e++
I|Q" Qxllo & sup (IA4) + ÁRl) +
+ > - 2.+1lloo
1-7
ef.efaa
2yrimax
(|A + (El) +
1-7 Epr. + +7llQ" - 111
sup
27
AR(t)= 1 1-2 Pen +~ 14+1llo:
(34)
where we define
sup Drv (rip))D)))(())))))))()((((()))) hit1
PP11
17

---

as the model difference at time step + 1 with the worst-case history. Meanwhile, we define
(E(()= sup (LA7)+
||" E(ppe( 1Q + ||Q E P 12)
bx)
(35)
measure the difference in the same value function with true sy and the expectation of sy conditioned on history lt.
In the following, we consider the step in worst case by taking supremum of t when t greater some to. and ensure the history is long and informative, as
sup lIQ Qr1loo S sup (() + sup 12te
2ymma max
1-7 PPets + sup I|@ Q4t1lloc
12to
12
12t
(36)
2yTmax
S sup (t) + sup
E + ssu IIQ Q.1100 12t
12
12
1 - 7
where we use inequality
sup
|1@T Q4t1/100 = sup
|1" - Qe1loc I sup 11Q Qcllx 12
f2
12t+1
Then we have
sup ||0- - Qello 5 12
SUP1210 At
sup 2yT'max 12
(1
(1 7)2 Pet1
(37)
Ag 27Tmmax
1-7 (1 7)2 cp.
where we define AE suP12to Ax(t) consider the step of worst case by taking supremum of t 2 fo. Mcanwhile, define the model error in worst case
cp
sup Cp + 1), 124
(38)
which be minimized by using a neural network represent P, and reducing the error through Monte Carlo sampling of transitions in training.
We remark that in A of Eq. (35), the term (i) in (35) captures how informative the history h, is in inferring necessary information of the hidden state sy. Intuitively, term (i) characterizes the error in estimating the optimal Q -avalue with the informative history h4, Similarly, term (ii) characterizes the error in predicting the hidden state si that follows the estimated model P based on the history. If the model P is deterministic and each &f E S has corresponding history hr, then term (ii) vanishes. Empirically, we use an informative history to make Ag sufficiently small.
18

---

B
Environment Setting
We will open-source our code once the paper is accepted. Below, we provide the details of the environment setup.
B.1 Details of Privileged DMC Benchmark
Based existing benchmarks 56), we develop three environments for privileged knowledge distillation, in- cluding pendulum, finger spin, and quadruped walk, that cover various training difficulties from easy to difficult. Each environment defines a series of privilege param- eters, which are indeed real-world physical properties, such as gravity, friction, mass of an object. Such privi- lege parameters define the behavior of the environments, and are only visible in training and hidden in testing. In order to train generalizable policy, we follow the domain randomization method and define randomization range for privilege parameters for training. The underlying transition dynamics will change since we sample these privilege parameters uniformly from a pre-defined randomization range every episode. For policy evaluation, we additionally define two testing randomization ranges (o.o.d and far o.o.d) that contain out-of-distributt parameters compared to the training range. The privilege parameters and the corresponding ranges of each environment are listed in Table Table and Table 6, respectively.
(b) morphology shift Figure 8: An example of morphology shift.
(a) origin
In addition to the aforementioned shifted dynamic parameters, which belong to external privilege, we consider shifted morphology parameters, classified as internal privilege, in the quadruped walk task. Specifically, we reset the lengths of the front legs back legs the beginning of each episode. The lengths are sampled uniformly from the ranges indicated in Table 7. Visualization of morphology shift can be seen in Fig. 8.
Table 4: Privileged knowledge randomization range pendulum. The testing range (ordinary) is the same as the training range. The privilege parameter d, refers the observation interval length, g gravity, I the pole length, the pole mass.
Privilege Parameter
Training Range(ordinary)
Testing Range (o.o.d.) Testing Range (far o.o.d.)
de
[0.0, 0.05]
[0.0, 0.1]
[0.0, 0.2]
g
[-2.0, 2.0]
[-3.0, 3.0]
[-4.0, 4.0]
[-0.5, 0.5]
[-0.8, 0.8]
[-0.9, 1.5]
[-0.5, 0.5]
[-0.8, 0.8]
[-0.9, 1.5]
Table 5: Privileged knowledge randomization range on finger spin. The testing range (ordinary) is the training range. friction_tange Langentala friction_torsional and triction_rolling refer to the scaling factors for tangential friction, torsional friction, and rolling friction of all ohjects, respectively. hn and ka are hyper-parameter, for the PD controller for all joints. actuator actuator.strengtt strength refers to the scaling factor for all actuators in the model. Viscosity refers to the scaling factor of viscosity used to simulate viscous forces. Wind_x, wind_y, and wind_z are used to compute viscous, lift and drag forces acting the body. respectively.
the
Privilege Parameter Training Range(ordinary)
Testing Range (o.o.d.) Testing Range (far o.o.d.)
friction tangential friction_torsional friction_rolling kp
[-0.8, 0.8]
[-0.9, 1.0]
[-0.9, 1.0]
[-0.8, 0.8]
[-0.9, 1.0]
[-0.9, 1.0]
[-0.8, 0.8)
[-0.9, 1.0]
[-0.9, 1.0]
[-0.8. 0.8]
[-0.9. 1.0]
[-0.9, 1.0]
&i actuator strength
[0, 1.01
J0.0, 1.5l
[0.0, 3.01
[-0.8, 0.8]
[-.., 1.0]
[-0.9, 1.0]
viscosity
0.8]
[0.0. 1.2]
[0.0, 2.0]
wind_x Lx
[-2,2]
[-3,3]
[-7, 7]
wind. y
[-2, 2]
[33,
[-7, 7]
wind_z
2-2,2
[-3, 3]]
[-7, 7]
19

---

Table 6: Privileged knowledge randomization range quadruped walk. The testing range (ordinary) is the same
the training range. Density is used to simulate lift and drag forces, which scale quadratically with velocity. geom, geom_density density is used infer masses and inertias. The other parameters have the meaning described in
Table
Privilege Parameter
Training Kangelordinary
Testing Range (o.o.d.)
Testing Range (far o.o.d.)
gravity
[-0.2. 0.2]
[-0.4, 0.4]
[-0.6, 0.6]
friction_tamee taaientiaa
[-0.1, 0.11
[-0.4, 0.4]
[-0.6, 0.6]
[-0.1,
[-0.4, 0.4]
[-0.6, 0.6]
friction. rolling kp &d
friction_rolling
0.1.
[-0.4, 0.4]
[-0.6, 0.6]
0-1.1.
[-0.4, 0.4]
[-0.6, 0.6]
[0. 0.11
[0.0, 0.4]
[0.0, 0.6]
actuator_strength
0.1.
[-0.4. 0.4]
[-0.6, 0.6]
density viscosity
10.0, 0.1]
[0.0, 0.4]
[0.0, 0.6]
[0.0, 0.1]
[0.0. 0.4]
[0.0. 0.6]
geom geom_density
[-0.1, 0.11
[-0.4, 0.4]
[-0.6, 0.6]
Table Privileged knowledge randomization range morph quadruped walk. The testing range (ordinary) the same as training range. Length is used to identify the length of the modified leg. The other parameters have the same meaning described in Table 5.
Privilege Parameter
Training Range(ordinary)
Testing Range (o.o.d.)
Testing Range (far o.o.d.)
gravity
[-0.2, 0.2]
[-0.4, 0.4]
[-0.6, 0.6]
friction tangential
[-0.1, 0.1]
[-0.4, 0.4]
[-0.6, 0.6]
tristion.torsional
[-0.1.
[-0.4, 0.4]
[-0.6, 0.6]
Diiciio_rollingg
1-0.1,0.1
[-0.4, 0.4]
[-0.6, 0.6]
kp ki
[-0.1.
[-0.4, 0.4]
[-0.6, 0.6]
10, 0.1]
J0.0, 0.4]
[0.0, 0.6]
actuator strrength
[-0.1,
[-0.4, 0.4]
[-0.6, 0.6]
density
10.0, 0.1]
[0.0. 0.4]
[0.0. 0.6]
viscosity
[0.0, 0.1]
[0.0. 0.4]
[0.0. 0.6]
geom density
[-0.1, 0.1]
[-0.4, 0.4]
[-0.6, 0.6]
length
[-0.1. 0.1]
[-0.15, 0.15)
[-0.25, 0.25]
B.2 Details of Legged Robot Benchmark
We develop codes based on Rudin et al. [3]. We create 4096 en- vironment instances to collect data in parallel. In each environment, the robot is initialized with random poses and commanded walk for- ward gnd 0.4m/s. The robot receives new observations and up- dates its actions every 0.02 seconds. Similar to Rudin et al. [3], we gen- erate different sets of terrain, includ- ing smooth slope, rough slope, stairs up. stairs down, wave, and discrete obstacle (see Fig. for visualiza- tion), with varying difficulty terrain levels. At training time, the envi- ronments are arranged in a 10 x 20 matrix with each row having terrain of the same type and difficulty in- creasing from left to right. We train with curriculum over terrain where robots are first initialized on easy terrain and promoted harder terrain if they traverse more than half its length. They demoted to easier terrain if they fail to travel at least half the commanded distance gondT where T is the maximum episode length.
(a) Smooth slope
(b) Rough slope
(c) Stairs up
(d) Stairs down
(e) Wave
(f) Discrete obstacle
Figure 9: Visualization of stains up, smooth slope, rough slope, stairs up. stairs down, and discrete obstacle.
20

---

The parameters of dynamic and heights of surroundings are defined as privilege parameters in this environment, as they can only be accessed in simulation but masked in the real world. The Heights of surroundings is an ego-centric map of the terrain around the robot. In particular, consists of the height values my {l(ar, ()) E P} at 187 points (-0.5, -0.4, ................................ 0.4, 0.53. The randomization range of dynamic parameters shown in Table 8. We the same reward function for method and all baselines, including task rewards following [3] and style rewards (given by Adversarial Motion Priors [62]) following [63, 64).
{-0.8, -0.7,.... 0.7, 0.8} x
Table 8: Privileged knowledge randomization range on Legged Robot Benchmark
Privilege Parameter Training Range
Testing Range
Added mass (kg)
[0. 3]
[0, 7]
kp
[22.4, 33.6] [0.56, 0.84]
[20, 60]
kd
[0.5, 1.0]
Table 9: Hyperparameters used in HIB
HyperParameter
Value
0.01
pondate,targe__interv
A
0.1
X2
0.1
history length &
50
B.3 Details of Experiments on Real A1 Robot
We apply our method HIB, and the strongest baseline, DreamWaQ, which uses asymmetric actor critic without using HIB, to a Unitree A1 robot without any fine-tuning in the real world. The computations are performed on an onboard NVIDIA Jetson TX2. The policy runs at 50Hz and the target joint angles were tracked by PD controller at a frequency of 200 Hz. The PD gains are k, = 28 and ka = 0.7, respectively. Policies rely on only the proprioception without any visual information for taking actions.
During the experiments, send 2-dimensional linear velocity and 1-dimensional angular velocity commands to the robot with remote controller. The performance of the robot is evaluated on different terrains, including plains, stairs, slopes and grass fields. The robot is commanded to perform basic moving skills like walking forward/backwarr and steering with different speed (maximum 1m/s)(see Fig. 10 for examples of real world terrains).
(a) Plain
(b) Grass
(c) Slopes
(d) Stairs
Figure 10: Examples of real-world terrains.
To further evaluate the robustness and the generalizability of the trained policy and also validate the effectiveness of the HIB method, we design two hard cases to test its performance. For the first case, the robot is required safely jump down from a 0.6m-high stair. HIB can go down this stair with 100% success rate while DreamWaQ completely fails. For the second case, the robot is required to maintain its balance when a leg is suddenly pulled back in dashing. HIB can achieve 75% success rate in 10 trials while Dream WaQ only obtains a 25% success rate. See Fig. 11 for examples.
21

---

(a) High stairs
(h) Pull leg in dash
Figure 11: Examples of two hard cases.
Algorithm 1 Historcal Information Bottleneck (HIB)
Training Process (sim)
Initialize: Buffer D = {is s/l, az, Te, 1,
1.. }
Initialize: Historical encoder fv. privilege encoder fwr projector go and target projector ga-.
1: while not coverge do
2: Interact to the environment to collect (sf, 0z, Tú 8441 with privileged state and save it to D
3: for j from 0 to
4: Sample batch of (s%, d4, s1+1) with history h;
5: Feed online Compute cosine similarity Lsim (Wh, Vi), KL regularization LKI. reconstruction loss fres
fuy (h:), y/ 4- go (z)), and target network ē; +- [u(())) yi 90 (E)
6:
and RL objective LRL
7: Update the HIB parameters via Eq. (13)
8: end for
9: end while
Evaluation Process (real)
1: Reset the environment and obtain the initial and ho
2: for from to /max do
3: Estimate the privileged state via ~ (he). and select action via f(als', z))
4: Interact with the environment, obtain the next state, set sf t- s/4 and update h4
5: end for
C Implementation Details
C.1 Algorithm Implementation
Empirically we find that the learned privilege representation 2; is requested to have similar and compact dimensions compared with another policy input s/. So when privileged state sf is high- dimensional, for example, sf in Legged Robot benchmark, the encoder fu is necessary to project privileged state in a lower dimension, keeping the dimension between 2 and é; being the same. In order to prevent information loss, we implement fa as an AutoEncoder [53]. Specifically, after Ja projecting of into representation é,, we have a decoder denoted as D to reconstruct the sp. Therefore, we leverage the following reconstruction loss to update encoder fw:
Crea = || Dwr (()
(39)
As result, employ Adam optimizer [55] to perform the following stochastic optimization step:
optimizer(w, Vulree)
(40)
In the Legged Robot benchmark. the privileged state consists of clevation information and some dynamic parameters, which are noisy and high- dige--imensionl. To project sf in lower-dimensional space, encoder fw can be a simple two-layer MLP. In the DMC benchmark. encoder fa can be an identity operator because the privileged state defined as dynamic parameters that are compact and have similar dimensions with s/. In this case, 2t has the same dimensions as si and é,
Considering there is historical trajectory at the beginning of the episode, padding and then gradually update it for practical implementation. In order to compute HIB loss
up it with zero
22

---

more accurately, we maintain another IB buffer Bib to sample h2 and sy practically, which only stores normal trajectories without zero padding.
C.2 Model Architecture and Hyperparameters
History encoder f& is TCN model, consisting of three conv ld layers and two linear layers.
The projector and target projector both have the same architecture, which is two-layered MLP.
Encoder fu is two-layer MLP on the Legged Robot Benchmark, and an identical operator on the privileged DMC benchmark.
Actor-Critic 3-layered MLPs (with Relu activation).
Hyperparameters used in HIB can be found in Table 9.
23


Reviews:
[{"review_id": "ArIwKiOmst", "reviewer": "Reviewer_iHnC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces Historical Information Bottleneck (HIB), a policy generalization approach that considers bridging the sim2real gap as an information bottleneck problem.   The method proposes to learn an encoding $f_\\varphi$ of the privileged state (only knowledge which is accessible in simulation) conditioned on the history of local states $h_t$. Such history encoding yields a latent variable $z_t$ which the generalist policy is conditioned on, next to the current local state $s_t^l$. To maximize mutual information $I(Z_t; S_t^p)$ between the latent representation $Z_t$ and the privileged knowledge $S_t^p$, a contrastive objective is proposed. Simplifying such objective by removing negative sampling, the proposed method uses cosine similarity with only positive samples $z_t\\sim f_\\varphi(h_t)$, resulting in a two-stream architecture consisting of an online and a target network. The online network learns the latent encoding of privileged information from the history of local states via a time convolutional neural network (TCN), the target network learns a representation $\\tilde{e}_t$ via an AutoEncoder of the privileged state. The cosine similarity between projections of both network outputs are used to evaluate $I(Z_t; S_t^p)$. At the same time, a regulizing loss term is used to ensure the information irrelevant to the privileged state information is compressed.  The authors demonstrate the performance of the proposed sim2real method on the DeepMind Control Suite benchmark and a real quadruped robot. Ablation studies over different HIB variants on the quadriped walking task are given. The learned latent representations are visualized through t-SNE which indicate that HIB is able to better recover the privileged information than previous policy distillation methods.", "summary_of_recommendation": "I tend to recommend the paper for acceptance. The approach is sound and the real robot results are strong. The paper is not a strong accept for me due to the remaining clarity issues and the potential limitation I raised in my review.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Why is it necessary to project $\\tilde{e}_t$ and use this projection to compute cosine similarity with the projection of $z_t$? Why can the autoencoder of the target network not directly learn a latent variable $\\tilde{z}_t$ whose cosine similarity to $z_t$ is maximized, i.e. without the projection step (given that both $\\tilde{e}_t$  and $z_t$ already have the same dimension)?\n\nSince the authors only mention the lack of support for recovering multi-modal privileged information, I wonder if there is a more significant limitation that should at least be mentioned:\nHow valid is the assumption that the local state space from the simulated environment is close enough for the policy to perform well in the real world?  Is it possible to quantify the closeness between such state representations (sim and real) within the proposed method is robust to generalize to the real world?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AkTC0H5Bwm", "reviewer": "Reviewer_jExp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a novel method called Historical Information Bottleneck (HIB) to address the sim-to-real gap in reinforcement learning. It assumes during training time, privileged knowledge is available in simulation, and agents only have access to local states in the real world. The proposed HIB method uses the Information Bottleneck principle to distill privileged knowledge from historical information by maximizing the MI between the historical representation and the privileged state via contrastive learning. The paper presents theoretical analysis and empirical results demonstrating HIB's efficacy in both simulated and real-world environments.", "summary_of_recommendation": "I recommend weak accept of the paper based on my evaluations.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Handling Multi-Modal Privileged Knowledge: As the authors have already pointed out in the limitation section, the method cannot handle multi-modal privileged knowledge such as RGB images. Are there any preliminary results or insights into this extension?\n2. Impact of Historical Trajectory Length: How does the length of the historical trajectory impact the performance of the HIB method? Currently the method uses a fixed length. Is there an optimal length, and how sensitive is the method to variations in this parameter? More analysis on this will be appreciated\n3. Noise and Uncertainty Handling: How does the HIB method handle noise and uncertainty in the historical data? Are there mechanisms to mitigate the impact of noisy or incomplete historical trajectories on the performance of the learned policy?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uvueci56Oa", "reviewer": "Reviewer_HU57", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces the Historical Information Bottleneck (HIB), a method that treats the sim-to-real transfer as an information bottleneck problem. HIB learns a privileged knowledge representation from historical data, improving generalizability and reducing value discrepancies between oracle and learned policies. Empirical results show that HIB outperforms previous methods in simulation tasks. The paper also shows HIB with good performance in real-world tasks.", "summary_of_recommendation": "In summary, I am leaning towards accepting the paper though it has some minor issues. Because it provides a detailed analysis of a knowledge distillation method for RL. The two-stream design and information bottleneck loss effectively reduce performance discrepancies, as demonstrated in both simulation and real-world experiments.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Some questions to address are:\n\n\n1) Figure 4's Reproducibility: What are the results if all baselines are trained for a longer duration?  Running additional experiments with more seeds could help verify whether the observed conclusions are consistent and reliable.\n\n2) Real-World Reward Function Definition: How is the reward function defined for real-world experiments?\n\n3) Real-World Performance of RMA Baseline: What is the real-world performance of the RMA baseline? It's better to show how good/bad are others in the real world instead of showing HIB only.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u5qb21CyLx", "reviewer": "Reviewer_jvAp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "To address the limitation in existing policy distillation framework which is computational expensive, lack of theoretical guarantee, and do not fully use privileged information. This paper introduces the Historical Information Bottleneck (HIB) method for reducing the sim-to-real gap in reinforcement learning.", "summary_of_recommendation": "see in the review", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "see in the review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses the sim-to-real gap by proposing a privileged knowledge distillation method called the Historical Information Bottleneck (HIB) that captures the changeable dynamic information. 

Strengths:
- Sound approach with a solid theoretical background 
- Empirical results and comparisons are strong, particularly the real-robot experiments 

Weaknesses:
- Lack of implementation details
- Limitations should be explained more clearly
- Limitations when dealing with high-dimensional observations
- Missing technical details of the method

The reviewers provided valuable suggestions to improve the paper. I encourage the authors to work with the reviewers in addressing their concerns.
=============================
Pos-rebuttal update

Thank you addressing the reviewers' comments. Great work!
Decision: Accept