Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective

Authors: haoran  he
peilin  wu
chenjia  bai
hang  lai
lingxiao  wang
ling  pan
xiaolin  hu
weinan  zhang
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 06:01:39

Abstract: Reinforcement Learning (RL) has recently achieved remarkable success in robotic control. However, most works in RL operate in simulated environments where privileged knowledge (e.g., dynamics, surroundings, terrains) is readily available. Conversely, in real-world scenarios, robot agents usually rely solely on local states (e.g., proprioceptive feedback of robot joints) to select actions, leading to a significant sim-to-real gap. Existing methods address this gap by either gradually reducing the reliance on privileged knowledge or performing a two-stage policy imitation. However, we argue that these methods are limited in their ability to fully leverage the available privileged knowledge, resulting in suboptimal performance. In this paper, we formulate the sim-to-real gap as an information bottleneck problem and therefore propose a novel privileged knowledge distillation method called the Historical Information Bottleneck (HIB). In particular, HIB learns a privileged knowledge representation from historical trajectories by capturing the underlying changeable dynamic information. Theoretical analysis shows that the learned privileged knowledge representation helps reduce the value discrepancy between the oracle and learned policies. Empirical experiments on both simulated and real-world tasks demonstrate that HIB yields improved generalizability compared to previous methods.

Main paper: 
 introduction: 
 Reinforcement Learning (RL) has made significant advancements across various simulated environments (e.g., games [1] , financial trading [2] ), but its applications in real-world scenarios still remain a challenge. The primary obstacle is the sim-to-real gap, an inherent mismatch between simulated and real environments that cause policies learned in simulation to perform sub-optimally in the real world. Previous works tackle this problem through more realistic simulation [3, 4] , adversarial training [5, 6] , and domain randomization [7, 8] to minimize the mismatch. However, building high-quality simulators is difficult, and excessive introduced assumptions can lead to an overly conservative policy.Recently, a more effective branch of methods proposes to utilize privileged knowledge to address the sim-to-real problem, where privileged knowledge is the information available in simulation (e.g., dynamics, surroundings, terrains) but inaccessible in real environments. Previous methods solve this problem via a two-stage policy distillation process [9] . Specifically, a teacher policy is first trained in the simulator with privileged states accessible, and then a student policy relying on local states (without privileged information) is trained by imitating the teacher policy. Nevertheless, such a two-stage paradigm is computationally expensive and requires careful design for imitation. An alternative approach involves gradually dropping the privileged information as the policy is trained [10] or conditioning the critic on privileged information [11] . However, these methods lack a theoretical understanding of the sim-to-real problem and do not fully exploit the available privileged information during training.In this paper, we present a representation-based approach, instead of policy distillation, to better utilize the privileged knowledge from simulation with a single-stage learning paradigm. Inspired by the Information Bottleneck (IB) method [12, 13] , which learns a minimal sufficient representation Z of a given input source X with the target source Y , we propose a novel method called Historical Information Bottleneck (HIB). Similar to the goal of the IB method, HIB aims to find a maximally compressed representation of the privileged knowledge while preserving sufficient information about the current environment for real-world decision-making. In particular, HIB takes advantage of historical information that contains previous local states and actions to learn a history representation, which is trained by maximizing the mutual information (MI) between the representation and the privileged knowledge. Theoretically, we show that maximizing such an MI term will minimize the privileged knowledge modeling error, reducing the discrepancy between the optimal value function and the learned value function. Furthermore, benefiting from the IB principle, we compress the decision-irrelevant information from the history and obtain a more robust representation. The IB objective is approximated by variational lower bounds to handle the high-dimensional state space.In summary, our contributions are threefold: (i) We propose a novel policy generalization method called HIB that follows the IB principle to distill privileged knowledge from a fixed length of history. (ii) We provide a theoretical analysis of both the policy distillation methods and the proposed method, which shows that minimizing the privilege modeling error is crucial in learning a near-optimal policy. (iii) Empirically, we show that HIB learns robust representation in randomized RL environments and achieves better generalization performance in both simulated and real-world environments than state-of-the-art (SOTA) algorithms, including out-of-distribution test environments. 
related work: 
 Sim-to-Real Transfer. Transferring RL policies from simulation to reality is challenging due to the domain mismatch. To this end, the previous study hinges on domain randomization, which trains the policy under a wide range of environmental parameters and sensor noises [14, 15, 16, 17, 18] . However, domain randomization typically sacrifices the optimality for robustness, leading to an over-conservative policy [19] . To address this problem, various works perform privilege distillation by teacher-student learning [20, 21, 22, 23, 24] , teacher demonstration exploration [25, 26, 27, 28] and teacher policy progressive imitation [29, 30] . However, these methods are sample inefficient due to the requirement of training an additional teacher policy. Aside from privileged policy, recent works exploit the privileged information by introducing privileged critic [31, 32, 11] or privileged world models [33, 34] . An alternative method gradually drops privileged knowledge [10] . Nevertheless, these methods are limited to fully leveraging historical knowledge for better generalization. Different from the above approaches, our method learns a privileged representation via informative historical trajectories from the IB perspective, resulting in better utilization of historical information.Information Bottleneck for RL. The IB principle [35, 13] was initially proposed to trade off the accuracy and complexity of the representation in supervised learning. Specifically, IB maximizes the MI between representation and targets to extract useful features, while also compressing the irrelevant information by limiting the MI between representation and raw inputs [36, 37] . Recently, IB has been employed in RL to acquire a compact and robust representation. For example, Fan and Li [38] takes advantage of IB to learn task-relevant representation via a multi-view augmentation. Other methods [39, 40, 41] maximize the MI between representation and dynamics or value function, and restrict the information to encourage the encoder to extract only the task-relevant information. Unlike the previous works that neither tackle the policy generalization problem nor utilize historical information, HIB derives a novel objective based on IB, which aims to learn a robust representation of privileged knowledge from history while simultaneously removing redundant decision-irrelevant information. 
preliminaries: 
 In this section, we briefly introduce the problem definition and the corresponding notations used throughout this paper. We give the definition of privileged knowledge in robot learning as follows.Definition 1 (Privileged Knowledge). Privileged knowledge is the hidden state that is inaccessible in the real environment but can be obtained in the simulator, e.g., surrounding heights, terrain types, morphology parameters like length of legs, and dynamic parameters like friction and damping. An oracle (teacher) policy is defined as the optimal policy with privileged knowledge visible.In this paper, we extend the concept of the sim-to-real gap to a general policy generalization problem with a knowledge gap. We define the MDP as M = (S l , S p , A, P, r, γ), where [s l , s p ] = s o represents the oracle state s o that contains s l ∈ S l (i.e., the local state space) and s p ∈ S p (i.e., the privileged state space), where s p contains privileged knowledge defined in Definition 1. A is the action space. The transition function P (s o t+1 |s o t , a t ) and reward function r(s o , a) follows the ground-truth dynamics based on the oracle states. Based on the MDP, we define two policies: π(a|s l , s p ) and π(a|s l ), for the simulation and real world, respectively. Specifically, π(a|s l , s p ) is a privileged policy that can access the privileged knowledge, which is only accessible in the simulator. In contrast, π(a|s l ) is a local policy without accessing the privileged knowledge throughout the interaction process, which is common in the real world. A thorough discussion about our problem and Partially Observable MDP (POMDP) [42, 43] is provided in Appendix A.1.Based on the above definition, our objective is to find the optimal local policy π * based on the local state s l that maximizes the expected return, denoted asEQUATION4 Theoretical Analysis & Motivation 
value discrepancy for policy generalization: 
 In this section, we give a theoretical analysis of traditional oracle policy imitation algorithms [44] . Note that previous works give similar analyses but in different formulations [22, 27, 30] . Specifically, the local policy is learned by imitating the optimal oracle policy π * . We denote the optimal value function of the policy π * learned with oracle states s o = [s l , s p ] as Q * (s l , s p , a), and the value function of policy learned with local states as Qπ (s l , a). The following theorem analyzes the relationship between the value discrepancy and the policy imitation error in a finite MDP setting. Theorem 1 (Policy imitation discrepancy). The value discrepancy between the optimal value function with privileged knowledge and the value function with the local state is bounded asEQUATIONwhereEQUATIONis the policy divergence between π * and π, and r max is the maximum reward in each step.The proof is given in Appendix A.2. Theorem 1 shows that minimizing the total variation (TV) distance between π * (•|s l , s p ) and π(•|s l ) reduces the value discrepancy. However, minimizing ϵ π can be more difficult than ordinary imitation learning where π * and π have the same state space. Specifically, if π * and π have the same inputs, D TV (π * (•|s)∥π(•|s)) will approach zero with sufficient model capacity and large iteration steps, at least in theory. In contrast, due to the lack of privileged knowledge in our problem, the policy error term in Eq. (3) can still be large after optimization as π * (•|s l , s p ) and π(•|s l ) have different inputs.Previous works [21, 20] try to use historical trajectoryEQUATIONwith a fixed length to help infer the oracle policy, and the policy imitation error becomes D TV (π * (•|s l , s p )∥π(•|s l , h)). However, without appropriately using the historical information, imitating a well-trained oracle policy can still be difficult for an agent with limited capacity, which results in suboptimal performance. Meanwhile, such two-stage policy imitation methods are also sample-inefficient.To address the above challenges, we raise an alternative theoretical motivation to relax the requirement of the oracle policy. Specifically, we quantify the discrepancy between the optimal value function and the learned value function based on the error bound in reconstructing the privileged state s p t via historical information h t , which eliminates the reliance on the oracle policy and makes our method a single-stage distillation algorithm. Specifically, we define a density model P (s p t |h t ) to predict the privileged state based on history h t in Eq. ( 4). Then the predicted privileged state can be sampled as ŝp t ∼ P (•|h t ). In policy learning, we concatenate the local state s l and the predicted ŝp as input. The following theorem gives the value discrepancy of Q * and the value function Q with the predicted ŝp . Theorem 2 (Privilege modeling discrepancy). Let the divergence between the distribution of privileged state model P (s p t+1 |h t+1 ) and the true distribution of privileged state P (s p t+1 |h t+1 ) be bounded asEQUATIONThen the performance discrepancy bound between the optimal value function with P and the value function with P holds, asEQUATIONwhere∆ E = sup t≥t0 Q * -E s p t ∼P (•|ht) [Q * ] ∞ + Q -E ŝp t ∼ P (•|ht) [ Q]∞ is the difference in the same value function with sampled s p t and the expectation of s p t conditioned on h t . We defer the detailed proof in Appendix A. 3. In ∆ E , we consider using a sufficiently long (i.e., by using history t greater than some t 0 ) and informative (i.e., by extracting useful features) history to make ∆ E small in practice. We remark that ∆ E captures the inherent difficulty of learning without privileged information. The error is small if privileged information is near deterministic given the history, or if the privileged information is not useful given the history. ϵ P measures the model discrepancy in the worst case with an informative history. In the next section, we provide an instantiation method inspired by Theorem 2. 
methodology: 
 In this section, we propose a practical algorithm named HIB to perform privilege distillation via a historical representation. HIB only acquires the oracle state without an oracle policy in training. In evaluation, HIB relies on the local state and the learned historical representation to choose actions.Theorem 2 indicates that minimizing ϵ P yields a tighter performance discrepancy bound. We then start by analyzing the privilege modeling discrepancy ϵ P in Eq. ( 5). We denote the parameter of Pϕ by ϕ, then the optimal solution ϕ * can be obtained by minimizing the TV divergence for ∀t, asEQUATIONwhere the true distribution P (•|h t ) is irrelevant to ϕ, and we convert the TV distance to the KL distance in Eq. ( 7) by following Pinsker's inequality. Since h t is usually high-dimensional, which is of linear complexity with respect to time, it is necessary to project h t in a representation space and then predict s p t . Thus, we split the parameter of Pϕ as ϕ = [ϕ 1 , ϕ 2 ], where ϕ 1 aims to learn a historical representation z = f ϕ1 (h t ) first, and ϕ 2 aims to predict the distribution Pϕ2 (z) of privileged state (e.g., a Gaussian). In the following, we show that maximizing I pred is closely related to maximizing the MI between the historical representation and the privileged state. In particular, we have where we denote the random variables for s p t and h t by S p t and H t , respectively. In Eq. ( 9), the upper bound is obtained by the non-negativity of the Shannon entropy and KL divergence. The bound is tight since the entropy of the privileged state H(S p t ) is usually constant, and D KL (P ∥ P ) can be small when we use a variational Pϕ with an expressive network.EQUATIONAccording to Eq. ( 9), maximizing the predictive objective I pred is closely related to maximizing the MI between S p t and f ϕ1 (H t ). In HIB, to address the difficulty of reconstructing the raw privileged state that can be noisy and high-dimensional in I pred objective, we adopt contrastive learning [45] as an alternative variational approximator [46] to approximate MI in representation space. Moreover, HIB restricts the capacity of representation to remove decision-irrelevant information from the history, which resembles the IB principle [35] in information theory.To optimize the MI in Eq. ( 9) via a contrastive objective [47] , we introduce a historical representation z t ∼ f ψ (h t ) to extract useful features that contain privileged information from a long historical vector h t , where f ψ is a temporal convolution network (TCN) [48] that captures long-term information along the time dimension. We use another notation ψ to distinguish it from the predictive encoder ϕ 1 in Eq. ( 9), since the contrastive objective and predictive objective I pred learn distinct representations by optimizing different variational bounds. In our IB objective, the input variable is H t , and the corresponding target variable is S p t . Our objective is to maximize the MI term I(Z t ; S p t ) while minimizing the MI term I(H t ; Z t ) with Z t = f ψ (H t ), which takes the form ofEQUATIONwhere α is a Lagrange multiplier. The I(Z t ; S p t ) term quantifies the amount of information about the privileged knowledge preserved in Z t , and the I(H t ; Z t ) term is a regularizer that controls the complexity of representation learning. With a well-tuned α, we do not discard useful information that is relevant to the privileged knowledge.We minimize the MI term I(H t ; Z t ) in Eq. ( 10) by minimizing the following tractable upper bound:EQUATIONwhere the inequality follows the non-negativity of the KL divergence, and q(z t ) is an approximation of the marginal distribution of Z t . We follow Alemi et al. [36] and use a spherical Gaussian q(z t ) = N (0, I) as an approximation.One can maximize the MI term I(Z t ; S p t ) in Eq. ( 10) based on the contrastive objective [47] . Specifically, for a given s p t , the positive sample z t ∼ f ψ (h t ) is the feature of corresponding history in timestep t, and the negative sample z -can be extracted from randomly sampled historical vectors. However, considering unresolved trade-offs involved in negative sampling [49, 50] , we simplify the contrastive objective by removing negative sampling. In HIB, the contrastive loss becomes a cosine similarity with only positive pairs, and we empirically find that the performance does not decrease. Such a simplification was also adopted by recent contrastive methods for RL [51, 52] .We adopt a two-stream architecture to learn z t , including an online and a target network. Each network contains an encoder and a projector, as shown in Fig. 1 . The online network is trained to use history to predict the corresponding privilege representation. Given a pair of a history sequence and privileged state (h t , s p t ), we obtain ẽt = f ω (s p t ) with an encoder f ω to get the representation of s p t . Here, f ω is used to project s p t into the same dimensional space as z t , so f ω can be a simple AutoEncoder [53] trained by a reconstruction loss L rec = ∥s p t -D ω ′ (f ω (s p t ))∥ 2 , where D ω ′ is an additional trained decoder (see Appendix C.1 for details of implementation). Then we use TCN as the history encoder f ψ to learn the latent representation z t ∼ f ψ (•|h t ). As ẽt and z t have the same dimensions, the projectors share the same architecture. The online projector g θ outputs y t = g θ (z t ) and the target projector g θ -outputs ỹt = g θ -(ẽ t ). We use the following cosine similarity loss between y t and ỹt , and use stop gradient (sg[•]) for the target value ỹ, asEQUATION)To prevent collapsed solutions in the two-stream architecture, we follow previous architectures [54] by using a momentum update for the target network to avoid collapsed solutions. Specifically, the parameter of the target network θ -takes an exponential moving average of the online parameters θ with a factor τ ∈[0, 1], as θ -← τ θ -+ (1 -τ )θ.At each training step, we perform a stochastic optimization step to minimize L sim with respect to θ and ψ. Meanwhile, we learn an RL policy π(a|s l t , z t ) based on the historical representation z t , and the RL objective is also used to train the TCN encoder f ψ . The dynamics are summarized asEQUATIONwhere L KL = D KL (f ψ (h t )∥N (0, I)) is the IB term in Eq. ( 11) that controls the latent complexity, and L RL is the loss function for an arbitrary RL algorithm. We use the Adam optimizer [55] . We summarize our algorithm in Alg. 1 in Appendix C.1. 
benchmarks and compared methods: 
 To quantify the generalizability of the proposed HIB, we conduct experiments in simulated environments that include multiple domains for a comprehensive evaluation, and also the legged robot locomotion task to evaluate the generalizability in sim-to-real transfer.Privileged DMC Benchmark. We conduct experiments on DeepMind Control Suite (DMC) [56] with manually defined privileged information, which contains dynamic parameters such as friction and torque strength, and morphology parameters such as the lengths of specific legs. The privileged knowledge is only visible in the training process. Following Benjamins et al. [57] , we randomize the privilege parameters at the beginning of each episode, and the randomization range can be different for training and testing. Specifically, we choose three randomization ranges for varied difficulty levels, i. Sim-to-Real in Blind Legged Robot. This experiment is conducted on a quadrupedal robot. In this domain, privileged knowledge is defined as terrain information (e.g. heights of surroundings) of the environment and dynamic information such as friction, mass, and damping of the quadrupedal robot. We leverage the Isaac Gym simulator [58] for training, which also provides multi-terrain simulation (e.g., slopes, stairs, and discrete obstacles). Details can be found in Appendix B.2. For experiments in the real robot, we utilize the Unitree A1 robot [59] to facilitate real-world deployment. Baselines. We compare HIB to the following baselines. (i) Teacher policy is learned by oracle states with privileged knowledge. (ii) Student policy follows RMA [21] that mimics the teacher policy through supervised learning, with the same architecture as the history encoder in HIB. We remark that student needs a two-stage training process to obtain the policy. (iv) DreamWaQ follows the implementation in Nahrendra et al. [31] , conditioning the critic on privileged information and utilizing history trajectories to learn a latent variable z for action. (iv) Dropper is implemented according to Li et al. [10] , which gradually drops the privileged information and finally converts to a normal agent that only takes local states as input. (v) DR agent utilizes domain randomization for generalization and is directly trained with standard RL algorithms with local states as input.For varying dynamic parameters, the results are shown in Table 1 . Our method achieves the best performance in almost all test environments, closely matching the returns of the oracle (teacher) method. The results in Table 2 demonstrate that HIB can also generalize to unseen environments with morphology shifts, highlighting its superior capability which benefits from our proposed IBstyle training. While the student baseline can perform slightly better than HIB, it exhibits larger variance because its performance heavily relies on the teacher policy and its exploration ability is restricted. The results on the Legged Robot benchmark (Table 3 ) further verify the advantage of HIB.Our method outperforms the two strongest baselines, student and DreamWaQ, on the three most challenging terrains: stairs, discrete obstacles, and slopes. From the simulation results, HIB can be seamlessly combined with different RL algorithms and generalizes well across different domains and tasks, demonstrating the efficacy and high scalability of HIB. To investigate the ability of HIB in modeling the privileged knowledge, we visualize the latent representation learned by HIB and student via dimensional reduction with t-SNE [60] . We also visualize the true privileged information for comparison. The visualization is conducted in finger spin task, and the results are given in Fig. 3 . We find that the student agent can only recover part of the privileged knowledge that covers the bottom left and upper right of the true privilege distribution. In contrast, the learned representation of HIB has almost the same distribution as privileged information. This may help explain why our method outperforms other baselines and generalizes to o.o.d. scenarios without significant performance degradation.We conduct an ablation study for each component of HIB to verify their effectiveness. Specifically, we design the following variants to compare with. (i) HIB-w/o-ib only uses RL loss to update history encoder f ψ , which is similar to a standard recurrent neural network policy. (ii) HIB-w/o-rl only uses HIB loss to update the history encoder without the RL objective. (iii) HIB-w/o-proj drops the projectors in HIB and directly compute cosine similarity loss between z t and ẽt . (iv) HIB-contra employs contrastive loss [61] instead of cosine similarity, using a score function that assigns high scores to positive pairs and low scores to negative pairs.From the result in Fig. 4 , we observe that HIB-w/o-ib almost fails and HIB-w/o-rl can get relatively high scores, which signifies that both HIB loss and RL loss are important for the agent to learn a well-generalized policy, especially the HIB loss. The HIB loss helps the agent learn a historical representation that contains privileged knowledge for better generalization. Furthermore, projectors and the momentum update mechanism are also crucial in learning robust and effective representation. Moreover, HIB-contra performs well at the beginning but fails later, which indicates that the contrastive objective requests constructing valid negative pairs and learning a good score function, which is challenging in the general state-based RL setting. To further evaluate the generalizability of HIB in the real world, we deploy the HIB policy trained in the Legged Robot benchmark on a real-world A1 robot without any fine-tuning. Detailed setup is referred to Appendix B.3. Note that the policy is directly run on the A1 hardware, and the local state is read or estimated from the onboard sensors and IMU, making the real-world control noisy and challenging. Fig. 2 shows the snapshots of our HIB agent traversing different terrains. The agent can generalize to different challenging terrains with stable control behavior and there is no failure in the whole experimental process. Fig. 5 further showcases the advantages of our proposed HIB, enabling the agent to navigate down high stairs with a 0.6m height, achieving a 100% success rate in 10 trials, while the strongest baseline (e.g. DreamWaQ) completely fail. These real experiments demonstrate that HIB enhances the ability to bridge the sim-to-real gap without any additional tuning in real environments. 
conclusion and limitation: 
 We propose a novel privileged knowledge distillation method based on the Information Bottleneck to narrow the knowledge gap between local and oracle RL environments. In particular, the proposed twostream model design and HIB loss help reduce the performance discrepancy given in our theoretical analysis. Our experimental results on both simulated and real-world environments show that (i) HIB learns robust representations to reconstruct privileged knowledge from local historical trajectories and boosts the RL agent's performance, and (ii) HIB can achieve improved generalizability in outof-distribution environments compared to previous methods. However, HIB is limited in recovering multi-modal privileged knowledge (e.g., RGB images), which is more high-dimensional and complex. 
 
 Appendix: We summarize common points and differences between POMDP and our setting as follows.1. The transition function and reward function for both problems follows the ground-truth dynamics of the environment.2. In POMDP, the agent cannot access the oracle state in both training and evaluation, while in sim-to-real adaptation, the agent can access the oracle in training (in the simulation). 𝑠 ! " 𝑠 ! # 𝑎 ! 𝑟 ! 𝑠 !$% # 𝑠 !$% " 𝑏 ! 𝑏 !$% 𝑎 !$% 𝑟 !$% POMDP train/eval 𝑠 ! " 𝑠 ! # 𝑎 ! 𝑟 ! 𝑠 !$% # 𝑠 !$% " 𝑎 !$% 𝑟 !$% Sim-to-real adaptation sim 𝑠 ! " 𝑠 ! # 𝑎 ! 𝑟 ! 𝑠 !$% # 𝑠 !$% " 𝑎 !$% 𝑟 !$% real belief stateWe begin by deriving the imitation discrepancy bound in an ordinary MDP, as shown in the following theorem.Theorem A.1 (General oracle imitation discrepancy bound). Let the policy divergence between the oracle policy π * and the learned policy π is ϵ π = sup s D TV (π * (•|s)∥π(•|s)). Then the difference between the optimal action value function Q * of π * and the action value function Q of π is bounded asEQUATION)where γ is the discount factor, and r max is the maximum reward in each step.Proof. For any (s, a) pair that s ∈ S, a ∈ A, the difference between Q * (s, a) and Q(s, a) isEQUATION)Then we introduce some intermediate terms and obtainEQUATIONThus,EQUATIONThen we haveEQUATIONwhich completes our proof.Next, we consider the value discrepancy bound in policy imitation with the knowledge gap. Specifically, we denote the optimal value function with oracle states s o = [s l , s p ] as Q * (s l , s p , a), and the value function with local states as Q(s l , a). Then we have the following discrepancy bound (restate of Theorems 1).Theorem A.2 (Policy imitation discrepancy). The value discrepancy between the optimal value function with privileged knowledge and the value function with local state is bounded asEQUATIONwhereEQUATIONis the policy divergence between π * and π, and r max is the maximum reward in each step.Proof. We note that the transition functions for learning π(a|s l ) and π * (a|s l , s p ) are the same and follow the ground-truth dynamics. Although we cannot observe the privileged state s p in learning the local policy, the transition of the next state P (s l t+1 , s o t+1 |s l t , s p t ) still depends on the oracle state of the previous step that contains the privileged state s p t . Since the agent is directly interacting with the environment, the privileged state does affect the transition functions. As a result, for policy imitation of the local policy, we have a similar derivation as in (15) because the transition function in (15) is the same for π * (a t+1 |s o t+1 )Q * (s o t+1 , a t+1 ) and π(a t+1 |s l t+1 ) Q(s l t+1 , a t+1 ), which leads to a similar discrepancy bound as in an ordinary MDP.We have defined the history encoder Pϕ which takes the history as input and the privileged state as output:EQUATIONIn the following, we give the performance discrepancy bound between the optimal value function with oracle state s o t = [s l t , s p t ] and the value function with local state s l t as well as the predicted privileged state ŝp t (restate of Theorem 2).Theorem A.3 (Privilege modeling discrepancy). Let the divergence between the privileged state model P (s p t+1 |h t+1 ) and the true distribution of privileged state P (s p t+1 |h t+1 ) be bounded asEQUATIONThen the performance discrepancy bound between the optimal value function with P and the value function with P holds, asEQUATIONwhere∆ E = sup t≥t0 Q * -E s p t ∼P (•|ht) [Q * ] ∞ + Q -E ŝp t ∼ P (•|ht) [ Q]∞ is the difference in the same value function with sampled s p t and the expectation of s p t conditioned on h t .Proof. For Q that estimates the value function of state (s l t , ŝp t ), the value is stochastic since the hidden prediction ŝp t is a single sample from the output of model P (•|h t ). We take the expectation to the output of P (•|h t ) and obtain the expected Q asEQUATIONwhere h t+1 = h t ∪ {s l t+1 , a t }. We remark that the reward function r(s l t , s p t ) is returned by the real environment, thus s p t follows the true P in the reward.Remark 1 (Explanation of the Bellman Equation). Our goal is to fit the Bellman equation in (24) .Fitting the Bellman equation in (24) has several benefits. Firstly, it is easily implemented based on the fitted privileged state model P . Specifically, solving (24) is almost the same as solving an ordinary MDP, with ŝp t generated by P based on the history in place of the true privileged state of s p t (which is unavailable). Secondly and most importantly, when the history is sufficiently strong in predicting s p t , solving (24) leads to approximately optimal solutions, as we show in this theorem.For Q * that represents the optimal value function with the true privileged state, we have Q * (s l t ,where the second equation holds since h t contains s l t . Then we take a similar expectation to the true hidden transition function P as E s p t ∼P (•|ht) [Q * (s l t ,where we add three terms with positive sign and negative sign, including Q * with the expectation of P (•|h t ), Qt with the expectation of P (•|h t ), and Qt with the expectation of P (•|h t ).For (i) and (ii), we take the absolute value and get ∆ * E -∆E ≤ |∆ * E | + | ∆E | = ∆ E , where ∆ * E -∆E ≤ |∆ * E | + | ∆E | ≤ sup s l t ,EQUATIONwhich represents the difference in the same value function caused by the expectation of s p t with respect to the history.For (iii) that represents the value difference in the optimal value function Q * and the current value function Q with the same distribution of state-action, we have the following bound as E s p t ∼P (•|ht) [Q * (EQUATIONFor (iv) that represents the model difference of hidden state with different distributions, we have the following bound by following ( 24), as E s p t ∼P (•|ht) [ Qt (s l t , s p t , a t )] -E ŝp t ∼ P (•|ht) [ Qt (s l t , ŝp t , a t )] = E[r(s l t , s p t )] + γE s l t+1 ∼P (•|ht,at),s p t+1 ∼P (•|ht+1) [max a ′ Qt+1 (s l t+1 , s p t+1 , a ′ )] -E[r(s l t , s p t )] -γE s l t+1 ∼P (•|ht,at),ŝ p t+1 ∼ P (•|ht+1) [max a ′′ Qt+1 (s l t+1 , ŝp t+1 , a ′′ )] = γE s l t+1 ∼P (•|ht,at) E s p t+1 ∼P (•|ht+1) [max a ′ Qt+1 (s l t+1 , s p t+1 , a ′ )] -E ŝp t+1 ∼ P (•|ht+1) [max a ′′ Qt+1 (s l t+1 , ŝp t+1 , a ′′ )] = γE s l t+1 ∼P (•|ht,EQUATIONAccording to derivation of (i) ∼ (iv), we take the supreme of s l t , s p t , a t and obtain the discrepancy between Q * (s l t , s p t , a t ) and Qt (s l t , ŝp t , a t ) as as the model difference at time step t + 1 with the worst-case history. Meanwhile, we defineEQUATIONEQUATIONto measure the difference in the same value function with true s p t and the expectation of s p t conditioned on history h t .In the following, we consider the step in worst case by taking supremum of t when t greater some t 0 , and ensure the history is long and informative, asEQUATIONwhere we use inequalitysup t≥t0 ∥Q * -Qt+1 ∥ ∞ = sup t≥t0+1 ∥Q * -Qt ∥ ∞ ≤ sup t≥t0 ∥Q * -Qt ∥ ∞ .Then we haveEQUATIONwhere we define ∆ E = sup t≥t0 ∆ E (t) to consider the step of worst case by taking supremum of t ≥ t 0 . Meanwhile, we define the model error in worst case asEQUATIONwhich can be minimized by using a neural network to represent P , and reducing the error through Monte Carlo sampling of transitions in training.We remark that in ∆ E of Eq. ( 35), the term (i) in (35) captures how informative the history h t is in inferring necessary information of the hidden state s p t . Intuitively, term (i) characterizes the error in estimating the optimal Q-value with the informative history h t . Similarly, term (ii) characterizes the error in predicting the hidden state s p t that follows the estimated model P based on the history. If the model P is deterministic and each ŝp t ∈ S p has a corresponding history h t , then term (ii) vanishes. Empirically, we use an informative history to make ∆ E sufficiently small. We develop our codes based on Rudin et al. [3] . We create 4096 environment instances to collect data in parallel. In each environment, the robot is initialized with random poses and commanded to walk forward at v cmd x = 0.4m/s. The robot receives new observations and updates its actions every 0.02 seconds. Similar to Rudin et al. [3] , we generate different sets of terrain, including smooth slope, rough slope, stairs up, stairs down, wave, and discrete obstacle (see Fig. 9 for visualization), with varying difficulty terrain levels. At training time, the environments are arranged in a 10 × 20 matrix with each row having terrain of the same type and difficulty increasing from left to right. We train with a curriculum over terrain where robots are first initialized on easy terrain and promoted to harder terrain if they traverse more than half its length. They are demoted to easier terrain if they fail to travel at least half the commanded distance v cmd x T where T is the maximum episode length. Interact to the environment to collect (s o i , a i , r i , s o i+1 ) with privileged state and save it to D 3:for j from 0 to N do 4:Sample a batch of (s o i , a i , r i , s o i+1 ) with history h i 5:Feed online z i ∼ f ψ (h i ), y i ← g θ (z i ), and target network ẽi ← f ω (s p i ), ỹi ← g θ -( ẽi )6:Compute cosine similarity L sim (y i , ỹi ), KL regularization L KL , reconstruction loss L rec and RL objective L RL 7:Update the HIB parameters via Eq. ( 13) 8:end for 9: end while # Evaluation Process (real)1: Reset the environment and obtain the initial s 0 and h 0 2: for t from t 0 to t max do 3:Estimate the privileged state via z t ∼ f ψ (h t ), and select action via a ∼ π(a|s l t , z t )Interact with the environment, obtain the next state, set s l t ← s l t+1 , and update h t 5: end for C Implementation DetailsEmpirically we find that the learned privilege representation z t is requested to have similar and compact dimensions compared with another policy input s l t . So when privileged state s p t is highdimensional, for example, s p t in Legged Robot benchmark, the encoder f ω is necessary to project privileged state in a lower dimension, keeping the dimension between z t and ẽt being the same. In order to prevent information loss, we implement f ω as an AutoEncoder [53] . Specifically, after f ω projecting s p t into representation ẽt , we have a decoder denoted as D ω ′ to reconstruct the s p t . Therefore, we leverage the following reconstruction loss to update encoder f ω :EQUATION)As a result, we employ Adam optimizer [55] to perform the following stochastic optimization step:EQUATIONIn the Legged Robot benchmark, the privileged state consists of elevation information and some dynamic parameters, which are noisy and high-dimensional. To project s p t in a lower-dimensional space, encoder f ω can be a simple two-layer MLP. In the DMC benchmark, encoder f ω can be an identity operator because the privileged state is defined as dynamic parameters that are compact and have similar dimensions with s l t . In this case, z t has the same dimensions as s p t and ẽt . Considering there is no historical trajectory at the beginning of the episode, we warm up it with zero padding and then gradually update it for practical implementation. In order to compute HIB loss more accurately, we maintain another IB buffer B ib to sample h t and s p t practically, which only stores normal trajectories without zero padding.• History encoder f ψ is a TCN model, consisting of three conv1d layers and two linear layers.• The projector and target projector both have the same architecture, which is a two-layered MLP.• Encoder f ω is a two-layer MLP on the Legged Robot Benchmark, and an identical operator on the privileged DMC benchmark.• Actor-Critic is 3-layered MLPs (with Relu activation).• Hyperparameters used in HIB can be found in Table 9 .† Correspondence to: Weinan Zhang (wnzhang@sjtu.edu.cn). 


Decision: The paper addresses the sim-to-real gap by proposing a privileged knowledge distillation method called the Historical Information Bottleneck (HIB) that captures the changeable dynamic information. 

Strengths:
- Sound approach with a solid theoretical background 
- Empirical results and comparisons are strong, particularly the real-robot experiments 

Weaknesses:
- Lack of implementation details
- Limitations should be explained more clearly
- Limitations when dealing with high-dimensional observations
- Missing technical details of the method

The reviewers provided valuable suggestions to improve the paper. I encourage the authors to work with the reviewers in addressing their concerns.
=============================
Pos-rebuttal update

Thank you addressing the reviewers' comments. Great work!

Reviews: review: title: title: A convincing approach to policy generalization for bridging the sim2real gap
paper_summary: paper_summary: The paper introduces Historical Information Bottleneck (HIB), a policy generalization approach that considers bridging the sim2real gap as an information bottleneck problem.   The method proposes to learn an encoding $f_\varphi$ of the privileged state (only knowledge which is accessible in simulation) conditioned on the history of local states $h_t$. Such history encoding yields a latent variable $z_t$ which the generalist policy is conditioned on, next to the current local state $s_t^l$. To maximize mutual information $I(Z_t; S_t^p)$ between the latent representation $Z_t$ and the privileged knowledge $S_t^p$, a contrastive objective is proposed. Simplifying such objective by removing negative sampling, the proposed method uses cosine similarity with only positive samples $z_t\sim f_\varphi(h_t)$, resulting in a two-stream architecture consisting of an online and a target network. The online network learns the latent encoding of privileged information from the history of local states via a time convolutional neural network (TCN), the target network learns a representation $\tilde{e}_t$ via an AutoEncoder of the privileged state. The cosine similarity between projections of both network outputs are used to evaluate $I(Z_t; S_t^p)$. At the same time, a regulizing loss term is used to ensure the information irrelevant to the privileged state information is compressed.  The authors demonstrate the performance of the proposed sim2real method on the DeepMind Control Suite benchmark and a real quadruped robot. Ablation studies over different HIB variants on the quadriped walking task are given. The learned latent representations are visualized through t-SNE which indicate that HIB is able to better recover the privileged information than previous policy distillation methods.
main_review: main_review: ### Strengths

The approach is technically sound and the authors clearly explain the theoretical background behind underlying historical information bottleneck and its application to policy generalization.

The experimental results are strong, particularly the real-robot experiments clearly indicate robust behaviors in the real world.
The ablation studies and investigation of the learned privileged state representation is convincing that the proposed approach achieves more robust policy generalization than previous works.
The supplemental video is well produced and demonstrates the experimental setups as well as detailed real-robot evaluations.

### Weaknesses

The motivation behind the two-stream architecture should be clarified (see more points in my first question for rebuttal), especially implementation details, such as why stop gradient operations are needed. The architecture appears close to the one in [54] but it would be helpful to have a more self-contained presentation of the entire approach and design choices.

The limitations should be explained more clearly. It seems that the generalization success of this work hinges on the closeness of the local state spaces in simulation and reality, and the realism of the simulated dynamics of these states (simulation states need to closely follow the same dynamics of the real world states), i.e. the sim2real gap already has to be sufficiently small for the policy to generalize to the real world.

It would be better if the authors clarified in the main paper in Sec. 6.4 that the only evaluated baseline (even though it could be the best performing baseline) on the real robot was DreamWaQ instead of claiming "the baselines completely fail".

### Small issues:
* Typo in line 90: "fraction" instead of "friction"
* Line 99: the term "oracle" policy is perhaps not appropriate since it does not have access to oracle states
* Variable $r_\max$ in Eq. (2) should be explained
* Line 316: I assume the authors mean limited "in" instead of limited "to" in this context?
questions: questions: Why is it necessary to project $\tilde{e}_t$ and use this projection to compute cosine similarity with the projection of $z_t$? Why can the autoencoder of the target network not directly learn a latent variable $\tilde{z}_t$ whose cosine similarity to $z_t$ is maximized, i.e. without the projection step (given that both $\tilde{e}_t$  and $z_t$ already have the same dimension)?

Since the authors only mention the lack of support for recovering multi-modal privileged information, I wonder if there is a more significant limitation that should at least be mentioned:
How valid is the assumption that the local state space from the simulated environment is close enough for the policy to perform well in the real world?  Is it possible to quantify the closeness between such state representations (sim and real) within the proposed method is robust to generalize to the real world?
limitations: limitations: 2
review_summary: review_summary: I tend to recommend the paper for acceptance. The approach is sound and the real robot results are strong. The paper is not a strong accept for me due to the remaining clarity issues and the potential limitation I raised in my review.
title: title: Review for Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
paper_summary: paper_summary: The paper proposes a novel method called Historical Information Bottleneck (HIB) to address the sim-to-real gap in reinforcement learning. It assumes during training time, privileged knowledge is available in simulation, and agents only have access to local states in the real world. The proposed HIB method uses the Information Bottleneck principle to distill privileged knowledge from historical information by maximizing the MI between the historical representation and the privileged state via contrastive learning. The paper presents theoretical analysis and empirical results demonstrating HIB's efficacy in both simulated and real-world environments.
main_review: main_review: **Strengths**
1. The proposed method is theoretically sound approach to tackling the sim-to-real gap in RL. The paper provides a rigorous theoretical analysis, demonstrating how the proposed HIB method reduces the value discrepancy between oracle and learned policies.
2. The extensive experiments on simulated and real-world tasks show that HIB outperforms state-of-the-art methods. The real-world deployment on a quadrupedal robot has impressive performance, achieving a 100% success rate in navigating challenging terrains.

**Weaknesses**
1. As mentioned in the limitation section, this paper deals with limited scope of privileged knowledge: The method's effectiveness is constrained when dealing with high-dimensional and complex space such as RGB images. This limitation could restrict the method's applicability to a broader range of real-world scenarios.
2. Currently the paper deals with stationary historical data. In highly dynamic environments where historical data patterns change frequently, this assumption might not hold, potentially degrading performance.
questions: questions: 1. Handling Multi-Modal Privileged Knowledge: As the authors have already pointed out in the limitation section, the method cannot handle multi-modal privileged knowledge such as RGB images. Are there any preliminary results or insights into this extension?
2. Impact of Historical Trajectory Length: How does the length of the historical trajectory impact the performance of the HIB method? Currently the method uses a fixed length. Is there an optimal length, and how sensitive is the method to variations in this parameter? More analysis on this will be appreciated
3. Noise and Uncertainty Handling: How does the HIB method handle noise and uncertainty in the historical data? Are there mechanisms to mitigate the impact of noisy or incomplete historical trajectories on the performance of the learned policy?
limitations: limitations: 3
review_summary: review_summary: I recommend weak accept of the paper based on my evaluations.
title: title: Review
paper_summary: paper_summary: This paper introduces the Historical Information Bottleneck (HIB), a method that treats the sim-to-real transfer as an information bottleneck problem. HIB learns a privileged knowledge representation from historical data, improving generalizability and reducing value discrepancies between oracle and learned policies. Empirical results show that HIB outperforms previous methods in simulation tasks. The paper also shows HIB with good performance in real-world tasks.
main_review: main_review: This paper introduces Historical Information Bottleneck (HIB), a new approach to make better use of privileged knowledge from simulations. It focuses on creating a compact representation of this knowledge to aid real-world decision-making by leveraging historical data that maximizes its connection to the privileged knowledge, reducing errors and improving performance.

Advantages:

1) Theoretical Analysis: It shows that minimizing privileged knowledge modeling errors leads to more effective policies.
2) Performance: HIB outperforms baseline methods, offering better generalization and robustness in both simulated and real-world tasks.

Issues:

1) For Figure 4, what happens if all baselines are trained for a longer period? Also, why does the vanilla RL method show such high variance? More seeds should be run to confirm whether the conclusions remain valid.
2) For Figure 1, the main training pipeline is unclear to readers who may not read the entire methods section. Consider revising the flow and style for better clarity, especially regarding gradient backpropagation and the different stages during training and deployment.
questions: questions: Some questions to address are:


1) Figure 4's Reproducibility: What are the results if all baselines are trained for a longer duration?  Running additional experiments with more seeds could help verify whether the observed conclusions are consistent and reliable.

2) Real-World Reward Function Definition: How is the reward function defined for real-world experiments?

3) Real-World Performance of RMA Baseline: What is the real-world performance of the RMA baseline? It's better to show how good/bad are others in the real world instead of showing HIB only.
limitations: limitations: 3
review_summary: review_summary: In summary, I am leaning towards accepting the paper though it has some minor issues. Because it provides a detailed analysis of a knowledge distillation method for RL. The two-stream design and information bottleneck loss effectively reduce performance discrepancies, as demonstrated in both simulation and real-world experiments.
title: title: novel and solid work
paper_summary: paper_summary: To address the limitation in existing policy distillation framework which is computational expensive, lack of theoretical guarantee, and do not fully use privileged information. This paper introduces the Historical Information Bottleneck (HIB) method for reducing the sim-to-real gap in reinforcement learning.
main_review: main_review: To address the limitation in existing policy distillation framework which is computational expensive, lack of theoretical guarantee, and do not fully use privileged information. This paper introduces the Historical Information Bottleneck (HIB) method for reducing the sim-to-real gap in reinforcement learning. By leveraging historical data, the method optimizes the representation of privileged information, aiming to enhance real-world decision-making. In my view, this paper seems to be novel and technically solid. However, I have no time to go through the proofs of theorem 1&2. 

While the paper presents a promising approach through the innovative use of the information bottleneck principle and historical data, there are several doubts about the technical parts:
1/ Around line 161-162, TV distance is convert to KL distance due to Pinsker's inequality. Reference is need.
2/ In equation 9, I cannot see how the second equality is derived. 
3/ In section 5.2, around line 180, another notation \psi is introduce to distinguished from \phi. however, this f(.) seems do the same job map h_t to z_t. It is confusing to me.
4/ I find Z_t and z_t, S_t and s_t sometimes are used interchangeably. It would be better make it consistent.

The proposed method is tested in both simulation and real world. In my view, the real-world test should be more quantitive and with more well-design scenarios. Overall the paper is well written. The whole paper is a bit mathematically dense. The punctuation in equation (12) should be period.
questions: questions: see in the review.
limitations: limitations: 3
review_summary: review_summary: see in the review