Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulation
Authors: Shuanghao Bai
Wanqi Zhou
Pengxiang Ding
Wei Zhao
Donglin Wang
Badong Chen
Venue: ICML 2025 poster (2025)
Keywords: Behavior Cloning
Robot Manipulation
Information Bottleneck
Primary area: applications->robotics
TL;DR: Integration of information theory with behavior cloning in robot manipulation

Abstract: Behavior Cloning (BC) is a widely adopted visual imitation learning method in robot manipulation. Current BC approaches often enhance generalization by leveraging large datasets and incorporating additional visual and textual modalities to capture more diverse information. However, these methods overlook whether the learned representations contain redundant information and lack a solid theoretical foundation to guide the learning process. To address these limitations, we adopt an information-theoretic perspective and introduce mutual information to quantify and mitigate redundancy in latent representations. Building on this, we incorporate the Information Bottleneck (IB) principle into BC, which extends the idea of reducing redundancy by providing a structured framework for compressing irrelevant information while preserving task-relevant features. This work presents the first comprehensive study on redundancy in latent representations across various methods, backbones, and experimental settings, while extending the generalizability of the IB to BC. Extensive experiments and analyses on the CortexBench and LIBERO benchmarks show consistent performance improvements with IB across various settings, underscoring the importance of reducing input data redundancy and highlighting its practical value for real-world applications.

Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulation
Shuanghao Bai
Wanqi Zhou'
Pengxiang Ding" 23
Wei Zhao
Donglin Wang?
Badong Chen
Abstract
Visual
Observation
Behavior Cloning (BC) is a widely adopted visual imitation learning method in robot manipulation. Current BC approaches often enhance generaliza- tion by leveraging large datasets and incorporating additional visual and textual modalities to capture more diverse information. However, these meth- ods overlook whether the learned representations contain redundant information and lack solid theoretical foundation to guide the learning pro- adopt an
0
(e.g-
((X,2)
((Z,))
Proprioceptive State
S
X
A
Input Representations
Latent Representations
Others
Action
(c.g. text)
Figure 1: Policy architecture of BC. Current BC methods (black arrows) do not impose restrictions on the latent rep- resentations Z, potentially allowing redundant information from the input representations X.
cess. To address these limitations, information-thenne thhoreetic perspective and introduce mutual information to quantify and mitigate re- dundancy in latent representations. Building on this, we incorporate the Information Bottleneck (IB) principle into BC, which extends the idea of reducing redundancy by providing structured framework for compressing irrelevant informa- tion while preserving task-relevan features. This work presents the first comprehensive study on redundancy in latent representations across var- ious methods, backbones, and experimental set- tings, while extending the generalizability the BC. Extensive experiments and analyses the ConcexBench and LIBERO benchmarks show consistent performance improvements with IB across various settings, underscoring the im- portance of reducing input data redundancy and highlighting its practical value for real-world ap- plications. Project Page: BC-IB Website.
from expert demonstrations. BC has been widely stud- ied in autonomous driving (Bain & Sammut, 1995; Torabi et al., 2018), robotics control (Argall et al., 2009) and game AI (Pearce & Zhu, 2022). In robot manipulation, BC has become foundational approach, enabling robots to repli- cate expert actions based on sensory inputs such as images or proprioception information like gripper states. To en- hance the generalization of robots, most BC methods focus incorporating large datasets of human or manipulation videos (Jang et al., 2022; Zhang et al., 2024; Cheang et al., 2024; Saxena et al., 2025; Cui et al., 2025), or integrating additional text and visual information (Jia et al., 2024; Wen et al., 2024a; Hu et al., 2024; Zhang et al., 2025b). While these methods have made significant progress in improv- ing generalization by leveraging more diverse information, they often neglect critical aspect: whether the learned representations contain significant redundant information.
1. Introduction
Why do we need to explore this? Firstly, the inherent chal- lenges of input data redundancy remain largely unexplored in BC for robot manipulation, despite their potential impact on policy performance and generalization. Secondly, most existing methods lack solid theoretical foundation guide the learning process. This raises a key question: how can we formally characterize and reduce redundancy in inputs or representations in theoretically grounded way?
Behavior Cloning (BC), one of the simplest and most widely used methods in Imitation Learning (IL), learns a map- ping from states to actions by training on state-action pairs
Inntittte of Artificial Intelligence and Roboties, Xi an Jiaotong University, China. *Westlake University *Zhejiang University. Correspondence Badong Chen <ehenbd @ mail.xjtu.cdu.. Donglin Wang <wangdongline @ westlake.edu.cn..
How to explore this? As illustrated in Figure 1, in BC, the inputs are typically encoded into individual representations and concatenated form the input representation X. This
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Behavior Cloning with Information Bottleneck
then processed through a feature fusion module produce the latent representation Z, which is subsequently decoded to predict the action A. The policy is optimized by mini- mizing the discrepancy between the predicted actions and the expert-provided actions. In information theory, mutual information between X and Z. denoted as [X, Z). mea- sures the amount of information gained about one random variable by knowing the other. In BC, if output Y can be well predicted by Z, reducing T T(,, Z) means continuously eliminating redundant information from X.
settings, indirectly highlighting the considerable redundancy present in current robot trajectory datasets. (3) We provide detailed theoretical analysis explaining why IB enhances the transferability of BC methods.
2. Related Work
Behavior Cloning in Robot Manipulation. Behavior Cloning (BC), first introduced by (Pomerleau, 1991). is a well-known Imitation Learning (IL) algorithm that learns policy by directly minimizing the diserepancy between the agent's actions and those of the expert in the demonstration data. To learn more generalizable representations, one class of visual representation learning methods pre-trains on large video datasets of robotics or humans, enabling rapid ap- plication of the pre-trained encoder to downstream robotic tasks. Notable examples include VC-1 (Majumdar et al., 2023), R3M (Nair et al., 2023), and Voltron (Karamcheti et al., 2023). Meanwhile, another line of research focuses on training on even more extensive and diverse datasets with larger models, such Internet-scale visual question answer- ing and robot trajectory data (Brohan et al., 2023), as well as a vast collection of Internet videos (Cheang et al., 2024). Additionally, some methods further enhance generalization by incorporating additional sources of information. These include inferring textual descriptions based on the robot's current state (Zawalski al., 2024), leveraging visual trajec- tories (Wen al., 2024a) and generated images (Tian et al., 2025; Hu et al., 2025; Zhang et al., 2025a), and integrating 3D visual information (Goyal et al., 2023). However, these methods have not deeply analyzed the redundancy in learned latent representations, and most also lack a solid theoretical foundation. Thus we extend the Information Bottleneck (IB) principle BC, addressing this fundamental gap.
Taking step further, an information thheoretccc approach that balances the trade-off between representation complexity and predictive power offers a natural framework to address the problem of latent representation redundancy and the lack of a solid theoretical foundation, namely information bottleneck (IB) principle (Tishby et al., 1999). IB regular- izes the representation Z by minimizing the mutual infor- mation XX, Z) between X and Z, while maximizing the mutual information I (22 A) between Y and A. The first term I [XX, Z) represents the compression of the representa- tion, where a smaller mutual information indicates a greater degree of compression and redundancy reduction, while [ZZ A) ensures predictive power is maintained.
Motivated by this information-thente thhoreetcc approach, we make the first attempt in this work to study the impact of latent representation redundancy in BC for robot manipulation and extend the IB method to this context, where redundancy in latent representations is quantified by I(X, Z). We conduct extensive experiments in various settings and analyses to val- idate its effectiveness, highlighting the benefits of reducing redundancy to enhance generalization in robotic tasks. Addi- tionally. we provide detailed theoretical analyses, including generalization error bounds, to validate its effectiveness.
Information Bottleneck in Robotics. The Information Bot- tleneck (IB) principle was first proposed in (Tishby et al., 1999) within the context of information theory. Since then, it has been widely applied in deep learning and various down- stream tasks to balance the trade-off between representation accuracy and complexity, including classification (Federici et al., 2019), segmentation (Bardera et al., 2009; Lee et al., 2021), and generative tasks (Jeon et al., 2021). In robotics learning, IB has found notable applications in reinforcement learning, where some works maximize the mutual informa- tion between the representation and the dynamics or value function, while restricting the information to encourage the encoder to extract only task-relevant features (Kim et al., 2019; Bai et al., 2021: He et al., 2024). In imitation learn- ing, it has been applied to alleviate the copycat problem from observation histories (Wen et al., 2020). Different from prior works, we introduce IB into Behavior Cloning to explore and empirically validate the redundancy in latent representations in robotics. Additionally, we demonstrate its effectiveness through detailed theoretical analyses.
How to apply IB to the BC architectures, and what are its potential applications? To ensure the generality of our findings. we categorize BC architectures based on their feature fusion methods into two types: spatial fusion and temporal fusion. This allows us to identify the applicable scenarios for each fusion method, and by incorporating IB, we uncover a series of interesting findings. Furthermore, our experiments reveal that regardless of the pre-traing pre-training stage, the final fine-tuning phase, or the size of the dataset, incorporating IB by reducing redundancy enables the model to learn more robust features and improve performance, suggesting its potential applicability in these scenarios.
Our contributions are three-fold. (I) We extend the IB to BC and provide a comprehensive study on the impact of latent representation redundancy in BC for robot manip- ulation. (2) We empirically demonstrate that minimizing redundancy in latent representations helps existing BC algo- rithms significantly improve generalization performance the Cortexbench and LIBERO benchmarks across various
2

---

Behavior Cloning with Information Bottleneck
Action
Action
I Language
King
Instruction
[Emb.]
Policy Head
Policy Head
Visual Observation
Enc,
Latent Representation
[Emba)
Temporal Fusion Module
Spatial Fusion Module
Pruprioceptive State
Bac,
X
[Emby]
[Embi]
(Fmbo) [kmbal
[CLS]
[Emby
[Embal
[Emby]
Feature Extraction
(b) Spatial Fusion
(c) Temporal Fusion
Figure 2: Model architectures used in this study. Based on feature fusion methods, we categorize the BC methods in robot manipulation into two types: spatial fusion and temporal fusion. After extracting features from each modality a), spatial fusion b) extracts spatial features at a given time step or concatenates features across multiple time steps using encoders like MLPs or CNNs. Temporal Fusion с) fuses input features by modeling dynamic relationships and dependencies between time steps using RNNs or Temporal Transformers. The latent representations are then decoded into actions via the policy head.
3. Preliminary
3.2. Mutual Information Neural Estimation
3.1. Problem Setting of Behavior Cloning
Estimating mutual information between variables directly is challenging, thus we use Mutual Information Neural Estima- tion (MINE) (Belghazi et al., 2018) to estimate it. MINE based on neural networks, which can efficiently handle high- dimensional, continuous, discrete, and hybrid data types without requiring assumptions about the underlying distri- butions. MINE estimates mutual information by training a classifier to differentiate between samples from the joint dis- tribution Pxz z and the product of the marginal distributions Px & Pz of the random variables and Z. MINE uses a lower bound for mutual information based on the Donsker- Varadhan representation (Donsker & Varadhan, 1983) of the Mallback-Leibbe (KL) divergence:
BC can be formulated the Markov Decision Process (MDP) framework (Torabi et al., 2018), which is often de- fined without an explicitly specified reward function, to model sequential action generation problems. The concept of rewards is replaced with supervised learning, and the agent learns by mimicking expert actions. Formally, in robot manipulation, the state at each timestep consists of visual observations o,, the robot's proprioceptive state st, and optionally a language instruction 1. Let (or, st, 1) represent the overall state. The policy maps sequence of states to an action: å = T T where t indicates the length the state history. For simplicity, we set r 1. The optimization process can be formulated as:
I(X; Z) := DKL. (Pxz(xx @ Pz) 2 (X;
(3)
== Exxx [To =)] log E
[
π argmin, E( (TT (e)]:
(1)
where To : X R is discriminator function modeled by neural network with parameters 9. We empirically sample from Pxz. and for Px & Pz, we shuffle the samples from the joint distribution along the batch axis.
where D. is expert trajectory dataset and a, is action labels. In vanilla BC, L typically represents the mean squared error (MSE) loss function for continuous action spaces, or cross- entropy (CE) loss for discrete action spaces. In this study, we adopt the continuous action spaces with MSE loss:
4. Pipeline of BC with IB
4.1. Model Architecture
LBc B [| (ax)
a, a.13]
Before introducing IB, we first define its input and latent representations. Traditional IB methods (Amjad & Geiger, 2019; Pacelli & Majumdar, 2020; Wan et al., 2021) typically apply the bottleneck to a single modality (e.g., images or states) and their corresponding latent features, following the information flow O + Z + A. contrast, BC for robot manipulation is more complex than earlier control or single-modal tasks, as requires models to process di-
Building this vanilla BC loss, some methods also intro- duce alignment loss (Jang et al., 2022; Ma et al., 2024) and reconstruction loss (Radosavovic et al., 2023; Karamcheti et al., 2023). However, in this study, to more clearly illus- trate the relationship with representation redundancy, we focus solely on the vanilla BC loss.
3

---

Behavior Cloning with Information Bottleneck
verse, multimodal data. This data not only includes RGB images but may also incorporate the robot's proprioceptive state, language instructions, and other modalities, making effective feature fusion essential. Strietly adhering to the conventional IB paradigm would involve constraining each input modality and its corresponding features separately, re- sulting in pipeline that is inelegant, difficult to scale, overly complex, and unable to capture cross-modal associations. Furthermore, previous work has shown that proprioceptive states can lead to overfitting (Wang et al., 2024).
and temporal information, is then passed through a policy head generate actions. Policy heads typically include MLPs, GMMs, diffusion policy (Chi et al., 2023; Reuss et al., 2024), and other designs (Gong et al., 2024). We choose an MI. for clarity and ease of empirical analysis.
4.2. Behavior Cloning with Information Bottleneck
The Information Bottleneck (IB) principle is an information- theoretic approach aimed at extracting the most relevant information from an input variable X with respect to an output variable, i.e., action A. The central idea is to find a compressed representation Z of X that retains the relevant information needed to predict A, while discarding irrelevant parts of X that do not contribute to predicting A. The relevant information is quantified as the mutual information [X;; A), and the optimal representation Z is the minimal sufficient statistic of X with respect to A. In practice, this can be achieved by minimizing a Lagrangian that balances the trade-off between retaining predictive information and compressing the input, which can be formulated as:
As a result, we do not treat image or other modalities sep- arately as inputs to IB, as done in previous studies. In- stead, we concatenate features extracted from all modalities through respective feature extractors as our input X, i.e.,
Zy = concat(Enc,,e)) Enc, (s,), (n),
(4)
where Enc..) denotes the feature extractor of each modality. This results in the information flow O + X + Z + A, and brings several advantages: (1) It enables unified redundancy reduction across all modalities. (2) In practice, encoders are often frozen, and this approach is more effective under such conditions. (3) It scales better to broader range of robotic algorithms. Then, regarding how to process the input X, or how to fuse information from multiple modalities into latent representations Z, we categorize BC methods in robot manipulation into two types based on their feature fusion strategies: spatial fusion and temporal fusion.
== (1x 2) - ((Z;
(5)
where 33 is the Lagrange multiplier that balances the trade- off between the compression ability and the predictive power. Thus Equation (2) can be modified as:
CBC-IB = ((sr,ac) [3I 2() + [|T(22)) - ael?
As illustrated in Figure 2 (b), spatial fusion involves ex- tracting spatial features from data at a given time step or concatenating features across multiple time steps along the feature dimensions. This approach does not explicitly dif- ferentiate between time steps but instead processes the ag- gregated features whole, emphasizing the modeling of inter-feature relationships. The spatial fusion module can be implemented using Multi-Layer Perceptrons (MLPs), Con- volutional Neural Networks (CNNs), Spatial Transformers, or even simple concatenation operations. These methods are primarily designed to learn highly generalizable visual encoders by leveraging large-scale human video datasets. The pretrained encoders are then fine-tuned for downstream robotic tasks (Majumdar et al., 2023; Zeng al., 2024).
(6)
where 2, F(x:) and F()) denotes the fusion module.
When B < O, the information from X to Z increases; when B = 0, the method reduces to vanilla BC; and when ,3 > O, the information from X to Z decreases.
4.3. Theoretical Analysis
We provide a theoretical analysis of our BC-IB objective in Equation (5). We adapt Theorem 4. 1 and Theorem 4.2 to reveal that the generalization error is upper-bounded by the mutual information between the input O and the latent rep- resentation Z, following the information flow O - 7 + A. Minimizing this mutual information tightens the bound and improves generalization. However, when O is diverse and multimodal, directly minimizing the mutual information between each modality and its corresponding Z is computa- tionally intractable and unnecessarily complex. To address this, we extract and concatenate features from all modalities into an intermediate feature X, obtain Z via a fusion net- work f. and minimize the mutual information between X and Z instead. To validate the compatibility of this paradigm with the original theorems, we present Theorem 4.3. The theorem establishes that, even if we optimize I(X; Z) by applying the bottleneck at an intermediate feature level X, as long as X preserves the essential structure of the original
On the other hand, as illustrated in Figure 2 (c), temporal fusion integrates input features by capturing dynamic rela- tionships and dependencies across time steps. This enables the modeling of both long-term and short-term temporal dynamics in sequential data. Temporal fusion modules can be implemented using Recurrent Neural Networks (RNNs), Long Thort-Term Memory networks (LSTMs), or Tempo- ral Transformers. These methods are commonly incorpo- rated into approaches that utilize Transformer-based back- bones (Wu et al., 2024; Li et al., 2024b; Liu et al., 2024).
The latent representation Z, which integrates both spatial
4

---

Behavior Cloning with Information Bottleneck
input O, we are effectively controlling I(0;)) with the difference bounded by a small constant 8.
Lel Je Epgg (o,))[](=; Assume the mutual information gap satisfies the following condition: for any we have
( Egy oe (o.x.z) (=; 0=)] J
Theorem 4.1. Generalization Bound Adapted
from (Shwartz-ZN, Ziv et al., 2019). Let S = {(x,, at)}i=1 denote the training data sampled from the same distribution as the random variable pair (X, A). Given the policy π trained on S. the generalization error is given by:
8
[o, 2:) T(o, 2; 2;000
(12)
3
Then, the gap between the two optimizations is bounded as:
(SS) = (xxx 1 Σ (m(()))..). (7)
|J - J S 6.
(13)
t=1
Using the Probably Approximately Correct (PAC) bound framework and the Asymptotic Equipartition Property (AEP) (Cover, 1999), with probability at least 1 8, the following upper bound on the generalization error holds:
The detailed proof can be found in Appendix A.
5. Experiments
21(X; Z) + log 22
5.1. Embodied Evaluation
(SS) S
2n
Simulation Benchmarks. We mainly evaluate BC with IB across two benchmarks, CortexBench (Majumdar et al., 2023) and LIBERO (Liu et al., 2024). CortexBench is single-task benchmark. For validation, imitation learning-related simulators, encompassing total of 14 tasks: Adroit (2 tasks) (Rajeswaran et al., 2018), Meta- World (5 tasks) (Yu et al., 2020), DMControl (5 tasks) (Tassa et al., 2018), and TriFinger (2 tasks) (Wuthrich et al., 2021). During evaluation, the number of validation trajectories is set 25, 10, 25, and 25, respectively. LIBERO is a manguage-condtioons multi-task benchmark. For evaluation, we select four suites: LIBERO-God (10 tasks), LIBERO- Object (10 tasks), LIBERO-Spare -Spaial (10 tasks), and LIBERO- Long (10 tasks), each focusing on the controlled transfer of knowledge related to task goals, objects, spatial information, and long-horizon tasks, respectively. During evaluation, the number of validation trajectories is 20.
where I(X; Z) represents the mutual information between the input X and the intermediate representation Z, and ō is the confidence level. Details of proof can be seen in Appendix A of (Shwartz-Ziv el al., 2019).
selected four
Theorem 4.2. Generalization Bound Adapted from (Kawaguchi et al., 2023). Let S = as))?= denote the training data sampled from the same distribution as the random variable pair (X, A). The generalization error is approximately bounded by:
A(S) x I(x:Z A) + 1(05:5)
n
where d3 is the encoder mapping the input X the inter- mediate representation Z. This bound indicates that the generalization error is:
Positively correlated with I(X; Z| A), which captures mutual information between the inpul X and the latent representation Z. conditioned on the actions A. This term reflects that the IB compresses X into Z while preserving the relevant information for predicting A.
Real-world Evaluation. As shown in Figure 4, our real- world experiments use 6-DOF UR5 arm equipped with a Robotiq 2F-85 gripper and a RealSense L515 base camera for RGB image capture. Following the simulation setup, we evaluate both a single-task setting and a more challenging language-conditiones conditioned multi-task setting. The latter intro- duces increased distractor objects, randomized object po- sitions, and unseen instances during evaluation to assess generalization. We design two tabletop manipulation tasks: Pick, where the robot lifts an object from the table, and Put (Pick and Place), where the robot picks up an object and places it into a bowl. Demonstrations are collected using 3D mouse with only the base camera. In the single- task setting, we use 25 demonstrations for Pick and 50 for Pick-and-Ple In the multi-task setting, we collect 800 demonstrations in total, with 200 per task. During evalua- tion, each task is tested over 10 trajectories.
Positively correlated with T( S), which reflects the information content of the representation ó for the given dataset S.
Theorem 4.3. Optimization Gap under Different Inpul Com- pression. Let o > -> 2 form Markov chain, where is transformed into by network, f, and x is further trans- formed into z by network c. Let do = foo optimization problems:
oo 6. Define two
log Po(2(() 1 P²(2)
(0,)55) = =arg min Ep, (o,x,)) 0,00
B [(:00
(10)
Po. (zl0)
1 (z;
Baselines. In CortexBench, we evaluate four visual imitation learning models: R3M (Nair et al., 2023), Voltron (Karamcheti et al., 2023), VC-I (Majumdar et al.,
(0", (%) = arg min Ep, (0,z)
log
0,00
(2)
3
(11)
5

---

Behavior Cloning with Information Bottleneck
2023), and MPI (Zeng al., 2024). Following the origi- nal papers, we use pre-trained models with frozen image encoders for downstream tasks. Additionally, we introduce two full fine-tuning baselines by replacing the encoders with partially uninitialized ResNet-18 (He et al., 2016) and ViT-S (Dosovitskiy, 2021), denoted as ResNet and ViT, respectively. All methods use the two fusion techniques from Section 4.1: an MLP for spatial fusion and Temporal Transformer for temporal fusion. In LIBERO, we imple- ment four vision-language policy networks. One of them uses a spatial fusion approach, which employs ResNet as the image encoder and an MLP as the fusion module, re- ferred to as BC-MLP. The other three use temporal fusion. Following the original paper, we rename them based on the combination of the image encoder and fusion module:
Resper-Temporal
0.6
ResNet Spatial
0.5
0.010
BC Loss
0.4
-0.0056
0.005
0.3
X10
0.0005
0.2
0.000
40
42
44
08
68
50
0.1
0.0
0
10
20
30
40
50
Epochs
(a) BC loss comparison for the two fusion methods
100
100
ResNet-Spatial ResNet-Temmoral
VC1 2pa-Spatill
VC1-Temporal
BC-RNN, BC-Transformer, and BC-VILT (Liu et al., 2024).
81.07
24.27
74.03
80
37.60
Success Reate (%)
The policy head for all methods is fixed an MLP. In real-
world evaluation, we adopt VC-1 (Majumdar et al., 2023)
35.00
60
40.30
53,82
for the single-task setting and CogAct (Li et al., 2024a) for
39.91
40-
33.00
35.73
the segguage-conditionee multi-task setting. Notably, all baselines with IB are referred to as BC+IB.
20
18.09
Implementation. In CortexBench, for four partial fine- tuning methods, we train for 100 epochs on each task using the Adam optimizer with learning rate of Ic-3, batch size of 512, and weight decay of le-4, with learning rate decay applied using a cosine annealing schedule. For two full fine-tuning methods, we train for 50 epochs with a learning rate of le-4 and a batch size of 256. In LIBERO, we train for 50 epochs using the AdamW optimizer with a learning rate of le-4 and batch size of 64, decayed using a cosine annealing schedule. In real-world evaluation, we train VC-1 for 200 epochs using the Adam optimizer with learning rate of 1e-3 and batch size of 512. CogAct is trained for 8k steps with the Adam optimizer, using a learning rate of 2e- and a batch size of 128. For BC+IB methods, the model used in MINE consists of two-layer MLP, with a learning rate of 1c-5. The Lagrange multiplier in Equation (6) ranges from 1e-4 to 1e-2 in this work.
metaworld
dmcontrol
metaworld
dmcontrol
(b) Success rate comparison for the two fusion methods
Figure 3: (a) BC loss variation for ResNet in spatial and temporal fusion methods on the bin-picking task of the Meta- World. (b) Averaged success rates of ResNet and VCI in spatial and temporal fusion methods across the WerldWorld and DMControl.
CortexBench, with the results shown in Figure 3.
Finding I: For simple single-task scenarios, spatial fusion is more efficient and effective than temporal fusion. As shown in Figure 3 (b), the performance of methods with temporal fusion drops significantly. From Figure 3 (a), this can be attributed to the slower loss reduction in methods using temporal fusion, which results in higher loss at the same training epoch. Therefore, we focus exclusively on presenting the results for methods employing spatial fusion.
Model Selection. For the single-task benchmark Cor- texBench, we test the model every 5 or 10 epochs and select the model with the highest success rate. For the multi-task benchmark LIBERO, we select the model from the final epoch. For real-world evaluation, we follow the correspond- ing strategy for each setting as described above.
Results. We next report the performance, i.e., success rate, of the baselines and baselines with IB on the single-task benchmark CortexBench in Table with a full-shot setting. Based on results, we derive the following findings.
The appendix provides detailed descriptions of each bench- mark (Appendix 1) all baselines (Appendix B.2), imple- mentation details (Appendix B.3), and the rationale behind the model selection (Appendix B.4).
Finding 2:
Whether using full fine-tuning or partial fine- tuning, all vanilla BC methods with different visual back- bones incorporating IB outperform their vanilla counterparts across the board. In some benchmarks, the improvements are substantial. For example, ResNet with IB achieves a 10.01% % improvement on DMControl, and VC-1 with IB shows a 4.80% improvement on Meta- World. In Ap- pendix C.I, we report the success rate for each task, where
5.2. Performance on Cortexbench
The Selection of Fusion Method. We first evaluate the effectiveness of the two fusion methods in the baselines on
6

---

Behavior Cloning with Information Bottleneck
Table 1: Performance in spatial fusion on single-task benchmark CortexBench. We evaluated 14 tasks across benchmarks
using 3 random seeds and reported the average success rate along with the standard deviation. denotes the use of only a small portion of the original model for feature extraction. The best performance is highlighted in bold.
Method
Image Encoder
Adroit
Meta- World
Meta-World
DMControl
TriFinger
Avg
Full Fine-tuning
ResNet (He et al., 2016)
66.000+.229
81.07
74.93+6..1
71,59
73.40
ResNet
ResNet+IB
72.00
83.20
84.9443354
1.730
78.11
ViT (Dosovitskiy. 2021) ViT+IB
35.33-13.06
31.73:1:1667
10,41:1.21
55.57
33.26
VII
37.33
36.00=66.7
12.53+2.17
55.93+2216
35.45
Partial Fine-tuning
R3M (Nair et al., 2023)
25.33=4643
53,07=1.67
40.31
59,87=0078
44,65
VIT-S
R3M+IB
27.33
54.1312.44
41.74.15.54
60.63
45.96
Voltron (Karamcheti et al., 2023) Voltron+IB
18.67#6.11
72.53+1:222
25.35
74.21+2.61
47.69
ViT-S
51.33
74.40++349
33.16:16.70
75.12
51.00
VC-1 (Majumdar et al., 2023)
24.67+47.02
77.60+2288
53.82-+5.03
72.05
57.04
ViT-B
VC-I+IB
26.00:t9.17
82.40
14.93
73.80
59.28
MPI (Zeng et al., 2024) MPI+IB
34.67
66.40+t2.12
59,45
61.91 63.34
55.61
ViT-S
36.67:6611
69.33
61.41+1311
57.69
Table 2: Performance on language-condition multi-task benchmark LIBERO. We evaluated 40 tasks of 4 suites using 3 random seeds and reported the average success rate along with the standard deviation. S-Trans. denotes Spatial Transformer and T-Trans. denotes Temporal Transformer. The best performance is bolded.
Image
Fuse
LIBERO-
LIBERO- Object
LIBERO-
LIBERO- Long
Method
Avg
Encoder
Module
Goal
Spatial
BC-MLP
16.50.13.7
19.00112.22
29.33
2.33
16.79
ResNet
MLP
BC-MLP+IB
27.67.1200
31,50 10.83
41.00
2.67
25.71
BC-RNN
15.17:10.91
13.93
30.67
2.33
15.38
ResNet
RNN
BC-RNN+IB
26.00
17.67 1.777
35.17.19.55
3.00
20.46
BC-Trans. BC-Trans.+IB
17.83++0.42
4.88
08.00:+11.00
15.83++2.52
48.37
ResNet
T-Trans.
74.1745.75
45.67 14.31
10.20
18.00
52.59
VCLV
76.17433.01
43.00
67.17:42.25
6.50+0087
48.21
S-Trans.
T-Trans.
BC+VILT+IB
83.83
52.00-:3.04
70.67:42.52
8.67
53.79
significant improvements can be observed in certain tasks.
5.3. Performance on LIBERO
Finding 3: Finding 2 implicitly suggests that the latent rep- resentation derived from input X is redundant. Therefore, compressing information from input essential, which can further enhance performance.
We report the performance on the multi-task bench- mark LIBERO with full-shot setting in Table 2.
Results.
Finding 6: For more complex language-conditioned multi- task scenarios, all baselines with different backbones incor- porating IB consistently show performance improvements across all LIBERO benchmarks. For example, BC-VILT achieves large gains of 7.66% and 9.00% LIBERO-Goo and LIBERO-Object, respectively, while BC-RNN shows sigmificant improvement of 10.83% on LIBERO Goal. IB proves to be more effective in more complex environments and settings. We attribute this to the difference task com- plexity: ContexBench, the history length is 3, while in LIBERO. it is 10. with LIBERO being multi-task bench- mark and CortexBench being single-task benchmark. The increased data complexity (task quantity and input informa-
Finding 4: In some benchmarks, particularly Trifinger, the improvement is minimal. We attribute this to the benchmark itself containing very limited redundancy in the visual input, which consists primarily of the robot arm and single object.
Finding 5: For simple single-task downstream tasks, full fine-tuning of a simple, uninitialized model (ResNet) is sufficient and may even outperform a pre-trained larger model. However, the latter is more efficient for faster fine- tuning and deployment, and proves to be more effective for more complex tasks (Burns et al., 2023).

---

Behavior Cloning with Information Bottleneck
Base Camera
Time
Tosk: Put corn into red bowl
(a) Illustration the Real-world Experimental Setup and an Example Task
100
100
80
20
Pick red
Put red bowl
Put into blue bowl Put red bowl corn into green bowl*
Put wood into blue bowl
Put wood block red bow!
Put wood block into green bow/
Air
(b) Success Rate in Single-task Setting
(c) Success Rate in Landuage-conddtioned Multi-task Setting
Figure 4: Real-world robot experiments conducted on a tabletop setup with two settings. (a) Left: the experimental setup. (a) Right: an example of predicted trajectories alongside policy execution. (b) and (c): quantitative evaluation results across two settings, where blue denotes the vanilla BC method and red denotes the method with IB. denotes the unseen tasks.
tion) suggests higher level of data redundancy, making IB even more effective.
Finding We observe that in complex multi-task scenarios with more intricate inputs, such as a greater number of input modalities and extended historical information, using the Temporal Transformer in temporal fusion proves to be more effective than both spatial fusion and RNN-based temporal fusion. The evidence lies in the fact that the average success rates of BC- Ce-Trnssformrr and BC- -C-VILT are over 30% higher than those BC-MLP and BC-RNN. This is likely because Temporal Transformers excel in handling long-range interac- tions and capturing dynamic dependencies across time steps, where RNNs and spatial fusion methods may struggle. This finding, together with Finding underscores the specific scenarios in which each fusion method is most applicable.
Success Rate (%)
75
LIBERO-Go
LIBERO Object
LIBERO- Spatill
0.0
0.2
0.6
1.0
B
Figure 5: Effect of the Lagrange multiplier / in BC- VILT+IB across three suites LIBERO. When 3=0, the method reduces to vanilla BC-VILT.
Finding 8: IB is particularly effective for tasks requiring diverse feature extraction, such as distinguishing distinct task objectives or differentiating between various objects, as in LIBERO-Goal and LIBERO-Object. By filtering out ir- relevant information, IB facilitates better generalization and more compact representations. However, the impact is less pronounced in spatial tasks such as LIBERO-S LIBERO-Spatial, Spatial, which rely heavily on structural information that can be disrupted by excessive compression. For long-hone morghooizon tasks, the main performance bottleneck lies in the lightweight baseline mod- els, whose limited capacity restricts their effectiveness on more complex tasks such as LIBERO-Long.
cases. In the single-task setting, VCI+IB significantly outperforms VCI in both the pick and put tasks. In the more challenging language-conditing CogAct+IB consistently outperforms CogAct across most tasks, including unseen object-bowl combinations, demon- strating enhanced generalization capabilities. These results suggest that reducing redundancy in latent representations leads to more robust grasping and more reliable overall execution in real-world scenarios.
multi-task setting,
5.5. More Analysis
5.4. Performance on Real World Experiments
Effect of the Lagrange multiplier /3 of Equation (6). This experiment evaluates how incorporating IB enhances perfor- mance. Since the MINE model's parameters are fixed, the key difference between BC+IB and BC lies in the param- eter 8, which balances compression and predictive power.
As shown in Figure (b) and (c), incorporating IB consis- tently improves success rates across both single-task and language-conditioned multi-task real-world settings in most
8

---

Behavior Cloning with Information Bottleneck
or
sr
1(X, Z) BC
2(X,2) 8C+18
LIBERO-Goal
LIBERO-Object
LIBERO-Spatial
LIBERO-Long
60
80
3.0
10
2.5
85
55
75
2.5
2.0
80
2.0
50
70
1.5
75
1.5
45
1.0
70
1.0
1.5
65
0.5
40
0.0
10
20
30 Epochs
40
50
60
0.0
0
10
20
30
40
50
35
0
10
20
30
40
50
1.0
10
20
30
40
50
Epochs
Epochs
Epochs
Figure 6: Comparison of vanilla BC and BC+IB the LIBERO benchmark in terms of success rate (sr) and mutual
information. BC-VILT is denoted as BC. BC+IB consistently achieves lower I [(X, 2) and higher success rates.
6. Limitations and Discussion
70
DC- BC-VILT-IB
53.3
60
58.8
While our work provides extensive experimental validation of the effectiveness of IB and the necessity of input redun- dancy reduction in robotics representation learning, several limitations remain. First, we do not comprehensively as- sess the scalability of the IB approach in large-scale robotic models. Most of experiments atively lightweight architectures, and only the real-world experiments employ CogAct. We have not systematically evaluated more complex models such as vision-language action (VLA) architectures (Brohan et al., 2023; Song et al., 2025; Zhao et al., 2025) due to the high computational and time requirements. Second, we not explore alternative policy head designs, including transformer-bast VLA models (Kim et al., 2024; Ding et al., 2024) that re- move the explicit policy module and instead treat actions as text tokens. These architectures are left for future in- vestigation. Third, although our method is evaluated on various benchmarks, its robustness to domain shifts, includ- ing changes environment and task configurations, remains insufficiently studied. We hope this work inspires future research and advances the development of these methods.
Success Rate (16)
50
40
38.5
34.5
30
conducted using rel-
20
10
3.5
UBERO-Goal
LIBERO IBERO-Objert
LIBERD Spatial
LIBERO Long
Average
Figure 7: Comparison of the success rates of BC-VILT+IB trained with 10 demonstrations against the vanilla BC-VILT across four LIBERO suites.
heads or
For LIBERO experiments, is explored within le-4, le-3, 5e-3, le-2. As shown in Figure 5, IB improves performance within a specific ,9 range, with peak observed at an unde- termined value. However, across all experiments, 3 around lc-4 consistently yields stable improvements.
Effect of the Number of Demonstrations. We evaluate IB's effectiveness in few-shot settings, as few-shot learn- ing is crucial for fine-tuning on domain-specific tasks in real-world applications. As shown in Figure 7, IB consis- tently improves performance even with limited data across multiple suites in LIBERO, highlighting its effectiveness in real-world scenarios where data is scarce. This further underscores the potential of IB improving model general- ization in practical settings.
7. Conclusion
In this study, we investigated the redundancy in latent repre- sentations for behavior cloning in robot manipulation and introduced the Information Bottleneck (IB) principle to mit- igate this issue. By incorporating IB, we aimed to filter out redundant information in latent representations while preserving task-relevant features. Extensive experiments across various visual representation learning methods on CorcexBench and LIBERO benchmark revealed insightful findings and demonstrated that IB consistently improves per- formance across diverse tasks and architectures. We hope this work will inspire future research to further integrate coormation-theoretii principles into robotics, not only as optimization tool, but also as framework for theoretical understanding and model design.
Visualizations of I(X, Z). As shown in Figure BC+IB achieves a larger reduction in T(X, Z) compared to vanilla BC, leading to improved performance and validating the effectiveness of IB. For example, in LIBERO Goal, IB re- duces [(X, Z) to one-quarter of its original value and yields a 7.7% increase in success rate.

---

Behavior Cloning with Information Bottleneck
Acknowledgements
Cheang, C.-L.,, Chen, G., Jing, Y., Kong, T., Li, H., Li, Y., Liu, Y., Wu, H., Xu, J., Yang, Y., et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06155 2024.
This work was supported by the National Natural Science Foundation of China (Grant No. U2IA20485), the National Science and Technology Innovation 2030 Major Project (Grant No. 20222D0208800), the NSFC General Program (Grant No. 62176215, 62436005), and the Fundamental Research Funds for Xi'an an Jiaotong University (Grant No. xzy022024012).
Chi, C., Xu, Z.., Feng. S., Cousineau, E.,, Du, Y., Burchfiel, B., Tedrake, R., and Song. S. Diffusion policy: Visuomo- tor policy learning via action diffusion. The International Journal of Robolics Research, pp. 02783649241273666 2023.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning in Robotics. There many potential societal consequences of our work, none of which we feel
Cover, T. M. Elements of information theory. John Wiley & Sons, 1999.
Cui, C., Ding, P., Song, W., Bai, S., Tong. X., Ge, Z., Suo, R., Zhou, W., Liu, Y., Jia, B., et al. Openhelix: A short survey, empirical analysis, and open-source dual- system vla model for robotic manipulation. arXiv preprint arXiv:2505. 0e0222555.039... 2025.
must be specifically highlighted here.
References
Amjad, R. A. and Geiger, B. C. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analy- sis and machine intelligence, 42(9):2225 2239, 2019.
Deng, J., Dong, W., Socher, R., L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and patlern recognition, pp. 248 245.,55. leee, 2009.
Argall, B. D., Chernova, S., Veloso, M., and Browning, B. survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483, 2009.
Ding, P., Zhao, H., Zhang, W., Song, W., Zhang, M., Huang. S., Yang, N., and Wang, D. Quar-vla: Vision-language- action model for quadruped robots. In European Confer- ence Computer Vision pp. 352-367. Springer, 2024.
Bai, C., Wang, L., Han, L.,, Garg, A., Hao, Liu, P., and Wang, Z. Dynamic bottleneck for robust self-supervised exploration. Advances in Neural Information Processing Systems, 34:17007-17020, 2021.
Donsker, M. D. and Varadhan, S. S. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications pure and applied mathematics, 36 (2):183-212, 1983.
Bain, M. and Sammut, C. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition scale. In International Conference on Learning Representations, 2021.
Bardera, A., Rigau, J., Boada, I., Feixas, M., and Sbert, M. Image segmentation using information bottleneck method. IEEE Transactions on Image Processing, 18(7): 1601-1612, 2009.
Federici, M., Dutta, A., Forré, P., Kushman, N., and Akata, Z. Learning robust representations via multi-view in- formation bottleneck. In International Conference on Learning Representations, 2019.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Ben- gio, Y., Courville, A., and Hjelm, D. Mutual information neural estimation. In International conference on ma- chine learning, pp. 531-540. PMLR, 2018.
Gong, Z., Ding, P., Lyu. S., Huang, S., Sun, M., Zhao, W., Fan, Z.,, and Wang, D. Carp: Visuomotor policy learn- ing via coarse to- to-fie autoregressive prediction. arXiv preprint urXiv:2412.06782 2024.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-aeoe web knowledge to robotic control. In Conference on Robot Learning, 2023.
models transfer
Goyal, A., Xu. J., Guo, Y., Blukis, V., Chao, Y.-.., and Fox, D. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pp. 694 710. PMLR, 2023.
Burns, K., Witzel, Z., Hamid, J. I., Yu, T., Finn, C., and Hausman, K. What makes pre-trained visual representa- tions successful for robust manipulation? arXiv preprint arXiv:23/2. 2023.
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. Ego4d: Around the world in 3,000 hours of
10

---

Behavior Cloning with Information Bottleneck
egocentric video. In Proceedings of the IEEF/CVF Con- ference Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E.,, Lam, G., San- keti, P., et al. Openvla: An open-source vision-language- action model. arXiv preprint arXiv:2406.09246 2024.
He, H., Wu, P., Bai, C., Lai, H., Wang, L.., Pan, L.,, Hu, X., and Zhang, W. Bridging the sim-to-real gap from the information bottleneck perspective. In 8th Annual Conference on Robot Learning, 2024.
Kim, Y., Nam, W. Kim, H., Kim, J.-H., and Kim, G. Curiosity-bottleneck. Exploration by distilling task- specific novelty. In International conference on machine learning, pp. 3379-3388, PMLR, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Lee, J., Choi, J., Mok, J., and Yoon, S. Reducing informa- tion bottleneck for weakly supervised semantic segmenta- tion. Advances in neural information processing systems, 34:27408-2747 2021.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Li, Q. Liang, Y., Wang, Z., Luo, L., Chen, X., Liao, M., Wei, F., Deng, Y., Xu, S., Zhang, Y., al. Cogact: A foundational vision- iitoo-llnguage-attin ing cognition and action in robotic manipulation. arXiv preprini arXiv:241 1. 19650, 2024a.
model for synergiz-
Hu, Y., Guo, Y., Wang, P., Chen, X., Wang, Y.-.., Zhang, J., Sreenath, K., Lu, C., and Chen, J. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412.148003 2024.
Li, X., Liu, M., Zhang, H., Yu, C., Xu, J., Wu, H., Cheang, C., Jing, Y., Zhang, W., Liu, H., et al. Vision-language foundation models as effective robot imitators. In Interna- tional Conference on Learning Representations, 2024b.
Hu, Y., Guo, Y., Wang, P., Chen, X., Wang, Y.J.,, Zhang, J., Sreenath, K., Lu, C., and Chen, J. Video prediction policy: A generalist robot policy with predictive visual representations. In International conference on machine learning, 2025.
Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q. Zhu, Y., and Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
Stone, P. Libero:
Ma, T. Zhou, J., Wang. Z., Qiu, R., and Liang, J. Con- trastive imitation learning for language Language-uiide multi-task robotic manipulation. In Conference on Robot Learning, 2024.
Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S., and Finn, C. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991-1002. PMLR, 2022.
Majumdar, A., Yadav, K., Arnaud, S., Ma, J., Chen, C., Silwal, S., Jain, A., Berges, V.-P., Wu, T., Vakil, J., et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Informa- tion Processing Systems, 36:655 677, 2023.
Jeon, I., Lee, W., Pyeon, M., and Kim, G. Ib-gan: Disentan- gled representation learning with information bottleneck generative adversarial networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35. pp- 7926-7934, 2021.
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3m: A universal visual representation for robot manipulation. In Conference on Robot Leurning, pp. 892- 909. PMLR, 2023.
Jia. Z., Thumuluri, V., Liu, F., Chen, L., Huang, Z., and Su, H. Chain-of-thouuht predictive control. In Forty-first International Conference on Machine Learning, 2024.
Pacelli, V. and Majumdar, A. Learning task-driven control policies via information bottlenecks. In Robotics: Science and Systems (RSS), 2020.
Karamcheti, S., Nair, S., Chen, A. S., Kollar, T., Finn, C., Sadigh, D.,, and Liang, P. Language-driven representation learning for robotics. In Robotics: Science and Systems, 2023.
Pearce, T. and Zhu, J. Counter-strike deathmatch with large- scale behavioural cloning. In 2022 IEEE Conference on Games (CoG), pp. 104-111, IEBE, 2022.
Kawaguchi, K., Deng, Z., Ji, X., and Huang, J. How does information bottleneck help deep learning? In Inter- national Conference on Machine Learning, pp. 16049- 16096. PMLR, 2023.
Perez, E., Strub. F,, De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with general con- ditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

---

Behavior Cloning with Information Bottleneck
Pomerleau, D. A. Efficient training of artificial neural net- works for autonomous navigation. Neural computation, 3(1):88-97, 1991.
Wan, Z., Zhang, C., Zhu, P., and Hu, Q. Multi-view information-bottllness representation learning. In Pro- ceedings of the AAAI conference artificial intelligence, volume 35, pp. 10085-10092, 2021.
Radosavovie, I., Xiao, T., James, S., Abbeel, P., Malik, J., and Darrell, Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pp- 416-426. PMLR, 2023.
Wang. L., Chen, X., Zhao, J., and He, K. Sccaling proprioceptive-visus learning with heterogeneous pre- trained transformers. In Advances in neural information processing systems, 2024.
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schul- man, J., Todorov, E., and Levine, Learning complex dexterous manipulation with deep reinforcement learn- ing and demonstrations, Robotics: Science and Systems, 2018.
Wen, C., Lin, J., Darrell, T., Jayaraman, D., and Gao, Y. Fighting copycat agents in behavioral cloning from ob- servation histories. Advances in Neural Information Pro- cessing Systems, 33:2564-2575, 2020.
Reuss, M., Yagmurlu, Ö. E., Wenzel, F., and Lioutikov, R. Multimodal difffusion transformer: Learning versatile behavior from multimodal goals. In First Workshop Vision-Langguge Models for Navigation and Manipula- tion at ICRA 2024, 2024.
Wen, C.,, Lin, X., So, J., Chen, K., Dou, Q., Gao, Y., and Abbeel, P. Any-point trajectory modeling for policy learn- ing. In Robotics: Science and Systems, 2024a.
Wen, J., Zhu, M., Zhu, Y., Tang, Z., Li, J., Zhou, Z., Li, C., Liu, X., Peng, Y., Shen, C., et al. Diffusion-vla: Scaling robot foundation models via unified diffusion and autoregression. arXiv preprint 24Xiv:2412.03299 2024b.
Saxena, V., Bronars, M., Arachchige, N. R., Wang, K., Shin, W. C., Nasiriany, S., Mandlekar, A., and Xu, D. What matters in learning from large-scale datasets for robot manipulation. In International Conference on Learning Representations, 2025.
Wen, J., Zhu, Y., Li, J., Tang, Z., Shen, C., and Feng, F. Dexvla: Vision-language model with plug-in diffu- sion expert for general robot control. arXiv preprint a5X5v:2502.05855, 2025.
Shwartz-Ziv, R., Painsky, A., and Tishby, N. REPRESEN- TATION COMPRESSION AND GENERALIZATION IN DEEP NEURAL NETWORKS, 2019. URL https: //openreiiww net / forum? id id- SkeL6sCqK7.
Wu, H., Jing, Y., Cheang, C.,, Chen, G., Xu, X., Liu, M.. Li, H., and Kong, T. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations, 2024.
Song, W., Chen, J., Ding, P., Zhao, H., Zhao, W., Zhong. Z.,, Ge, Z., Ma, J., and Li, H. Accelerating vision- languge- action model integrated with action chunking via parallel decoding. arXiv preprint arXiv:2503.00 02310,, 2025.
Wuthrich, M., Widmaier, F., Grimminger, F., Joshi, S.,
Agrawal, V., Hammoud, B., Khadiv, M., Bogdanovic, M., Berenz, V., Viereck, J., et al. Trifinger: An open- source robot for learning dexterity. In Conference on Robot Learning, pp. 1871-1882. PMLR, 2021.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L.., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A. et al. Deepmind control suite. arXiv preprini 2018
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, Meta-world: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference robot learning, pp. 1094-1100 PMLR, 2020.
Tian, Y., Yang, S., Zeng, J., Wang, P., Lin, D., Dong, H., and Pang, J. Predictive inverse dynamics models are scal- able learners for robotic manipulation. In International Conference on Learning Representations, 2025.
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. arXiv preprint physics/0004057, 1999.
Zawalski, M., Chen, W., Pertsch, K., Mees, O., Finn, C., and Levine, S. Robotic control via embodied chain-of- thought reasoning. In 8th Annual Conference on Robot Learning, 2024.
Torabi, F., Warnell, G., and Stone, P. Behavioral cloning from observation. Proceedings of the 27th Interna- tional Joint Conference on Artificial Intelligence, pp. 2018.
Ze, Y., Zhang, G., Zhang, K., Hu, C., Wang, M., and Xu, H. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In ICRA 2024 Workshop on 3D Visual Representations for Robot Ma- nipulation, 2024.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
12

---

Behavior Cloning with Information Bottleneck
Zeng, J., Bu, Q. Wang, B., Xia, W., Chen, L., Dong, H., Song, H., Wang, D., Hu, D., Luo, P., et al. Learning ma- nipulation by predicting interaction. In Robotics: Science and Systems, 2024.
Zhang, H., Ding, P., Lyu, S., Peng, Y., and Wang. D. Gevrm: Goal-expressive video generation model for robust visual manipulation. In International Conference Learning Representations, 2025a.
Zhang, J., Guo, Y., Chen, X., Wang, Y.J., Hu, Y., Shi, C., and Chen, J. Hirt: Enhancing robotic control with hierarchical robot transformers. In 8th Annual Conference on Robot Learning, 2024.
Zhang, J., Guo, Y., Hu, Y., Chen, X., Zhu, X., and Chen, Up-vla: A unified understanding and prediction model for embodied agent. arXiv preprint arXiv:2501.18867, 2025b.
Zhao, W., Ding, P., Zhang. M., Gong, Z., Bai, S., Zhao, H., and Wang, D. Vlas: Vision-language-action model with speech instructions for customized robot manipulation. In International Conference on Learning Representations, 2025.
Zhu, H. Yang, H., Wang. Y., Yang, J., Wang, L., and He, T. Spa: 3d swateal-swareness enables effective embodied representation. arXiv preprint arXiv: 241000200, 2024.
13

---

Behavior Cloning with Information Bottleneck
A. Proof of Theorem 4.3
Proof. The first optimization problem optimizes [(x, 2), which imposes a looser constraint on I(o, 2), as it does not directly regulate the information flow from o to 2. In contrast, the second optimization problem directly constrains I(o, 2). which may result in a smaller [( 2; ot). Therefore, we have:
(1,,;;0) 2 1(0, (0)))
(14)
From the optimization objectives of the two problems, it follows that:
I(0, 3 I(0, 200)) J.
(15)
Rearranging this inequality gives:
|J |J (1 00) - 1(0, 2; 0:))
(16)
According to the assumption that the mutual information gap is bounded:
I(o, 2; (5) 20) - 5
(17)
3
We substitute this bound into the inequality:
| - J4 S = 6.
(18)
3
Thus, the performance gap is bounded as:
J J91 19156 S 8.
(19)
This completes the proof.
B. Details of Experiment Setting
B.1. Details of Benchmarks
B.1.1. CORTEXBENCH
We provide a detailed overview of the four imitation learning benchmarks used in CortexBench (Majumdar et al., 2023). CortexBench is single-task benchmark that includes 7 selected simulators, collectively offering 17 different embodied Al tasks spanning locomotion, navigation, and both dexterous and mobile manipulation. Three of the simulators are primarily designed for reinforcement learning and are therefore excluded from our analysis. The remaining four simulators, with a total of 14 tasks, are retained for validation: Adroit (2 tasks) (Rajeswaran et al., 2018), Meta- (5 tasks) (Yu et al., 2020), DMControl (5 tasks) (Tassa et al., 2018), and TriFinger (2 tasks) (Wuthrich et al., 2021).
First, Adroit (Rajeswaran al., 2018) a suite of dexterous manipulation tasks in which an agent controls 28-DoF anthropomorphic hand. It includes two of the most challenging tasks: Relocate and Reorient-Pen. In these tasks, the agent must manipulate an object to achieve specified goal position and orientation. Each task consists of 100 demonstrations.
Second, Meta World (Yu et al.. 2020) is a collection of tasks in which agents command a Sawyer robot arm to manipulate objects in a tabletop environment. Cortex includes five tasks from MetaWorld: Assembly, Bin-Picking, Button-Press, Drawer-Open, and Hammer. Each task consists of 25 demonstrations.
Third, DeepMind Control (DMControl) (Tassa et al., 2018) is widely studied image-based continuous control benchmark, where agents perform locomotion and object manipulation tasks. CorcexBench includes five DMC tasks: Finger-Spin, Haad,,r-Hard, Cheetah-Run Stalker-Stand and Walker- Walk. Each task consists of 100 demonstrations.
Lastly, TriFinger (TF) (Wuthrich et al., 2021) is robot consisting of a three-finger hand with 3-DoF per finger. CortexBench includes two tasks from TriFinger: Push-Cube and Reach-Cube. Each task consists of 100 demonstrations.
Although only Meta-World is strictly a robot manipulation benchmark, we include all tasks to demonstrate the effectiveness of IB comprehensively. We provide visualizations for one task from each benchmark, as shown in Figure
14

---

Behavior Cloning with Information Bottleneck
a) Relocate task in Adroit of CortexBench
b) Walker stand task in DMControl CortexBench
Bin picking task in Meta-Norld Cortexbench
d) task in Trifinger CortexBench Figure 8: Visualizations for task from each suite in CortexBench.
a) Open the middle drawer of the cabinet task in LIBERO-Goal
b) Pick up the butter and place it in the basket in LIBERO-Object
Pick up the black bowl the place on the plate task in in LIBERO- LIBERO-Spatial
the and put the moka pot on it task in LIBERO-Long
Figure Visualizations for one task from each suite LIBERO.
B.1.2. LIEBRO
LIBERO a language-conditioned multi-task benchmark comprising 130 tasks across five suites. LIBERO (Liu et al., 2024) has four task suites: LIBERO-Gos (10 tasks), LIBERt-Object (10 tasks), LIBERO-Spaie (10 tasks), and LIBERO- 100 (100 tasks).
LIBERO-Goal tasks share the objects with fixed spatial relationships but differ in task goals, requiring the robot continually acquire
knowledge about motions and behaviors. Examples include (1) opening the middle drawer of the cabinet, (2) opening the top drawer and placing the bowl inside, (3) pushing the plate to the front of the stove, (4) placing the bowl on the plate, (5) placing the bowl on the stove, (6) placing the bowl on top of the cabinet, (7) placing the cream cheese in the bowl, (8) placing the wine bottle on the rack, (9) placing the wine bottle on top of the cabinet, and (10) turning on the stove.
LIBERO-Objec tasks involve the robot picking and placing unique objects, requiring it continually learn and memorize new object types. Examples include (1) picking up the alphabet soup and placing it in the basket, (2) picking up the BBQ sauce and placing it in the basket, (3) picking up the butter and placing it in the basket, (4) picking up the chocolate pudding and placing in the basket, (5) picking up the cream cheese and placing it in the basket, (6) picking up the ketchup and placing in the basket, (7) picking up the milk and placing it in the basket, (8) picking up the orange juice and placing it in the basket, (9) picking up the salad dressing and placing it in the basket, and (10) picking up the tomato sauce and placing it in the basket.
LIBERO-Spatin requires the robot to place bowl, selected from the same objects, onto a plate. The robot must continually learn and memorize new spatial relationships. Examples include (1) picking up the black bowl between the plate and the ramekin and placing it on the plate, (2) picking up the black bowl from the table center and placing it the plate, (3) picking up the black bowl in the top drawer of the wooden cabinet and placing it the plate, (4) picking up the black bowl next to the cookie box and placing it on the plate, (5) picking up the black bowl next to the plate and placing it on the plate, (6) picking up the black bowl next to the ramekin and placing it on the plate, (7) picking up the black bowl on the cookie box and placing it on the plate, (8) picking up the black bowl on the ramekin and placing it on the plate, (9) picking up the black bowl on the stove and placing it on the plate, and (10) picking up the black bowl on the wooden cabinet and
15

---

Behavior Cloning with Information Bottleneck
placing it on the plate.
LIBERO-100 consists of 100 tasks involving diverse object interactions and versatile motor skills. It can be divided into LIBERO-10 (10 tasks) and LIBERO-90 (90 tasks), where we use LIBERO-10, also referred to as LIBERO-Long, as our benchmark. LIBERO-Long requires the robot to learn long-horizon tasks, demanding it to plan and execute actions over extended periods to accomplish complex objectives. Examples include (1) turning on the stove and placing the moka pot on it, (2) putting the black bowl in the bottom drawer the cabinet and closing it, (3) putting the yellow and white mug in the microwave and closing it, (4) putting both moka pots on the stove. (5) putting both the alphabet soup and the cream cheese box in the basket, (6) putting both the alphabet soup and the tomato sauce in the basket, (7) putting the cream cheese box and the butter in the basket, (8) putting the white mug on the left plate and the yellow and white mug on the right plate, (9) putting the white mug on the plate and the chocolate pudding to the right of the plate, and (10) picking up the book and placing it in the back compartment of the caddy.
We provide visualizations for one task from each suite, as shown in Figure 9.
B.2. Details of Baselines
B.2.1. BASELINES IN CORTEXBENCH
In CortexBench, the classification of baselines is primarily based on the visual encoder used.
For full fine-tuning baselines, ResNet (He et al., 2016) and ViT (Dosovitskiy, 2021) are baselines built from the original ResNet- 18 and ViT-S models, using only a portion of their architecture and with uninitialized parameters.
For partial fine-tuning baselines, R3M (Nair et al., 2023) pre-trains ResNet model on human videos (Grauman et al., 2022) using time contrastive learning and video-language alignment. For direct comparison, we use the version reproduced with ViT. VC-1 (Majumdar et al., 2023) pre-trains VIT using Masked Auto-Encoding (MAE) (He et al., 2022) on a mix of human-object interaction videos, navigation, and the ImageNet (Deng et al., 2009) datasets. Voltron (Karamcheti et al., 2023), a framework for language-driven representation learning from human videos and associated captions, pre-trains a ViT using MAE. MPI (Zeng et al., 2024), a framework for interaction-oriente representation learning, directs the model to predict transition frames and detect manipulated objects using keyframes as input. It learns from human videos and associated captions.
If a proprioceptive state is available, it is first transformed into embeddings using linear layer. Depending on the fusion method, these embeddings are then combined with the visual embeddings. For spatial fusion, an MLP is used, while for temporal fusion, a temporal transformer is employed. The fused features are ultimately processed through an MLP-based policy head to generate actions.
B.2.2. BASELINES IN LEBERO
Similar to previous work (Zhu et al., 2024), the baselines in LIBERO largely follow the three architectures outlined in the original paper (Liu et al., 2024), which we have renamed as BC-RNN, BC-Transformer, and BC-VILT. These three baselines part of the temporal fusion methods.
BC-RNN uses a ResNet as the visual backbone to encode per-step visual observations, with an LSTM as the temporal backbone to process sequence of encoded visual information. The language instruction is incorporated into the ResNet features using the FiLM method (Perez et al., 2018), and is added to the LSTM inputs.
BC-Transformer employs a similar ResNet-based visual backbone but instead uses a transformer decoder (Vaswani, 2017) as the temporal backhone to process outputs from ResNet, which are temporal sequences of visual tokens. The language embedding is treated separate token alongside the visual tokens in the input to the transformer.
BC-VILT utilizes a ViT as the visual backbone and a transformer decoder as the temporal backbone. The language embedding is treated as a separate token in the inputs of both the ViT and the transformer decoder. All temporal backbones output a latent vector at each decision makinn step.
Additionally, we introduce a spatial fusion method, BC-MLP, which uses similar ResNet-based visual backbone. The visual and language embeddings are directly concatenated and input into an MLP for fusion. After feature fusion, all methods use an MLP-based policy head to generate actions.
16

---

Behavior Cloning with Information Bottleneck
Table 3: Training hyperparameters of all (full partial) fine-Table 4: LIBERO.
Training hyperparameters of all baselines in
tuning baselines in CortexBench.
Hyperparameters
Training
Hyperparameters
Training
epoch batch size optimizer learning rate weight decay Ir scheduler warm up clip grad augmentation history length
50 100
epoch batch size optimizer
50
256 512
64
AdamW
Adam W
lc-4| le-3
learning rate weight decay Ir scheduler warm up clip grad augmentation history length
le-4
le-4
le-4
Cosine
Cosine
0
0
100
100
Resize, CenterCrop, Normalize
Normalize, ColorJitter
3
10
Table 5: IB-related Hyperparameters of all baselines in both CortexBench and LIBERO.
IB-related Hyperparameters
Training
MINE model
architecture
layer MLP
hidden size output size optimizer
512
Adam
learning rate
1e-5
loss weight
0.1
IB loss
Lagrange multiplier 8
[lc-4, 1e-2]
B.3. Details of Implementations
For Cortexbench and LIBERO experiments, we use a single NVIDIA V100 A100 GPU (CUDA 11.8) with 12 CPUs. For real-world experiments, the single-task setting uses one V100 GPU with 12 CPUs, while the language-conddiionee multi-task setting is trained on 8 A100 GPUs with 100 CPUs and evaluated on a single A A100 GPU.
B.3.1. CORTEXBENCH
We largely adhere to the original parameter settings from the CortexBench paper (Majumdar et al., 2023). For both full fine-tuning and partial fine-tuning methods, as shown in Table 3, training parameters are presented with full fine-tuning on the left and partial fine-tuning on the right. For model architecture parameters, spatial fusion employs a 4-layer MLP, where the input dimension matches the output dimension of the image encoder. The features are first downsampled and then upsampled to maintain consistency with the input dimension. For temporal fusion, each modality's feature dimension is first projected to 64, then processed through four-layer, six-head Transformer. For dataset configurations, we adopt a full-shot setting, training with 100 demonstrations for Adroit, 100 for DMControl, 25 for MetaWorld, and 100 for TriFinger. During evaluation, we assess performance using 25, 10, 25, and 25 test trajectories, respectively.
B.3.2. LIBERO
We largely follow the original parameter settings from the LIBERO paper (Liu et al., 2024). The training parameters provided in Table 4. Regarding model architecture, Appendix A. of the original LIBERO paper (Liu et al., 2024) describes the model parameters for BC-RNN, OC-Trrnsformer, and BC-VILT. Here, we present the model parameters for BC-MLP, which shares the same architecture as -CCTransforrmrr except for the fusion module. Specifically, BC-MLP employs a four-layer MLP with a hidden size of 256 as its fusion module. For dataset configurations, in the full-shot setting, we use five demonstrations for evaluation, leaving the remaining 45 for training, which is considered the full-shot setting in
17

---

Behavior Cloning with Information Bottleneck
80
70
60
60
Success Rate (%6
Success Rate (%)
50
40
40
30
20
20
10
0
40
80
90
100
20
30
40
Epoch
Epoch
Success rate of VC-1 in Assembly task of Meta-World
b) Averaged Success rate of BCTTransformer in LIBERO-Goal
Figure 10: Comparison of success rate curves between single-task and multi-task training.
experiments. However, full-shot training typically refers to utilizing all 50 demonstrations without allocating any for evaluation, robotic systems can operate without separate validation data.
For BC+IB, all training and model parameters remain identical to those of BC, except for the IB-specific parameters.
B.4. Details of Model Selection
B.4.1. CORTEXBENCH
In the single-task dataset CortexBench, we observed that the learning curves of certain tasks exhibit significant oscillations, such as the assemble task in MetaWorld, as shown in Figure 10 (a). Previous studies often record performance at intervals of many epochs or steps, selecting either the highest value (Majumdar al., 2023) or the average of multiple peak values (Ze al., 2024). Following (Majumdar al., 2023), we directly use the highest value explore the model's full potential on the given task.
B.4.2. LIBERO
For the multi-task dataset LIBERO, we observed that while the learning curve for individual tasks may still oscillate, decrease in success rate for one task is often accompanied by an increase for another. This trade-off results in smoother overall learning curves across multiple tasks, as shown in Figure 10 (b). To make model selection more practical and representative, we directly select the model from the final epoch.
C. Additional Experiment Results
C.1. Details of Simulation Experiments
We provide the task-wise wise results and corresponding 3 values for CortexBench. The results for Adroit and TriFinger in Table 6, DMControl in Table 7, and World are shown in Table 8. Across almost all tasks in CortexBench, incorporating the IB consistently improves performance compared to vanilla BC methods. Notably, models such as ResNet+IB, VC-1+IB, and MPI+IB often achieve the highest success rates, demonstrating the benefits of redundancy reduction in latent representations. In most cases, properly tuning 3 (c.g., selecting values in the range of lc-4 to 1c-2) leads to noticeable improvements.
C.2. Attention Map Visualizations
As shown in Figure 11, IB helps the model focus on task-relevant regions (e.g., the arm and target object) by suppressing attention to redundant background features- an effect less evident without redundancy reduction, highlighting IB's distinct role beyond standard regularization.
18

---

Behavior Cloning with Information Bottleneck
Table Task-wise Performance on Adroit and Trifinger of CortexBench. We evaluated tasks of benchmarks using random seeds and reported the average success rate (sr). The best performance bolded.
Adroit Relocate
TriFinger Move-Cule Cube
Method
Reorient-Pen
Avg
Reach-Cube
Avg
ResNet
65.33
66.67
66.00
87.12
56.06
71.59
ResNettIB
69.33
74.67
72.00
87.14
57.45
72.30
VIT
61.33
9.33
35.33
78.77
32.37
55.57
VIT+IB
64.00
10.67
37.33
77.83
34.04
55,93
R3M
45.33
5.33
25.33
74.29
45.45
59.87
R3M+IB
52.00
2.67
27.33
75.04
46.23
Voltron
32.00
5,33
18.67
86.37
62.04
74.21
Voltron+IB
38.67
4.00
21.33
86.62
75.12
VC-1
38.67
10.67
24.67
84.19
59.90
72.05
VC-1+IB
37.33
14.67
26.00
84.69
62.91
73.80
MPI
60.00
9.33
34.67
44.13 46.78
61.91
MPI+IB
61.33
12.00
36.67
79.91
63.34
Table 7: Task-wise Performance DMControl of CortexBench. We evaluated 5 tasks using 3 random seeds and reported the average success rate (sr). The best performance is highlighted in bold.
Method
Cheetah-
Finger-Spp Spin
Reacher-Eaad Easy
Walker-Staie
Walker Walk
Avg
ResNet
38.32
88.37
92.20
91.42
64.34
74.93
ResNet+IB
50.75
90.42
99.78
96.39
87.37
84.94
ViT
7.22
3.39
18.97
18.05
4.43
10.41
VIT+IB
4.27
12.34
24.11
17.45
4.46
12.53
R3M
17.01
65.31
53.73
49.25
16.27
40.31
R3M+IB
16.19
72.10
57.34
46.48
16.60
41.74
Voltron
1.65
8.56
44.06
46.27
26.19
25.35
Voltron+IB
6.93
23.85
35.89
56.48
42.68
33.16
VC-1
20.02
85.35
74.01
64,65
25.07
53.82
VC-I+1B
21.52
80.91
74.80
67.10
30.34
54.93
MPI
38.76
88.43
75.87 86.99
68.92 69.29
25.27
59.45
MPI+1B
33.82
86.44
30.50
61.41
Table 8: Task-wise Performance on Meta-Workee CentexBench. We evaluated 5 tasks using random seeds and reported the average success rate (sr). The best performance is highlighted in bold.
Method
Assembly
Bin- lin-Picking
Button-Press
Drawer-Open
Hammer
Avg
ResNet 40.00
74.67
94.67
100.00
96.00
81.07
ResNet+IB
49.33
76.00
94.67
100.00
96.00
83.20
VIT
13.33
13.33
21.33
37.33
73.33
31.73
ViT+IB
13.33
9.33
18.67
62.67
76.00
36.00
R3M
42.67
56.00
38.67
66.67
61.33
53.07
R3M+IB
38.67
53.33
38.67
68.00
72.00
54.13
Voltron
60.00
58.67
68.00
82.67
93,33
72.53
Voltron+1B
57.33
74.67
54.67
93.33
92,00
74.40
VC-1
68.00
60.00
65.33
100.00
94.67
77.60
VC-1+IB
70.67
76.00
69.33
100.00
96.00
82.40
MPI
61.33
40.00
100.00
72.00
66.40
MPI+BB
61.33
53.33
58.67
100.00
73.33
69.33
19

---

Behavior Cloning with Information Bottleneck
BC-VILT
BC-VILT+IB
BC-VILT
IC-
(a) open the top drawer and put the bowl inside task
(b) put the wine bottle on top of the cabinet task
BC-VILT
BC-VILT-IB
BC-VILT
BC-VILT+IB
(c) put the bowl the plate task
(d) put the wine bottle on the rack task
Figure 11: IB encourages the model to focus more selectively tesk-relevant regions, specifically the robotic arm and the larget object, by suppressing attention to redundant background features.
C.3. More Explanation the LIBERO Experiments
C. MORE EXPERIMENTS ON LIBERO-LONG
In the main text, we point out that the experimental results LIBERO-Long affected by the limited capacity of baseline models (e.g.. BC-VILT with 10M parameters), leading only marginal improvements. While these models perform well on simpler tasks such as LIBERO-Goal achieving an 80% success rate with an 8% improvement, they show limited gains on more complex tasks like LIBERO-Long, with only a 2% increase in performance. This is primarily due to the performance ceiling imposed by the lightweight baselines. To further illustrate this limitation, we conduct experiments on LIBERO-Lone using Diffusion Policy (DP), where the 2.14M-paramtee MLP head of BC-Transformer is replaced with the more expressive MoM-parameter DP head. We train DP from scratch on single 100 GPU with a batch size of 64, learning rate of 1e-4, and for 50 epochs. The training setup for DP+IB is identical to that of DP, except for the IB-related components. For the IB specciiii hyperparameters, the Lagrange multiplier / is set to le-5.
Table 9: Performance LIBERO
Method
LIBERO-Long
DP
78.0
DP+IB
84.0
Our experimental setup differs from that papers (Kim et al., 2024; Wen et al., 2024b; 2025) like OpenVLA. In those works, the image observations are saved at resolution of 256x256 (instead of 128x128) and undergo additional filtering, such removing "no-op" (zero) actions and unsuccessful demonstrations. In contrast, our setting uses the raw LIBERO data with lower-resolution images and filtering.
Attention:
C.3.2. PERFORMANCE ON LIBERO-OBJECT
On LIBERO-Object, we observe that the success rate does not consistently improve with an increasing number of demon- strations. Specifically, in the 10-shot setting, BC-VILT achieves success rate 56. 56,17% whereas in the full-shot setting, its performance drops to 43.00%. We hypothesize that this decline is due to inherent data distribution characteristics and suboptimal data quality within the benchmark.
20

---

Behavior Cloning with Information Bottleneck
C.4. Extension to Few-shot Setting
80
70
BCMILT+
We further assess the effectiveness IB in few-shot set- tings by evaluating BC-VILT under varying numbers of demonstrations in LIBERO-Goal, as shown in Figure 12. The results consistently show that incorporating IB im-
Success Rate (%)
50
40
proves success rates across all LIBERO suites, demon- strating its efficacy in few-shot learning scenarios.
30
10
Number of Training Demos
Figure 12: Comparison of the success rates of BC- +ILT+I+ trained with 1, 5, 10, and demonstrations against the vanilla BC-VILT in the LIBERO-Goal suite.
21


Reviews:
[{"review_id": "SfASoDG9iM", "reviewer": "Reviewer_a6Qr", "summary": "The manuscript investigates the redundancy in representations for Behavior Cloning in robot manipulation and\nintroduces the Information Bottleneck principle to mitigate this issue. By incorporating IB, we aimed to filter out redundant information in latent representations while preserving task-relevant features.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your affirmative review of this paper! We address your questions as follows.\n\n### **Comment 1 on Writing and Organization: Some of the less important findings (e.g. finding 9) are unnecessarily highlighted.**\n\nWe sincerely appreciate the reviewer’s feedback on refining the presentation of our findings.\n\nOur intention in structuring the findings as they are was to provide a comprehensive and systematic analysis of both the impact and potential applications of the IB in BC.\nFor Finding 9, in robot manipulation, it is common practice to fine-tune a generalist model on domain-specific data to better adapt to new tasks. Our intention was to emphasize the practical value of our method in this context, showing that it can be effectively integrated into such fine-tuning processes commonly encountered in real-world scenarios.\n\nThat said, we acknowledge the reviewer’s concern and **will revise the manuscript accordingly by highlighting only the most essential findings** and integrating more general observations into the discussion section to improve focus and readability.\n\n### **Comment 2 on Theoretical Clarification: Finding 2 implies that the latent representation Z derived from input X is redundant, making information compression essential for performance improvement. Can it be understood that feature learning is key for action decoding, and that redundancy between X and Z can be addressed through optimized feature learning?**\n\n1. Yes, feature learning itself is crucial for action decoding.\n\nMost existing approaches aim to learn a generalizable representation and typically focus on scaling up data and model capacity. For example, many recent methods leverage larger datasets to train larger models, such as vision-language-action (VLA) models [1]. Fundamentally, these methods aim to learn more generalizable features Z by ensuring that the input space X covers a sufficiently broad distribution in the mapping X→Z.\n\nOthers incorporate additional sources of information to guide the learning process—for instance, CoTPC [2] introduces richer textual context, and ATM [3] integrates visual information from robot trajectories to guide robotic actions.\nThese methods essentially introduce additional information to guide or constrain the learning of Z.\n\nIn contrast, our approach takes a different perspective. Instead of adding more information, we focus on reducing redundancy in the input to learn more compact and task-relevant representations.\n\n2. Yes, the redundancy between the input X and the latent representation Z can be mitigated through optimized feature learning. \n\nFor instance, we use I(Z;X) to measure the amount of information transferred from X to Z, and aim to minimize I(Z;X) while ensuring that I(Z;A) increases, in order to ensure that the information being reduced is redundant. \nThis approach is essentially based on the IB framework. It encourages the model to retain only task-relevant information, while discarding irrelevant or redundant information, thereby reducing overfitting and improving generalization (as illustrated in **Figure 3 in [Link](https://anonymous.4open.science/r/rebuttal_4879-FD3A/README.md)**).\n\nFinally, thank you again for your time and effort in reviewing our work. We will revise the paper based on the rebuttal to better present our work.\n\n### Reference\n### Reviewer a6Qr\n[1]Rt-2: Vision-language-action models transfer web knowledge to robotic control.\n[2]Chain-of-thought predictive control.\n[3]Any-point trajectory modeling for policy learning.\n### Reviewer xKwq\n[1]RT-2: Vision-language-action models transfer web knowledge to robotic control.\n[2]Openvla: An open-source vision-language-action model.\n[3]Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.\n[4]Spa: 3d spatial-awareness enables effective embodied representation.\n[5]Learning manipulation by predicting interaction.\n### Reviewer H7Dt\n[1]MeMo: Meaningful, modular controllers via unformation bottlenecks.\n[2]Infobot: Transfer and exploration via the information bottleneck\n[3]Learning task-driven control policies via information bottlenecks\n[4]Learning representations for neural network-based classification using the information bottleneck principle.\n[5]Multi-view information-bottleneck representation learning.\n[6]Protecting your llms with information bottleneck.\n### Reviewer PrSW\n[1]Learning representations for neural network-based classification using the information bottleneck principle.\n[2]Multi-view information-bottleneck representation learning.\n[3]Protecting your llms with information bottleneck.\n[4]Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.\n[5]CALAMARI: Contact-aware and language conditioned spatial action MApping for contact-RIch manipulation.\n[6]Predictive inverse dynamics models are scalable learners for robotic manipulation.", "claims_and_evidence": "The manuscript is articulated clearly, with well-defined supporting provisions and clear conclusions.", "methods_and_evaluation_criteria": "Yes, proposed methods and evaluation criteria make sense for the problem.\nTheoretical analysis enhances the credibility of the algorithm.", "theoretical_claims": "They seem no problem.", "experimental_designs_or_analyses": "The experimental validation is thorough, and the results effectively substantiate the experimental hypotheses.", "relation_to_broader_scientific_literature": "no", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "see comments for authors.", "other_comments_or_suggestions": "see comments for authors.", "questions_for_authors": "1. There are a total of 9 Findings in the paper. Some of the findings are not particularly striking; it is recommended to refine them further and present the most significant conclusions as key findings. Others may be discussed as general experiment results, for example, Finding 9.\n2. \"Finding 3: Finding 2 implicitly suggests that the latent representation Z derived from input X is redundant. Therefore, compressing information from input is essential, which can further enhance performance.\"\nCan it be understood that feature learning itself is crucial for action decoding, and that the redundancy of information between X and Z can be resolved through optimized feature learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lhLWvQWVPl", "reviewer": "Reviewer_xKwq", "summary": "This manuscript explores latent representations in behavior cloning for robot manipulation by applying the Information Bottleneck principle to address redundancy in learned representations. It introduces an innovative perspective by applying mutual information to quantify and mitigate redundancy, supported by theoretical analysis and extensive empirical evaluation on CortexBench and LIBERO benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review of this paper! We hereby address your concerns as follows.\n\n### **Comment 1 on Writing: The claimed improvements from incorporating IB are overstated.**\nThanks for the feedback. We will emphasize the consistency of improvements rather than their magnitude in the revised manuscript.\n\n### **Comment 2 on Experiment Performance: Why is the performance on LIBERO so low, and why does IB underperform in complex, long-horizon tasks?**\nThe relatively low performance on LIBERO, especially on the LIBERO-Long, is attributed to the following factors:\n\n1. Differences in data processing:\n\nIn OpenVLA, the image observations are saved at a resolution of 256×256 (instead of 128×128) with additional filtering (e.g., removing \"no-op\" (zero) actions and unsuccessful demos).\nWe use the raw LIBERO data with lower-resolution images without filtering.\n\n2. Differences in model capacity:\n\nOpenVLA has around 7B parameters, and diffusion policies may have around 100M parameters. Our models such as BC-ViLT+IB contain only around 10M parameters.\n\nAs a result, our models are less effective in complex, long-horizon tasks like LIBERO-Long.\n\n### **Comment 3 on Method: The performance gains on LIBERO are uneven, raising concerns about the effectiveness of the approach.**\nThis limitation arises not from the IB theory itself, but from the mismatch between model capacity and task complexity. Our lightweight baseline models perform well on simpler tasks like LIBERO-Goal, but their limited capacity constrains performance on more complex tasks like LIBERO-Long. As a result, the potential gain from incorporating IB is limited. We **illustrate the issue with experiments on LIBERO-Long in Comment 5**. (Due to word limit, please refer to our response to Comment 2 from reviewer PrSW for further discussion.)\n\n### **Comment 4 on Essential References: This paper does not adequately contextualize IB in relation to recent advances in large-scale pretrained vision-language-action models like RT-2[1] and OpenX-Embodiment.**\nFirst, we clarify that VLAs like RT-2[1] and OpenVLA[2] are mentioned in the introduction to highlight our motivation, and we also discuss them in the limitations section.\nSecond, our work focuses on improving generalization within a given model architecture by optimizing the training paradigm through information compression, rather than designing a new large model. The IB method can also be incorporated into VLA-based models.\n\n### **Comment 5 on Baselines: Include more established baselines rather than design baselines all by themselves.**\nWe include two more baselines. OpenVLA[3] (4 A100 GPU, 80K steps, 32 batch size) and Difusion Policy (The MLP head (1.14M parameters) of BC-Transformer is replaced with the DP head(90M)). Each task is rolled out 20 times during testing.\n\n|Method|LIBERO-Long|\n|-|-|\n|OpenVLA|51.5|\n|OpenVLA+IB|**54.0**|\n|DP|78.0|\n|DP+IB|**84.0**|\n\nRationale for baselines: Our most baselines are consistent with previous works like[4,5]. Our custom baseline, BC-MLP, complements the experimental setup by introducing a spatial fusion approach, as the first three baselines rely on temporal fusion.\n\n### **Comment 6 on Experiments: The manuscript lacks sufficient empirical evidence to support the claim that redundancy is the primary limitation.**\nThe attention map in Figure 3 of [anonymous material](https://anonymous.4open.science/r/rebuttal_4879-FD3A/README.md) shows that without redundancy reduction, the model focuses on areas beyond the robotic arm and target object.\n\n### **Comment 7 on Experiments: Providing a systematic sensitivity analysis of key hyperparameters.**\nThe complete experimental hyperparameters are in Appendix Tables 3 to 5. All IB-related hyperparameters, except for the Lagrange multiplier β, remain consistent across experiments. The range of β is reported in Finding 8, with exact values provided in Appendix Tables 6 to 9.\n\n### **Comment 8 on Experiments: More diverse real-world examples.**\nWe conduct more challenging real-world experiments focusing on:\n1. Increased distractor objects and random positions.\n2. Training and evaluating a multi-task policy.\n3. Investigating generalization ability (*denotes unseen, which refers to instances not observed during the SFT stage).\n\nWe use 8 A100 GPUs to train both CogAct[3] and CogAct+IB for 8K steps on our newly collected dataset. The batch size is 128. Each task is tested 10 times. Videos are in [anonymous material](https://anonymous.4open.science/r/rebuttal_4879-FD3A/README.md).\n\n|Method|Put corn into blue bowl|Put corn into red bowl|Put corn into green bowl*|Put wood block into blue bowl|Put wood block into red bowl|Put wood block into green bowl*|Avg|\n|-|-|-|-|-|-|-|-|\n|CogAct|80|60|20|100|60|40|60|\n|CogAct+IB|100|80|60|80|60|40|**70**|\n\nThank you again for your time and effort in reviewing our work. We will revise the paper based on the rebuttal to better present our work.\n\n### Reference\nPlease refer to Reviewer a6Qr's reference list.", "claims_and_evidence": "The authors claim that incorporating Information Bottleneck substantially improves Behavior Cloning (BC) performance across various settings and benchmarks.\nWhile some performance improvements are shown, this claim is overstated. Particularly, the improvements in LIBERO multi-task benchmarks appear inconsistent, and the results on the LIBERO-Long tasks are relatively marginal (e.g., just around 2%~ improvement), thus casting doubt on the effectiveness and generality of the IB approach in complex multi-task settings.\n\nThe authors claim that the proposed IB approach achieves robustness and stable improvements across different scenarios and methods.\nThis claim is problematic. The LIBERO benchmark results clearly reveal that performance gains are uneven, sometimes marginal.", "methods_and_evaluation_criteria": "The proposed methods and evaluation benchmarks are appropriate for addressing redundancy in latent representations. The adoption of the Information Bottleneck is justified, and the selected benchmarks align effectively. However, the manuscript lacks sufficient empirical justification for the claim that redundancy is the primary limitation affecting current behavior cloning approaches, and it doesn't thoroughly discuss why IB underperforms in complex, long-horizon tasks. The authors should strengthen the manuscript by clearly demonstrating the specific negative impacts of redundancy and providing a systematic sensitivity analysis of key hyperparameters.", "theoretical_claims": "No explicit mathematical errors or mistakes in the proofs were found.", "experimental_designs_or_analyses": "I don't know if I missed something, but I really wonder why the performance on LIBERO is that low. The numbers provided in the paper are like in the last century. Current BC methods like openvla or diffusion policy (trained from scratch) can achieve promising performance, and I think the authors should consider how to apply the IB theory to them. If the training split is different from theirs, the authors could consider using theirs instead.", "relation_to_broader_scientific_literature": "The manuscript makes itself within the broader context of representation learning in BC for robotic manipulation, specifically emphasizing the Information Bottleneck (IB) theory. The authors effectively situate their work within existing research.", "essential_references_not_discussed": "The manuscript cites VC-1, R3M, and Voltron, yet fails to deeply contextualize IB in relation to recent advances in large-scale pretrained vision-language-action models like RT-2 and OpenX-Embodiment, which are increasingly central to robot imitation learning and generalization. The manuscript would benefit significantly by clearly differentiating IB's effectiveness from these existing approaches.\n\nBrohan, A., et al. (2023). RT-2: Vision-language-action models transfer web knowledge to robotic control.", "other_strengths_and_weaknesses": "Please see above.", "other_comments_or_suggestions": "The Supplementary Material shows some real-world examples, which is nice to check. However, the case is not complex enough to demonstrate the method. I recommend the authors to show more diverse examples.", "questions_for_authors": "See above. \n\nBasically, I think the authors could include more established baselines and serve IB theory as a plugin module to see if the proposed method could improve the existing methods rather than design baselines all by themselves.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VI8aElyBGj", "reviewer": "Reviewer_H7Dt", "summary": "This paper proposes an information bottleneck approach to learn policies via BC for robot manipulation. To achieve this the paper uses the mutual information via neural estimation (MINE) objective and uses that to add an additional mutual information based regularization loss during training. Theoretical results are presented but these are just mostly derived from prior works. Experiment results are presented in two domains and shows that adding the regularization loss slightly improves performance on simulation based benchmark tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Please open [link to anonymous material](https://anonymous.4open.science/r/rebuttal_4879-FD3A/README.md) for easy reference.\n\n### **Comment 1 on Novelty of Method: simply apply the IB in BC and other works had done it [1-3].**\n1. Differences with [1-3]\n\n(1) Papers [1] and [2] differ significantly from our idea of reducing mutual information to eliminate redundancy. Paper [1] loosely references the IB concept but lacks theoretical grounding, simply adding noise to control signals to shift reliance toward local perception. Paper [2] uses IB to identify critical states based on goal dependence, aiming to improve exploration rather than reduce redundancy.\n\n(2) Paper [3] uses the IB to obtain task-relevant information for control, which is similar to our idea. We explain the differences in Point 2.\n\n2. Our contribution\n\n(1) As shown in Figure 1 (a) of the *anonymous material*, traditional IB methods [3-6] typically apply the bottleneck to a single modality (e.g., images) and its latent features, with the information flow being O→Z→A.\n(2) In contrast, BC for robot manipulation is more complex than earlier control or single-modal tasks, requiring models to handle *diverse, multimodal data and inevitably necessitating feature fusion*. If strictly following the previous IB paradigm (as shown in Figure 2 of the *anonymous material*) would require constraining each input modality and its features individually, resulting in a pipeline that lacks elegance and scalability, is highly complex, and lacks the ability to capture cross-modal associations.\n(3) Thus, we propose redefining the \"input\" at the representation level and treating the features before the policy head as latent features, with the information flow being O→X→Z→A. The advantages of this approach are: 1) It allows us to apply redundancy reduction in a unified way across all modalities. 2) In practice, encoders are often frozen, making this approach more effective in such cases. 3) It better scales to a wider range of robotic algorithms.\n\n### **Comment 2 on Novelty of Theory: Simply applying theoretical claims to robotics does not constitute much novelty.**\n\nAs shown in Figure 1 of the *anonymous material*, we adapt Theorems 4.1 and 4.2 to support the robot task with information flow of O→Z→A. To demonstrate that our paradigm remains compatible with these theorems, we present Theorem 4.3: Even if we optimize I(X;Z) by applying the bottleneck at an intermediate feature level X, as long as X preserves the essential structure of the original input o, we are effectively controlling I(O;Z), with the difference bounded by a small constant δ.\n\n### **Comment 3: \"Rethinking Latent Representations\" is confusing.**\n\nThank you for the feedback. We agree that the original title \"Rethinking Latent Representations\" may be confusing, and we will revise it to \"Rethinking Latent Redundancy\" in the revised version.\n\n### **Comment 4: Slight improvements in success rate. For example, in Table 1 (if we remove the MetaWorld tasks), the improvements in all the other tasks are very small, often less than 1%.**\n\nDue to word limit, please refer to our response to Comment 2 from reviewer PrSW.\n\n### **Comment 5: Slight improvements in success rate do not really show that the representations are more compact and more robust. And this paper does not really tell us much about the redundancy.**\n\nAs discussed in Comment 4, Trifinger contains minimal redundancy, limiting IB’s impact (less than 1% gain). In contrast, LIBERO-Goal includes more clutter and complex backgrounds, where IB yields a larger improvement (~8%). This observation suggests that **the representations in certain environments, especially those with more visual distractors and complex backgrounds, tend to contain higher redundancy, allowing IB to be more effective.**\n\nAs shown in the **attention map** (Figure 3 in *anonymous material*), IB helps the model focus on task-relevant regions (e.g., the arm and target object) by suppressing attention to redundant background features—an effect less evident without redundancy reduction, highlighting IB’s distinct role beyond standard regularization.\n\n### **Comment 6: LIBERO benchmark is not really a challenging multi-task benchmark since it has little task instruction diversity.**\n\nWhile LIBERO may have limited instruction diversity, it is a widely used benchmark in goal-conditioned manipulation and is sufficiently challenging for our study. The perceptual redundancy present in LIBERO allows IB to demonstrate clear benefits, as reflected in the significant performance gains observed in our experiments.\n\nFurthermore, We conduct more challenging **real-world experiments**. Due to word limit, please refer to our response to Comment 8 from reviewer xKwq. The videos are in *anonymous material*. \n\nThank you for your time and effort in reviewing our work. We will revise the paper based on the rebuttal to better present our work.\n\n### Reference\nPlease refer to Reviewer a6Qr's reference list.", "claims_and_evidence": "I do not think all claims are properly evaluated. First, as can be seen in the title, the paper talks about “Rethinking Latent Representations.” But I am a bit lost on how we are doing any re-thinking. The paper for the most part simply applies the information bottleneck objective during BC. There are some slight performance improvements and the paper claims that this shows that compressing latent space is essential. However, it is hard for me to agree with this claim. Slight improvements in success rate do not really show that the representations are more compact and more robust. Most of the tasks used in the paper have very little environment variation hence a lower-dim representation (or additional regularization) would probably be sufficient  to complete the task and boost the success rate, but this does not really tell us much about the redundancy in the inputs or representations. This can also be seen in the numbers in Table 1 (if we remove the Metaworld tasks) the improvements in all the other tasks is very little often < 1%. The variance of the base result is already pretty high. So I am quite unsure about the very strong claims being made in the paper.\n\nI am also a bit unsure about the multi-task results. The LIBERO benchmark while useful is not really a challenging multi-task benchmark since it has little task instruction diversity. But the robust representations and information bottleneck are both more interesting in multi-task settings, since the fusion of representation (across vision and language) would result in more complex and interesting representations.", "methods_and_evaluation_criteria": "The method and evaluations make sense. However, the evaluations are rather limited and it’s hard to conclude some of things that the paper concludes (please see above).", "theoretical_claims": "The theoretical claims are almost all derived from previous works. There isn't much novelty in them besides just applying them to robotics but since they have already been applied to supervised learning problems it isn't clear what is new here.", "experimental_designs_or_analyses": "I looked through and tried to carefully analyze both single-task (cortex-v1) and multi-task results (libero).", "relation_to_broader_scientific_literature": "The paper applies the well known information bottleneck principle to behavior cloning training for robot manipulation. In terms of novelty and contribution this is a straight forward extension of many prior works, just that the domain (robot manipulation) is different. There are may other papers that have focused on using information bottleneck for many different design choices in control tasks. These papers often focus on dm_control tasks which are slightly different than robot manipulation benchmarks but the principles are the same and they often discover similar things (e.g. applying IB in some interesting way can improve robustness/transfer/performance etc.) However, the lack of novelty could be alleviated with more extensive experiments and careful analysis. Unfortunately, the paper does not really do this, it simply uses the success rate criterion to show that IB uniformly improves representations (please see above). \n\n[1] MeMo: Meaningful, Modular Controllers Via Information Bottlenecks\n[2] InfoBot: Transfer and Exploration via the Information Bottleneck\n[3] Learning Task-Driven Control Policies via Information Bottlenecks", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "The paper is well written and easy to follow.", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FvpQO9NXBU", "reviewer": "Reviewer_PrSW", "summary": "This paper presents a method for incorporating the principal of information bottleneck (IB) into behavior cloning. They accomplish this by adding a mutual information objective to the loss function alongside the standard action regression term. This extra objective works to reduce the overfitting in low-data regime.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Please open [link to *anonymous material*](https://anonymous.4open.science/r/rebuttal_4879-FD3A/README.md) for easy reference.**\n\n### **Comment 1 on Novelty: Limited novelty beyond adding a loss term to BC.**\nOur contributions go beyond simply adding a loss term to BC. Specifically, we highlight the following:\n\n1. **Method Novelty**\n\n(1) As shown in Figure 1 (a) of the *anonymous material*, traditional IB methods [1-3] typically apply the bottleneck to a single modality (e.g., images) and its latent features, with the information flow being O→Z→A.\n(2) In contrast, BC for robot manipulation is more complex than earlier control or single-modal tasks, requiring models to handle diverse, multimodal data and inevitably necessitating feature fusion. If strictly following the previous IB paradigm (as shown in Figure 2 of the *anonymous material*) would require constraining each input modality and its features individually, resulting in a pipeline that lacks elegance and scalability, is highly complex, and lacks the ability to capture cross-modal associations.\n(3) Thus, we propose redefining the \"input\" at the representation level and treating the features before the policy head as latent features, with the information flow being O→X→Z→A. The advantages of this approach are: 1) It allows us to apply redundancy reduction in a unified way across all modalities. 2) In practice, encoders are often frozen, making this approach more effective in such cases. 3) It better scales to a wider range of robotic algorithms.\n\n2. **Theoretical Novelty** \n\nAs shown in Figure 1 of the *anonymous material*, we adapt Theorems 4.1 and 4.2 to support the robot task with information flow of O→Z→A. To demonstrate that our paradigm remains compatible with these theorems, we present Theorem 4.3: Even if we optimize I(X;Z) by applying the bottleneck at an intermediate feature level X, as long as X preserves the essential structure of the original input o, we are effectively controlling I(O;Z), with the difference bounded by a small constant δ.\n\n3. To support our claims, we conduct extensive experiments across various models and settings.\n\n### **Comment 2 on Experiment Performance: The improvements are marginal in most cases.**\nThe marginal improvements are not due to limitations of the IB theory, but rather due to:\n\n1. Model capacity and task complexity:\nOur models, such as BC-ViLT+IB, have limited capacity (e.g., 10M parameters). They perform well on simpler tasks like LIBERO-Goal (80% success, 8% improvement), but show limited gains (2%) on more complex tasks like LIBERO-Long, due to the performance ceiling of the baselines.\nWe **illustrate this issue with Difusion Policy** (The MLP head (1.14M parameters) of BC-Transformer is replaced with the DP head(90M)) experiments on LIBERO-Long.\n\n|Method|LIBERO-Long|\n|-|-|\n|DP|78.0|\n|DP+IB|**84.0**|\n\n2. Dataset and task complexity:\nIn benchmarks like Cortexbench, particularly Trifinger (Figure 7 in the appendix), the visual input contains only the robot arm and objects, with minimal redundant information, so applying IB yields minimal gains. In contrast, LIBERO environments feature more distractors, complex backgrounds, and diverse tasks, where IB leads to more noticeable improvements.\n\n### **Comment 3 on Experiments: No testing of out-of-distribution generalization.**\n1. **Object position.** We modify the positions of manipulated objects in MetaWorld of Cortexbench and perform 25 rollouts during testing.\n\n|Method|Assembly|Bin-Picking|Button-Press|Drawer-Open|Hammer|Avg|\n|-|-|-|-|-|-|-|\n|VC1|65.33±6.11|62.67±8.33|66.67±2.31|100.00±0.00|94.67±6.11|77.87|\n|VC1+IB|69.33 ± 12.86|76.00±4.00|70.67±8.33|100.00±0.00|96.00±4.00|**82.40**|\n\n2. **Generalization on unseen tasks with more distractors.** Refer to Comment 4.\n\n### **Comment 4 on Experiments: Limited/unconvincing real-world evaluations.**\nDue to word limit, please refer to our response to Comment 8 from reviewer xKwq. The videos are in *anonymous material*. \n\n### **Comment 5 on Method: In Figure 2.c, why does only the [CLS] embedding go into the policy head?**\nWe follow the common practice in transformer-based architectures, where the [CLS] token serves as a global summary of the input sequence, as seen in prior works [4-6].\nThe [CLS] embedding aggregates information through attention layers, making it ideal for downstream tasks. It is then passed to the policy head to represent the fused multimodal information.\nThis design is computationally efficient and empirically effective, as shown in prior vision-and-language modeling works.\n\n### **Comment 6 on Writing: Explaining the transition from I(Z;A) to the regression loss in Equation (6) more clearly would improve the presentation.**\nWe will revise the manuscript to explain this connection more clearly.\n\nThank you for your time and effort in reviewing our work. We will revise the paper based on the rebuttal to better present our work.\n\n### Reference\nPlease refer to Reviewer a6Qr's reference list.", "claims_and_evidence": "The paper claims that reducing latent redundancy via the IB principle significantly enhances generalization in BC for robot manipulation. This claim is supported by solid theoretical analysis, including generalization bounds, and by experiments across multiple benchmarks. The improvements in success rates and ablation studies. Additional experiments on out of distribution generalization would be informative.", "methods_and_evaluation_criteria": "The proposed approach of integrating an IB regularization term into the standard BC loss using Mutual Information Neural Estimation (MINE) is simple, yet novel for this particular application. \n\nThe choice of benchmarks (CortexBench for single-task and LIBERO for multi-task, language-conditioned settings) is well justified. \nAlthough LIBERO does not actually test language generalization (the language input is merely a task indicator), this baseline is still useful for showing that the IB objective doesn't prevent the network from scaling to multi-task settings.", "theoretical_claims": "I did not check the correctness of the proofs in the appendix.", "experimental_designs_or_analyses": "The paper compares several BC models (R3M, Voltron, VC-1, MPI, and more) with and without the IB enhancement, and it evaluates both spatial and temporal fusion methods. The set of baselines and tasks was comprehensive, and the results show a clear benefit to using IB across all model types. Additionally, detailed analyses on the effects of different β values and the performance under few-shot conditions further strengthen the study.\n\nThe real-world experiments are somewhat limited to a few rollouts on very simple tasks with no statistical analysis.", "relation_to_broader_scientific_literature": "The Information Bottleneck (IB) principle is a well-established concept in neural networks, supported by extensive theoretical foundations. This paper adapts the IB principle to behavior cloning, a field where data scarcity is common, making it a particularly promising candidate for regularization using an IB loss term.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- Combines strong theoretical analysis from the IB literature with extensive empirical validation.\n- Demonstrates how IB can help with in-distribution generalization in data-scarce regimes like behavior cloning for robot manipulaton\n- Uses a diverse set of benchmarks and ablation studies to support the claims.\n\nWeaknesses\n- Limited novelty beyond adding a loss term to BC, and the benefits are marginal in most cases.\n- No testing of out of distribution generalization\n- Limited/unconvincing real-world evaluations", "other_comments_or_suggestions": "N/A", "questions_for_authors": "In figure 2.c, why does only the [CLS] embedding go into the policy head?\n\nI think the jump from I(Z;A) to the regression loss in equation 6 could be better explained. I realize they maximize the same thing and that such a substitution is common in IB, but it would still be helpful to acknowledge it for those unfamiliar.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces the Information Bottleneck (IB) principle to reduce redundancy in representations for behavior cloning in robotic manipulation. It received two Accepts, one Weak Accept, and one Weak Reject after the rebuttal.
The  initial main concerns raised by reviewers centered around the novelty of the approach, the limited performance improvements, and the narrow evaluation scope. While the IB principle itself is not new, its application to robotic manipulation constitutes a novel contribution. The authors provided additional explanations and results in the rebuttal, including stronger performance with more recent robot models and improved real-world experiments. 
Given the clarified contributions and strengthened empirical results, the AC recommends acceptance.
Decision: Accept (poster)