HiRT: Enhancing Robotic Control with Hierarchical  Robot Transformers

Authors: jianke  zhang
yanjiang  guo
xiaoyu  chen
yen-jen  wang
yucheng  hu
chengming  shi
jianyu  chen
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 12:42:18

Abstract: Large Vision-Language-Action (VLA) models, leveraging powerful pre-trained Vision-Language Models (VLMs) backends, have shown promise in robotic control due to their impressive generalization ability. However, the success comes at a cost. Their reliance on VLM backends with billions of parameters leads to high computational costs and inference latency, limiting the testing scenarios to mainly quasi-static tasks and hindering performance in dynamic tasks requiring rapid interactions. To address these limitations, this paper proposes \textbf{HiRT}, a \textbf{Hi}erarchical \textbf{R}obot \textbf{T}ransformer framework that enables flexible frequency and performance trade-off. HiRT keeps VLMs running at low frequencies to capture temporarily invariant features while enabling real-time interaction through a high-frequency vision-based policy guided by the slowly updated features. Experiment results in both simulation and real-world settings demonstrate significant improvements over baseline methods. Empirically, we achieve a 58\% reduction in inference time delay while maintaining comparable success rates. Additionally, on novel dynamic manipulation benchmarks which are challenging for previous VLA models, HiRT improves the success rate from 48% to 75%.

Main paper: 
 introduction: 
 Large Vision-Language-Action (VLA) models [1, 2] provide a principled way to combine large vision-language models (VLMs) [3, 4, 5, 6] with end-to-end training on embodied tasks. Building on the top of pre-trained VLMs, existing VLA models [1, 2] propose to tune VLMs on massive robot data, which enables the direct end-to-end robot control while enjoying the benefits of VLM pretraining. Existing works mostly focus on multi-task generalization, enhancing performance in zero-shot and few-shot learning across various tasks.Though the VLM backends with billions of parameters bring superior generalization advantages, it comes at the cost of the heavy computational burden. During deployment, it results in low control inference speed and high latency. This can slow robot movements and extend task completion times, impairing performance and safety in dynamic tasks like manipulating fast-moving objects in cluttered environments [7, 8] . The control frequency limitations of large VLA models remain a significant obstacle to deploying these advanced models on real-world robots.Inspired by the dual process theory of human cognition [9] , we propose HiRT, a hierarchical interactive imitation learning framework for VLA models. Dual process theory posits that there are two systems Given a task language instruction, the VLM encodes the observations into features that integrate multimodal information, and then a lightweight action policy conditions this latent to generate low-level actions asynchronously. As shown in (c), our method can achieve higher performance and significantly improve inference speed. in human cognition: System 1, responsible for fast, intuitive reactions, and System 2, responsible for slow, analytical planning. Current VLA models can be seen as relying solely on System 2, using computationally expensive VLMs for inference and action generation. However, we argue that VLA models can benefit from merging these systems. HiRT utilizes System 2 to extract high-level, slowly changing information that guides a lightweight System 1 module. This System 1, implemented by a smaller model, can react swiftly to environmental changes. Though lightweight, System 1 in HiRT can leverage the guidance from System 2 to maintain performance comparable to the original VLM while obtaining notable speed gain.We term this approach HiRT, a hierarchical interactive imitation learning framework designed for rapid execution across a variety of instructions, scenes, and tasks. HiRT consists of two primary components: the understanding module and the execution module. The understanding module, InstructBLIP (7B) [5] , is a pre-trained large visual-language model that transforms visual information and language instructions into latent features enriched with commonsense knowledge for long-term scene understanding, including task planning and error correction. The execution module is a compact visual-based action policy that processes short-term scene cognition, utilizing prior observations and latent features from the visual-language model. To enhance focus on global instruction and visual data, we incorporate novel conditioning layers within the execution module. HiRT leverages the slower visual-language model to guide the swift low-level policy, enabling efficient performance in both quasi-static and dynamic tasks at high frequencies. Additionally, we achieve further speed optimizations by adjusting the asynchronous frequency of the modules. 
related works: 
 Language-Conditioned Imitation Learning for Robot Manipulation. The study of integrating language with robotic actions [10, 11, 12] through imitation learning has a long history, where language is commonly used as goal specification [13, 14, 15, 16] or intermediate representation for planning [17, 18, 19] . Some prior works have employed reinforcement learning techniques [20, 21, 22, 23, 24] to solve certain types of downstream tasks. To address incapability in generalization of these RL methods, recent works concentrate on prompting Large Language Models (LLMs) [17, 25, 26, 27, 28] for high-level task planning and fine-tuning vision-language models (VLMs) on expert robotic datasets for low-level robotic control [20, 13, 24, 29, 30, 31] . Different from previous works that explore how to generalize to new tasks, we focus on solving low-level manipulation tasks by leveraging the extensive visual-linguistic knowledge within VLMs more efficiently and effectively.Vision-Language Models for Robotics. Applying pre-trained VLMs [3, 4, 5, 6, 32] to various embodied scenarios is a recent focal area of research. Most of the prior works focus on using VLMs for high-level planning or reasoning [27, 33, 34, 35, 36, 37, 38] . To effectively connect visual or linguistic information with the physical environment, embodied models need to fine-tune pre-trained VLMs on embodied data [1] including video data containing task-level planning in linguistic form [39, 17, 27] , simple text descriptions [40, 41] , low-level actions [42, 43, 44] (known as vision-language-action models). However, deploying such large VLA models often results in slow inference speeds [45] , which makes embodied models unsuitable for scenarios requiring precise operations or quick execution. Our approach focuses on addressing this limitation by using a novel policy model, which can effectively retain the robust visual-linguistic capabilities of the larger models.Hierarchical Action Planning. Hierarchical action planning [17, 46, 27, 47, 48] involves decomposing a task into multiple simpler tasks that can be executed directly, enabling strategies to tackle more complex, long-horizon tasks. Previous works have demonstrated the role of inputting prompts into LLMs as a bridge to low-level actions. Specifically, this can be implemented through task-level planning [49, 39, 46] , code execution [50, 51, 52] , or other planning representations such as 3D scene graph [53] , affordance function [54] , and action pattern for locomotion [55] . However, these approaches are typically agnostic to physical embodiment, preventing the high-level models from directly interacting with the physical environment. In contrast to these methods, we ground VLMs to a specific robot's physical form in an end-to-end manner, enabling it to learn hierarchical task planning through continuous intermediate representations. 
method: 
 In this section, we first establish the problem in Sec.3.1. Then, we present HiRT, a hierarchical policy architecture that supports multi-task learning and fast inference in Sec.3.2. The key intuition is to draw help from pre-trained VLMs to extract rich semantic representations from multi-modal inputs, and then apply these representations to lightweight action policies that can operate asynchronously and independently of the VLM. Specifically, HiRT explores a popular vision-language model, InstructBLIP [5] , utilizing its open-source model as the backbone. We aim to output low-level actions with a latent-conditioned policy that leverages historical observations and latent encoded by VLM. This small-scale policy should operate independently of the large model at a higher frequency, necessitating a compact architecture composed of lightweight visual encoder. Following BC-Z [15] and RT-1 [13] , we design a latent-conditioned model as the lower-level policy, capable of independently performing behavior cloning for a limited number of tasks at high frequency.The language-conditioned manipulation problem can be considered a decision sequence under the environment modeled by Markov decision process: (S, A, R, P, ρ 0 ), where S, A, ρ 0 represents state space, action space and initial state distribution respectively, R : S × A × S → R represents the reward function, indicating whether a wanted state or task has been completed, P : S ×A×S → [0, 1] represents probabilistic forward dynamics function of the environment. Specifically, given a free-form language instruction l specifying a certain task, the control policy receives a visual observation o which is typically composed of a series of images. Then an action a ∈ A, incorporating the relative position and pose of the end effector, is sampled from an action distribution π(•|o, l) modeled by control policy.For HiRT, the policy π(a|o, l) is parameterized by F θ from the vision language model and S ϕ from the swift latent-conditioned policy. At certain time steps in the trajectory tk ∈ {t i } T i=1 , k ≤ T , the VLM backbone takes in a visual observation ōt k = Sample(o : tk ) obtained through asynchronous sampling and a natural language instruction l, and outputs a fused embedding: z tk = F θ (ō tk , l). Simultaneously, at each time of step, the latent-conditioned model predicts actions with recent context of visual observations and the latest latent: a t = S ϕ (o :t , z tk ). The specific details of the modules will be explained in the following section. The instruction is transformed into a continuous latent with sampled visual observation with a vision-language model and is cached into a latent buffer. At each step of inference, the pre-trained vision encoder encodes visual observations conditioned on the latest latent, and then the reduced vision-language tokens are decoded to low-level action with a conditioned action head.In HiRT, InstructBLIP [5] encodes the instruction l using a visual signal ō in the form of a single image. InstructBLIP comprises a pretrained visual encoder, a large language model (LLM), learnable query tokens, and a Q-Former [3] . At each execution time step tk , the visual observation (from either the wrist or third-view camera) is encoded by a Vision Transformer (ViT) [56] into a sequence of visual tokens:Xo t k = V iT (ō t k ) ∈ R N ×dwhere N denotes the token length and d the token width. Subsequently, Xo t k is concatenated with the instruction tokens X l t k and learnable query tokens X Q , and encoded by the Q-Former (a lightweight transformer) into an image representation fused with semantic information:X o t k = QF ormer( Xo t k , X l t k , X Q )Finally, these visual query features are used as prompts for the pre-trained LLM (LLaMA [57] ). Set the embeddings at layer i as X i t k , the output at layer i + 1 is computed as follows:EQUATIONEQUATIONEQUATIONwhere L denotes depth of transformer layers in the LLM, M SA represents the multi-head attention module, M LP stands for the multi-layer perceptron, and LN denotes LayerNorm. Instead of generating language tokens from the final layer output X L+1 t k , we aim to use the informative language embeddings to guide action generation. We employ a MAP module [58] , a single layer of attention block, to aggregate these representations:x t k = M AP (X L+1t k ), which will be used for conditioning the action policy in Sec.3.2.2.Following the BC-Z [15] and RT-1 [13] , which uses instructions and video as task embeddings, we encode the image context o :t into visual tokens X v :t using a lightweight visual encoder, i.e., EfficientNet [59] and Vision Transformer [4] . Then, we use a MAP block to aggregate all the tokens into the continuous action space. To further integrate the informative task embeddings encoded by VLM, we make use of the following conditioning strategies on either the visual encoder or action head:FiLM-Condition. For visual encoder based on convolutional network (CNN), each hidden layers are conditioned on the VLM latent variable x t k . In EfficientNet, We use FiLM layers to compute the conditioned features:Ĥ = F iLM (H | x t k ) = W γ x t k • H + W β x t k, where H represents the hidden features, and W γ , W β are the learnable parameters in the FiLM layer.Condition with Cross-Attention Layers. In each self-attention layers of Transformer, we insert an additional cross-attention layer for conditioning: Ĥ = CrossAttn(H, W h x t k ) + H, where W h represents a learnable parameter that projects x t k to the space of hidden tokens H.Condition with Prefix Tuning. To better enable VLM to regulate low-level actions, we utilize the VLM latent variable x t k as a prefix prompt for the MAP block in the action head. Specifically, the actions are computed by a t = M LP (M AP ([x t k , X v :t ])).Asynchronous Operation and Sampling. During the inference phase, we can accelerate the model by adjusting the execution frequency of the VLM. Specifically, at the initial time step t = 0, the VLM encodes multi-modal information with visual contexts and stores it in a cache. In subsequent steps, the latent-conditioned policy use the most recent latent variable from the cache to quickly output actions while the VLM runs asynchronously in parallel with the latent-conditioned policy. This asynchronous mechanism allows the policy to operate at nearly the same speed as the latent-conditioned policy, avoiding delays due to the VLM's slower inference. However, the asynchronous operation may cause the policy to use latent variables that reflect scene and instruction information from several steps earlier, which is misaligned with signals used in training. Therefore, during training stage, HiRT randomly selects a step from the past observation contexts o :t and uses the corresponding third-view image as the VLM's visual input. This technique can enhance robustness of the policy to the time-inconsistant latent variable.Training Objective During training, the VLM part is finetuned with LoRA [60] while rest of the network is fully finetuned. Concretely, we utilize maximum likelihood imitation learning objectives. The desired relative position a pos of the end-effector (or continuous joint action) is optimized via regression loss (e.g. MSE loss). The discrete status a end of the end-effector is optimized with binary cross-entropy loss:L = B 1 |B| (||a pos -âpos || 2 2 + BCE(a end , âend ))where âpos , âend denote the demonstration for relative position and status of the end-effector in a sampled mini-batch B. 
experiments and analysis: 
 In this section, we conduct extensive experiments across three domains, including two simulated benchmarks Metaworld [61] and Franka-Kitchen [62] , and a real-world panda manipulation environment to verify the effectiveness of our HiRT framework. We first introduce experiment setups in Sec.4.1. Then we present a quantitative analysis of the performance on quasi-static tasks, evaluating HiRT's capability to enhance inference speed while preserving generalization performance in Sec.4.2. Additionally, we test the performance in real-world dynamic tasks in Sec.4.3. Finally, we discuss design choices for implementing HiRT and perform ablation studies on key modules in Sec.4.4.Figure 3 : Visualization of the tasks in three domains. The left is Metaworld [61] in which we focus on the ability to learn multi-tasks. The middle depicts Franka-Kitchen [62] in which we study the ability to generalize to new scenes. The right shows our real-world settings, in which the model is trained on simple quasi-static tasks and tested on much more complex scenarios with unseen objects. Table 1 : Success rates on quasi-static manipulation tasks.Simulation Setup. The Metaworld benchmark provides 50 distinct tabletop manipulation tasks, in which we use 20 tasks (each with 50 expert demonstrations) for multi-task learning. Franka-Kitchen includes 5 kitchen manipulation tasks. Following Nair et al. [19] , we train policy models on 100 expert demonstrations for each task and test on tasks in origin and two new scenarios (alter the color scheme of the scene). We record the success rate to assess task performance: 20 attempts for each task in Metaworld and 100 for each task in Franka-Kitchen. To evaluate inference speed, we directly measure average time the policy takes to process 100 frames (avoiding influence of rendering).Real World Setup. Our real-world experiments involve multiple quasi-static manipulation tasks on the Franka Emika Panda robot, involving picking and placing various objects, routing cables, pressing buttons, and opening drawers. Specifically, we collect 2000 trajectories including image observations from wrist and third-view cameras. For quasi-static tests, we place many other objects on the table to introduce distractions and we also test whether the model can grasp entirely new objects it has never seen before to verify its semantic grounding capabilities. Besides, we test the policy's performance on dynamic tasks by moving the target object at a roughly constant speed while the robotic arm executes its actions. All tasks involve randomization (e.g. the object's position, type, number of distracting objects, and the initial state of the gripper). We report success rate of each task over 20 attempts and the average time cost during real-world roll-out. More details on the design of experimental scenarios can be found in Appendix A.2, which can better demonstrate our testing of the generalization ability of semantic grounding in real scenarios.We train HiRT on 20 tasks from Metaworld, 5 tasks from Franka-Kitchen, and 4 skills from the real world (as shown in Figure 3 ). For comparison, we evaluate Diffusion Policy (DP) [63] , Vanilla-VLA, which directly outputs actions from VLM (reimplementation of RT-2 [1] ), and RT-1 [1] method under the same settings. Imitation and Zero-Shot Generalization Performance. Table 1 presents the experimental results in simulation and real-world environments. HiRT achieves the highest success rates in simulated tasks and a high level of generalization capability similar to Vanilla-VLA in real-world environments.Compared to RT-1, which uses language embeddings for conditioning, HiRT, which utilizes vlm latent for conditioning, shows an average 20% higher success rate on seen tasks and a 30% higher success rate on new task scenarios. This demonstrates that VLM can leverage visual scenes to provide better instruction embeddings, aiding the small action policy in generalizing to new tasks.Balancing between Performance and Efficiency. To further demonstrate HiRT's ability to balance performance and execution frequency, we evaluate the model's inference speed and task success rate. Results are shown in Figure 4 . Notably, HiRT's performance is comparable with Vanilla-VLA, while its inference speed significantly increases to 9.8Hz, nearly doubling the original speed. This indicates that HiRT can significantly enhance inference speed while maintaining model generalization capabilities.We evaluate models with varying frequencies and performance levels on a series of dynamic tasks in real-world scenarios. Specifically, as illustrated in Figure 5 , we simulate realistic operational tasks by moving target objects at a constant speed of 1 cm/s, presenting different levels of generalization. Table 2 : Success rates on real-world dynamic manipulation tasks. With our hierarchical design, HiRT achieves the highest success rate and finishes the task in the least time.Results are shown in Table 2 , where the column of Time represents the duration taken by the model to complete the quasi-static task in the same scene without moving objects, serving as a reference of the model's efficiency. HiRT achieves the shortest completion time in quasi-static tasks and the highest task success rate in dynamic action tests within both in-domain and out-domain scenarios, indicating its strong generalization capability while maintaining a high execution speed. Overall, the HiRT approach effectively applies VLM-based methods to various dynamic tasks. A comparision example is visualized in Figure 6 . Our method successfully tracks the movement of the blue block and catches it.While the baseline method misses the blue block due to long inference time and high latency.In this experiment, we seek to understand the different components of HiRT. Specifically, we compare the full HiRT with HiRT(-IC), which uses the most recent single frame for latent-conditionedpolicy, HiRT(-CD), which replaces the conditioned-transformer and conditioned MAP (keep FiLM block which has been validated as effective in RT-1 [13] ) with the original module. We primarily conduct module ablation in the Franka-Kitchen environment because it allows for rapid testing on generalization capability. More ablations can be found in Appendix A. Does VLM-Latent Conditioning Improve Multi-Task Performance? As shown in Table 3 , the full HiRT model achieves the highest task completion rates in both multi-task and new task settings. Removing the extra conditioning layers (HiRT-CD) results in a 20% decrease in success rate, highlighting the importance of conditioning for multi-task learning.How Does the Latent-Conditioned Policy Perform with Different Visual Inputs? In Table 3 , regardless of the model structure (e.g. full, -CD), using multiple consecutive visual inputs significantly outperforms using a single image (results with -IC). Although using multiple visual contexts as input can reduce inference speed, it greatly enhances the policy's understanding of the scene, enabling it to generate more accurate actions using the same VLM-Latent output. 
conclusions, limitations and future works: 
 In conclusion, this study addresses the limitations of VLMs in handling complex dynamic tasks due to high computational costs and inference delays. By proposing HiRT, a hierarchical imitation learning framework, we enhance execution speed and multi-task generalization. However, due to data constraints, we have not yet employed HiRT on more complex dynamic tasks, such as grasping high-speed flying objects or rapidly adjusting object postures. This could be an area for our future research with more specific task data and adaptive adjustments to certain modules.A. To determine the impact of sampling images for VLM inputs, we compared the performance of HiRT with HiRT-AS under VLM settings of interval 1 and interval 6. As shown in Table 4 , when VLM synchronizes with the action policy at every step during testing, HiRT-AS slightly outperforms HiRT. This is expected since HiRT-AS uses the most recently updated latent at each step during training. However, when VLM operates with an interval of 6 steps for inference, HiRT-AS shows nearly a 10% decrease in success rate compared to the original method, indicating that random sampling helps HiRT maintain better generalization performance during asynchronous operation. 
 
 Appendix:  


Decision: Strengths:
+ HiRT tackles an important issue of slow inference speed in using VLMs for robotics.
+ The paper is well-written and easy to follow.
+ The figures are informative.
+ The experimental setup is clear and includes a good set of ablations. 


Weaknesses:
- Evaluations do not extensively test for semantic generalization to highlight the generalization capability from VLM. The simulation experiments include only 2 unseen tasks on Franka-Kitchen. Franka-Kitchen also has limited pose-randomization of objects, which makes it easy to overfit.
- Evaluations are missing dynamic tasks that could highlight the benefit of fast inference speeds. 
- Some details on adapter layers, data collection, and asynchronous sampling are missing.

Post rebuttal:  
The authors provided an extensive rebuttal with additional experiments. One reviewer feels that some details are still missing, but the AC believes they can be added for the camera-ready.

Reviews: review: title: title: Review of HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers
paper_summary: paper_summary: This paper focuses on the problem of enabling pretrained vision-language models (VLMs) to be used for robotic tasks with a high frequency. The overall goal is to be able to use pretrained VLMs (with potentially billions of parameters) to accomplish fast action inference which can in turn allow the robot to perform dynamic manipulation tasks. The paper accomplishes this using a heirarchical approach with a high-level VLM which uses InstructBlip (+  LLAMA) to provide instruction context. Similarly, a small low-level vision (and proprioceptive) uses this context to output robot actions at a high-control frequency. Experiments are performed on the meta-world and franka-kitchen tasks in simulation and some real-world experiments.
main_review: main_review: **Pros:**

The paper aims to address a challenging problem in trying to use large VLM models for manipulation tasks with fast inference. Overall, the paper is well written and easy to follow. 

**Cons / Questions:**

The overall approach seems reasonable. However, some of the design decisions are not fully ablated and it is unclear how significant these are. 

First, the overall VLM approach to use InstructBLIP’s vision encoder followed with Q-former to get semantic vision-language representations makes sense. However, the query tokens from this model are then further sent through LLAMA (again together with the instruction tokens) to get the final instruction (or context) embedding. Since, the output of Q-former already has some vision-language grounding are all of the LLAMA layers important? or was it empirically found that Q-former’s grounding is not particularly good. The paper also mentions that LoRA was used for updating this VLM model. Can the authors add details on where were the adapter layers inserted (e.g .just for LLAMA or for InstructBLIP as well)?

This approach is somewhat broadly similar to RT-2 (which is the vanilla-VLA baseline according to the paper), wherein images are first processed by a VIT and then processed by a large LLM (together with the instruction tokens). However, importantly, in RT-2 the outputs of the LLM are the actual robot actions (tokenized). By contrast, in the current paper, the output of the LLM (LLAMA) is just the context for the low-level policy. This to me seems like a very important distinction. This is because, since RT-2 uses the LLM as the policy (and co-finetunes) it gets all the robustness and grounding from the VLM. However, in the current paper, the low-level policy is still trained on small robotics datasets and the LLM output is only used as context. Further, it is unclear if the low-level policy’s image encoders are pretrained (maybe the authors can comment on this). But in any case the low-level policy cannot really generalize or have very good semantic grounding since it hasn’t been exposed to large amounts of pre-training data. Further, the policy is being finetuned (without co-finetuning on VLM data) which means that its robustness will be lost. 

Experiments: The above point is also reflected in the experimental section. The paper performs experiments on Metaworld and Franka-Kitchen. Neither of these environment suites test for any semantic generalization. However, the paper does mention that “test on tasks in origin and two new scenarios.” (Line 180). However, no other details are provided. I think this raises many questions —  How were these new scenarios created, what do they test? Is the vision model for low-level policy pre-trained? How do we get semantic generalization when we train our low-level policy on such few tasks. 

Figure-3 in the paper shows some color variations for the franka-kitchen tasks. However, these variations seem minor. Another big problem with the Franka-Kitchen task is that the object positions/poses/geometry is fixed which means that even if semantics are changed the policy completely ignore and just use propriceptive input to complete the task. It is unclear, how any of these generalization questions were evaluated in these settings. 

Dynamic real-world tasks:  The paper seems to mention that the policy is robust to moving the target object at a constant 1cm/s speed. How was this accomplished? It seems from some of the figures that this was done manually (please correct me if I am wrong). If this is indeed true, do you not see issues with the hand making the observations out-of-distribution (or was  data collected in this setting). Additionally, I wouldn’t really call a task where there is a slow-moving target a “dynamic task” since the dynamics of the task haven’t really changed because of motion. The task may require slightly more faster inference but clearly the dynamics of the task are still mostly the same. 

Related work: I think the paper misses some of the related works, especially, which are concerned with dynamic tasks. For example, Saxena et al. look at a very similar problem of using VLMs for real-time control (i.e. fast inference). Similarly, there are other works which focus on using MPC for real-time control (although these works do not focus on VLMs), but I think they should still be cited and provided in context (since there is a huge body of work in enabling real-time control in robotics).

Saxena et al, MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Models

Abeyruwan et al, Agile Catching with Whole-Body MPC and Blackbox Policy Learning
questions: questions: see above
limitations: limitations: 2
review_summary: review_summary: The paper aims to address a very important problem in robotics specifically, how can we get generalization from VLM while having fast inference. It is really unclear to me how generalizable the policies trained in the paper are. More details around experiment design, evaluation are needed to precisely understand the strength and weaknesses of the proposed approach.
title: title: Review of HiRT
paper_summary: paper_summary: This paper presents an implementation of the system1-system2 concept, where a large vision language model is used for high level reasoning and better generalization, while a small model is used for outputting actions. The key appeal to this approach is that one may have the best of both worlds by having better generalization through the pretraining of the large model, while still having fast inference by running the small action model at a fast rate. The method presented in this paper uses a pre-trained 7B InstructBLIP model which is connected to the action model asynchronously through latent embeddings. The authors achieved both generalization and speed in the selected tasks, presenting a convincing case for the effectiveness of this approach.
main_review: main_review: Overall, this paper is well presented and showcases a nice execution of the system1-system2 idea. I appreciate the clear presentation of the method, the experimental results and ablation studies. Specifically,

Pros
- Concise, unambiguous illustrations e.g. Fig.1, Fig.2
- Well presented Section 3 (with some clarification questions below)
- The asynchronous inference setup provides a tunable execution frequency for the VLM, which helps balancing speed and performance given different VLM size, hardware etc.
- Experiment setup is overall relevant and of reasonable difficulty

Cons
- I would appreciate more diverse unseen tasks in both sim and real. This will help provide better understandings on the gains in generalization by using the proposed approach
- One main appeal for the proposed method is the ability to execute dynamic tasks, for which fast execution speed is a necessity. However, the experiments presented in the paper are limited to tracking a moving target object. It would be good to include more realistic tasks
questions: questions: - 134: Which vision encoder is used in the experiments?
- 161-167: Clarification: Are the VLM and action model co-trained where for every training step, gradients are propagated from the action model to the VLM?
- Table 2: How do you explain the poor performance of RT-1 here?
- 222: IC: is the vision encoder completely removed?
- 223: Re: CD, when both conditioned transformer and conditioned MAP blocks are removed/replaced, how does the VLM still connect with the action model?
limitations: limitations: 3
review_summary: review_summary: Overall a well presented paper with strong experimental results. Some additional experimental results on generalization and dynamic tasks will help provide stronger arguments for the advantages of the approach.
title: title: This well-structured paper presents a compelling experimental section and convincing results, demonstrating notable advancements in model architecture.
paper_summary: paper_summary: The paper tackles the issue of slow inference speeds in Vision-Language Models (VLMs) for robotic actions. Using the RT-2 paper as a baseline VLM for robotic actions, the authors assert their approach achieves comparable performance on quasi-static tasks. Their method excels in dynamic manipulation tasks where an object is moved adversarially by an operator, emphasizing the need for fast inference to enable quick replanning. HiRT significantly improves the success rate in these scenarios from 48% to 75%.  Their approach leverages an open-sourced InstructBLIP pretrained VLM to generate embeddings of language-conditioned observations at a low frequency. A separate model with fewer parameters is then conditioned on the latent representation coming from the VLM. This allows the slower VLM to infer the current global state less frequently, while the second model operates at a higher frequency, enabling effective action in dynamic tasks.
main_review: main_review: **Quality and clarity**

The paper is very well-written, striking a balance between clarity and providing sufficient technical detail. The authors' ability to replicate and train nuanced models like RT-1 and RT-2, which demand significant computational resources, is particularly impressive. The Related Works section is also commendable. Some minor details are missing, which I will inquire about in the rebuttal.

**Originality and significance**

Building upon RT-2, a VLM model fine-tuned for robotics actions, the paper proposes a seemingly obvious yet unexplored idea: splitting the model into "slow" and "fast" components. The community may have hesitated to implement this due to the perception that faster inference speed isn't a particularly attractive metric. However, the authors convincingly demonstrate that this approach is beneficial for dynamic tasks where rapid replanning is crucial for success. The insights, proposed architecture, and novelty of this work are valuable contributions to the field.

**Strengths**

The paper presents a well-chosen hypothesis with excellent execution. The experimental section is thorough, and the ablation studies are sufficient. The authors establish strong baselines and comparisons to well-known models (RT-1, RT-2).

**Weaknesses**

Given the formulated problem (best use of a VLM for robotics actions), there are no major weaknesses. Initially, my primary concern was the asynchronous nature of the VLM's interaction with the policy, potentially leading to stale latent representations. The authors address this with Asynchronous Sampling during training, explicitly accounting for this scenario. However, this crucial architectural choice was not ablated, which I recommend investigating.
questions: questions: Hello, thank you for your paper.

I have a few questions and suggestions to further enhance the paper:

1. Performance Comparison: It would be beneficial to include a performance comparison with the diffusion policy, even if it outperforms HiRT. This would provide valuable insights to the research community.
2. Appendix: Do you intend to provide the appendix mentioned in the paper?
3. Video Demonstration: A video showcasing dynamic tasks and a head-to-head comparison of inference speeds would significantly enhance the paper's impact.
4. Data Collection Details: The paper could benefit from more information about the collected data, including the data collection protocol and method (scripted policy or teleoperation).
5. Fast Policy Details: Please provide additional information about the fast policy, such as the number of parameters used.
6. Section 3.3 Clarification: The first paragraph of Section 3.3 could be rephrased to clarify the mechanics of the asynchronous interaction between the VLM and the policy.
7. Asynchronous Sampling Ablation: The paper mentions Asynchronous Sampling in Section 6 to address the issue of stale latents during training. This architectural choice seems crucial for the success of dynamic tasks, yet it wasn't ablated. I recommend including an ablation study for this.
8. Dataset and Model Release: I encourage you to release the dataset and the model. Additionally, consider releasing the replicated RT-1 and RT-2 models.
limitations: limitations: 3
review_summary: review_summary: Overall, I appreciate the proposed paper, its quality, and the results presented. The concept of splitting large VLM models into slow and fast components, while seemingly obvious, has not been thoroughly explored before, to my knowledge. The authors certainly deserve credit for their contribution in this area.  My "Weak Accept" rating stems from the possibility that other reviewers might perceive this as a small, incremental step within the vast design space of VLMs for robotics. Admittedly, from that perspective, this work doesn't represent a significant departure from the ideas presented in RT-1 and RT-2.