Multi-Resolution Sensing for Real-Time Control with Vision-Language Models
Authors: Saumya Saxena
Mohit Sharma
Oliver Kroemer
Venue: CoRL 2023 Poster (2023)
Keywords: Manipulation
Learning for manipulation

Abstract: Leveraging sensing modalities across diverse spatial and temporal resolutions can improve performance of robotic manipulation tasks. Multi-spatial resolution sensing provides hierarchical information captured at different spatial scales and enables both coarse and precise motions. Simultaneously multi-temporal resolution sensing enables the agent to exhibit high reactivity and real-time control. In this work, we propose a framework for learning generalizable language-conditioned multi-task policies that utilize sensing at different spatial and temporal resolutions using networks of varying capacities to effectively perform real time control of precise and reactive tasks. We leverage off-the-shelf pretrained vision-language models to operate on low-frequency global features along with small non-pretrained models to adapt to high frequency local feedback. Through extensive experiments in 3 domains (coarse, precise and dynamic manipulation tasks), we show that our approach significantly improves ($2\times$ on average) over recent multi-task baselines. Further, our approach generalizes well to visual and geometric variations in target objects and to varying interaction forces.

Multi-Resolutton Sensing for Real-Time Control with Vision-Language Models
Saumya Saxena "t,, Mohit Sharmai', Oliver Kroemer Robotics Institute, Carnegie Mellon University! (saumyas, mohitsl, okroemer @cmu. edu
Abstract: Leveraging sensing modalities across diverse spatial and temporal res- olutions can improve performance of robotic manipulation tasks. Multi-spatial resolution sensing provides hierarchical information captured at different spa- tial scales and enables both coarse and precise motions. Simultancously multi- temporal resolution sensing enables the agent to exhibit high reactivity and real- time control. In this work, we propose a framework for learning generalizable reguage--cnditionee multi-task policies that utilize sensing at different spatial and temporal resolutions using networks of varying capacities to effectively per- form real time control of precise and reactive tasks. We leverage off-the-shelf pretrained vision-language models to operate on low-frequency global features along with small non-pretrained models to adapt to high frequency local feed- back. Through extensive experiments in domains (coarse, precise and dynamic manipulation tasks), we show that our approach significantly improves (2x on av- erage) over recent multi-task baselines. Further, our approach generalizes well visual and geometric variations in target objects and to varying interaction forces.
Introduction
Performing robotic manipulation tasks in the real world often requires using sensing modalities at different spatial resolutions. For instance, for peg-insertion, the robot can use a statically matucelly-mounted third-person camera (low spatial resolution global information) reach close to the hole, use wrist-mounted first-person camera for finer alignment, and finally use proprioception and force- feedback for insertion (high spatial resolution or local information). Additionally, each sensing modality can be utilized at a different temporal resolution. For example, for coarse quasi-statie subtasks ("reach hole"), using third-person camera images at a low frequency can be sufficient. However, finer reactive subtasks ("insert peg"), might require high-frequency force-torque feedback. Based on this insight, we propose multi-resolution (spatial and temporal resolution) sensor fusion approach for coarse quasi-static as well as precise reactive manipulation tasks.
Multi-resolution sensor fusion can enable generalization to novel visual-semantic targets. For in- stance, by utilizing global information from third-person camera images only for coarse localization and relying on local information from in-hand cameras and force-torque feedback for finer motions, the policy can learn to generalize to novel objects. Previous approaches to learning generalizable policies either require extensive data collection [1, 2, 3] or rely on pretrained models [4, 5, 6, 7] [8]. However, such approaches typically utilize single sensory modality, while others that incorporate multiple sensors do not prioritize generalization [9]. In our work, we avoid extensive data collection and instead leverage pretrained vision-language models in our multi-resolution approach to learning generalizable canguage-conditionn multi-task policies.
for policy adaptation
Although pretrained vision or lision-language models (VLMs) provide impressive generalization capabilities and enable learning language-condiing conditiood multi-task policies, using large VLMs can have certain disadvantages. First, given their large size (e.g. Flamingo has 80B parameters [6]), they
' equal contribution.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

rultiresollaionn pulicy
Multiressoation tasks
Generalization
Thir pinze
Coarse
Shapp
Protzined
manipulation
Trandoma
Precise
Fisd
Hz Q
ResNetl8
Pollcy head
action
Hir
manipulation
"Juuer elack nog"
pag"
Dynamic manipularios
Pravtssopnin Farcedaqua
MLP
Real-time control loop
Figure 1: Our proposed approach uses sensing different spatial and temporal resolutions for real time control of coarse, precise and dynamic tasks while enabling generalization novel visual features and interactions.
have slow inference which makes them unusable for real- closed-loop control which is necessary for reactive tasks. Second, since pre-trained models are often trained on out-of-domain data, using them to solve in-domain manipulation tasks (especially precise tasks) may require finetuning [10]. However, task-specific finetuning can make models less robust with reduced generalization [11].
To overcome the above challenges of utilizing large pretrained VLMs for real-time control of reac- tive tasks, we propose a framework that incorporates different capacity networks (that operate on dif- ferent sensing modalities) at different frequencies. Specifically, we use large pretrained VLMs with slow inference at a lower frequency while small networks with fast inference Our fow-frequency pretrained VLMs operate on statically mounted third-person views and can pro- vide global coarse feedback (such approximate object locations) that is usually only needed at low rate. On the other hand, we propose using small frrasscccr models with first-person camera views and force-torque data to obtain the high-frequency fine-grained feedback necessary to perform precise and reactive tasks. Further, to overcome the challenge of loss in generalization when finetuning pre-trained VLMs, we freeze the pretrained VLMs to avoid losing their robustness and maintain their generalization abilities. Overall main contributions include:
higher frequency.
a framework for learning generalizable multi-task policies that incorporates multiple sen- sory modalities to capture global to local spatial information,
combine sensor modalities at different frequencies to avoid bottlenecks and enable reactive control which we show empirically is essential for dynamic tasks,
comprehensive experiments across domains (and 2 real-world tasks) that include coarse, precise and dynamic manipulations tasks, and
effective generalization across semantic task variations in both simulation and real- world.
Related work
Vision-Language Pretrained Models for Robot Manipulation: Many prior works combine vi- sion and language for robotic tasks. While early works focus on tabula-rasa learning [12, 13, 14], more recent works, use pretrained large language models (LLMs) and show efficient learning and improved generalization for robotics tasks [15, 16. 17, 18, 19]. Many recent works also combine large general-purpost pretrained vision or vision-language models (VLMs) [4, 6, 20] for manipula- tion [21, 22, 8, 10, 23, 24, 25, 26, 27]. Our work is more closely related to these latter works in that we also use pretrained VLMs for robot manipulation. Among these works, many works only use language for ssssfppeciiiiation and do not focus on the generalization provided by pretrained mod- els [26, 27]. Additionally, other works adapt the pretrained representation for the downstream task [24, 10, 28). However, as we show empirically, such updates lead representation drift and loss of robustness for the pretrained general-purpose VLM. Hence, we propose not updating the pretrained representations. While [25, 8] frozen VLMs, [25] only uses pretrained VLM an open- world object detector to get pixel targets for the task at the first episode step. On the other hand, [8] uses

---

Thrad
Visson +
Amunioon
Hi
Langunge
Cros
PPick block"
Files
Vimet
ResNet18
Afteniion
Croo
PlNy MLP
actions
20
Proprioccption Force-tonque
MLPP
75
Figure 2: Overall architecture: Global low frequency information is extracted from third-person camera im- ages using slow inference networks, local high frequency information is extracted from first-person camera images and proprioceptive, foree-torque feedhack using fast inference networks. These sensing modalitics then fused different frequencies enable real time high frequency control.
the pretrained VLM with templated pick-and-place actions for manipulation. By contrast, we use VLMs in our multi-resolution framework with continuous feedback for reactive manipulation tasks.
Multi-Spatial Resolution for Robot Manipulation: Many prior works use multiple sensor modal- ities for robot manipulation, wherein each modality operates at a different spatial resolution. For instance, prior works often combine visual (low spatial resolution) and proprioceptive (high spatial resolution) feedback [29, 30, 31], use wrist-mounted cameras for visual servoing [32, 33, 34] or for contact-rich manipulation tasks [35, 36, 37, 38), while other works focus on combining vision and haptic sensing [39, 40, 41, 42]. Our work is similar the first set of works i.c. we use both third person and first person cameras for precise manipulation. However, unlike most prior works [35, 38] which focus single-task settings, focus multi-task settings and fuse multiple sens- ing modalities at different resolutions.
Multi- Tempo-ll Resolution for Robot Manipulation: Learning reactive policies requires the robot operate high frequencies. Some recent works in robot manipulation focus on learning policies at different temporal resolutions. For instance, [43] decompose manipulation task into different phases (e.g. visual reaching phase and tactile interaction phase) and learn separate policies for each phase well as blending policy. While [44] avoid the discrete formulation of an MDP and instead learn continuous differential equation [45, 46] to model the low resolution features. By contrast, we use the discrete formulation and instead of decomposing policies into different phases we reuse features from low-resolution signals while operating at a high temporal resolution.
Dynamic Reactive Manipulation: Many prior works in robot manipulation focus quasi-static tasks [17, 1]. However, there has been increased interest in solving tasks that are reactive and dynamic in nature [47, 48, 49]. Previous works focus explicitly learning the dynamics [49] or using analytical models [47, 50] of such systems for achieving reactivity. These works often assume access to the ground truth object pose and are limited to a single-task setting. In our work, we learn how to perform such dynamic and reactive tasks using visual inputs multi-task setting.
3 Proposed Approach
In this section, we discuss our approach for learning generalizable language-conditioned multi- resolution multi-task policy for precise and reactive manipulation tasks. Below, we provide details on how we utilize different sensing modalities and then delineate our training/inference and discuss how our approach enables real time control for reactive tasks while generalizing novel tasks.
3.1 Multi-Resolution Architecture
Figure shows the architecture of our multi-resolutie approach. Our model takes as input multi- ple sensing modalities with different spatial resolutions, i.e., statically-mounted third-person cam- era view, first-person camera view and high frequency force-torque feedback. Each input is first processed separately before being fused together different temporal resolutions to output high frequency robot actions. Below we expand on each component of our architecture.

---

Square Insert
Pick & Lift Small
Pickup Blocks
Shape Sort
Take
Insert Blocks
MT-Precise
MT-Dynamic
MT-Coarse
Real-Worid
Figure 3: Task settings for evaluating our proposed approach. Left: Precision tasks. Middle-left: Dynamic lasks. Middle-right: Coarse tasks. Right: Real world pick and insertion tasks.
Low-Spatial Resolution Model: We use low-spatial resolution sensor (third peridd camera) to provide global task information our agent. We use pretrained visual-language models extract this global information from third-person views well as to enable langage-conditionigg in multi-task setting. Such pretrained models enable generalization to novel semantic features such as new objects or novel language commands. However, to ensure the pretrained model maintains its robustness we keep it frozen. However, using large VLMS to extract this generalizable global information comes with the drawback that the inference speed very slow (= 5Hz). We experiment with two models CLIP [4] and MDETR [51] (languag-coondttioned DETR [52]), which use image- level and object-level information respectively.
High-Spatial Resolution Model: To ensure reactivity in the face of slow inference of pretrained VLMs, we use smaller non- non-pretrained vision model (ResNet-18) [53] to process the first-person camera view at higher frequency ( 20Hz). This view provides us with high-resolution local spa- tial information. To provide appropriate task-context to the first-person view we use small FiLM lay- ers [54] for language conditioning. We train this model from scratch with augmentations (explained in the next paragraphs) extract local spatial features that useful for precise tasks. While using small vision model enables faster processing it can still be insufficient for some highly dynamic tasks. Hence, we process the force-torque feedback and proprioceptive information at a much higher frequency (+= 75Hz) using a small linear layer.
We combine local and global sensing information (spatial reso- lutions) mentioned above at different temporal resolutions based on the capacities of the respective networks. Specifically, we reuse features (network activations) from lower frequency (third-person and first-person views) networks to match the frequency of the highest frequency (force-torque feed- back) network. Doing this ensures that the policy network outputs actions at a high frequency (equal to the frequency of the force-torque feedback network), thus enabling real- miletime control.
Restlttenn Sensor Fusion:
In addition to temporal-sensee fusion we also spatially fuse local and global sensing information, i.e, fuse information extracted from third-person views with first-person view information and vice- versa. We achieve this using two small camera-specific transformers together with cross-attention. Each transformer uses self-attention within each modality (for its associated camera view) and cross- attention with the other modality (other camera view). As shown in Figure 2, we readout the CLS token from each transformer and concatenate them with the force-torque and proprioception embed- ding. This concatenated embedding is then processed using a 2-layer MLP policy head to output the robot actions. Please refer to the Appendix for further details on the architecture.
Data Augmentations: Data augmentations have been shown to be helpful for single-task learning of manipulation tasks [55, 38). However, naively using image augmentations can be detrimental for learning generalizable multi-task policies. This is because pixel-level augmentations, such as color- jitter, grayscale etc., can result in semantic changes in the overall scene. Such semantic changes can lead to mismatch between the input image and language instruction provided for the given task. For instance, demonstration shows "move red block" but pixel augmentations can change the red block's 's color. To avoid this while being able utilize the benefits of augmentations we propose to two different sets of augmentations. First, for third-pprssa cameras only use image-level aug- mentations (c.g. random crops, shifts). This avoids mismatch between image-and-tex instructions

---

and allows visual- -anguaee grounding from pretrained VLM to be utilized. Second, for first first-person camera we use both image-level and pixel-level augmentations (color-jitter, grayscale). Since these augmentations lead to image-text mismatch this further enforces our agent to use the third-person camera view for coarse localization, while only relying on the in-hand view for finer precise mo- tions. Using strong pixel-Ievel augmentations on first-person view further make the in-hand model invariant to texture but rely more on edges and corners [56]. This, the generalization performance of our model on heldout object variations.
show empirically, improves
Training and Inference: We use behavior cloning from expert demonstrations to train our model. We record data from each sensor at their respective frequencies. Specifically, camera images are recorded at 30 Hz and trreetorque feedback at 250Hz. To match slower processing times of larger models during inference we sub-sample the third-person camera images to 5Hz and first-person camera images 20Hz. We use AdamW [57] optimizer with learning rate e-4 and weight decay 0.01. We train our model for 60 epochs, using a linear warmup, starting with learning rate O, for
for 5 epochs and then decay the learning rate using a cosine-scheduler We use inference. Overall our architecture has R 250M parameters. The pretrained vision-language model has ~ 150M parameters (for MDETR) with an inference time of Rs 0.1 seconds. The first-person camera model has 2 25M parameters with an inference time of 0.04 seconds. Finally, the force- torque and proprioception model along with the policy head have a total of z 250K parameters with an inference time of ~ 0.005 seconds. This allows the actions to be inferred a max frequency of R 200Hz although we use it at reduced frequency of x 75Hz which was sufficient for our tasks.
Experimental Setup
We first identify the key research questions that we aim to evaluate:
Q1: How does multi-spatial resolution sensing benefit learning language-condee conditioned multi-task (MT) manipulation polices for precise tasks? Specifically, we aim evaluate the utility of multi- spatial resolution sensing for tasks that involve visual occlusions, partial observability, and precision.
Q2: How does multi-temporal resolution sensor fusion benefit learning reactive manipulation tasks? Specifically, we evaluate how our architecture enables closed loop control for reactive tasks.
Q3: How well does our approach generalize tasks with novel visual-semantic targets? Specifi- cally, we evaluate our approach's 's robustness to distribution shifts, e.g., object colors and geometries.
4.1 Environments
To evaluate the above questions we use three task settings, 1) MT--Preiis:: Precise manipulation tasks, 2) MT-Dynamic: Dynamic manipulation tasks, and 3) MT-Coarse: Coarse table-top manip- ulation tasks. Below we detail each environment and discuss its usage to answer above questions.
MT-Precise For precise manipulation we use 4 spatial precision tasks from RLBench [58] (see Fig- ure 3 (Left)) sqqure block insertion, pick up small objects, shape sorting, and unplug usb. We use this task domain to answer Q1. Specifically, we evaluate the need for multi-spatial resolution sens- ing in manipulation tasks that require precise feedback and have partial observability, i.c., objects can go out of view of the first-person camera.
MT-Dynamic: We use the CMU ballbot [59] platform to perform dynamic pickup tasks in sim- ulation (Figure 3 (Middle-Right)). We choose ballbot since it is highly dynamic robot with an omnidirectional base (ball) capable of performing fast, reactive and interactive tasks. We consider the task of dynamically picking up an object, which requires quick reaction to contact with the object and grasping it to prevent toppling the object over. We use this setting to answer Q2.
MT-Coarse: We consider a canonical table-top manipulation setting ([60, 61)) involving coarse pick-and-place manipulation tasks with diverse objects blocks, shoes, mugs, cylinders. We use this environment to answer Q1 and Q3. Specifically, for Q1 we contrast these coarse manipulation tasks with high precision tasks to evaluate the utility of multi-spatial resolution sensing.
5

---

Frosen
Finctuned
Q
VLM
SA
SA
B3
VLM
VLM
"Issirsctio"
CA
Ho
RadNett
Reahetl
CA
MLP
31tr
MLP
FT
Ni+
MLP
MLP
Regh
Temporal resolution baselines
Robustaess baselines
Figure 4: Temporal resolution and robustness baselines used to compare multi-resolution approach.
World Setup: We evaluate our approach two real-world tasks. For precise manipulation (Q1) we use an insertion task to insert different blocks into cylindrical pegs (Figure (Right top)). We also evaluate generalization abilities (Q3) using a pickup task, wherein we use train objects and evaluate the learned policy on 8 objects with different geometry (shape, size) and visual (color, texture) features. Additional details on each environment are provided in Appendix
4.2 Baselines
We compare our approach against recent methods which focus on learning generalizable policies in multi-task settings. We compare against RT-1[1] which proposes a transformer based policy and also against BC-Zero [2] which uses language conditioning using FiLM [54]. However, both [1, 2] focus on coarse manipulation tasks and operate at a single-resolution (both temporal and spatial). To the best of our knowledge prior work focuses on Hence, highlight the benefit of each component of our approach and answer the questions posed in Section 4 we modify our approach along different axes and propose additional baselines below.
approach for multi-task learning.
Spatial Resolution baselines: To verify the utility of multiple spatial resolutions (Q1) we modify our approach and remove one sensory modality a time. We use --th- T- T- FT refer to policies which remove first-person (hand view), 3rd person view and force-torque respectively.
Temporal Resolution baselines: To answer Q2 we compare against single comporal-resolutin approaches (Figure 4 (Left)), i.e., where all modalities (including force-torque) operate at the same frequency. introduce two baselines, 1) Thighrres : small models with fast inference for both cameras (20Hz), and 2) Tom-rrs larger models with slow inference for both cameras (5Hz).
Robustness baselines: We compare visual-semantse generalization ability of our approach (Q3) against two baselines (Figure (Right)): 1) Fm. Finetune the pretrained VLM model, 2a) TiS-Frozen Uses only third-person camera (and force-torque) and keeps the pretrained model frozen. 2b) WBI.T: Uses only third-person camera (and force-torque) but finetunes the pretrained model.
Metries: We use task success as the evaluation metric and report mean success over all tasks. During training. we evaluate the policy every epochs and report average top-5 mean success rates across all evaluation epochs. For task generalization (Q3) evaluate the train policy novel sesuati tasks not seen during training. For all evaluations we use 20 rollouts per task. Further training details are provided in Appendix B.
Experimental Results
First, we evaluate the effectiveness our multi-resolution approach against common multi-task baselines, RT-1[1] and BC-Zero[2]. We then present results for each research question. For qualita- tive results see: https://sitee http:://sites.google. google.
real temall-mmmecooottt control..
5.1 Comparison to Multi-Tass Baselines
Table 1 shows the results for the multi-task baselines RT-1[1] and BC-Zero[2] across all task We note that for coarse manipulation tasks (MT-Coarse) these baselines, that use single cam- era views, can perform quite well. This because these tasks only require coarse local- ization of the target object for task completion. However, for precise manipulation tasks (MT-Precise), such baselines perform quite poorly since these tasks require fine-grained grasp- ing (as many objects z 1cm in size) and insertion for successful task completion.

---

R-Th
-1
T- FT
Ours
now-ses
Thigh-res
Ours
MT-Coarss 74.5
41.0
81.8
82.0
MT-Coarse
82.0
81.0
82.0
MT-Precise 7.7
29.6
56.1
55.0
MT-
53.4
56.2
55.0
MT-Dynamic 65.8
27.5
33.2
73.6
MT-Dynamic 4.2
12.2
73.6
Table 2: Results for multi-spatial resolution experi- ments (Section 5.2). Here, - implics that we remove this input from policy. Thus, Æー implies that the policy only operates third-person camera views and force-torque feedback.
Table 3: Results for multi-temporal resolution periments (Section 5.2). Here, both Mow-re and Thigh-ss are single-resolution approaches which run at 5 Hz and 20 Hz respectively, while ours is a multi-resolution approach.
Xip--Phoenn
Mmobi-ooPP
Ours
MT-Coarse (Visual)
74.5/7.1
8
82.0/72.3 82.4/45.6
MT-Course (Geometry)
44.2/16.
56.4/184
60.7/3199 56.4/319
38.9/44.6
MT-Precise (Visual) 7.5/4.5
15.6/9.2
55.0/48.1
Table 4: Robustness experiment results, each cell shows train/heldout success rate (Section 5.2)
domains. On the other hand, our multi mustiiresolution approach, performs much better as it uses the first- person camera view and force-feedback for finer grasping and insertion. For dynamic tasks (MT- Dynamic), our method considerably outperforms the baselines (1.5x). This is because dynamic tasks require reactive response to contact events. Only our multi-temporal resolution approach uti- lizes high spatial and temporal resolution sensing, enabling fast response to contact events.
MT-Coarse
MT-Precise
MT MT-Dynamic
RT-1
81.0
12.5
4.5
BC-Z
74.1
7.8
4.8
Ours
82.0
55.0
73.6
Table 1 Task success comparison for multi-task baselines across all task domains.
5.2 Additional Baseline Comparisons
Q1 Spatial Resolution Experiments: We now compare against the spatial resolution baselines discussed in Section 4.2. For this set of baselines all methods use multi-temporal resolution sensing with high-frequency force-torque feedback. Table shows results across all task settings. For MT- Coarse we see that only using first-person camera (m_13) performs poorly. This is because of partial observability in this view, i.e., the target object can be out of view and lead task failure. On the other hand, for MT-Precise (Row 2), only using first-person camera (T- performs better (N 2x) than using only the third- -errdnn camera (π-I1). This is because MT-Precisc tasks require finer motions which are hard to perform from low spatial resolution (hirdd-erson) view only. Further, for dynamic tasks (Row 3), using first- first-person views alone again suffers because of partial observability.
Q2 Temporal Resolution Experiments: Table compares against single-temporal resolution baselines (Flow-es and Thigh-res ). Table shows that for coarse and precise domains single-resolution perform as well as multi-resolution approach. This is because tasks in both domains quasi- static and hence fast reaction to contact events is not critical for task success. On the other hand, for dynamic tasks (Table 2 bottom row), since fast response to contact events is necessary (to avoid failures such as object toppling, see Figure 7 Appendix) our multi-resolution approach performs better than both Tlow-res (5Hz) and Thigh-res (20Hz) since it incorporates force feedback at 75Hz.
Q3 Robustness Experiments: Table 4 compares results (train heldout) for visual-semantie generalization against the robustness baselines in Section 4.2. As noted previously, for these exper- iments we evaluate the trained policies on heldowt environments (see Appendix B.1 for details). We note that our approach, with frozen pretrained model, generalizes better than the finetuned model Fm. This shows the ability of our approach to maintain the generalization capabilities of the pretrained VLM as compared to the finetuned model that suffers from 'forgetting' and representation drift towards the training tasks. Additionally, from column-1 and column-2, we again note that the finetuned NT3-PT model suffers a larger decrease in performance as compared to Mi--Frzzen Finally, comparing тF4T against Tmulti-rs-FF. we see that even with finetuning our multi-spatial resolution approach generalizes better because it can utilize first-person views for improved task success.

---

Tesk 130
Tran
Heldout
Finetuning
No Augmentation
No Cross-Attention
Ablatton using pre-trained CUP model
[c] Robustness results
(a) Ablation results
real-worid Pickup
Figure 5: Left: Ablation results (see Section 5.3). Right: Robustness result for real-world pickup.
Real-World Experiments: We evaluate approach in the real-world two tasks, pickup and peg-insertion [62]. Table 5 shows comparison against the spatial resolution baselines. We note that our approach, with multi-spatial resolution, performs z 3x better than the baselines on both tasks. We see that given limited demonstrations both T-13 and - IA fail to perform well (across both tasks). On the other hand, removing force- torque feedback I- only affects performance on insertion task (<s 20% less) since this task relies more contact feedback. Additionally, Figure 5 (c) figure plots the robustness result for pickup task. As before we see that our ap- proach with frozen model performs better. See website for qualitative results.
T-
T-
W-FT
Ours
Pickup
(3.5)
20.0 (14.1)
67.5 (3.5)
75.0 (7.0)
Peg-Insert
10.0 (0.0)
12.5 (4.6)
42.5 (3.5)
67.5 (3.5)
Table 5: Mean (stdev) results (using 2 seeds) for multi- spatial resolution for real world tasks.
5.3 Ablations
We further ablate the different components of our proposed approach. Due space limitations only summarize key findings and provide details in Appendix C.2.
Pixel-Level Augmentations: For pixel-level augmentations (Figure (a) blue bar) we see little difference training performance but larger increase in generalization performance z 15%.
Spatial Sensor Fusion using Cross-Attention: Figure 5 (a) (green bar) shows that using concate- nation instead of cross-attention reduces performance (=x 10%) on both train & heldout tasks.
Effect of Pretraining: We also evaluate the effects of using pretrained-VLMSs Figure 5 (a) (yellow bar) shows that not using a pretrained model (no vision-language grounding) suffers little drop in train performance but significant drop (3x worse) in generalization, i.e. heldout performance.
Conclusion and Limitations
Our work proposes using sensing modalities at multiple spatial and temporal resolutions for learning multi-task manipulation policies. Our multi-resolution approach captures information multiple hi- erarchies and allows the robot to perform both coarse and fine motions with high reactivity and real- time control. To learn generalizable multi-task policies we further leverage off-the-shel pretrained vasion-lamguage models and freeze them to maintain their robustness. Our work has several limi- tations. While our proposed framework is general for multi-spatial sensing we only rely on global hird-person camera and local first-person camera view. Further local sensing using vibro-tactile sensors [63, 64, 65] was not explored. Further, for sensor fusion will be optimal for more than 2 sensors. Additionally, while our multi-resolution policy allows to leam robust policies not all sensing modalities will be available for all tasks. Thus, future work should explore adapting to scenarios with missing sensing modalities.
unclear our approach of using cross-attention

---

Acknowledgements
This project was supported by NSF Grants No. CMMI-1925132 and IIS-1956163 ONR Grant No. N00014-18-1-2775, ARL grant W911NF-18-2-0211 as part of the A212 program.
References
[1] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, Haus- man, A. Herzog, Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 2022.
[2] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: shro task generalization with robotic imitation learning. In Conference Robot Learn- ing, pages 991-1002. PMLR, 2022.
[3] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06173. 2022.
[4] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning, pages 7748-8763. PMI., 2021.
[5] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
[6] J.-B. Alayrac, J. Donahue, Luc, A. Miech, I. Barr, Hasson, K. Lenc, Mensch, K. Mil- lican, M. Reynolds, al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
[7] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[8] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[9] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. arXiv preprint arXiv:2209.05451, 202..
[10] M. Sharma, C. Fantacci, Y. Zhou, S. Koppula, N. Heess, J. Scholz, and Y. Aytar. Lossless adaptation of pretrained vision models for robotic manipulation. In The Eleventh International Conference on Learning Representations.
[11] M. Wortsman, G. Ilharco, J. W. Kim, M. Li. S. Korbblith, R. Roelofs, R. G. Lopes, H. Ha- jishirzi, A. Farhadi, H. Namkoong, et al. Robust fine-tuning of zero-shot models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959 7971, 2022.
[12] Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 1507-1514, 2011.
[13] M. R. Walter, S. M. Hemachandra, B. S. Homberg, S. Tellex, and S. Teller. Learning semantic maps from natural language descriptions. Robotics: Science and Systems, 2013.
[14] C. Matuszek, Bo, L. Zettlemoye, and D. Fox. Learning from unscripted deictic gesture and language for human-robor interactions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 28, 2014.

---

[15] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209. 11302, 2022.
[16] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng. Do as can and not as i say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.00 01691, 2022.
[17] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Clip on wheels: Zero-shot object navigation object localization and exploration. arXiv preprint arXiv:2203.1042. 2022.
[18] J. Liang, W. Huang, Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07553, 2022.
[19] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303. 12153, 2023.
[20] A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela. Flava: A foundational language and vision alignment model. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 15650, 2022.
[21] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprini arXiv:2203. /2601, 2022.
[22] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. arXiv preprint arXiv:2203.03580 2022.
[23] K. Zheng, X. Chen, O. C. Jenkins, and X. E. Wang. Vimbench: A compositional benchmark for sasion-and-languae manipulation. arXiv preprint arXiv:2206.08552, 2022.
[24] T. Xiao, H. Chan, P. Sermanet, A. Wahid, A. Brohan, K. Hausman, S. Levine, and J. Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:221 11766, 2022.
[25] A. Stone, Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, B. Zitkovich, Xia, C. Finn, and K. Hausman. Open-world object manipulation using pre-trained vision- language model. In arXiv preprint, 2023.
[26] O. Mees, L. Hermann, E. Rosete- Beas, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3)773277334, 2022.
[27] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 120-112205-112.. 2022.
[28] M. Sharma, C. Fantacci, Y. Zhou, S. Koppula, N. Heess, J. Scholz, and Y. Aytar. Lossless adap- tation of pretrained vision models for robotic manipulation. arXiv preprint arXiv:2304.06600 2023.
[29] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-133,, 2016.
10

---

[30] D. Kalashnikov, A. Irpan. P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
[31] A. X. Lee. C. Devin, Y. Zhou, T. Lampe, K. Bousmalis, T. Springenberg, A. Byravan, A. Abdolmaleki, N. Gileadi, D. Khosid, C. Fantacci, E. Chen, A. Raju, R. Jeong, M. Neunert, A. Laurens, S. Saliceti, F. Casarini, M. Riedmiller, R. Hadsell, and F. Nori. Beyond pick-and- place: Tackling robotic stacking of diverse shapes. In Conference on Robot Learning (CoRL), 2021. URL httpp:///ppeaeeeee wttp://wwww.....o.mmmmoooooooo/o
[32] B. Yoshimi and K. Allen, Active, uncalibrated visual servoing. Proceedings of the 1994 IEEE International Conference on Robotics and Automation, pages 156-161. IEEE, 1994.
[33] D. Kragic, H. I. Christensen, et al. Survey visual servoing for manipulation. Computational Vision and Active Perception Laboratory, Fiskartorpsv, 15:2002, 2002.
[34] B. Nelson, J. D. Morrow, and P. K. Khosla. Improved force control through visual servoing. In Proceedings of 1995 American Control Conference-ACC"95, volume 1, pages 380-386. IEEE, 1995.
[35] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:/707.0833 2017.
[36] A. S. Morgan, B. Wen, J. Liang. A. Boularias, A. M. Dollar, and K. Bekris. Vision- driven compliant manipulation for reliable, high-precision assembly tasks. arXiv preprint arXiv:2106.1 14070, 2021.
[37] E. Johns. Coarse-to-fine imitation learning: Robot manipulation from single demonstration. In 2021 IEEE international conference on robotics and automation (ICRA), pages 4613 4619. IEEE, 2021.
[38] O. Spector and D. Di Castro. Insertionnet-a scalable solution for insertion. IEEE Robotics and Automation Letters, 6(3):5509-5516, 2021.
[39] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International Conference on Robotics and Automation (ICRA), pages 8943-8951 IEEE, 2019.
[40] N. Fazeli, M. Oller, Wu, Z. Wu, J. B. Tenenbaum, and A. Rodriguez. See, feel, act: Hicrar- chical learning for complex manipulation skills with multisensory fusion. Science Robotics, (26))aaav3123 2019.
[41] R. Calandra, A. Owens, D. Jayaraman, J. Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4):3300-3307, 2018.
[42] Q. Li, O. Kroemer, Z. Su, F. Veiga, M. Kaboli, and H. Ritter. A review of tactile informa- tion: Perception and action through touch. IEEE Transactions on Robotics, 1666):1619-1633 2020.
[43] T. Narita and O. Kroemer. Policy blending and recombination for multimodal contact-rich tasks, IEEE Robotics and Automation Letters, 6(2):2721-2728, 2021.
[44] S. Singh, F. M. Ramirez, Varley, A. Zeng, and V. Sindhwani. Multiscale sensor fusion and continuous control with neural edes. In 2022 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 10897-10904, IEEE, 2022.

---

[45] K.-i. Funahashi and Y. Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural networks, 6(6):801-806, 1993.
[46] P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
[47] J. Shi, J. Z. Woodruff, P. B. Umbanhowar, and K. M. Lynch. Dynamic in-hand sliding manip- ulation. IEEE Transactions on Robotics, 33(4))777-775, 2017.
[48] C. Mucchiani and M. Yim. Dynamic grasping for object picking using passive zero-dof end- effectors. IEEE Robotics and Automation Letters, 6(2):388-3006. 2021.
[49] S. Saxena, A. LaGrassa, and O. Kroemer. Learning reactive and predictive differentiable con- trollers for switching linear dynamical models. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 7563-7569. IEEE, 2021.
[50] R. Shu and R. Hollis. Momentum based whole-body optimal planning for a single-spherical wheeled balancing mobile manipulator. In 2021 IEEE/RSJ International Conference on Intel- ligent Robots and Systems (IROS), pages 3221-3226. IEEE, 2021.
[51] A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEF/CVE Inter- national Conference on Computer Vision, pages 1780-1790, 2021.
[52] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end ob- ject detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt I 16, pages 213-229. Springer, 2020.
[53] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[54] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[55] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 19:14494-18895, 2020.
[56] N. Somavarapu, C.-Y. Ma, and Z. Kira. Frustratingly simple domain generalization via image stylization. arXiv preprint arXiv:2006. 2020.
[57] D. P. Kingma and Ba. Adam: method for stochastic optimization. arXiv:1412.6980, 2014.
arXiv preprint
[58] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
[59] U. Nagarajan, G. Kantor, and R. Hollis. The ballbot: An omnidirectional balancing mobile robot. The International Journal of Robotics Research, 33(6):117-93, 2014.
[60] T. Yu, D. Quillen, 7. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning, pages 1000. PMLR, 2020.
[61] Y. Zhou, S. Sonawani, M. Phielipp. S. Stepputtis, and H. B. Amor. Modularity through atten- tion: Efficient training and transfer of language-conditiones policies for robot manipulation. arXiv preprint a:X1r:2112.04573, 2022.
12

---

[62] K. Zhang, M. Sharma, J. Liang, and O. Kroemer. A modular robotic arm control stack for research: Franka-interface and frankapy. arXiv preprint 2tt::::01110223900 2020.
[63] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017. 2017.
[64] A. Yamaguchi and C. G. Atkeson. Combining finger vision and optical tactile sensing: Re- ducing and handling errors while cutting vegetables. In 2016 IEEE-RAS 16th International Conference Humanoid Robots (Hummanoids), pages 1045-1051, IEEE, 2016.
[65] K. Zhang, M. Sharma, M. Veloso, and O. Kroemer. Leveraging multimodal haptic sensory data for robust cutting. 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), pages 409-416. IEEE, 2019.
[66] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh, and V. Vanhoucke. Google scanned objects: high-quality dataset 3d scanned household items. 2022 International Conference on Robotics and Automation (ICRA), pages 2553- 2560. IEEE, 2022.
[67] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/SI International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012. 11011..110//RR0222222..3330
[68] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. attp://pybullet org, 2016-2022
[69] Lüddecke and A. Ecker. Image segmentation using and image prompts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7086-7090 2022.
13

---

A Environment Details
In this section we provide further details on the different environments used in our experiments.
A.1 MT-Coarse Manipulation
For coarse manipulation tasks we focus on variety of objects including blocks, mugs, cups, and shoes (both men and women shoes). As noted in the main paper, for these set of objects we focus on pick-and-place skills. However, we note that did experiment with more complex contact-rich skills (e.g. pushing, stacking). However, we found the physics to be unstable with more complex objects (e.g. cups). For instance, pushing cups would almost always topple them and roll over. For future work, we hope to make our skills more robust.
Specifically, we use fixed size blocks with different semantic colors, 4 mugs, cups and shoes. We use google scanned objects [66] to collect non-block objects and use mujoco [67] to simulate our environment. We use the latest mujoco environments to import meshes into the simulator. Each environment in this set of tasks is created by first selecting a target object-type and then selecting a target object from the set of objects. We then select 3-5 distractor objects to fill the scene. These objects are uniformly selected from the remaining objects.
A.2 MT-PPrecie Manipulation
As noted in the main paper for precise manipulation tasks we use the spatial precision set of tasks from RLBench [58]. Overall, use 4 tasks (see Figure 3 (Left)) square block insertion, pick up small objects, shape sorting, and unplug usb from computer. We avoid using the motion-planner augmented approach for solving these tasks and instead opt for learning reactive closed-loop control policies. We use the delta end-effector actions for our tasks. Additionally, we use standard front and wrist mounted camera. along with proprioceptive and force-torque feedback as policy input.
However, directly using end-effector actions increases the policy horizon significantly. Moreover, naively using the original input distribution for each task also requires learning full 6-DOF policies. Both of these can significantly increase the data requirements to learn the manipulation policy. To avoid this we restrict the starting distributions for each task such that the objects are spawned in slightly narrow region infront of the robot. We further make other task-specific changes, detailed below, such that the robot can perform each task without changing hand orientations.
Insert Onto Square Peg: For this task restrict the orientations of the square ring (blue object) and the peg on which to insert. This allows the robot to perform the task without changing gripper orientations. Further, we use region of 40cm x 30cm infront of the robot to spawn both the base and ring. Finally, the default task configuration provides 20 different peg colors, of which we use the first 10 colors for training and remaining 10 colors for robustness experiments.
Pick and Lift Small: For this task, again a region of 40em 30cm infront of the robot to spawn both all objects. We also restrict the orientation of each object such that it can be grasped directly without requireing gripper orientation changes.
Shape-Sorting: The default configuration for the shape-sorting task considers different shaped objects (see Figure Bottom-Left) - square, cylinder, triangle, star, moon. In the default RLBench configuration most objects directly stick to the robot finger and are simply dropped into the hole for task completion. However, with closed loop control we find that non-symmetric objects (star, trian- gle, and moon) can have significant post-grasp displacement such that it is impossible to insert these objects without changing gripper orientation. Hence, we exclude these two objects from evaluation and only use symmetric square and cylinder objects.
Take USB Out: This task requires the robot to unplug a USB inserted into the computer. However, the default configuration for this task requires 6-dof control. To avoid this, we create smaller com- puter and USB assets and mount them vertically on the table such that the USB can be unplugged without changing hand orientation. See Figure 3 (Bottom-Right) for visualization.
14

---

Traie
Test
(8) Examples objects for real-world pickup task
(C) Example objects coarse manipulation task
(A) Real-world setup for pickup and insertion tasks.
Figure 6: Left: Real World env setup with hird-pprsoon (red) and first-person (blue) camera views. Middle: Example objects set used for real-world pickup task. Right: Example objects used for MT- coarse.
A.3 MT-Dynamic Manipulation
This task involves using the CMU Ballbot in simulation (PyBullet [68]) to perform a dynamic pick up task. The task involves picking up a block that is placed on table in front of the ballbot. We use two blocks (red and blue) in this task and use language instructions to specify which object to pick up. The initial conditions are such that the table and objects are always ballbot arms and the ballbot has to roll forward to pick up the objects. We use statically mounted camera looking at the table and the ballbot the third-person camera and the camera located on the turret of the ballbot as the first-person camera. The turret tilt is adjusted such that the objects on the table are initially out of the view of the turret camera and only when the ballbot starts moving towards the table, the objects come into view. The third person camera is always able to view both the objects and the ballbot. We use task space control control the ballbot and-effector while center of mass balancing controller is always running in a figh-freeuency feedback loop balance the ballbot.
of the reach of the
B Architecture Details
Section 3 discusses the overall architecture used in our work. To recall, our proposed architecture uses a multi-resolutio approach with multiple-sensors, each with different fidelity. We process each sensor with a separate network which is conditionally initialized using pre-trained vision- language model. The output of each vision model is flattened to create set of patches. For DETR [51, 52] based model we use ResNet- 101 backbone and flatten the output layer into 49 patches and add positional embedding to it. For CLIP [4] we use ViT-B model and use hierarchical features from the 5'th, 8'th and 11'th layer. Since MDETR already does vision-language fusion using transformer directly use its output. However, since CLIP only weakly associates vision and language at the last layer, we additionally use FiLM layers to condition the output. Our use of FiLM is similar to previous models [69]. For each camera modality we use small transformer with multi-head attention. Each transformer uses an embedding size of 256 and heads. We use post layer-norm in each transformer layer. Further, in each transformer layer we use cross-attention with the other camera. Overall we use transformer layers for each camera modality. Our force- torque and proprioceptive input concatenated together and mapped into 256 dimensions using linear layer. We concatenate the readout tokens from each camera transformer and the force-torque embedding. This 256 x 3 size embedding is then processed by 2 linear layers of size 512 which output the robot action.
Input:
For each of our camera sensor we use an image of size 224 224. For proprioceptive input use the end-effector position of the arm. While for force-torque input use the 6 dimensional force-torque data. We use cropping augmentation for both camera sensors. Specifically, first resize the image to 226 and then do random crop with shift 8. For, more aggressive pixel level
15

---

Key
Value
batch size
16
proprio force torque embedding
comera-transforme embedding Dim.
256
camera-transformer fecdForward Dim.
768
Number of transformer layers
learning rate warmup epochs total epochs optimizer
0.0001
60
Adam
weight decay scheduler
0.01
cosine
Table 6: Hyperparameters used for our architecture and model training.
augmentations we stochastically apply grayscale and use color jitter with brightness E (0.4,0.8) contrast E (0.4, 0.8), saturation E (0.4, 0.6) and hue E 0.5). These augmentations significantly change the underlying visual semantics of the task.
B.1 Training Details
In this section we provide details on the demonstrations (for each environment type) used to train our approach. Further, we also provide details on the train and heldout configurations used for robustness evaluation.
MT-Coarse: As noted above in Appendix A, we use multiple different objects to train and evaluate our policy. Each environment is created by first sampling a target object and then a set of distractor objects. For each environment and skill combination we collect 20 demonstrations. Overall, this gives us R 1000 demonstrations across all tasks. We then learn one policy across all tasks.
MT- MT-Preccse: For spatial precision tasks from tasks from RLBench [58] we use 4 different tasks. As discussed in Section A.2, each task has it's own set of variations. For training our multi-task policy we use try to balance the number of demonstrations from each task. For square peg insertion (insert onto square peg) task we use first 10 variations for training and gather 25 trajectories per variation. Each other task has less than variations hence for each task we use 100 demonstrations each for training. To test sesual-semantic robustness for these tasks Section 5.2 we use the insert- onto-square-pes task since only this task has any semantic variations. We use the remaining peg colors (i.e. 10 heldout variations) test each approach.
MT-Dynamie: To collect expert demonstrations, we sample the locations of the objects on the table in a 72cm+ region and sample the initial ballbot location in a 50cm*50cm region. We collect 50 demonstrations for each task setting (each block). As noted earlier, the third-person camera is used frequency of 5Hz, the turret camera used 20Hz and proprioception and force-torque feedback is used at 75Hz.
Real-World:
For real-world tasks we collect data using teleoperation with leap-motion device which can track hand movements upto 100Hz. We map these movements to robot movements and collect proprioceptive and force-torque data at 75Hz, while both cameras are recorded at 30Hz. To collect data for pickup tasks we use two blocks with different shapes and different colors. The green and pink blocks Figure 6 (Right) were used collect all training data. While evaluation happened 8 other blocks, each with different shape and color. For training our policies collect 60 demonstrations for each pickup variation and 50 demos for the insertion task. We note that the initial state distribution for insertion was narrower than pickup and hence it required fewer demonstrations.
We use task success as the evaluation metric. Since we use a multi-task setting we report mean success over all tasks. During training, we evaluate the policy every 4 epochs on all train tasks. We report the average over top-5 mean success rates across all evaluation epochs. For task generalization results (Q3) we use the trained policy and evaluate it novel visual-semante tasks
Metrics:
16

---

Figure 7: Example failure case for MT-Dynamic (Ballbot) task. As can be seen in the figure, if the robot approaches the object but does not react fast enough to the object contact, the block can topple resulting, in task failure.
which were never seen during training. Hence, for Q3 we report task success on novel unseen tasks. For all evaluations we use rollouts per task. Further training details are provided in Appendix B.I.
B.2 Implementation Details
In this section, discuss our real-robot implementation details. In implementation, the real- time control loop is composed of low-level task-space impedance controller and high-level neural network controller. The low-level controller operates at IKHz using a real-time time kernel and sends control commands to Franka Panda's control interface (FCI) [62]. Our neural-network controller implementation can operate up to a maximum of 100Hz given communication latency. Specifically, for our experiments we run neural network controller at Hz. We use fixed low impedance values (Kp: 350) avoid damaging the robot during fast execution of contact-rich tasks.
Neural network controller implementation: For our real-robot neural-network controller imple- mentation we follow a multi-threaded architecture. Robot state information such proprioceptive data and force-torque data is published at 100Hz, while camera images are published at 30Hz. Each sensor modality is appended a separate fixed size time-stamped buffer. We process each modal- ity independently multi-threaded manner by extracting the latest time-stamped data from respective buffer.
Camera images are processed on separate threads using their respective neural networks and we save the network outputs for future processing. More specifically, we process images from third-person camera using a large VI.M and save a set of visual-language representations from its output in a buffer. This thread is limited by the inference speed the large VLMs and operates at 5Hz. We process the image from the in-hand camera in a separate thread using a small ResNet based model to get hand-camera image representations. On the same thread, further process these hand-camera image representations with the existing cached layers to get multi-modal fused visual-language output which is added to fixed size buffer. This thread operates at 20Hz.
representations using cross-attention
Finally, the high-level neural network controller (which runs on the main thread 75Hz) concate- nates the cached robot state information (force-torque, proprioceptive) with the latest fused multi- modal features. The concatenated features processed through small multi-layer perceptron to get the final action output which is sent to the low-level impedance controller.
C Additional Results
C.1 Additional Real- World Comparisons
In addition to real-world results in Table we also tried out BC-Z and RT-1 on the pickup task the real world. Table 7 reports the average success rate and compares them to our method. We find that BC-Z's performance is much worse than our proposed approach. This is because BC-Z operates single-resolution (both spatial and temporal) as it uses only third-person camera. In the absence
17

---

Setup
BC-Z [2]
RT-1 [1]
Ours
Tow-res
Thigh
Ours
Train
12.5
0.0
75.0
RealWorld Poginsert
45.0
62.5
67.5 5
Eval
5.0
0.0
71.1
Table 8: Additional Results for multi-temporal resolution experiments. As before, both Mow-ves and Thigh ass are single-resolution approaches which run at Hz and 20 Hz respectively, while ours is a multi-resolution approach.
Table 7: Real-World results for using commonly used imitation learning (single- spatial resolution baselines) for Pickup task.
of a first-person camera view often unable to accurately localize the target object and fails to perform the final fine-grained motion grasp the object and lift up. Further, for RT-1 we find the performance to be very poor. We believe this is because RT-1 1 uses tokenized actions which requires us discretize our continuous robot actions. Since we operate in the low data regime (120 trajec- tories) such discretization leads to token imbalances during training and deteriorates the model's performance. Additionally. since RT-1, similar to BC-Z, uses single-resolution (i.e. third-person camera only) we believe its performance suffers from similar challenges of inaccurate localization. Furthermore, we evaluate visual generalization the BC-Z and RT-1 policy on novel unseen objects (and instructions). Since both BC-Z and RT-1 do not use pre-trained vision- langugee model and thus have visual grounding for the text instructions they fail perform well unseen novel objects. By contrast, our approach that utilizes pretrained VLM generalizes well.
C.2 Additional Ablations
We further ablation on the different components of our proposed approach. For these set of results instead of using all 3 environment suites for evaluation, we choose the most appropriate environment suite for each component of our approach and evaluate on it.
Pixel-Level Augmentations: We evaluate the effect of pixel-level augmentations (color jitter, gray- scale) the training and generalization of our MT-policies MT-Coarse. Figure 5 reports results on both training and heldout (novel) evaluation configurations. We see that while there is very little difference in training performance, extensive pixel-level augmentations helps generalization by close to 2 15%. While pixel-level augmentations change the semantics of the task, our multi- modal approach still able to complete the task because of visual-language grounded provided from pretraining.
Multi-Modal Fusion using with late fusion using concatenation. Figure 5 shows that using cross-attention improves the per- formance by around & 8% on both train and heldout configuration. Thus, using cross-attention for multi-modal fusion is more effective than concatenation. However, we note that cross-attention requires more parameters and has slower inference.
We compare use of early fusion using cross-attention
Effect of Pretrained-VLMM. We also evaluate the effects of using Setrained-VLMMs Figure 5 shows the training and heldout performance using ImageNet initialization which only has visual pretraining and no vision-language pretraining. We see that while training performance matches our approach the heldout performance decreases tremendously. This large decrease is due to missing visual-language grounding since we use separately trained visual and language models.
World Temporal- Cemporal-Resolutiin Resolution Comparison: We also ablate the effect of temporal resolutions on real-word robot performance. Specifically, we evaluate single temporal-resolution approaches (Noww-es) and Thigh-res for the peg-insertion task in the real-world. As before, to evaluate the learned policy we run each episode for a fixed duration of 60 seconds. However, we use early termination the episode is solved successfully or the robot violates the desired workspace. Table shows our results. Given that the insertion task is not dynamic, Thigh performs similarly to our approach. However, by comparison, (Mow-ses) performs much more poorly (45% only). This is because a low- temporal resolution policy is not very reactive and hence doesn't 't respond fast contacts made with the wooden peg. Thus, it is often unable to find the appropriate location to insert the block into the
18

---

wooden peg. This can also be seen from qualitative videos (see success and failure videos), where both success and failure scenarios are much less reactive.
Cemporal-Resolutione Finally, we also ablate the temporal frequencies for the MT-Dynamic tasks. We ablate the effect of using camera inputs low-resolution (third-person and in-hand camera inputs at 5H2) while only force-torque feedback used at high-resolution (75Hz).
Table 9 below shows our results. From the table below, we observe that the performance on MT-Dynamic tasks drops significantly when using the camera views very low temporal resolution. From our qualitative observations where the ballbot is sometimes unable to reach the block to pick up. This is because, due to latency in the camera inputs (5 Hz), the policy outputs sub-optimal actions. Upon receiving updated camera inputs the policy tries to correct the trajectory. The overall resulting trajec- tory is noisy and fails to reach the target object. Second, again due to camera latency, the end effector does not align well with the target object and ends up toppling the object while trying to grasp it.
new-res-high FT Ours
33.4
73.6
note common failure cases. First,
Table 9: Results using
low-temporal resolutions for camera-inputs (5Hz) and high- temporal resolutions for force- torque only (75Hz).
19


Reviews:
[{"review_id": "ZRbENpUAA8", "reviewer": "Reviewer_FtkG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies generalization in policies that operate at different spatial and temporal resolutions. The main motivation is that different parts of a task may require different sensors (e.g., third-person camera images for global information versus a wrist-mounted camera for finer feedback) and, may either require high-frequency or low-frequency information processing (i.e., at different temporal resolutions). This work leverages pre-trained vision-language models to process the global information at a lower-frequency to avoid the need for large amounts of data collection.", "summary_of_recommendation": "This work contributes a framework that leverages the strengths of pretrained VLMs while mitigating their weaknesses, i.e., slow inference time and inability to handle non-third person view inputs. The experiments show that augmenting them with trained-from-scratch networks to handle high-resolution spatial and temporal information improves performance on dynamic and precise manipulation tasks.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- How does performance vary with the size of the training dataset?\n- Is there a benefit to processing inputs at a higher frequency than studied right now?\n- In the real robot setup, the robot partially occludes the workspace in the third-person view. Is a first-person view still needed if the camera is placed so that there is less occlusion?\n- Could this approach enable handling of dynamic instructions?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- (Significance) This work presents a framework for tackling dynamic and precise tasks with pre-trained vision-language models. This allows the method to be sample-efficient and tackle challenging robotics tasks.\n- (Originality) This is the first work that I'm aware of that leverages VLMs for dynamic/precise tasks.\n- (Quality) There are some results on a real (albeit, relatively simple) robotic task.\n\nWeaknesses:\n- (Clarity) I think the messaging could be improved. Currently, there are two axes of information discussed: spatial and temporal. However, it ties together the two, i.e., low-resolution spatial info is to be processed at a low frequency, which is not always the case. For some tasks, global information may need to be processed at a high frequency. What would be clearer is to first focus the problem on designing a general-purpose (static) manipulation policy; then motivate the use of VLMs for this domain; then explain the need for high-frequency processing of finer feedback such as force.\n\n- (Significance) While the simulated results are quite varied, the generalization studied in the real robot setup is considerably narrower.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "TKElBrHSZ0r", "reviewer": "Reviewer_pRJN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a framework for learning generalizable language-conditioned multi-task policies that utilize sensing at different spatial and temporal resolutions. The proposed framework consists of two main components: \n\n1. A multi-modal sensory fusion module that combines information from multiple sensing modalities at different spatial and temporal resolutions.\n\n2. A multi-task policy learning module that learns to control the robot to perform a variety of tasks using the fused sensory information.\n\n\nThe authors conducted extensive experiments in three domains of manipulation tasks: coarse, precise, and dynamic. They showed that their approach significantly outperforms recent multi-task baselines.\nThe authors also showed that their approach generalizes well to visual and geometric variations in target objects and to varying interaction forces.", "summary_of_recommendation": "Based on my comments above, I think the paper is of high quality despite lacking some additional comparison. Therefore, I vote for a weak accept.\n\nPost-rebuttal update:\n\nI have read the author response and appreciate the additional experiments. I would like to keep my rating as weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Compare the method to RT-1 and BC-Z in the real world experiments.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The framework proposed by the authors is quite novel and addresses a very important and practical problem, multi-resolution sensing in real robot learning. The empirical results in the work are very extensive and clearly show the advantages of the proposed method. The authors also included rigorous ablation studies to validate the importance of each design choice.\n\nI have one concern regarding the work. I think the authors should also include comparisons to prior works like RT-1 and BC-Z in the real-world evaluations since RT-1 and BC-Z are designed to be real robot learning algorithms. Including such comparisons would make the paper more convincing.1", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "GCu8kM7kdcj", "reviewer": "Reviewer_VHHQ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper has three main ideas around imitation learning: (1) incorporating multiple spatial resolutions (in this case, hand and third person views) is better, (2) incorporating multiple frequencies of control for different modalities (high frequency f/t) is better, and (3) VLMs provide more robust low resolution representations for visual and geometric variations. Their model consists of a low-frequency VLM encoder for the third person view fused with medium-frequency Resnet encoding of the hand camera view using a cross attention mechanism. Additionally, force/torque and proprioceptive info is encoded at high-frequency, and actions are also predicted at this high frequency conditioned on the concatenated outputs of each module. They show experiments in all three areas, with baselines involving changing the input modalities and changing the frequencies of each. There are some real world experiments that ablate only the spatial resolutions.", "summary_of_recommendation": "The three separate claims in the paper could each use a bit more attention, especially the temporal resolution and VLM generalization points. It would also be good if the authors could use a more diverse range of dynamic tasks to test out the benefits of the high frequency inputs, and to try recurrent policy architectures since these are more commonly used in practice.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See above points.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Originality*: To my knowledge, the proposed method is a novel combination of individually well known techniques, but I still believe their analysis to be valuable. The use of hand cameras and third person cameras together is a common choice in the literature.\n\n*Clarity*: The paper is well written and organized, it was simple for me to understand the key contributions. However I do find the paper to be a bit crowded, the temporal and spatial aspects for example are somewhat independent, and I overall think the paper should be separated out and analyzed in more detail for each claim.\n\n*Quality*: The methods and claims are technically sound.\n\n*Strengths*\n- The results support the claims made in the paper. More dynamic tasks require F/T information at a higher frequency, and occlusion is partially prevented with the use of both hand and third person views. Pre-trained VLMs offer better generalization than models trained from scratch or fine-tuned.\n- The authors provided many interesting baselines and a good range of experiments in simulation.\n\n*Weaknesses*\n- I'm curious why the authors did not consider any recurrent architectures for their method? The use of recurrence might affect claims about the need for various temporal and spatial resolutions, and so this would be good to see in the rebuttal.\n- It is unclear to my why the tasks in MT-Dynamic are considered \"dynamic\" given that they are just pick up tasks. The provided justification of potentially knocking over objects in my mind is not a strong enough justification for calling these dynamic. I'd be interested to see more traditionally dynamic tasks, like scooping objects or throwing trash away, or anything that requires complex f/t and end effector signals. In general the provided tasks were somewhat simple.\n- I think there should be a bit more experimentation with temporal frequencies of each input modality. Some examples: what happens when 3rd person is run at 20Hz but 1st person is run at 5Hz (testing if there's any benefit to higher frequency hand camera views)? or when both are run at 5Hz but F/T is still run at 75Hz (testing if we can get away with slower visual perception but faster proprioception)? In general, as a consequence of having three separate main ideas in this paper, I felt like there were a lot of experiments but not enough evidence for each claim (e.g., choice of VLM, where to incorporate it, how to fuse sensory feedback and how that changes at different frequencies, etc.)", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "CFaE-yqKfCl", "reviewer": "Reviewer_7F3D", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This strong paper introduces a framework for learning real-time language-conditioned visuomotor manipulation policies that balance global reasoning with high frequency reactive control. The key insight of this work is simple — different input modalities shape the frequency of control actions in different ways. VLMs provide generalization and global reasoning, but are slow. Features from wrist cameras are less necessary for broad task execution, but require higher-frequency handling to compensate for changes in immediate visual surroundings. Finally, proprioceptive + force-torque features dictate the most fine-grained behaviors during execution, and require higher frequency (but generally “lower capacity”) handling.\n\nResults on precise and coarse manipulation tasks in simulation and the real-world validate the necessity of the multi-resolution handling compared to ablations and state of the art BC algorithms.", "summary_of_recommendation": "This is a clean and strong paper. I advocate for acceptance at this time, with a plan to push for strong acceptance after resolving the above questions!", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Can you provide more details about the implementation / real-robot stack? How are you reading/parallelizing I/O inputs from the various modalities, and how are you processing inputs through each of the different neural modules? What’s the delta in ceiling control frequency if we were to move up to even one generation later GPU (a 2080 Ti)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is extremely well motivated, has crisp hypotheses, a simple implementation, and clean experiments that validate every part of the proposed approach. The RLBench and CMU Ballbot results in simulation are convincing, and are a great way for future work to build on this approach. The real-world experiments while a bit toy (peg insertion, block manipulation) highlight the precise/coarse manipulation distinction, and validate that the timing/multi-resolution nature of the approach holds up in the real-world.\n\nThe ablations on module capacity, effect of pretraining, fusion mechanisms, and data augmentation all work to further complement this paper.\n\n---\n\nThe one concern I have with this work is around the extent of the engineering/push to get temporal resolution experiments running at high frequency. At the scale of the models used (< 300M parameters), I would expect an ability to hit at least 50 Hz control on a 1080Ti with modern CPU compute; if you could run the entire policy at 50 Hz/75 Hz, then the gains seen in this work would probably go away, right?\n\n---\nEDIT (Post-Rebuttal): In light of the new experiments and comparisons, I think this is very strong work, and am raising my score to reflect that!", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received mixed reviews.  After the rebuttal, most reviewers became more positive, though reviewer VHHQ did not participate in the discussion.  The AC has read the paper, reviews, and rebuttals, and found that reviewer VHHQ's concerns are reasonably addressed by the rebuttal.  Therefore, the AC agrees with the majority of reviewers and recommends acceptance.  The authors are encouraged to further revise the paper based on the reviews in the camera-ready version.
Decision: Accept (Poster)