CALAMARI: Contact-Aware and Language conditioned spatial Action MApping for contact-RIch manipulation
Authors: Youngsun Wi
Mark Van der Merwe
Pete Florence
Andy Zeng
Nima Fazeli
Venue: CoRL 2023 Poster (2023)
Keywords: Contact-rich Manipulation
Visual-language guided policies

Abstract: Making contact with purpose is a central part of robot manipulation and remains essential for many household tasks -- from sweeping dust into a dustpan, to wiping tables; from erasing whiteboards, to applying paint. In this work, we investigate learning language-conditioned, vision-based manipulation policies wherein the action representation is in fact, \textit{contact itself} -- predicting contact formations at which tools grasped by the robot should meet an observable surface. Our approach, Contact-Aware and Language conditioned spatial Action MApping for contact-RIch manipulation (CALAMARI), exhibits several advantages including (i) benefiting from existing visual-language models for pretrained spatial features, grounding instructions to behaviors, and for sim2real transfer; and (ii) factorizing perception and control over a natural boundary (\ie contact) into two modules that synergize with each other, whereby action predictions can be aligned per pixel with image observations, and low-level controllers can optimize motion trajectories that maintain contact while avoiding penetration. Experiments show that CALAMARI outperforms existing state-of-the-art model architectures for a broad range of contact-rich tasks, and pushes new ground on embodiment-agnostic generalization to unseen objects with varying elasticity, geometry, and colors in both simulated and real-world settings.

CALAMARI: Contact-Aware and Language conditioned spatial Action MApping for contact-RIch manipulation
Youngsun Wi
Mark Van der Merwe' Andy Zeng² Pete Florence2
Nima Fazeli
"Roootic Department, University of Michigan Google Deepmind
(yswi, nfz)iumich. edu (andyzeng, peteflorence) https: htops://www... www...mintaabb...mmmmmmm..m.m... com/calamars
www.eeneeeeoeeee.....eee @googgl. com
Making contact with purpose is central part of robot manipulation and remains essential for many household tasks - from sweeping dust into a dust- pan, to wiping tables; from erasing whiteboards, to applying paint. In this work, we investigate learning language-conditione fact, contact itself predicting contact for-
Abstract:
vision-based manipulation policies
wherein the action representation mations which tools grasped by the robot should meet an observable surface. Our approach, Contact-Aware and Language conditioned spatial Action MAp- ping for contact-RIch manipulation (CALAMARI), exhibits several advantages including (i) benefiting from existing visual-language models for pretrained spa- tial features, grounding instructions to behaviors, and for sim2real transfer; and (ii) factorizing perception and control over a natural boundary (i.e., contact) into two modules that synergize with each other, whereby action predictions can be aligned per pixel with image observations, and low-level controllers can optimize motion trajectories that maintain contact while avoiding penetration. Experiments show that CALAMARI outperforms existing sfate-of-the-art model architectures for broad range of contact-rich tasks, and pushes new ground on embodiment- agnostic generalization to unseen objects with varying clasticity, geometry, and colors in both simulated and real-world settings.
Keywords: Contact-rich Manipulation, Vanual-language guided policies
1 Introduction
Contact-rich manipulation is ubiquitous in our day-to-day lives, encompassing broad range of tasks including sweeping dust into dustpan, wiping tables, erasing a whiteboard, and applying paint with brush. A key challenge in performing these tasks lies in controlling the interactions between tools and their environments. For instance, when sweeping, it is crucial to ensure continuous contact between the bristles and the surface while directing the collected dust towards the dustpan.
Language-condittonee representations and policies are a promising approach to addressing the chal- lenges of contact-rich manipulation, particularly for domestic applications. For one, language is powerful tool for creating abstractions that enable generalization for wide variety of tasks and environments. Secondly, language will be among the most common methods to command robots such as when performing tasks in the home. Recent work has demonstrated how large pretrained visual-language models (VLMs), such as CLIP [1] and PaLM-E [2], enable zero-shot transfer of visual-semantic reasoning based on language prompts and well-structured visual and language em- bedding spaces [3, 4, 5, 6, 7, 9, 10]. However, previous efforts have predominantly focused on rearrangement-bann tasks and have not adequately addressed the reasoning involved in contact-rich manipulations, thereby limiting their applicability tasks like wiping, sweeping, or scooping.
In this paper, we introduce novel resentation that predicts planar contact affordances contact formations at which tools grasped by
conditioned and contact-aware spatial action map rep-
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

(A)
(B)
Figure 1: CALAMARI is contact-aware and language-conditioned spatial-action mapping for contact-rich manipulation. We show that (A) wiping, sweeping, and pushing tasks, trained solely on simulations with single task and prompt, can be (B) directly transferred to the real world and applied to new environments with unseen tools, robot setups, table clevations, and prompts.
the robot should meet an observable surface in order to perform a tabletop task. Our novel multi- modal spatial action maps specifically for contact-rich manipulation where each pixel represents binary indication of extrinsic contact between object and the environment, and the entire map is implicitly linked to the tool pose and robot configuration. Notably, our extrinsic contact policies remains agnostic to intricacies of specific tools and physical robot platforms, unlocking possibilities for generalization to unseen objects with distinct elasticity, geometry, and colors, both in sim and the real world.
The key contributions of this paper are: 1) multi-modal extrinsic contact policy with novel spatial- action maps that outperforms SOTA language-guides manipulation methods for contact-rich tasks; 2) an MPPI controller algorithm compatible with the predicted contact goal and contact constraint; and 3) generalization to unseen objects with various elasticity, geometry, and colors in both simula- tion and the real world without requiring fine-tuning of the extrinsic contact policy.
2 Related Works
Language Grounding for Manipulation: The recent advancements in large language models (LLMs) [11, 12, 13] and visual-language models [1, 2] have enabled language grounded manip- ulation. Numerous approaches [4, 9, 3, 14, 8, 15, 6, 16), have emerged and demonstrated remark- able multi-task performance, particularly for pick-and-place tasks. These end-to-end methods map directly from RGB voxel observations to robot configurations. However, drawback of end- to-end approaches is their reliance extensive real-world data collection, which span sev- eral weeks or months [15, 6]. To address this real-world data efficiency challenge, [9] achieved significant improvements by predicting only key frames and discretizing input and action spaces. Nevertheless, like end-to-end models, [9] faces the challenge of effectively handling novel tool ma- nipulations, as the tool variations must be captured in the training demonstrations. We show that our approach CALAMARI can efficiently handle contact-rich tasks while reducing the burden on real-world data collection via zero-shot sim2real transfer.
language-grounded
Controlling extrinsic contacts for planning and manipulation has been an active area of research, with several notable contributions recent years[17, 18, 19, 20, 21, 22, 23, 24]. Kim et al. [17] developed a method for simulatancously estimating and controlling extrinsic contacts of rigid objects using tactile signitures, while [18] focused on planning with ex- trinsic point contact for planar manipulation scenarios. Van der Merwe et al. [21, 22] demonstrated extrinsic contact detection for deforming tool manipulation, specifically scraping with spatulas, in- corporating predefined contact goals and learned dynamics. Wi et al. [23, 24] presented technique
Planning with Extrinsic Contact:

---

action (Kyzzyaw)
Architecture
grayscale Image decoder
MPPI
temporal transformer
mlp
elll-attention
position
F-3
1--2
[+0
Pooling
contact patch goal
visnal language transformer
CALAMARI
CLIP + Heatmap Extractor
"wipe up
language
"vipe up the
"wijer, "xp". "Mhe" "dea""
wiiee
"spp" beatmaps
word strings
RGB history
Figure 2: Overview. shown the left panel, our method utilizes the history of RGB and language instructions as inputs and predicts the contact patch goal as a binary mask from the input image frame. The three yellow blacks (e.g.. "CLIP") represent the pretrained models, which are not updated during training.
for predicting dense contact patches for compliant tools using learned tool dynamics. CALAMARI provides a framework for policies with contact goals, allowing for seamless integration of contact dynamics models [21, 22, 23] into broader context of planning and manipulation.
3 Methodology
3.1 Problem Statement
Our objective is to learn function F that predicts the next desired contact patch goal, denoted as C.aal, at time t. This function takes as input sequence of RGB key frames and language instructions compute contact goals: F([I==011 lo), (I,, 40)) C where is RGB image, to is language instruction, and w dictates the observation time window considered for making predictions. The output is the contact patch goal C.ocal R a 2D binary mask in I,'s camera frame and is particularly well-suited to for contact-rich planar tasks [5, 25]. The contact patch Cgoal can be de- de-projected to a point cloud by overlaying the predicted mask onto a depth map that excludes objects/tools (D (oominal). The point cloud conversion enables the utilization of model predictive controller achieve the desired contact formation. this work, data is provided in the form of demonstrations consisting of variable-length T key frame trajectories, denoted by (Lo, 4o, ... (Ir, 4o)) Here, C, =x is the contact patch between the object grasped tool) and environment represented by binary mask from the camera perspective. The contact patch from the demonstrations serves as the ground truth contact goal for the function F.
3.2 Behavior Cloning Yool-Ennironmen Interactions
Vision-Language Pre-Processing: CALAMARI has two key vision-language pre- mre-processing steps to convert observations into CLIP features. Firstly, generate word-wise heatmaps that high- light the spatial locations in the RGB images corresponding to specific words in the language in- struction. The Heatmaps, denoted as H, = f(I, I), are grayscale images with dimensions w x h, obtained through the image-language relevancy extraction introduced in [26]. Using heatmaps in- stead of raw RGB input particularly advantageous for sim2real and generalization novel objects and environments, as long as similar heatmap distributions are present. This is because heatmaps provide abstract representation that is less sensitive to variations in visual appearance caused by factors like object colors and lighting conditions. The heatmap is subjected to further abstraction through a pretrained heatmap encoder [27], resulting in word-wiss heatmap features. These features are organized based on the word sequence and serve as the query input for the Visual-language trans- former, which will be discussed detail in the subsequent section. Additional information about the selection of heatmap encoders can found Appendix-A.4.1.
Similarly, convert the language prompts into embeddings using CLIP's language encoder. In this case, we combine both sentence and word embeddings to capture the contextual information

---

Contact goal generation
MPPI
Result
contact goal
Figure 3: We generate the key contact goal via CALAMARI (magenta) and reach the contact goal via MPPI, which is linked with corresponding low-level actions. We visualize the tool's contact trajectory in bluc until it reaches the contact goal. Once have reached
generate new contact goal until the task terminates.
conveyed by the language prompt while focusing on individual words. These embeddings are also arranged in accordance with the word order and serve as inputs for the subsequent transformer networks in the following section.
Architecture: The CALAMARI architecture (Fig. consists of of two types of transformers, visual-language (v-l) transformer and temporal transformer. Drawing inspiration from the LAVA structure [14], our v-l transformer is responsible for encoding inputs into multi- molli- features and our temporal transformer fuses latent observation over time to generate spatial-actions. In contrast to [14], the language query is comprised of set of sentence and word embeddings, denoted as Q E IR ( Here, represents the sentence length and dys the feature dimension. The keys and values, denoted as K and V E Rdxxdy respectively, are word-wise heatmap embeddings. The temporal transformer takes into account the history latent observations from the v-l trans- former, considering the w most recent time stamps. Using the history of observation is important in contact-rich manipulation as it often involves occlusion caused by tools and the robot itself. The temporal transformer utilizes self-attention multi-modal features stacked with time, represented as Q K V E Rwxdy, The outputs of the temporal transformer are decoded using grayscale image decoder, similar to the decoder architecture of the UNet model proposed by [28], without incorporating skipping layers.
Training Loss: The loss function is standard supervised behavioral cloning loss similar to the prior works [6, 29, 30]. Specifically, use L2 regression between predicted contact patch Cgoui and the ground truth contact patch Cfº the following: |[Cgoal C/
3.3 MPPI controller and Contact Goals
We use the Model Predictive Path Integral (MPPI) [31] controller to plan sequence of robot actions with corresponding contact patches to reach the desired contact goal Cgoal The input to MPPI consists of the current pose of the end-effector and initial guess for the action trajectories. We define an action as the displacement in Cartesian SE(3) pose of the end effector and denote an action trajectory (av, a1....,
-w- Next, define the controller cost:
dist(C C
Dnominab Pa-e) + A(1 - IoU (Cgoal C(+i))
where P.+i the predicted contact pointcloud from applying the end-effector delta change in pose a, to the object. We estimate P.- by transforming objects with known geometry to the world frame and identifying intersections with environment (Appendix A.5.4). C++: is a 2D projection of P P the camera frame. The first term of our cost function minimizes the mean Euclidean distance between the center of Pitt and the contact goal center. To do get the contact goal pointcloud by overlaying contact goal mask with the nominal depth map. The second term promotes matching the shape of future contact to the goal using Intersection over Union (loU). We align the center of the prediction and the goal of contacts by subtracting the mean to focus on matching the contact shape (Appendix A.5.6).
uses C.oall D Dominal to
Our MPPI has contact constraints, implemented via penalty costs. The first constraint is to maintain contact via ||Ca]l 0. This penalizes any actions that makes contact. The second constraint is max Doominal - Ptti) > Cp- This lower bounds the distance in axis from the environment to the transformed objects with epsilon Cp > 0. The entire control algorithm we use is described in Appendix Alg. I.

---

wipe desk
sweep_ to dusipan
press button
start
"wipe up the dots"
cnd
start
"sweep det to distpan"
md
push the red button"
RGB
Contact Paich
Figure 4: We visualize the dataset by displaying the language prompt alongside the RGB and contact patch sequences extracted from key contact frames in the demonstrations. The wiping task typically includes 7 to 12 key contact frames, sweeping involves key contact frames, and the press button tasks have contact patch.
Experiments and Result
Datasets: Both our model and baseline were trained using 100 demonstrations per task on Cop- peliaSim [32, 33j. Each task involved manipulating a single object and single language prompt, as described in Fig. 4. In Sec. 4.1, we investigate the generalization performance using various unseen objects and prompts. This paper focuses on three different types of contact-rich tasks: multi-step patch contact "wipe desk', three-step patch contact 'sweep to dustpan', and single-step point con- tact 'press button'. For the wiping task, improvised our closed-loop demonstration where the sponge moves towards the center of dust clusters computed from DBScan [34]. The other two tasks use open-loop demonstrations predefined in [33]. Each demonstration consists of a language prompt and a sequence of RGB observations along with the corresponding contact patches obtained from the key contact frames (Fig. ). The key contact frames were identified when the robot reached the waypoints defined in the demonstrations. However, frames where the contact patch was the same as the previous waypoints were removed to eliminate redundancy. Ground truth contact patches are computed via CoppeliaSim'ss 's contact detection algorithm.
4.1 Simulation Results
The task performance of our model across the tasks is presented in Table 1, where the scores averaged over 25 test episodes. Here, utilized 4 DoF action space (It, U, 2, yaw) to focus on planar manipulation, and we have included full 6DoF manipulation result in Appendix A.5.3, We conducted three different objects/tools for evaluations: one being the training object and the other two being held out objects (Fig. 5). Our objective in directly transferring to these held out ob- jects is twofold: 1) To demonstrate the robustness of our contact goal policy in effectively adapting to shifts in heatmap distribution from variations in object's structural/visual features. 2) To show- case the flexibility of our MPPI controller in accommodating previously unseen contact formations from unseen object geometries. Note that we employed new task- language prompts for the push_bution task's heldout objects to accommodate color change.
Evaluation Metrics: The evaluation metrics are the task success rates ranging from 0% to 100%. We evaluate wipe_desk and sweep_to dusttpn tasks with the percentage of dust removal. For the push button tasks, binary metrics were used, where 0% indicated failure and 100% represented success without partial credits.
wipe desk: We evaluated the wiping performance after 20 contact goal generations, which is the point at which task performance plateaus (see Appendix Fig. 14). This task involves clearing one hundred dust particles with size 0.6x0.6xx. that are randomly spawned within the bounding box of size 15emx We show in Tab. I that CALAMART achieves 98% of dust removal with the training object. To assess the performance with heldout tools, we conducted two tests. Testl examined the ability of our MPPI controller to manipulate objects with 55% decreased contact patch area. Test2 evaluated the robustness of CALAMARI against out-of-distribution broom geometry with a long handle. As shown in Tab. 1, our method performance only decreased by 2% for testl when handling smaller contact patches. Additionally, our goal generation exhibited robustness to heatmaps variations as test2 results are comparable to training object performance.
sweep 1o dustpan: We evaluated sweeping task with contact goal generations in the demon- stration. sweep dustpan task involves 5 dust particles with the size of Ixlxlem. Details of the task environments are in Appendix 1.2.1. Using training object, we achieved 93% success rate in
5

---

wipe desk
sweep to dustpan
push button
Method
train
testl
test2
train
test
test2
train
test
test2
Ours
98%
96%
90%
93%
73%
84%
92%
60%
60%
PerAct
97%
97%
9%
86%
0%
0%
63%
0%
24%
CLIPORT
92%
89%
45%
88%
0%
13%
84%
8%
20%
Table RI.Bench success rate each tasks in test cases using to train objects denoted as 'train' and heldout objects denoted as "testl" and 'test2'
wipe_desk
sweep_ ouduspnn
button
Daining
tesil
tesi2
training
testl
tesi2
treining
test]
lest2
7x15s3lem
4x12xclom
7xt5efem
1x10s15cm
4124s42cm
St54t8cn
2.8cn
2.8cm
2.0cr
Figure 5: Train and test ohjects in simulation of four different tasks. The heldout objects exhibit variations not only in color, size, and shape but also structural attrihutes, such as handle locations. The third row indicates object dimensions in centimeter either as width x depth x height or as radius.
test cases for sweeping to dustpan. We then transferred the pretrained model to two test objects: one with a longer handle, similar to the original, and another with a handle on the side. We noticed that larger visual discrepancies between the objects resulted greater performance drops as test shows worse performance than testl.
push button: We altered the prompt "push the {} button" from the word "red" to "green" and "blue". This change resulted in varying CLIP heatmap intensities and word embedding inputs ,or the queries of our visual-language transformer. Nevertheless, our test results revealed that even with training based on a single word, our model achieved contact goal accuracy of less than 2.5cm in 60% of the two heldout cases. These findings emphasize the robustness and effectiveness of our approach in accurately pushing buttons of different colors.
Baseline-PerAct: In this section, we compare our methods with PerAct, state-of-the-art language- conditioned manipulation study that also utilizes CLIP features. Details of implementation are de- scribed in Appendix A.3.1. Our method outperforms PerAct mostly across the three tasks (Tab. 1), both in training and testing with heldout objects. For the wiping task, PerACT shows significantly worse performance for test2 when compared to testl and the training object. There are two main factors that explain the performance drop. First, PerAct is sensitive to larger changes in the transfor- mations between the grasped position and the contact surface. We can observe this trend consistently across the sweeping tasks for the test objects well. Second, PerAct is sensitive to variation in vi- sual cues of the objects, as further supported by the sweep task. Our model is more robust these changes have less impact on our contact planning than on the robot configuration. The results of push button task show that, while PerAct fails to detect buttons not in the training prompts, CALA- MARI leverages VLM's generalization ability for interacting with unseen prompts of different colored buttons.
Baseline- CLIPORT: CLIPORT is particularly suited for tabletop manipulation with 2D affordance prediction [5]. The CLIPORT baseline shares number of important similarities to CALAMARI, in- cluding using an image action space and known tool geometry and pose. Details of implementation are described in Appendix A.3.2. Tab. 1 demonstrates that CALAMARI consistently outperforms the CLIPORT baseline. Wiping and sweeping results show that CLIPORT also struggles to gener- alize unseen objects and tools with significant visual and geometrical variations. This is because CLIPORT encodes RGB-D without further abstraction unlike CALAMARI. Moreover, CLI- PORT also faces difficulty in generalizing to unseen prompts with color variations for the pushing task, similar to PerAct with significant drop in performance for these tasks.
BaselineCCLIPORE
4.2 Sim2Real
In this section, we directly transferred the pretrained model from simulation the real world without fine-tuning. The quantitative analysis (Tab. 2) was conducted using 10 runs for each object with consistent resetting across all tasks. We utilized 2 Franka-emike robots and an Intel Realsense D435 camera for our real-world setup. The distance between the robot and camera between the real-world

---

start
end
Tool
Figure 6: We demonstrate the ability of model generate goals for non-rigid tools. We repeat the
sweeping task with a compliant tool, using a learned dynamics model servo contact the tool with tabletop. The predicted contact goal is visualized in magenta while the contact feature predictions from the dynamics model are overlaid in blue.
start
Target
Test
Test 2
15am
Figure 7: We visualized CALAMARI's contact patch goals in magenta. The first represents the predictions when using the testl object, while the second row corresponds to test2 object. Our model can navigate back and forth until all the dots are erased using a closed loop policy. In real-world scenarios where the pressure exerted by sponge on the board is not evenly distributed, this ability becomes particularly significant as the sponge may fail to erase certain dots even when it passes over them.
and simulation environments was [0.18m, 0.02m, -0.2 0.] in the x, y, and z directions. We found that the spatial-action map allows us to accurately predict contact patches the camera frame regardless of the differences in camera positioning between simulation and real-worke set-ups.
wipe desk: For the reset, we draw one hundred dots within bounding box measuring 17cmx 17em with an black marker. We generated 20 contact goals per run and counted the number of erased particles for evaluation. Fig. 7 shows our contact patch goals, where visualize the first and last three frames when contacts goals are generated. We also conducted experiment erase different distributions of dots with the testl object in Appendix Fig. 15.
sweep 1o dustpan: We performed resets by arranging the dustpan in 10 different configurations within bounding box measuring 9emx14cm Each configuration included 10 1em cubes placed in front of the dustpan. For evaluation, we counted the number of cubes successfully swept into the dustpan. Fig. 8 visualizes our contact goals, which directed the broom align with the dustpan and sweep towards it. Interestingly, our real-world results show dust sweeping rates of 91-92% from testl and test2 brooms, which surpassed performance in simulation with unseen tools.
Target
start
end
Test1
Test2
Figure 8: The real-world sweeping results were obtained using brooms with small width margins (1-3 cm) with the target dustpan. Left panel shows
contact goal generations, visualized magenta. The results in the last column demonstrate that our policy exhibits excellent zero-shot transfer to the real-world, even with unseen tool geometries. Furthermore, we demonstrate that our method is agnostic to the arrangement of the robot, the same policy applied to both the right and left arm.
7

---

"Push the (red) button"
Unseen elevation
Unseen environment
uaseen premme
38cm 20cm
contact goal
Result
Figure 9: Our model, trained in CoppeliaSim on the prompt "press the red button,"" can be directly transferred world without fine-tuning (first columa). Our spatial action space is not limited the tabletop (second column) and is robust to visual distribution (second and third column). also handles unseen prompts like "press the green/blue button" (last two columns). The contact goal is visualized in magenta in the first row, while the second row shows the actual execution results
test 91%
push button: We positioned three buttons (red, green, blue) in 10 predefined line and triangular arrangements on the desk. For evaluation, assessed whether the end-effector successfully made contact with the target button. The real-world outcomes aligned with the sim- ulation results for the red and green buttons (training and testl), but pressing the blue button exhibited a per- formance discrepancy of 20%. Fig. 9 demonstrates the adaptability of our pretrained model diverse, unseen setups involving variations in elevation and the number of buttons in the scene.
wipe
test2 98%
testl 91%
Jweep.oo.uusppon
test2 92%
comp.
85%
red 90%
push button
green 60%
blue 40%
Table Direct transfer to real-world using test objects described in Fig. 7, 8, 9. 'comp' utilizes learned compliant tool dynamics as in Sec. 4.3.
4.3 Compliant Tool Manipulation
In this section, demonstrate model performance real world, compliant tool manipu- lation task. By decoupling goal generation and dynamics, our method can generate valid contact goals for tasks using compliant tools, so long as dynamics of the tool are available. We execute the sweep to dustpan task with compliant brush (compare tool deformation in Fig. to Fig. 8). Note that the goal generation is learned on rigid tools CoppeliaSim and transferred without finetuning a deformable tool in the real-world. We replace rigid dynamics with a learned contact feature dynamics to estimate pdgn [21]. Full details of the contact feature dynamics can be found in Ap- pendix A.2.5. Quantitative results shown in Tab. 2 and qualitative sweep is shown in Fig. 6. Our method can effectively predict contact goals for a deforming tool, yielding 82% performance.
5 Limitations
Our approach offers versatility in manipulating objects on various planar manipulation scenarios, including elevated and potentially inclined planes. However, our ability to predict contact limited 2D binary contact patch, therefore, is challenging directly apply our method for more intricate contact-rich manipulation scenarios like screwing bulbs peg insertions. Moreover, assume the region of interest (e.g., areas wipe, sweep, or push) is already within the camera's 's field of view. Lastly, our approach lacks support for discontinuous contact. As a future work to enable CALAMARI to effectively address more complex contact-rich scenarios, we suggest an extension to 3D contact mask prediction, potentially leveraging state-of- of-the----e--r surface reconstruction techniques (e.g., [26) ). Additionally, we suggest integrating binary contact prediction with contact mask to inform the controller switch between free space motion and in-contact mode.

---

Acknowledgments
This work was supported by the National Science Foundation (NSF) grant NRI-2220876. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
References
[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[2] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
[3] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01699 2022.
[4] A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, Sindhwani, Lee, V. Vanhoucke, and P. Florence. Socratic models: Compos- ing zero-shot multimodal reasoning with language, 2022. URL https://arxiv: /rg//bs/ 00598.
[5] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[6] E. Jang, A. Irpan, M. Khansari, Kappler, F. Ebert, C. Lynch, Levine, and C. Finn. Bc-z: seoo task generalization with robotic imitation learning. In Conference on Robot Learn- ing, pages 991-1002. PMLR, 2022.
[7] W. Liu, C. Paxton, Hermans, and D. Fox. Structformer: Learning spatial structure for language-guided semantic rearrangement of novel objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 6322-632222 2022. 1oi:10.1 100011//////A44.9..000000
9811931.
[8] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXi::2207.05600, 2022.
[9] M. Shridhar, I. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation, 2022. URL https: http:p::///xxxv.oaaa//ss
05451.
[10] Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, J. Andreas, and
D. Fox. Correcting robot plans with natural language feedback, 2022.
[11] Т. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners. Advances in nearul information processing systems, 33:1877-1901, 2020.
[12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini arXiv: 1810.0080 2018.
[13] I. Solaiman, M. Brundage, J. Clark, A. Askell, A. Herbert-Voss. J. Wu, A. Radford, G. Krueger, J. W. Kim. S. Kreps, al. Release strategies and the social impacts of language models. arXiv preprint /908. 09203, 2019.
[14] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. arXiv preprint arXiv:2210.66400 2022.

---

[15] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenherg, et al. generalist agent. arXiv preprint
arXiv:2205.06/73 2022.
[16] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, B. Zitkovich, F. Xia, C. Finn, et al. Open-world object manipulation using pre-trained vision-languag mod- 200::333300000 2023.
els. arXiv preprint
[17] S. Kim, D. K. Jha, D. Romeres, P. Patre, and A. Rodriguez. Simultaneous tactile estimation and control of extrinsic contact. arXiv preprint arXiv:2303.03388 2023.
[18] B. Accituno and A. Rodriguez. hierarchical framework for long horizon planning of object- contact trajectories. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 189-196. IEEE, 2022.
[19] A. Sipos and N. Fazeli. MultiSCOPE: Disambiguating In-Hand Object Poses with Propriocep- tion and Tactile Feedback. Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi:10. 15677/SS 2023.XIX.07.
[20] M. Wilson and T. Hermans. Learning to manipulate object collections using grounded state representations. In Conference on Robot Learning, pages 490-502. PMLR, 2020.
[21] M. Van der Merwe, D. Berenson, and N. Fazeli. Learning the dynamics of compliant tool- environment interaction for visuo-tactile contact servoing. In Conference Robot Learning, pages 2052-2061. PMLR, 2023.
[22] M. J. der Merwe, Y. Wi, D. Berenson, and Fazeli. Integrated Object Deformation and Contact Patch Estimation from Visuo-Tactile Feedback. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023, doi: 00.11.66////SS5202222 2023S..222..XX22 222..00X...00000
[23] Y. Wi, A. Zeng. P. Florence, and N. Fazeli. Virdo++: Real-world, visuo-tactile dynamics and perception of deformable objects. K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1806-1816. PMLR, 14-18 Dec 2023. URL attpps:/pprocat //ppoeee////oceiings.... mlr.
Fress/v205/wi22 html.
[24] Y. Wi, P. Florence, A. Zeng, and N. Fazeli. Virdo: Visio-tactile implicit representations of deformable objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 3583-3550. IEEE, 2022.
[25] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747. PMLR, 2021.
[26] H. Ha and S. Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision- language models. Conference on Robot Learning, 2022.
[27] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[28] O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Cnervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234 231. Springer, 2015.
[29] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imita- tion learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628-5635. IEEE, 2018.
10

---

[30] C. Lynch and P. Sermanet. Grounding language in play, arXiv preprint 2002v::0005007648,, 40: 105, 2020.
[31] G. Williams, N. Wagener, B. Goldfain, Drews, M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 1714-1721, IEEE, 2017.
[32] E. Rohmer, S. P. Singh, and M. Freese. V-rep: versatile and scalable robot simulation framework. In 2013 IEEE/RSJ international conference intelligent robots and systems, pages 1321-1326. IEEE, 2013.
[33] James, Ma, Arrojo, and A. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
[34] M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. kdd, volume 96, pages 226-231, 1996.

---

start
Onrs
Point Contact
Using contact goals points instead of patches results in loss of control the contact orientation. As demonstrated in this figure, CALAMARI successfully reorients the broom to maximize dirt sweeping. The magenta color indicates contact goal in all figures.
Figure 10:
A
Appendix
A.1
Ablation Studies
A.1.1
Contact Patch Verses Point Predictions
Predicting contact patches is crucial for precise contact-rich manipulation. For instance, optimiz- ing contact patch orientation can greatly increase efficiency sweeping up larger amount of dust simultaneously. Our proposed model-predictive control approach tracks desired contact patches, in particular matching contact patch size and orientation. demonstrate the importance of contact patch control, present validation in the following ablation section and Fig. where the initial broom orientation is set to rad the world-frame Here, we extracted the center of the contact patch goal and executed our MPPI algorithm without the Intersection over Union (IoU) cost. Our results show that using the point contact goal results in a 14 % success rate, which is significantly lower than CALAMARI's performance with contact patch goal and the same orientation offset (93 %).
A.1.2 Temporal Transformer
To evaluate the impact of CALAMARI's temporal transformer, conducted comparison between the wiping task performance with and without the temporal transformer (Tab. 3. We chose the wiping task because it is characterized by the longest planning horizon among our tasks and exhibits the most significant scene occlusions due to the robot arm and tool. In the 'without temporal' experiment, we directly used the latent state from the visual-language transformer as an input to the grayscale contact goal decoder. We kept the hyperparameters fixed across trials. The training loss curve shown Fig. 11. We note that the task success rate using train object without temporal transformer was 93%, which is 4% lower than that of our proposed architecture.
A.2 Results Details
A.2.1 CoppeliaSim Sweep Task
12

---

wipe Lesk
train
test
test2
CALAMARI
97
93%
78%
without temporal
93 %
86%
74%
Table 3: Performance analysis with without temporal transformer.
CALAMARI
Without temporal
13
500
Figure 11: Training loss curve of CALAMARI with and without temporal transformer.
Among tasks, sweep_oo Sacpp_o_ddlll _dusppn involves two different objects in the scene: broom and dustpan. To assess the robustness of CALAMARI and the baselines towards unseen scenarios, both objects were modified, as illustrated in Fig. 12. Each broom was created from scratch, designed after real- world brooms commonly found in retail stores. As for the dustpan, we employed the same mesh used for training, but altered its color to match that the broom. Additionally, we introduced geometric variance by scaling the dimensions along the x, y, and directions. We note the dustpan dimension for training was 25emx30emx7enn while for Testl, the dimension was 20cmx35emx1 and for Test2, the dimension 20emx25ernx3em..
A.2.2 Real-world Setup Details
Fig. 13 shows our experimental setup using a single accessible vision sensor from the front view. We note that only one arm was used for manipulation. We found that RGB image noise could adversely influence the scale of the heatmap signal, which in turn affect goal generation of our method the sweep_o_dustpen task. This is due to the heatmap's sensitivity to high-frequency RGB noise, not present in simulation, resulting in mild divergence in subsequent contact goal predictions for nearly identical inputs. As such, for the sweep to dustpan task, and compliant variant, we repeated each trial twice and reported the better performing result.
Train
Test 1
Test 2
Figure 12: For the sweeping, we not only used unseen brooms, but also augmented the dustpan to different colors and dimensions.
13

---

Realsense D435
Franka- Cmika
Figure 13: We used Franka-emika rohots and Intel Realsense D435 for real-world setup.
Success with numbers of goul
Percentage of dust removal time
KDE final
KDs mettcc with mein 92 200 and 7.023
40
28-
031
1000
PS3CC
Details on wiping performance with training object over the number goal generated (first) and time (second). Second plot visualizes each 25 tests' dust removal rate over time. Finally we show the distribution of success rate after 20 goals.
Figure 14:
A.2.3 Real-world Wipe Desk
Fig. 15 shows generalization of our wiping task to unseen dot arrangements. This results different heatmap distributions. As in the main experiment, run the wiping until it generates 20 contact goals.
Initial observation
Result
Initial observation
Result
Initial observation
Result
Figure 15: We examined the wiping results using the testl object different dot arrangements, leading to unseen heatmap distribution for the model, which was originally trained only on square-shaped dot clus- ters. From top
the bottom right, we conducted tests on circle, triangle, star, quatrefoil, infinite, and hourglass shapes. Our observations show the first four shapes were nearly entirely erased after the wiping. In contrast, we observed disparity in performance with the last two shapes, characterized by unfilled holes in their centers, leaving behind 10% and 54% of the dots, respectively.
14

---

"Push the red button" T
"Push the red button" "
"Push the green button"
Figure 16: First row indicates generated contact goal indicated magenta pixels. The second row indicates robot execution results.
A.2.4 Real-world Push Button
Fig. 16 shows more example results on button pushing task with unseen numbers of buttons in the scene, unseen table elevation, and unseen prompts.
A.2.5 Compliant Sweep Task
Our method decouples contact goal generation from the low-level controller responsible for real- izing the contact goals. To control contact between deforming tool and the tabletop, we use the contact feature dynamics model from Van der Merwe al. [21]. The model predicts contact geome- tries (represented as lines in 3D) given candidate actions, conditioned on point cloud and wrench observations. The contact point cloud P1+; is obtained by sampling evenly between the end points of the predicted contact line.
15

---

(A)
(B)
Test
Test 2
Test
Test
(B) depicts the rigid object geometries utilized in our experiment, corresponding to the actual objects shown in (A). While the object models do not encompass the intricate details of the object geometry. our research demonstrates that relying solely on the coarse dimensions of the geometry such as the object's collision model, commonly employed for collision avoidance during robot planning proved sufficient successfully executing our task.
Figure 17:
We train the model by performing randomized actions and label contact lines using heuristic the observed point clouds. Specifically, threshold for points near the surface, then fit a line to the resulting points projected onto the tabletop. Point cloud observations are obtained by Photoneo Phoxi 3D scanner (I.) and a Photoneo MotionCam-3D Color (M+) scanner. Wrench observations are obtained by an ATI gamma force torque sensor, attached between the end effector and compliant tool. The dynamics model is trained on 3236 sampled transitions.
A.2.6 Grasped Tool Geometry
For the object geometries, we crafted their meshes via trimesh Python package utilizing rough object dimensions as shown in Figure 17(B). Constructing meshes required less than 10 minutes to create each for the objects we employed. While we haven't specifically explored automating this process in our current work, is plausible that SOTA 3D shape completion techniques from partial object point cloud measurements can be used to generate meshes.
A.2.7 Task Performance Robustification
We achieved robustification of contact goal generations by applying robot workspace mask to the RGB images using a known transformation from the camera to the robot and by employing heatmap augmentations. These heatmap augmentations include flipping and translation( both horizontal and vertical) ranging from to pixels. Furthermore, we introduced four different levels Gaussian noise the RGB before generating the heatmap, enhancing the robustness of our method against high-frequency noise in real-world RGB images.
A.3 Baseline Details
A.3.1 PerAct
PerAct employs voxel inputs from a multiple calibrated RGB-D camera setup, while our method relies solely on single front camera view. In contrast, our MPPI controller requires additional information about the model's geometry and object pose, whereas the baseline does not require such object-related details. For this experiment, PerAct was trained from scratch with 512 latent dimension. To align the baseline with experimental setup, trained PerAct with single-tas using the same dataset. During testing, we reduced the action space to (x, y. yaw) by providing
16

---

800
Blue is training loss curve when training image encoder from scratch and the red is the trainind loss curve when training with frozen pretrained ResNet18. Other than the encoder, we used the same training settings. The datasct used for this experiment is 50 demonstrations of press breto button dataset.
Figure 18:
ground truth values for the other actions (pitch, roll, gripper state, and collision prediction) well ground truth grasping.
A.3.2 CLIPORT
CLIPORT's inputs consist of single RGB-D and language instruction, whereas the outputs three affordances for picking, placing, and discrete end-effector angle for placing. To elaborate, CLIPORT outputs (u,v) for picking and (u,v.yaw) for placing, where and v denote pixel coordinates in the tabletop view. These table-top pixel coordinates are then projected into the world frame (x.y.z) using a known transformation. Subsequently, we employ the known tool geometry and pose to transform the target point (x.y.z) into the robot end effector frame, ensuring that the bottom center of the tool reaches the target point with the desired wrist rotation (yaw). We provide ground truth corresponding to the table height.
A.4 Input Processing Details
A.4.1 Heatmap Encoder
We implemented two gcal- image encoder. The first was based pytorch's 's ResNet implemen- tation by changing the first encoder's 's input dimension instead of 3. We used 4 residual blocks and 2 convolution layers for each residual blocks following the original pytorch implementation. We trained this encoder from scratch along with other modules of CALAMARI. The second was the pretrained ResNet18 [27] trained on ImageNet. The network for RGB image, such that we repeated the grayscale heatmap inputs for three times and stacked in depth match the desired the input dimension. We note that the pretrained image encoder's parameters are frozen without fining tuning. Our experiment shows that the second encoder gives faster convergence and lower training loss as shown in Fig. 18.
A.5
Controller Details
A.5.1
Control Pipeline
Alg. 1 explains our hierarchical controller in the context of other components of CALAMARI. When given the contact patch goal, we repeat a control process consisting of the MPPI (Alg. line 8) and the impedance controller (Alg. line 9). The control loop continues until the grasped objects achieves the contact goal, measured as cost using the same MPPI cost function (Alg. 1 line 5). If the current pose of the object resulting from the impedance controller has a smaller cost than the threshold 6, a observation(A)) Alg. 1 line 5). Task is finished when the task objective is achieved (Alg. I line 12); for example, when all the dust is swept, when all the dots are erased, and when the button has been pushed correctly. Tasks are also finished when the number of contact goals exceed the number of goal threshold (Alg. line 14), where the contact goal threshold (gthres) is set to 20 for wiping, for sweeping, and 1 for pushing.
contact goal is generated via generate contact goal using visual-language
Next, provide additional details about our MPPI controller, building upon the description pro- vided in Sec. 3.3. The role of MPPI is to compute sequence of robot actions that result in the
17

---

Algorithm
1: t 0
2: g+0
3: complete +- False
4: while not complete do
5: if (c,s((s)) 6 then Csool
6:
14-9 --1++ +1
7: end if
8:
a, f mppi(5t, Cgoal)
9:
env.step(a.)
10:
++1+1
11:
if task..gt. fask..ett.comple completed then
12:
complete t- True end if if g
14:
> Othres then complete True
16: end if
end while
desired contact goals given by representation, Fig. 3. Here, we define action the change in Cartesian SE(3) pose of the end effector and denote action trajectory The input to MPPI consists of the current pose of the end-effector and initial guess for the action tra- jectories. Given the cost function described in Section 3.3, the output of MPPI is the action sequence with the lowest cost. To predict contact locations using action samples, given a goal at each time step, we predict contact patches of tool given sampled end-effector actions. For rigid object contact estimation, we apply the end-effector Cartesian change in position to the grasped tool (assuming fixed grasp) and compute the intersecting geometry with the environment resulting in the contact patch. This is achieved by comparing the transformed tool mesh/pointcloud with the D Dominaal- For the compliant tool, we use the Extrinsic Contact Servoing approach [21] which directly yields the contact lines, utilizing wrench measurements and partial point cloud given the action to be executed. One may substitute of models including [23].
(ao, ai....s aw-1):
A.5.2 Controller Parameters
For the mppi, the actions were sampled from NN o) and clipped using the action bound. Our MPPI framework is based the external repository following the prior works [21] and we set the parameters for the mppi as follows: action high =|z, y, z, r. =]=00.44, 0.04, 0.001, O., 0., 0.3],
action low=(x, p. =/=000004 -0.04, -0.001, O., 0., -0.3], num samples=1000, nx=6, lambda= 0.000001, horizon=1, and
00.01
0.
0.
0.
0.
0.
0.
0.01
0.
0.
0.
0.
0.
0.
0.001
0.
0.
0.
0.
0.
0.
0.0005
0.
0.
0.
0.
0.
0.
0.0005
0.
0.
0.
0.
0.
0.
0.01
A.5.3 6-DoF Manipulation
In this section, we present the results of the full 6DoF manipulation by expanding the action bounds of and p. well the noise sigma. Now, all rotation components share the same action bounds: action high- [z, y, 2, p. y] [0.04, 0.04, 0.001, 0.3, 0.3, 0.3], and action low [z, y, 2, r. p, yl [-0.04, -0.04, -0.001, -0.3, -0.3, -0.3]. Other MPPI settings, such as the number of samples, lambda,
18

---

(Tr: By)
wipe_desk
sweep_Jo Juustan
(0.01, 0.01)
95 %
92 %
(0.05, 0.05)
87% 94 %
(0.1, 0.1)
91 %
84%
Table 4:
6-DoF manipulation experiments with different roll and pitch action sampling parameters.
horizon, and more, remain unchanged. Consequently, update the action noise sigma
0.01
0.
0.
0.
0.
0.
0.01
0.
0.
0.
0.
0.001
0.
0.
0.
0.
0.
0.
0.
0.
0.
0.
0.
0p
0.
0.
0.01
Here, d, and op are scalar values associated with the standard distribution of roll and pitch actions, respectively. In Tab. 4, evaluate different combinations of (αr,ūp) in of our more intri- cate control tasks, using the "train object" in Coppeliasim. The final row, which utilizes (Ør, σy) = (0.1, 0.1), representing scenario where all rotation components share the same action sampling distribution. The Tab. 4 shows the task performance may decrease as we increase the action noise. Keeping the number of samples constant while increasing action noise for MPPI results in sampling deficiencies, which lead to suboptimal control result. Finally, our robot's Cartesian impedance con- troller stiffness parameters was set to [3000.0, 3000.0, 3000.0, 300.0, 300.0, 300.0] for rigid tool manipulation and [2000.0, 2000.0, 1000.0, 100.0, 200.0, 200.0] for complaint tool manipulation.
A.5.4 Contact Patch Prediction
To obtain the intersections between the object's point cloud and the environment, first extract the lowest 10% of the point cloud, which serves as the contact candidates. Using contact candidates enhances calculation efficiency. If the z-coordinate of the contact candidate is lower than or equal to that of the closest point in the environment in Manhattan distance on the x and y axes, we consider the candidate pixel to be in contact.
A.5.5 Environment Geometry
We define the enviromment as point cloud or the depth map without tools and non-collidable objects. In the CoppeliaSim simulation, utilized rendeeeblee False) for the tools and non-collidable objects in the scene during task initialization. In the real world, set aside the tools from the camera's 's angle at the task initialization and capture depth map obtain the environment geometry.
A.5.6 Intersection over Union
Before calculating the loU, we align the centers of contact patch predictions with the contact goal mask to ensure that the IoU metric focuses purely on shape matching. This alignment is achieved by subtracting the mean pixel values of each 2D mask. We found that using IoU metrics without this center offset can lead undesirable yaw movements of the object. This is because yaw motions can lead to higher IoU scores by creating larger overlaps between the tool's actual contact and the contact goal, especially when the tool can only partially reach the goal with the action. Consequently, IoU scores without this alignment do not accurately reflect the deviations from the desired contact's shape and orientation.
19


Reviews:
[{"review_id": "yqC5bDWBH9I", "reviewer": "Reviewer_hB1X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes Contact-Aware and Language conditioned spatial Action MApping for contact-RIch manipulation (CALAMARI) that outperforms SOTA models for contact-rich tasks in simulation and in the real world (with zero-shot sim2real transfer). The model takes advantage of existing VLM for pretrained spatial features to ground instruction to spatial actions (also benefit sim2real transfer) and low-level controller (MPPI) to optimize motion trajectories that maintain contact while avoiding penetration. \n", "summary_of_recommendation": "Based on the review above, I recommend Weak Reject. I am open to changing my mind during the rebuttal. \n_________________\nPost-Rebuttal\n\nI have changed my recommendation to Weak Accept. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Is it assumed that the tool is already firmly grasped at the beginning of each episode? \n- Why is sweep_to_dustpan a three-step task? \n- What’s the resolution of the waypoints that determine key contact frames? This resolution seems to be slightly arbitrary. How does this resolution affect model performance?\n- MPPI knows object geometry and ground truth object pose, which makes generalization to novel objects much easier compared to the PerAct baseline. During test time evaluation with novel objects, what if you feed the training object geometry to MPPI? How much will the performance degrade?\n- CLIP heatmaps are probably only meaningful for nouns/adjectives. For instance, in Fig 2, the first three words “wipe”, “up”, “the”, produce meaningless heat maps. Is this a problem? If the instruction contains many words that result in irrelevant heat maps, will the model performance be affected? \n- Ablate temporal transformers (using only a single frame)\n- Ablate contact area estimate versus contact point estimate\n- Demonstrations in simulation are cheap. If more demos are used during training, can performance be further improved? \n- (Potentially) try a new, more challenging task of “applying paint”, in which the exact “shape” of the past contacts matters a lot. \n- (Potentially) add another baseline comparison against methods that focus on contact-rich tasks\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting robot learning system that is suitable for contact-rich tasks and demonstrates its superior performance over baselines in simulation and in the real world with zero-shot sim2real transfer. The model is able to generalize to novel objects (tools), robot setups, lighting conditions, table elevations, and prompts. \n\nThis work has a few noteworthy strengths. First of all, the algorithmic design seems intuitive but pretty effective. By leveraging the CLIP heatmaps, the model is robust to variations in object shape/texture/lighting conditions/sim2real gap, etc. The two layers of transformers to fuse multimodal features and temporal features seem pretty effective as well. The model also seems to be fairly reactive and can accommodate different types of elasticity of the tool (repetitively attempt to accomplish the task if not done yet).\n\nSecondly, because of the algorithmic design, the model is able to achieve zero-shot sim2real transfer with minimal performance drop. One of the main benefits of this is that only demonstrations in the simulation need to be (programmatically) collected. The experimental evaluation is well designed and three evaluation tasks are representative contact-rich tasks. \n\nThirdly, the paper is relatively well-written. Different components of the system are well explained and the figures are pretty illustrative. \n\nThis work, however, is not without limitations. One of the main limitations is that low-level MPPI controller requires ground truth object geometry and pose during inference time. This is okay and the authors made it very clear this is the case in the paper. However, this made it difficult or basically unfair to compare CALAMARI with PerAct, which is the only baseline that the proposed method is compared to. Also, the three tasks being evaluated, although representative, don’t seem to be too challenging. In fact, it’s a bit unclear if a contact patch area prediction is needed, as opposed to a contact “point” prediction. It would be interesting to evaluate the model on a more challenging task of “apply paint” (which is mentioned in the intro of the paper), which requires the robot to draw a specific shape/alphabet on the whiteboard. In this case, I would imagine a precise contact patch area prediction is needed (a point estimate is too coarse). Lastly, more ablation studies can make the paper stronger, e.g. ablating the temporal transfer (using just a single frame), predicting contact area versus contact point, etc. Lastly, to make the paper stronger, the authors can consider adding another baseline comparison against methods that focus on contact-rich tasks from previous literature.\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "X_OL92ANjgC", "reviewer": "Reviewer_eq6y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents CALAMARI, a language-conditioned manipulation agent that models actions as contact interactions. CALAMARI encodes a history of RGB observations, and predicts contact heatmaps in image-space. These heatmaps are initialized with CLIP saliency, and fine-tuned with in-domain data. These contact heatmaps are then used with an MPPI controller to optimize 4-DoF actions that achieve the desired contact interactions. Simulation experiments are set in Coppeliasim, where CALAMARI is benchmarked against PerAct. The results show that CALAMARI significantly outperforms PerAct, especially with unseen test objects. Additional experiments show good sim-to-real transfer and some preliminary capabilities with compliant tool manipulation.\n", "summary_of_recommendation": "CALAMARI is a novel framework that predicts contact heatmaps, and then uses an MPPI controller to achieve those contact interactions. CALAMARI achieves compelling results both in simulated and real-world manipulation tasks. The experiments could be improved by evaluating on more diverse tasks and conducting ablation studies, but overall, CALAMARI is a good addition to the venue. \n\n**Post-Rebuttal**\nThe authors' response is very helpful in clarifying initial doubts and questions. The reviewer thanks the authors for conducting additional experiments and adding more details to the paper. Despite this, the reviewer is still concerned about the limited diversity of tasks (which could have been scaled up in simulation), and some strong assumptions about object geometry for the MPPI controller. Based on these two key issues, I am keeping my original score of Weak Accept. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Summarizing the questions from above:\n- Have you tried evaluating on more simulated tasks?\n- Does the MPPI controller need a perfect 3D reconstruction and an accurate pose of the object?\n- How important are the individual components of CALAMARI? \n- What if the true contact regions are occluded by objects in the scene?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n+ Overall, the idea of fine-tuning large vision-language models to predict contact heatmaps, and then using model-based controllers to achieve those contract interactions, is novel and interesting. Vision-language models do not reason about actions, and end-to-end behavior-cloning agents fail to generalize beyond the training distribution. Prior works like CLIPort have attempted to reconcile these two aspects, but they are often limited to 2D pick-and-place tasks. CALAMARI cleverly adapts a vision-language model to predict contact interactions.\n+ CALAMARI achieves compelling empirical results both in simulation and real-world environments. CALAMARI outperforms PerAct (a state-of-the-art language-conditioned manipulation agent) in a reproducible setup. Specifically, CALAMARI generalizes better to unseen objects during test time. \n+ Generally, the paper is well-written and easy to understand. The figures are informative. The result tables are easy to parse. The supplementary material provides additional details for reproducibility.  \n\nWeaknesses:\n- The diversity of evaluation tasks is very limited. Is there a particular reason why there are only 3 task categories: wiping, sweeping, and push button? It’s understandably difficult to conduct large-scale evaluations in real-world settings, but CALAMARI seems applicable to a lot more RLBench tasks. More diverse tasks might also provide better insights into the capabilities and limitations of CALAMARI’s approach.\n- It seems like the MPPI controller makes additional assumptions about object geometry and pose (Line 192). Does the MPPI controller need a perfect 3D reconstruction of the object? If so, how were these reconstructions provided in the real-robot setup? Assuming good geometry and pose somewhat limits the generality of predicting contact patches in RGB observations.   \n- The experiments are missing some ablation studies. For instance, how important is providing a history of RGB observations, and how sensitive is the model to history length? How important are the positional encodings? These are some suggestions, but it would be great to see which design choices were crucial for CALAMARI’s success.\n- For certain tasks, what if the contact region is occluded in the RGB input? For example, when wiping a surface that is facing away from the camera, it would be difficult to predict contact patches without any visual feedback. If this is a limitation, it should be discussed in the limitations section. \n- Minor: It would be nice to have a supplementary video showcasing the system. The agility and robustness of contact-rich systems are much easier to understand with videos and demos. \n- Minor:  Section 5 “Limitation” -> “Limitations”. Also, it would be nice to have a conclusion section to wrap-up with main takeaways from the paper. But this section was understandably cut due to space limitations. \n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "IQafNZeMo3I", "reviewer": "Reviewer_GLce", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present a method which can extract language-guided heatmaps from RGB inputs through the usage of vision-language models. The heatmaps are then converted to binary masks which encode where in the image a contact should occur between objects/surface in the scene and a tool that a robot is holding. Those masks are then fed into an MPPI controller which estimates a robot end-effector trajectory which would satisfy the required contacts. The method is shown to work in simulation but to also transfer on a real robot seamlessly (for the chosen tasks).", "summary_of_recommendation": "I recommend weak accept and would be happy to change to accept depending on discussion during rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* How easy would it be to extend the method to tasks where we should not only estimate where contact occurs but also what type of contact - i.e. how much to press/squeeze. The authors identify this in the limitations section but is unclear if there's a way to extend the method.\n* How does the subgoal identification part in MPPI work? How do we decide on the number of subgoals - 20 for the dot wiping, 3 for the sweeping task, etc. Are those somehow automatically detected in a human observation?\n* Using point clouds from real sensors is usually very noisy - i.e it's common for pixels from the table to be considered part of some of the objects. Was that an issue for you and is that why we do an additional heat map encoding step, to discretise the continuous heat maps?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is very well-written with a clearly-explained methodology and well-devised and analysed experiments. We've identified the following strengths and weaknesses:\n\nStrengths:\n* The paper identifies and tackles a problem that is of high importance to the robot-learning community.\n* The learned policies are vision-based, making them more easy to deploy in a real-world scenario, assuming a small sim2real gap.\n* Due to the simplicity of the method, the sim2real gap seems to be easier to overcome when training in simulation and deploying to a physical robot. Namely, treating contact prediction as a 2D binary mask prediction problem, which can then be overlayed on top of a point cloud and given to a planner.\n\nWeaknesses:\n* The technical contribution is rather small and incremental - both the VLM model and the MPPI controller come off-the-shelf. What is more, the method heavily depends on MPPI but it is not completely clear to the reader how does the controller work exactly, specifically for cases where multiple subgoals have to be identified - the sweeping task. How does the controller find out that contact with the surface should happen in subgoal 1, where the broom touches the surface?\n* The method can only be used to train for tasks that can be demonstrated in simulation first, to have access to the ground-truth contact information.\n* It is worth elaborating what do the authors mean by contact-rich? Technically, the tasks demonstrated have rather simple contact dynamics. They are undoubtedly contact-related tasks but there's nothing complex about them - i.e. there's either contact or not - there's no friction or elasticity parameter estimation occurring. What if some sponges are harder than others and we should press less when wiping the dots?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "9mwEv-LHCe", "reviewer": "Reviewer_ceQE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a new approach for language-conditioned robotic manipulation by predicting “contact patches” — sequences of 2D image masks that are used as information to guide an MPPI planner. The idea here is similar in nature to prior work in language-conditioned imitation that use object-centric or voxel-based representations. By restructuring the action space to focus on more “high-level” or “semantically meaningful” features, we can use off-the-shelf pretrained models to efficiently learn new tasks.\n\nThe approach uses CLIP to produce word-level heatmaps that are then combined to predict “contact masks” given a language prompt and current (visual) state (e.g., “wipe the dots”). These masks are then deprojected through a calibrated camera, and the resulting point cloud is used to guide an MPPI planner to produce 4-DoF robot actions. The approach is validated on two simulation environments (from RLBench) and in the real world.", "summary_of_recommendation": "While I like some of the ideas in this paper, I currently advocate for rejection, following the weaknesses documented above. I’d love extra clarity from the authors about the assumptions behind the approach, and general applicability of CALAMARI to more complex manipulation tasks during rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "It’s not immediately clear from the paper what assumptions you’re making over the environment to facilitate action planning (via MPPI); I understand that we’re taking predict contact masks and deprojecting to get a point cloud — but do you need further information about dynamics? What about robot arm/tool masks per timestep?\n\nThe formalism in Section 3.1 is a bit misleading; your framework seems to only allow for a single language instruction per demonstration; however, the formalism seems to suggest that language can change per timestamp... is this really true?\n\nWhen generating initial heatmaps, why is it useful to use CLIP to generate heatmaps for filler or non-object words? For example, a heatmap for the word “the” is really underspecified/ambiguous — this also true for individual verbs as well, without any additional preprocessing. Can you provide more intuition for what’s actually going on here? Why not just predict heatmaps for individual objects/nouns, using off-the-shelf detectors?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Language-conditioned manipulation in high-contact tasks is generally underexplored; I like this paper’s approach to the problem, reasoning over a more “semantically meaningful” action space (contact masks) in order to facilitate learning from demonstration and generalization. I also like that the overall approach is generally easy to understand (build word-specific heatmaps, then learn to aggregate to predict a contact mask — as a segmentation task).\n\n---\n\nHowever, many aspects of this paper give me cause for concern. First, I am not sure how fair the comparison is between this approach and the sole baseline — PerAct. The paper argues that PerAct needs multiple RGB-D cameras to generate calibrated point clouds, while the proposed approach (CALAMARI) only requires a single viewpoint. However, for CALAMARI to work, all parts of the scene must also be visible — undercutting this weakness. As for CALAMARI outperforming PerAct, I think this is more to do with the underlying action space dimensionality than anything else — learning in voxel space is difficult because it’s huge relative to 2D image space → wouldn’t a more fair baseline here be something along the lines of CLIPort? Especially since we’re already limiting robot actions to 4-DoF (instead of 6-DoF).\n\nFurthermore, it’s not clear how easy it is to collect supervision in the real-world for the contact patches for the various different tasks. How is this done for real-world experiments? Is this approach scalable to larger, more long-horizon tasks?\n\nFinally, the biggest concern I have is that of generalizability of this approach. CALAMARI assumes that 1) the single camera image captures the majority of the interactable objects in a scene, 2) that a single 2D contact mask is enough to guide manipulation, and 3) that 4-DoF actions are enough to solve the tasks. It’s very hard for me to swallow all three of these assumptions given that many real-world manipulation tasks are subject to partial observability and complex motions (e.g., pouring, twisting, moving along arcs). I don’t see how the proposed approach can scale to such applications.\n\n---\nEDIT (Post-Rebuttal): The authors did a tremendous job during rebuttal, running out many of the experiments I had originally suggested, clearing up mis-conceptions; the results from the CLIPort baseline is especially meaningful to me.\n\nReading the rebuttals for the other reviewers, its clear how this paper is providing a novel contribution, with strong experiments that back up the original hypotheses. I will be significantly raising my score to a weak accept!", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This works presents a new approach for language-conditioned robotic manipulation. The approach fine tunes a pre-trained vision-language model to predict contact heat-maps in simulation and then leverages MPPI to generate end-effector trajectories that connect the contact-rich patches. The approach is evaluated in simulation against SOTA baselines and performs successfully in a zero-shot sim2real transfer on a real robot platform.

The authors have been very responsive during the rebuttal period, provided clarifications and additional evaluations and comparisons.
I believe this work will be of great interest to the CoRL Community.
Decision: Accept (Poster)