Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models
Authors: Nils Blank
Moritz Reuss
Marcel Rühle
Ömer Erdinç Yağmurlu
Fabian Wenzel
Oier Mees
Rudolf Lioutikov
Venue: CoRL 2024 (2024)
Keywords: Foundation Models
Language-conditioned Imitation Learning
Data Labeling
TL;DR: A novel framework to label unucrated long-horizon robot demonstrations without any model training our human annotation for langauge-conditioned policy learning.

Abstract: A central challenge towards developing robots that can relate human language to their perception and actions is the scarcity of natural language annotations in diverse robot datasets. Moreover, robot policies that follow natural language instructions are typically trained on either templated language or expensive human-labeled instructions, hindering their scalability. 
To this end, we introduce NILS: Natural language Instruction Labeling for Scalability. NILS automatically labels uncurated, long-horizon robot data at scale in a zero-shot manner without any human intervention.
NILS combines pre-trained vision-language foundation models in a sophisticated, carefully considered manner in order to detect objects in a scene, detect object-centric changes, segment tasks from 
large datasets of unlabelled interaction data and ultimately label behavior datasets.
Evaluations on BridgeV2 and a kitchen play dataset show that NILS is able to autonomously annotate diverse robot demonstrations of unlabeled and unstructured datasets, while alleviating several shortcomings of crowdsourced human annotations.

Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models
Nils Blank' Moritz Reuss'
Marcel Rühle'
Ömer Erding Yagmurlu'
Fabian Wenzel
Oier Mees?
Rudolf Lioutikor'
1 Karlsruhe Institute of Technology UC Berkeley
central challenge towards developing robots that can relate human language to their perception and actions is the scarcity of natural language anno- tations in diverse robot datasets. Moreover, robot policies that follow natural lan- guage instructions typically trained either templated language expensive human-labeled instructions, hindering their scalability. To this end, we introduce NILS: Natural language Instruction Labeling for Scalability. NILS automatically labels uncurated, long-horizon robot data at scale in a zero-shot manner without any human intervention. NILS combines pretrained vision-language foundation models in order to detect objects a scene, detect object-centric changes, seg- ment tasks from large datasets of unlabelled interaction data and ultimately label behavior datasets. Evaluations on B2, Fractal and kitchen play dataset show that NII.S can autonomously annotate diverse robot demonstrations of unla- beled and unstructured datasets, while alleviating several shortcomings of crowd- sourced human annotations, such as low data quality and diversity. We use NILS to label over 115k trajectories obtained from over 430 hours of robot data. We open-source our auto-labeling code and generated annotations on our website: http://rrobottaskllbelinn thub.
Abstract:
Keywords: Lunguage-Conditione Imitation Learning, Data Labeling
Introduction
Unlabeled
Foundation Models
Labeled Dataset
Inference: Language Conditioned Policy
Dataset
Object
Train Policy
Object-Centric Scene Annotations
state-actions, language
state-actions
Keystate Detection & Crounding
"Move the pot to the right"
Figure 1: A framework label long-horizon robot demonstrations without human annotations or model training from RGB videos. NILS leverages ensemble of frozen pretrained models to segment and annotate uncurated, long-horizon demonstrations, The resulting labeled and segmented dataset can be used to train language-conditioned policies without human annotation.
Natural language is an intuitive and flexible interface for humans to communicate tasks to robots. Recent works have shown promising results training language-conditiones policies using large
Correspondence to: nils.blank kit.edu
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

datasets of robot trajectories paired with language annotations. However, the performance and diversity of behaviors learned by language-conditiones policies hinge critically on the quality and quantity of the language annotations. Unfortunately, most available robot interaction datasets do not contain any associated language annotations due to the costly nature of generating these. Concretely, most labeled datasets often rely on post-hoc crowd-soureed language labeling to generate labeled data [1, 2, 3, 4]. which has several limitations: high cost in terms of time and money, variable quality of generated labels, reduced diversity when using templated instructions, and inconsistent granularity of annotations. The quality of the generated labels can vary significantly, from detailed descriptions to url links'. This issue can be addressed through templated language instructions [5], which in turn reduces the the diversity of the annotations, i.e., the same phrasing and instruction used for every instance of the same skill, object or spatial relation. The least apparent downside of manually labeled behavior is the granularity of the annotations. The behavior description along the temporal axis can vary significantly from individual actions to whole multi-step tasks, e.g. move left and clean the kitchen, hence directly affecting the learned skills. This raises the question, can we find more scalable and cost-effective way to generate semantically meaningful, detailed, and adjustable-granulariy language annotations for existing robot interaction datasets?
One option is leveraging the semantic knowledge of Vision-Language Models (VLMs) pretrained on Internet-scale data [6, 7. 8]. These foundation models have found applications in different robotic contexts [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]. However, none of the existing models accu- rately label robot demonstrations. Current VLMs struggle segmenting long-horizon demonstrations and tempo-spatial reasoning, especially in challenging robotic domains 20].
To address these limitations, we introduce Natural language Instruction Labeling for Scalability (NILS), the first comprehensive system for zero-shot labeling of hong-horizon robot videos without human intervention or additional model training. NILS employs ensemble of pretrained foun- dation models to identify relevant objects and generate potential task labels. NILS first identifies object-centric keystates, segmenting long videos with multiple tasks into smaller single-action windows. Next, a Large Language Model (LLM) generates free-form language annotations based on structured observation deseriptions. These descriptions are based on templated language generated from scene changes tracked by the pretrained foundation models. As a result, NILS converts videos long-horizon robot interaction data into segmented and annotated datasets, which can be utilized for training language-conditioned policies without manual labeling. Furthermore, NILS addresses all of the above shortcomings, i.e., cost, quality, diversity, and granularity.
We demonstrate through extensive experiments, that NILS efficiently annotates unlabeled robot in- teraction data with appropriate task deseriptions, surpassing state-of-the-art closed-source VLMs like Gemini-Pro. NILS reliably finds important keystates in long-horizon demonstrations better than prior zero-shot methods [21]. We demonstrate the scalability of NILS by labeling over 115k trajectories from different datasets, totaling 430 hours of robot data. Finally, we demonstrate the effectiveness of our approach by training Iinguage-conditoons manipulation policies the auto- matically labeled data on real robot.
Method
NILS consists of three stages: Stage I (Identifying Objects in the Scene), Stage (Object-Centric Scene Annotation), and Stage 3 (Keystate Detection and Label Generation). Figure 2 depicts com- prehensive method overview. The subsequent sections elaborate each stage. NILS uses different frozen pretrained models across each state, which enable modular replacement. A detailed list of the models used in NILS, including explanations and usage, is provided in Appendix A.
V2 dataset contains this annotation: http::///www..uuuuuuemmm/////aee

---

NILS: Natural Language Instruction Labeling for Scalability
Robot
Video
Stage Object Retrieval
Detected Objects: Table, Spinning Top. Towel, Cup, Tin Can
VLMs
Stage Object-Centric Annotation
Towell is mont to ton the right can nojon Object Relations
Gripper Positions
Segmontation Maskes
Bounding
Predicted Depth
Object
Object States
Gripper
Stage 3: Keystate Detection Label Generation
Templated Language Observations
Mate Changes
movements:
54.0
Gripper Poaltion
Object relation changes:
Mayetatec 09/00,309
96,102,110
Final relations: irewed food is neet spiiningg togs
Initial relations: uinee food on the towel
LLM
Object Novement Keyittee
Gripper prosimity: rottnt gripper cos close
food
object Relacions
Global changese
Language
Keystate: 102
Move Relocate the tin can to the left of the spinning
table to left of table
Meve the can the left and then forward
Figure 2: Overview of the proposed NILS framework for labeling long-horizon robot play sequences zero-shot manner using an ensemble of pretrained expert models. NILS consists of three Stages: First, all relevant objects in the video are detected. In the second step. object-centric changes are detected and collected. Stage object change information used detect keystates and an LLM is prompted to generate language label for the task.
2.1 Stage Identifying Objects in the Scene
In Stage I, NILS deploys a set of VLMs to detect all objects and their relevant properties across mul- tiple video frames. This ensures comprehensive object detection even with occlusion. NILS prompts the model output concise and unique names for all objects in each frame. However, this approach results in inconsistent naming across frames for the same object. To resolve this, NILS leverages temporal consensus, co-occurrence, and object defection alignment. It uses GroundingDINO [22] for bounding box generation and computes Intersection over Union (IOU) for co-occurrence. NILS combines the detector confidence with a SigLIP alignment score based on cropped image regions of the object. Finally, an LLM assigns the properties movable, is_container, states, and interactable to each object help filter invalid robot-object interactions in later Stages.
2.2 Stage 2: Object-Centric Scene Annotations
Next, NILS generates object-centric scene annotations that allow reason about robot-object inter- actions. For each object detected in Stage 1, NILS computes bounding boxes, segmentation masks, movement, relations, and state throughout the video. The extracted information is used in Stage to segment and annotate the demonstrations. First, NILS identifies the object bounding boxes and segmentation masks to track various object changes.
Object Annotations and Segmentations. NILS computes bounding boxes and segmentation masks for the robot manipulator and all objects from Stage using ensemble of open-vocabulas vocabulary detec- tion and segmentation models. To address the detector confidence misalignment and class struggles,

---

Figure 3: Overview of the environments used in our experiments. From left to right: Toy kitchen setup, two scenes from the Bridge V2 dataset [1], one example task from the Simpler Eval [26] and one scene from Fractal [5].
NILS ensembles the models by extracting bounding boxes for objects from Stage 1 (Subsection 2.1). Next, it computes the agreement between the detector and dense predictor inside each bounding box together with a temporal consensus approach. two-stage filtering process identifies the most repre- sentative bounding box to improve the detection robustness [23]. To address temporal misalignments and missing detections, NILS utilizes mask-tracking model [24] to capture temporal correlations between objects. Detailed explanations are provided in Appendix A.
NILS monitors four main signals: (1) object relations and movement using segmentation masks, bounding boxes, and depth maps, (2) object state changes over time, (3) gripper position relative objects, and (4) gripper closing and opening actions. These signals are used to generate tem- plated language descriptions and identify key states throughout the video, which are then used to describe the overall task. Detailed descriptions on how track individual signals provided in Subsection 2.4.
2.3 Stage 3: Keystate Detection and Label Generation
Determining critical states that mark task boundaries key challenge in labeling long-horizon robot demonstrations. NILS addresses this through a novel heuristic consensus method, using object-centric representations from Stage 2 to detect individual manipulation tasks. This method combines multiple heuristics to filter noise-induced false positives and identify reliable keystates. NILS acquires object-centric keystates by having each heuristic monitor changes for objects to min- imize the noise impact. An object-centric keystate o, is considered valid if its score exceeds a threshold 8,
S(a,)
Σ as * Sa(o.). K
where k is the heuristic index, is the heuristic weight, and Sk(o:) is the confidence of the cur- rent heuristic. NILS uses equal weights for all heuristics. The keystate score S& (ox) controls the quality generated keystates and language annotations (Figure 10), providing an advantage over VLM captioning [25]. NILS then aggregates nearby keystates and selects the highest-scoring one to minimize noise.
Action Retrieval and Grounding After identifying keystates, NILS generates natural language descriptions of performed tasks. It constructs templated language prompts based on the object- centric scene anmotations in between detected keystates. This information is used to query a LLM. The prompt focuses on a single object to help the LLM to concentrate on task-relevant information only. The LLM reasons about detected object movements and relation changes determine possible robot actions. NIL.S detected low-level keystates, and templated language observations can thus be aggregated. Combining multiple low-level language observations in single prompt allows the LLM to reason about higher-level tasks, allowing control over the task granularity. We provide detailed explanations and examples in Subsection B.2. A list of example prompts is given in Appendix D.
2.4 Object-Centric Information Retrieval of Stage 2
We provide further details on the object-centric information retrieval in Stage Using bounding boxes and the segmentation masks, NILS monitors various signals for labeling. NILS tracks the following signals to generate templated language with timestamps:

---

Kitchen Play
BridgeV2
Uniform Sampling
UVD
Ours
Ours Gripper
Uniform Sampling
UVD
Ours
Recall
4.6
Precidion
Predeton
Recall
6.4
MAP!
Valun
mAP
Predisod
Recall
02
02
0.0
16
Figure 4: Keystate accuracy for different frame distance tolerances on: Kitchen Play and Bridge V2. We report the precision and recall of our method at two different keystate thresholds. NILS generates relevant keystates on both datasets and surpasses both baselines.
Object Relations and Object Movement. NILS tracks object movement and relations using seg- mentation masks and bounding boxes. It detects movement through bounding box displacement and flow-based detection [27] for small movements. NILS constructs an object-relation graph with nodes representing objects and edges denoting spatial relations [28]. To capture depth-dependent relations (e.g., inside, behind), objects are projected onto a point cloud using zero-shot depth map [29]. NILS generates templated language describing movement and spatial relations, e.g., "[] moved to the left of []". NILS generates diverse and informative labels based on these grounded scene observations, outperforming simple language-based paraphrasing methods.
Object State Prediction. NILS predicts object states over time, which is crucial for static objects like drawers. It crops object images from each frame and compares state text embeddings with image embeddings to determine the current object state. To handle occlusion during robot interac- tions, NILS omits predictions when relevant objects are partially obscured. The system generates templated language like "Drawer changed from open closed.".
Gripper Position. The gripper position over time indicates robot-object interactions. NILS com- putes gripper-object proximity using object and robot segmentation masks and predicted depth map. To account for end-effector position inaccuracies, a per-object threshold used to determine con- tact. A keystate is generated if the gripper-object distance below the threshold for three frames with templated: "The gripper was close to ...".
Gripper Closing, When available, NILS uses gripper close signals potential keystates. NILS identifies keystate when a previously closed gripper opens.
NILS collects these object changes and templated language observations throughout the video.
3 Evaluation
In this section, we address the central question: Can NILS generate semantically meaningful, de- tailed, and adjustable fejjustable-gramalaring granularity language annotations for existing robot interaction datasets? To answer this question, we assess NIL.S' accuracy in labeling long-horizon robot data, keystate quality, and grounding performance against state-of-the-a VLMs and study the generated task labels with respect to various properties. NILS was evaluated on three datasets: (1) the Bridge V2 [1] containing 1200 long-hore horizon trajectories with 28K short horizon pick and place and state-mant satte-maiiplltiinn tasks with diverse scenes, tasks, and objects; (2) Fractal [5). dataset consisting of 87K short horizon demonstrations; (3) self-collected one-hour long-horizon, uncurated play dataset in robot play kitchen, containing 439 short-horizon demonstrations of 12 tasks. The dataset consists of multiple long-horizon demonstration videos where the robot solves at least 10 manipulation tasks in a row.

---

3.1 Keystate Evaluation
We quantitatively evaluate the keystates produced by NILS on both datasets using precision, recall, and mean-average precision (mAP) metrics. A predicted keystate is considered correct its dis- tance to a ground truth keystate is smaller than a threshold based on the average short-horizon task length. Figure 4 shows that our framework outperforms UVD [21] and Uniform Sampling on the Kitchen Play and V2 further improves performance, highlighting the importance of including proprioceptive information for detecting keystates, if available. These experiments show NILS can extract valuable keystates in challenging environments,
keystate threshold of 0, 0.25. Incorporating gripper-close signals
3.2 Grounding Evaluation
We evaluate the grounding capabilities of NILS mixture of the BridgeV2 dataset and our Kitchen Play dataset. Specifically, we select 14 different long-horizon play demonstrations from various settings, each containing 20 tasks. Of these. 12 trajectories are sampled from Br.dgeV2. We chose trajectories from diverse settings, including kitchens, tables, and washing machines, with a wide variety of objects to ensure maximal task diversity. Further, we added two long-horizon demonstrations from our Kitchen Play Dataset with 24 tasks each. Given the ambiguity of language annotations, we employ human evaluators assess whether a language label proposed by method correct given the segmented task video.
Baselines. We compare against four baselines. First, Gemini Vision Pro and GPT-4o, two large vision lan- guage models with video-understanddnng capabilities [30]. The prompt used to query these models provided in Appendix D. We further test against XCLIP, video- language retrieval model [31] and VideoLLava, recent open-source VLM [32]. Since VLMs can not segment long-horizon demonstrations out-of-the-boxx we provide the baselines with ground-truth keystates to assess their grounding capabilities.
Method
GT Keystates
Accuracy
XLIP
0.05
VidcoLLava
0.03
Gemini 1.5
V
0.16
GPT-4o
V
0.51
NILS
0.66
NILS
0.84
NILS TA NILS MFOR
0.46
0.38
NILS DI
0.47
Results. NILS outperforms all baselines in both settings The baselines struggle to ground the robot's actions due to the domain shift between the challeng- ing static camera robot environment and pretraining data, consistent with previous findings [17.33, 34, 35J. GPT-4o the best-performing baseline, highlighting the perfor- mance difference compared to open-source VLMs. We found that the VLMs often correctly capture the inter- acted object but fail to perform correct temporal reason- ing. Additionally, we observed that VLMs tend to suc- cessfully predict high-level state changes, such as opening or closing objects, but struggle with more nuanced interactions.
(Table 1).
Table 1: Grounding accuracy of NILS on our mixed dataset compared against baselines and ablations. NILS outper- forms all baselines by a wide margin. TA: Temporal Aggregation, MFOR: Multi Frame Object Retrieval, DI: Depth Information. GT: Ground Truth Key States.
3.3 Labeling Results for Bridge Dataset
To assess the scalability of NILS, we applied it to a larger, more diverse dataset. Therefore, we label a subset of V2 that consists of 31 K trajectories in diverse environments. The cost for NILS to label this subset was approximately 200 USD, compared to about 5000 USD for crowd-sourced labels?. This demonstrates the low-cost of NIL.S. To analyze the diversity of generated labels, we visualized the top labels, sorted by frequency, in Subsection B. This visualization confirms the diversity of labels generated by NILS compared crowd-sourced annotations. Additionally,
Based on the reported labeling cost of approximately 10, 000 USD for the full dataset.

---

ablation experiments in Subsection B.2 showcase NILS's ability to generate task labels at various granularities, further highlighting its flexibility.
3.4
Languuge--onnditooee Policy Training
Simulation Bridge
Bridge Real Robot
Method
Fractal Success
Spoon on Towel
Sushi in Bowl
Average
Success
CG
Success
CG
Success
CG
Success
CG
Gemini
0%
11%
GT
4.2%
41.7%
1.9%
60%
93%
60%
73%
60%
83%
NILS
2.8%
29.6%
3.1%
66%
100%
60%
80%
63%
90%
Table 2: Success rates for simulated (left) and real environments (right) across different tasks. We
evaluate all methods on grounding ability (CG) and task success rate. Results are averaged over 15 rollouts for each task in the real robot environments.
To study the generalization of NILS for large-scale, diverse datasets, we train a language-conddtioned policy with labels from NILS on the BridgeV2 [1], and Fractal [5] datasets. We use Octo [36], recent open-source diffusion policy and train it using 3 different label sets: crowd-sourced labels (GT), NILS labels, and Gemini labels. We train Octo [36] different data mixtures evaluate the capabilities of our labeling framework for downstream policy learning. In particular, we train Octo on the subset of BridgeV2 V2 labeled by NILS and Fractal fully labeled by NILS. We always train the baseline policies with the same datasets, but switch out the language labels with their respective counterparts. Due to labeling costs and Gemini's low performance, we omit labeling Fractal with Gemini. Further evaluation details are provided in Appendix
Results. The results of this experiment are summarized in Table 2. In the simulated environment, our method outperforms the policy trained on labels generated by Gemini by 19% in terms of correct grounding and 2.8% in terms of success rate while being slightly behind the crowd-sourced anno- tated dataset for V2. On Fractal, the policy trained with NILS labels outperforms the policy trained on ground- truth labels by 1.1%. For the real robot tasks, the policy trained with labels from NILS achieves the highest average performance with a 63% success rate. These results align with findings in the previous section label accuracy. Our experiments demonstrate that currett VLMs alone are insufficient to produce high-quality labels for effective policy training. Further- more, these results highlight that NILS generates labels of high quality even for diverse, large-scale datasets, making it a promising approach for scalable rohot learning.
3.5 Ablation Studies
What are the most crucial components of NILS?
We conduct ablation studies to investigate the importance of various design decisions in NILS. Results are summarized in the lower half of Table 1. Without multi-frame object retrieval in Stage 1, NILS achieves only half the grounding accuracy, highlighting the importance of accurate and complete initial object labels. Our experiments also indicate that the depth in- formation provided by the zero-shot depth prediction model (DI) in Stage 2 is crucial for accurate spatial reasoning, en- abling more precise task labeling. Given the current limita- tions of foundation models, we found that temporal aggre- gation (TA) is essential for obtaining reliable and consistent bounding boxes and segmentation masks across frames. Thor- ough ablations for every design decision in all Stages of NILS are provided in Appendix C, confirm- ing the importance of each Stage for achieving accurate labels.
Method
Captured Objects
Ours (Gemini)
0.94
Ours (Gemini) SOM
0.74
Ours (GPT4V) Ours (Prismatic (35))
0.67
0.48
Single Frame (Gemini) OWLv2 + SigL.IP
0.70
0.63
Table 3: Ablation of the effective- ness of initial object retrieval measured in recall.
How to retrieve all relevant objects in the scene? To reason about robot-object interactions, it is crucial to detect all relevant objects in the scene. We investigated various VLMs and combina-
7

---

tions for object retrieval in Stage 1. We compare NILS design using Gemini with co-occurence and detection alignment against several variants using other VLMs like GPT-4V and open-source base- line Prismatic [35]. Additionally, we tested a combination of popular open-source open-vocabulary models: SigLip [7] + OwIV2 [37]. To assess the effectiveness of all approaches, we create a di- verse dataset consisting of the V2 and our Kitchen Play dataset. In total, the ablation dataset comprises 185 objects in various scenes. The results for this experiment
summarized in Ta- ble 3. Naively querying VLM for a single frame does not reliably capture all objects, especially in cluttered scenes. NILS without co-occurrence and detection alignment, denoted as "Single Frame", only captures 70% of the objects present in the scenes, emphasizing the importance of filtering and improving initial object detections. While OWLv2+SggLIP presents a viable alternative, it requires prior knowledge about objects that might appear the scene. We also ablate Set of Mark Prompts (SOM) [38], with our method, but found that this harms performance.
4 Related Work
Key State Identification. Identifying important keystates from long-horizon tasks crucial for efficiently learning goal-conditioned policies. Some approaches use proprioceptive observations [39, 40, 41, 42] waypoint reconstruction loss [39]. UVD utilizes the latent space of pretrained image embedding models such as CLIP [21] to detect keystates. Similar to NILS, it also does not require any robot signals. However, NILS additionally incorporates more specific information, which significantly improves keystate quality. REFLECT [28] constructs a scene graph with object relations and states, labeling frames as keyframes when the graph changes. However, this method relies on ground truth state information and object positions, which are usually only available in simulated environments. NILS detects keystates in real-world environments from robot videos only.
Action Recognition. Video action recognition involves retrieving actions performed over multiple frames, either through dense video action recognition (extracting multiple actions and their time frames) or video action classification (assuming a single action per video). Generalist VLMs can solve these tasks in zero-shot, open-vocabulary manner [43, 44, 45, 46, 47]. RoboVQA [17] finetunes a video VLM a labeled robot demonstration VQA dataset answer questions about robot actions in robotics. REFLECT [28] extracts scene graphs that could be used for action retrieval but assume known robot plan to analyze robotic failure cases. Several works fine-tune CLIP [48, 49, 50, 51] using small subset of domain-specifir data. These CLIP models then retrieve actions for a larger, unlabeled dataset. These approaches assume known key states and require labeled in-domain finetuning data. To the best of our knowledge, NILS the only method that accurately labels real-world long-horizon robotic data without any model fine-tuning.
Discussion
Limitations. Despite NILS' ability to generate high-quality labels, some limitations remain: Us- ing multiple models for generating scene representations results significant computational cost, requiring 7 minutes to label one 8-minute long-horizon trajectory consisting of 50 tasks on 3090 RTX GPU, Correlated heuristics can sometimes lead to high-confidence keystates triggered by noise, resulting in incorrect labels. Grounding accuracy is limited by the performance of pretrained mod- els. NILS has high objectness assumptions for labeling, making it challenging to label tasks with granular objects and distinguish between visually similar objects due to the limitations of object detectors.
This work introduces NILS, the first framework to autonomously label long-horizon robot datasets without human intervention any model training. NILS offers cost-effective alter- native to annotate robot demonstrations compared to crowd-sourcing. Our experiments and exten- sive ablations showcase NIL.S' capability to generate labels of high quality and diversity, enabling efficient and scalable language-conditioned robot learning.
Conclusion.
8

---

Acknowledgments
The work presented here was funded by the German Research Foundation (DFG))448648559 The authors acknowledge support by the state of Baden-Wol Württemberg through HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden-Württenn Württemberg and by the German Federal Ministry of Education and Research.
References
[1] H. R. Walke, K. Black, T. Zhao, Q. Vuong, C. Zheng, P. Hansen- Estrucc,, A. W. He, V. My- ers, J. Kim, M. Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pages 1723-1736. PMLR, 2023.
[2] O. Mees, L. Hermann, E. Rosete-Beatt Beas, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3)7332777334, 2022.
[3] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. CoRR, 2024.
[4] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. JEEE Robotics and Automation Letters, 2023.
[5] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, Leal, K. Lee, S. Levine, Lu, U. Malla, D. Manju- nath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, Perez, K. Pertsch, J. Quiambao, K. Rao, M. S. Ryoo, G. Salazar, R. Sanketi, K. Sayed, J. Singh, S. Sontakke, Stone, C. Tan, H. T. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-1: robotics transformer for real-world control at scale. In K. E. Bekris, K. Hauser, S. L. Herbert, and J. Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14 14, 2023, 2023. 00..0..15560//SS.023....
[6] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, D. M, J. Peralta, et al. Scaling Robot Learning with Semantically Imagined Experience, Feb. 2023.
[7] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986, 2023.
[8] T. Lüddecke and A. Ecker. Image segmentation using text and image prompts. In Proceedings of the IEFF/CVF conference computer vision and pattern recognition, pages 7086-7096, 2022.
[9] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam. CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory. 2022. 400:1044850//AXX12220006 arXiv Version Number: 3.
Publisher:
[10] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler. -pen-uocybylary queryable scene representations for real world planning. In 2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 15509-11522. IEFE, 2023.
[11] M. Reuss, Ó. E. Yağmurlu, F. Wenzel, and R. Lioutikov. Multimodal diffusion transformer: Learning versatile behavior from multimodal goals. In Robotics: Science and Systems, 2024.
[12] P. Liu, Y. Orru, C. Paxton, N. M. M. Shafiullah, and L. Pinto. Ok-robot: What really matters in integrating open- spen-knowledge models for robotics. arXiv preprint arXiv:2401. 2024.

---

[13] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
[14] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, et al. Do i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01666 2022.
[15] C. Huang, O. Mees, A. Zeng, and W. Burgard. Audio visual language maps for robot naviga- tion. Proceedings of the International Symposium on Experimental Robotics (ISER), Chiang Mai, Thailand, 2023.
[16] W. Chen, O. Mees, A. Kumar, and S. Levine. Vision-language models provide promptable representations for reinforcement learning, arXiv preprint arXiv:2402.02657,, 2024.
[17] P. Sermanet, T. Ding, J. Zhao, F. Xia, D. Dwibedi, K. Gopalakrishnan, C. Chan, G. Dulac- Arnold, S. Maddineni, N. Joshi, et al. RobotVQA: Multimodal long-horizon reasoning for robotics. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 645-652, IEEE, 2024.
[18] M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine. Robotic control via embodied chain-of-thougeee reasoning. In Conference on Robot Learning, 2024.
[19] N. Hirose, C. Glossop, A. Sridhar, D. Shah, O. Mees, and S. Levine Lelan: Learning a Linguag-coondtioone navigation policy from wold video. In Conference on Robot Learning, 2024.
[20] B. Chen, Z. Xu, S. Kirmani, B. lchter, D. Sadigh, L. Guibas, and F. Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455 14465, 2024.
[21] Z. Zhang, Y. Li, O. Bastani, A. Gupta, D. Jayaraman, Y. Ma, and L. Weihs. Universal visual decomposer: Long-horizon manipulation made easy. 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6973-6980. IEEE, 2024.
[22] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Li, J. Yang. H. Su, J. Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023.
[23] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD'96. '96. AAAI Press, 1996.
[24] K. Cheng. S. W. Oh, B. Price, A. Schwing, and J.-Y. Lee. Tracking anything with decoupled video segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1316-13266 2023.
[25] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, Xu, L. T. Takayama, F. Xia, Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. In Conference on Robot Learning (CoRL). Proceedings of the Conference on Robot Learning (CoRL), 2023.
[26] X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sich, S. Kir- mani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941, 2024.
[27] H. Xu, J. Zhang. Cai, H. Rezatofighi, and D. Tao. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8121-8130, 2022.
10

---

[28] Z. Liu, A. Bahety, and S. Song, REFLECT: Summarizing robot experiences for failure expla- nation and correction. In Conference on Robot Learning, pages 3468-3484. PMLR, 2023.
[29] L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings the Vision and Pattern Recognition, pages 10371-10388 2024.
the IEEE/CVF Conference on Computer
[30] G. Team. Gemini: A family of highly capable multimodal models, 2023.
[31] Y. Ma, G. Xu, X. Sun, M. Yan, J. Zhang, and R. Ji. X-Clip: End-to-end multi-grained con- trastive learning for video-text retrieval. In Proceedings of the 30th ACM International Con- ference on Multimedia, pages 638-647, 2022.
[32] B. Lin, Y. Ye, B. Zhu, Cui, M. Ning, P. Jin, and L. Yuan. Video-Ilava: Learning united visual representation by alignment before projection, 2023.
[33] X. Wang, Y. Zhou, X. Liu, H. Lu, Y. Xu, F. He, J. Yoon, T. Lu, G. Bertasius, M. Bansal, al. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences, 2024.
[34] L. Guan, Y. Zhou, D. Liu, Y. Zha, H. B. Amor, and S. Kambhampati. "Task Success" is not enough: Investigating the use of video-language models as behavior crities for catching undesirable agent behaviors, 2024.
[35] S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh. Prismatic VLMs: Investigating the Design Space of Visually-Conditionn Language Models, Feb. 2024.
[36] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, et al. Octo: An open-source generalist robot policy. https://octo-modee github. io, 2023.
[37] M. Minderer, A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection. Ad- vances in Neural Information Processing Systems, 36, 2024.
[38] J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-Mark Prompting Unleashes Ex- traordinary Visual Grounding in GPT-4V, Nov. 2023.
[39] J. Borja-Diaz, O. Mees, G. Kalweit, L. Hermann, J. Boedecker, and W. Burgard. Affordance learning from play for sample-efficient policy learning. In 2022 International Conference Robotics and Automation (ICRA), pages 6372-6378. IEEE, 2022.
[40] O. Mees, J. Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data, 2023.
[41] S. James and A. J. Davison. Q-attention: Enabling efficient learning for rision-based robotic manipulation. IEEE Roboties and Automation Letters, 1(2):1612-169, 2022.
[42] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMLR, 2023.
[43] A. Yang, Nagrani, H. Seo, A. Miech, Pont-Tuset, I. Laptev, Sivic, and C. Schmid. Vid2seq: Large-scale pretraining of visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10714-10726, 2023.
[44] S. Yan, T. Zhu, Z. Wang, Y. Cao, M. Zhang, S. Ghosh, Y. Wu, and J. Yu. VideoCoCa: Video- Text Modeling with Zero-Shot Transfer from Contrastive Captioners, Mar. 2023.
[45] Z. Wang, M. Li, R. Xu, L. Zhou, J. Lei, X. Lin, Wang, Z. Yang. C. Zhu, D. Hoiem, al. Lan- guage models with image descriptors strong few-shot video-language leamner. Advances in Neural Information Processing Systems, 35:8483-8493 2022.

---

[46] H. Fang, P. Xiong, L. Xu, and Y. Chen. CLIP2Video: Mastering Video- Text Retrieval via Image CLIP, June 2021.
[47] D. Ko, S. Lee, W. Kang, B. Roh, and J. Kim. Large Language Models are Temporal and Causal Reasoners for Video Question Answering, Nov. 2023.
[48] T. Xiao, H. Chan, P. Sermanet, A. Wahid, A. Brohan, K. Hausman, S. Levine, and J. Tompson. Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models, July 2023.
[49] V. Myers, A. W. He, K. Fang, H. R. Walke, P. Hansen-Estruch, C.-A. Cheng, M. Jaloheanu, A. Kolobov, A. Dragan, and S. Levine. Goal representations for instruction following: A semi- supervised language interface to control. In Conference on Robot Learning, pages 3894-3908. PMLR, 2023.
[50] Y. Ge, A. Macaluso, L. E. Li, P. Luo, and X. Wang. Policy adaptation from foundation model feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19059-19069, 2023.
[51] Y. J. Ma, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image repre- sentations and rewards for robotic control. In International Conference on Machine Learning, pages 23301-23320. PMLR, 2023.
[52] H. K. Cheng and A. G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In European Conference on Computer Vision, pages 640- 658. Springer, 2022.
[53] J. Zhang, K. Pertsch, J. Zhang, and J. J. Lim. SPRINT: Scalable policy pre-training via lan- guage instruction relabeling. In 2024 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 9168-9175. IEEE, 2024.
[54] Y. Xiong, B. Varadarajan, L. Wu, X. Xiang, F. Xiao, C. Zhu, X. Dai, D. Wang, F. Sun, F. Ian- dola, et al. EfficientSAM: Leveraged masked image pretraining for efficient segment anything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 161 16111-16121, 2024.
[55] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training, 2023.
[56] A. O' O'Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta,
A. Mandlekar, A. Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration O. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892-6903, IEEE, 2024.
[57] M. Reuss, M. Li, X. Jia, and R. Lioutikov. Goal conditioned imitation learning using score- based diffusion policies. In Robotics: Science and Systems, 2023.
[58] J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh. Physi- cally grounded vasion-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462-12469, IEEE, 2024.
[59] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMLR, 2020.
[60] E. Rosete- Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task- agnostic offline reinforcement learning. In Conference Robot Learning, pages 1838-1849.
PMLR, 2023.
[61] C. Lynch and P. Sermanet. Language Conditioned Imitation Learning Unstructured Data, July 2021.
12

---

[62] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1205-11212, 2022.
[63] S. Sontakke, J. Zhang, S. Arood, K. Pertsch, E. Bıyık, D. Sadigh, C. Finn, and L. Itti. Robo- clip: One demonstration is enough to learn robot policies. volume 36, 2024.
[64] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task- agnostic offline reinforcement learning. In Conference Robot Learning, pages 1838-1849. PMLR, 2023.
13

---

A Additional Method Details
A.1 Overview of Pretrained Models for NILS
We summarize all used pretrained foundation models for NILS. All models are used without any fine-tuning. We want to highlight that all models can be exchanged for other pretrained models of the same category. Further, an overview of the usage all different models provided in Table 4.
Model Class
Stagel
Stage2
Stage3
VLM
Contrastive Vision-Language Encoder
Open Vocab. Detec.
Open Vocab. Detec. 2
Mask Tracking
Bounding Box Clustering
Depth Estimation
Large-Language Model
Flow Estimation Model
Table 4: Summary of each pretrained model usage across each Stage of the NILS framework.
VLMs for Object Retrieval: Gemini V 1.5 [30] NILS detects all objects in a scene by prompting a VLM for multiple frames. We found that Gemini 1.5 surpasses other VLMs for initial object retrieval, as shown Table
Contrastive Vision- Language Model (CVLM) due to it showing strong performance across many benchmarks. Given an image and several text descriptions, SigLip computes a score that measures the compatibility of text and images. NILS uses SigLiP in Stage 1 to improve the object detection accuracy when multiple labels are available for the same object. Stage 2, SigLip is used to track object changes of static objects, such as drawer, by cropping the image part of the scene with the relevant object and computing scores for all possible object changes.
Contrastive Vingun-L Encoder: SigLiP [7] NILS uses SigLiP
Open Vocabulary Detection 1: GroundingDINO [22]
NILS uses DIO as the main open- vocabulayy detector for object detection. We found that GroundingDINO is better at detecting more specific object descriptions containing colors, while OWLv2 better detecting simple object descriptions. This can be attributed to the token-matching loss of GroundingDINO. As the initial object list is generated by generative VLM, the object descriptions tend to be more specific, and OWLv2 often fails to capture these relations, so opt for GroundingDINO.
Class-Agnostic Bounding Boxes: OWLv2 [37] During the initial object retrieval stage, NILS uses class-agnostic bounding boxes and their corresponding objectness scores from OWLv2 to filter out low-objectness objects. IOU matching employed associate the class-agnostic bounding boxes with the objects detected by the VLM.
Mask Tracking: DEVA [24] DEVA is a mask tracking framework based on XMem [52], which merges new incoming detections with propagated detections for temporal consistent masks. We utilize DEVA with few simple extensions to obtain temporally robust masks. This especially important for cases in occlusion, where the object detector would falsely detect another object if the actual object is not visible.
Bounding Box Clustering: DBSCAN [23] Often, static objects can have different states. To ensure robust state predictions for static objects, such as a drawer or an oven, NIL.S performs static object bounding box refinement as described See. A.3
Depth Estimation: DepthAnything [29] Depth is important metric for capturing spatial rela- tionships between objects. Since NILS labels demonstrations from RGB videos that do not contain
14

---

depth information, we approximate the depth using DeythAny [29] as an approximation. NILS uses the model in Stage to monitor relevant object relations when the robot manipulates objects and changes in their positions.
Large-Language Model: Gemini Pro In Stage I, NILS uses LLM to generate possible object states, and in Stage 3, it employs the LLM to generate language labels for the short sequence. NILS utilizes Gemini Pro for both tasks due to its integration with Gemini V, which is used as a VLM in the earlier stages, and its lower costs compared to GPT-4.
Flow Estimation Model: GMFLOW [27] To detect object movements that do not show bounding box displacement, such as cabinet opening, NILS incorporates optical flow information in addition to the object's ss bounding box.
A.2 Stage I: Identifying Objects in the Scene
Initially, NILS utilizes 8 equally distributed frames from a long-horizon demonstration video to query Gemini. For each frame, NILS prompts the model to output concise, specific, and unique names for all objects the scene, as well as their colors. Figure 16 in the Appendix shows the prompt used for initial object retrieval. Querying the VLM for multiple frames significantly in- creases the number of detected objects, particularly in cases of occlusion. However, the same object is now likely referred to under different names in different frames. Thus, NILS leverages tempo- ral consensux, co-occurrence, and object detection alignment to select consistent and representative object names across frames. For temporal consensus, NILS prompts GroundingDINO with all ob- ject descriptions generated by the VLM for all 8 frames to generate a set of bounding boxes for each frame. Next, co-oceurrence is measured for each individual frame by computing bounding box IOUs, capturing different names referring to the same object. If the IOU is above a certain threshold, NILS assumse that the corresponding object names refer the same object. The co-occurring objects counted and grouped for all 8 frames, and a representative object name is chosen according to object detector confidence. Object names with the highest confidence scores per group are stored in synonym lists to enrich the instruction. This approach generates contextualized grounded object synonyms, increasing the overall data diversity. Furthermore, this approach can be considered as au- tomatic prompt engineering. Some object descriptions can result in better, more consistent bounding boxes produced by the object detector. By selecting the object name with the highest confidence, we also select the object name that is most likely produce correct detections for the other frames. Fi- nally, an LLM assigns the properties movable, is container, states, and interactable each detected object. These properties help to filter wrong robot-object interactions in subsequent Stages.
As simple baseline, we employ a combination of OWLv2 [37] and SigLIP [7]. Given a predefined list of objects commonly appearing in robot environments, we compare class-agnostic bounding boxes generated by OWLv2 with class embeddings of all objects in the list. We further cut out objects based on their class-agnostic bounding boxes and compare text-image similarity with SigLIP, Finally, average the scores of OWLv2 and SigLIP for each class-agnostic bounding box and retrieve the object with the highest score. This approach is promising when we have pretrained list of objects but falls short for similar objects, such as different colored objects of the same instance.
For set-of-mark prompting, we annotate the frames with enumerated class-agnostic bounding boxes generated by OWLv2 and task the VLM assign an object name to each numbered bounding box.
A.3 Stage 2: Object-Centrie Scene Annotations and Information Retrieval
Object Filtering and Mask Refinement through Temporal Aggregation.
Detecting keystates accurately based on object masks and boxes requires temporally consistent ob- ject masks. However, initial object detections may suffer from temporal misalignments, such as missing detections for certain frames or an object being classified with synonym for different frames. To address this challenge, NILS utilizes DEVA [24], mask-tracking model, capture temporal correlations between objects. DEVA propagates masks with X-Mem [52] while frequently
15

---

incorporating new segmentations. The new segmentations are fused with the propagated segmen- tations through alignment. NILS further extends DEVA to incorporate class score for each prop- agated mask. The resulting final mask belonging to each object then has multiple class scores of possibly different classes associated with it. NILS obtains the most confident class and labels the object as the determined class. NILS extends DEVA incorporate class scores for each propagated mask, obtaining the most confident class for each object. This results in bounding boxes for all timesteps and more robust predictions. NILS also applies techniques to mitigate DEVA issues, such as merging masks of the same class, keeping the highest iitersection-ooer-uniooo mask component, and filtering masks for temporally consistent and coherent class labels. In particular, DEVA masks often split if objects of similar visual features overlap. We mitigate this issue by computing indi- vidual mask components and removing the component with low area. After applying DEVA and filtering, the masks are temporally more consistent and have consistent class labels.
(a)
(b)
(c)
(d)
Figure 5: Static object detection refinement. (a) shows the initial noisy labels. The boxes are then filtered by removing statistical outliers (b) and by obtaining the highest confidence cluster (c). The final averaged box is visible in (d)
Increasing Detection Robustness for Static Objects. For non-moving objects, NILS employs a temporal consensus approach to enhance detection robustness and accuracy, especially in scenarios prone to occlusion. A two-Stage filtering process identifies the most representative bounding box for static objects over time, first by eliminating statistical outliers and then by clustering the remain- ing boxes using DBSCAN [23]. which also detects noise labels. The final bounding box for each static object is derived from the cluster with the highest overall confidence, representing the object consistently across frames.
A.3.1 Object-Centric Information Retrieval
Object Relations and Object Movement We generate an object-relation graph from segmentation masks generated in the first step. First, the objects are projected into a point cloud with depth map generated by Depth-Anythingy2 [29]. We further perform additional filtering steps, such as outlier detection and downsampling. To reason about object movement in natural language in camera space, we project the pointeloud into a coordinate frame that aligns with flat surfaces in the scene. First, NILS detects surfaces, such as floor, stove top. table, counter in the scene. The detected surface projected into the pointcloud, followed by plane segmentation and normal estimation. The normal acts as the upvector the new coordinate system, whereas the front-vector points towards the camera. Furthermore, many tasks include changing the position of an object with respect the surface it is located on. To detect the position of objects on a surface, NILS uses the previously detected surface object and performs homography transform account for camera perspective. To do this, we fit quadrilateral to the surface's 's segmentation mask and compute transformation matrix from the detected corners to image corners. We then project the objects bounding boxes with this matrix and compute object position on the surface by categorizing positions a 3x3 grid. Object State Prediction. NILS crops objects from images based on their bounding box. We add a small padding to the bounding box, to ensure all relevant information is present in the cropped image. Robot mask IOU determines occlusion with the cropped region. Then, NILS compares the CLIP similarities of state text embeddings and the cropped images. The prompts have the form "A picture of a <state> <object>".
16

---

B Additional Experiments
B.1 Diversity Comparison of Generated Task Labels
Comparison of Tasks from NILS and default Bridge Labels
NILS Tasks
30000
Bridge Tasks
8gi Eploodes
100
400
do
prases
cottrr
Tasks
(a) Comparison of the top 60 tasks labels generated by NILS and the top 60 souresd annotation on the BridgeV2 dataset.
Compartoon of Objects from 5 and defa.lt Brstge Labeis
7500
$30g
NILS Object
Bridge Object
1500
3300
2000
Objects
(b) Comparison of the top 60 object labels generated by NILS and the top 60 crowd-sourced annotation on the BridgeV2 V2 dataset.
3300
Comparison of Spotial Relations from NLS and defoult Bridge Labels
6300
NILS Spatial Relation
538-
Bridge Spatial Relation
4300
3300
750-
S00e Mof
Cood pue audi and
S0 120 Ro
and
samu
940
tt wa
Spatial Relation
(c) Comparison of the top spatial arrangement labels generated by NILS and the top 60 crowd-sourced annotation on the B2idgeV2 dataset.
Figure 6: Comparison of labels generated by NILSS and crowd-sourced annotation on the Bridge V2 dataset. The Bridge version used by NILS is around 4 times bigger then V2.
17

---

B.2 Hierarchical Instruction Generation
Fine-Graine Grei-Gr Tasks: Move the toy mouse to the center of the washing machine Place the plastic egg next the toy mouse High-Level Tasks: Put the toys in the washing machine.
Fine-Grained Tasks: Reposition the pan the kitchen towel Pick up the chicken wing and place the the pan Move the spoon the bottom left the stove top High-Level Task: Prepare the stove top for cooking the chicken wing.
Fine-Grained Tasks: Open the red drawer of the wooden box Move the green pickle toy from the wooden box the center of table Close the drawer of the wooden box Hige-Level Task: Fetch green pickle toy from box and place with other toys.
Figure 7: Examples of different levels of tasks granularity generated by NILS for two trajectories from the Bridge dataset.
NILS produces keystates and language annotations, which are contextually grounded, for single tasks. By combining the templated language descriptions of consequent keystates into single prompt, the framework can generate high-level language instructions. This allows for granularity control over the produced tasks, which can be beneficial for policy learning. [53]. We illustrate three examples from the Bridge Dataset Figure 7.
C Ablations
C.1 Stage 1: Initial Object Retrieval Ablations
To reason about robot-object interactions, it is crucial to initially capture all objects in the scene reliably. Our initial object detection based on a multiple-frant frame consensus that reliably works in different settings and for different objects. To assess the effectiveness our approach, we create a diverse dataset consisting of the Bridge V2 and our Kitchen Play dataset. In total, the evaluation dataset comprises 185 objects. We compare our approach with different VLMs and baselines.
The first baseline queries& VLM based single frame. In OWLv2 SigLIP, utilize prede- fined list of about 1400 objects commonly appearing in kitchen environments. Then, generate
18

---

class- agnostic bounding boxes with OWLv2 and align all class embeddings with the class- class-agnottic box embeddings generated by OWIv2. Furthermore, an ensembling approach combines the OWLv2 scores with cropped image-text alignment scores generated by SigLIP. Figure shows examples of multi-frame retrieval outperforming naive, single-frame retrieval.
C.2 Stage 2: Scene Annotation
(c 8)
(c = 16)
Amb.
Single
Amb.
Single
Naive
0.59
0.34
0.60
0.33
Grounding
Naive SG
0.62
0.27
0.59
0.26
Temporal Aligmment
0.76
0.53
0.68
0.51
Detection ensembling
0.59
0.50
0.59
0,50
F.F.
0.80
0.61
0.75
0.57
Precision
Recall
Precision
Recall
mAP
Naive Temporal alignment
0.46
0.36
0.67
0.53
0.36
Keystates
0.41
0.45
0.70
0.77
0.45
Detection ensembling
0.46
0.48
0.69
0.73
0.48
F.F.
0.50
0.46
0.75
0.69
0.51
Table Ablation for the effectiveness our perception filtering on our Kitchen Play Dataset. For Naive, simply OWL-v2 and SAM to extract masks and bounding boxes without additional filtering or temporal aggregation. In Naive-SG, we provide full object-relation prompt to the LLM instead of object-centric prompt when retrieving the action. F.F. depicts full filtering.
Detecting all relevant objects in the scene in Stagel is crucial for the performance of NILS. In Table 5, we provide ablations of our perception module. To assess the effectiveness of our percep- tion module, we compare against simple box generation with OWLv2 and object segmentation with Efficient-SAM [54]. We perform ablations by disabling several components: ensembling with a dense open vocabulary predictor, statistical mask outlier filtering, temporal aggregation, state pre- diction without occlusion, and static object box aggregation. We conduct all experiments with keystate threshold of 0.3 and Gemini the LLM. When omit our heavy postprocessing Stages, we observe significant decline in keystate quality and grounding accuracy. Although the drop in keystate precision is not substantial, the recall shows notable decrease. Additionally, the ground- ing accuracy drops significantly, especially when only unambiguous prompts are considered valid. We observed that constraining the prompt information to specific object and its relations helps to reduce hallucination and results in more precise predictions. These findings underscore the neces- sity of robust post-processing techniques to effectively leverage current state-of-the-art perception models in novel and challenging domains.
C.3 Stage 3: Keystate Ablations
Next, we analyze the importance of our keystate heuristic to determine critical states. Figure 8a shows the performance of method when incorporating different keystate heuristics. Gripper close signals present a very strong baseline. However, as mentioned before, gripper close signals are not always avail- able and can not represent all different kinds of tasks. This is shown by the increased precision and recall when incorporat- ing additional heuristics. Especially for a smaller threshold, observe significantly increased performance when incor- porating additional heuristics. Incorporating additional heuris- tics usually results in increase in precision and decrease in
1.
Figure 10: precision and recall for different threshold values for our Kitchen Play dataset.
Keystate detection
19

---

16
gripper
0.73
0.65
Gripper
gripper close object state
0.38
0.36
0.73
0.69
all
0.50
0.46
0.75
0.69
all
0,42
0.40
0.66
0.62
object relations
0.21
0.13
0.56
0.35
pos.
0.29
0.40
0.52
0.71
RGB
gripper pos.
0.51
déjjct movernent
0.31
0.40
0.52
0.68
grapper pos.
gripper pos. state
0.65
0.59
pripper pas.
0.50
(a) Keystate precision and recall when using different keystate heuristics. For all experiments, the keystate detection threshold is set to 0.3, if applicable.
recall. Always using heuristics desired, as the precision- recall tradeoff can then be best controlled by setting an appro- priate threshold. Figure 10 depicts the relation between thresh- old, keystate precision and recall, and grounding accuracy. With increasing threshold, the grounding accuracy and keystate precision increase. This indicates that with our scoring method, the quality of samples can be controlled effectively.
C.3.1 Stage 3: Simulation Keystate Ablation
16
Method
Precision
Recall
Precision
Recall
UVD
VIP
0.16
0.06
0.28
0.10
NILS
0.37
0.21
0.53
0.31
Table 6: Keystate accuracy for different frame distance tolerances on CALVIN Simulation Bench-
mark [2] with RGB Image-Data Only. We compare against UVD, which VIP [55] detect
keystates. NILS outperforms UVD this experiment.
We further evaluate NILS on low-resolution simulation environment that has a high domain shift compared real-world setting. For this experiment, test framework the CALVIN simulation benchmark [2]. CALVIN is challenging benchmark that has long-horizon play data. In Table 6 we show the keystate detection precision and recall on the CALVIN [2] benchmark. NILS utilizes off-the-shelf models trained on real-world data. Thus, these models struggle significantly in simulated environments that contain abstract objects. We had to perform prompt engineering to make the detection and OV-segmentation models detect any objects in the scene. After these changes, the method performs reasonably well.
C.4 Stage 3: Choice of LLM for Label Generation
We further compare the annotations produced by NILS with ground truth language annotations obtained through human labeling on the Kitchen Play dataset. To allow for quantitative analysis of the generated language annotations, we phrase the problem as a multiple-choice problem. Instead of generating language instruction directly, we provide a list of possible tasks from which the LIM has to select up two tasks, given the observations made in Stage Table 7 shows the grounding accuracy of our framework compared to Gemini, S3D, and XCLIP. NILS outperforms all baselines by large margin, whereas GPT-4 is the best LLM.
20

---

0000000
istwwasher
peta
0519
with
des
Stowberry
arood
Multi-fran initial object retrieval
Single-frame frame initial object retrieval
Figure 9: Comparison of multi-frame single-frame object retrieval. The single-frame retrieval often misses objects, and the overall label quality is worse than those of the multi-frame aligned approach.
21

---

Method
LLM
Amb.
Accuracy Is
Single
8)
Amb
Acconne (4
Single
161
S3D XCLIP
0.04
0.00
0.07
0.09
Gomini
GPT-3-5
0.70
0.53
NILS
Gemini (Lang)
0.80
0.61
0.75
0.57
Mistral8s
0.66
0.52
0.62
0.49
Table 7: Grounding accuracy of our framework on our Kitchen Play Dataset. For Amb., the pre-
diction is labeled correct if the list of answers contains the ground truth. In Single, ambiguous predictions are wrong.
D Example Prompts
We give example prompts used to generate the list of potential tasks given a list of objects in Fig- ure 17. In Figure 18, an example prompt used to label the task a robot solved in between two keystates is visualized.
E Qualitative Examples
We show qualitative examples of our framework's produced natural language instructions on BridgeV2 [1] in Figure 21.
F
Additional Experiments
F1
Evaluation Details
To evaluate policy performance on the BridgeV2 dataset, we use both sim and real robot experi- ments using a 6-DoF WidowX robot arm. To evaluate perfomance on the Fractal robot, rely on simulation only. Simulation experiments and real robot experiments are performed using scenes inspired by the Bridge V2 dataset [1]. One Simpler Task is also visualized in the right image of Figure 3. All Octo variants are trained for 300k steps on their respective label sets to ensure fair comparison. In the real robot environment two tasks were tested: Move the spoon inside the towel and Place the sushi inside the [color] bowl, both with various distractors and different starting positions for each object. The tasks are visualized in Figure 11 in the Appendix. We report success rate and correct grounding, where correct grounding refers the robot approaching and interacting with the task-relevant object.
conducted using the Simpler Environments [26],
F.2 Bridge V2 Experiments
Spoon on Tablecloth
Carrot on Plate
Eggplant Basket
Average
Method
CG
Full
CG
Full
CG
Full
CG
Full
NILS
43.1%
6.9%
12.5%
0%
33.3%
1.4%
29.6%
2.8%
Bridge Gemini Baseline
18%
0%
16%
0%
0%
0%
11%
0%
GT
37.9%
11.1%
40.3%
1.4%
27.8%
0%
41.7%
4.2%
Table 8: Success rates for simulated environment methods across different tasks.
CG (Correct
Grounding) refers the robot successfully grasping the task-relevant object.
We conduct the experiments for Bridge V2 in SIMPLER [26] and on a real robot. For the real robot setup. we perform 15 rollouts per task with different initial states and distractors. For simulation, we follow the setup provided in SIMPLER. The real-robot and simulation setups can seen in Fig. Table shows partial and full success rates for three different tasks in the simulated environment. The simulation additionally provides partial success metric, which requires the robot grasp the task-relevant object. This metric can be interpreted similarly correct-grounding metric.
22

---

Place the green spoon on top of the
Place the sushi inside the bowl
Place the spoon on the cioth
Figure 11: Tasks used for evaluation in Bridge real-worke (left) and simulation (right)
E3
Fractal Experiments
Open Drawer
Close Drawer
Move Near
Pick Coke Can
Average
Method
VM
VAR
VM
VAR
VM
VAR
VM
VAR
NILS
1.5%
0.5%
16.3%
1.5%
6.2%
2.9%
2%
0.7%
3.1 %
GT
0%
0%
9.3%
1.6%
6.7%
2.1%
1%
0%
1.9%
Table Success rates for simulated environment methods across different tasks. VM refers to visual matching, where the surfaces and environments are similar to the environment in the training data. VAR refers to variant, where scene located in different rooms and the surfaces have different textures.
Furthermore, we conduct experiments with dataset collected on a Google Robot [5]. We obtain the dataset from Open-X Embodiment [56] and label 87k trajectories with NILS. We train Octo the labeled dataset and additionally co-train policy with Bridge labeled by NILS, where choose the dataset weights so each dataset appears equally often during training. We found that cotraining with B2idge improves the performance Fractal. We again evaluate the policies in SIMPLER [26], where the robot has to solve 3 tasks with several different variations. Table 9 shows the success rate for different tasks SIMPLER. The policy trained with NILS outperforms the policy trained on woodd-oouces annotations.
23

---

F4 Real Robot Experiments in the Toy Kitchen
The following section describes our real-world play kitchen environment conducted in a toy kitchen environment detail. We collect play data through teleoperation in our robot kitchen environ- ment. The setup is illustrated in Figure 19. The robot can solve 12 different tasks, as shown in Fig- ure 20.
1.5
Success Rate
0.89
0.89
0.47
0.5
0.28
Success
Correct Grounding
We evaluate the performance of the trained poli-
cies based perform each task three times and calculate the average number of successful task completions. We then compute the average success rate over all tasks. Correct Grounding. We evaluate whether the policy correctly understands language instruc- tions. The task does need be completed successfully. The robot only has to show that it correctly understood the task. For instance, if the robot approaches the oven and tries to open it but fails, we label the task as correctly grounded. We again compute the average over all possible tasks.
metrics: Success Rate. We
NIL. NILS Noisy
Homan
Figure 12: Policy performance trained on a play dataset with ground-truth labels, seg- mented trajectories and labels from NILS and noisy labels from NILS. The policy based on NILS' labels competitive with the one trained on human-annotated labels. Incorpo- rating low-quality labels hurts the policy's per- formance.
We train three policies: Human, policy trained on ground truth, annottted data; NILS, policy trained on data labeled by NILS, where discard ambiguous labels; and NILS Noisy, where, if the LLM is unsure about the tasks and outputs multiple language instructions, incorporate the demonstration in the training dataset multiple times with each generated instruction. The grounding accuracies and success rates for each task are shown in Table 10.
Task
Human
NILS
NILS Noisy
Grounding
Success
Grounding
Success
Grounding
Success
banana in sink
1.0
0.67
1.0
1.0
1.0
1.0
pot right
1.0
1.0
1.0
1.0
1.0
0.67
pot left
1.0
0.67
1.0
0.67
1.0
0.33
open microwave
1.0
1.0
0.67
0.67
0.0
0.0
open oven
1.0
0.0
1.0
0.0
1.0
0.0
open fridge
1.0
0.0
1.0
0.0
1.0
0.33
close microwave
1.0
1.0
1.0
1.0
0.67
0.67
close oven
1.0
1.0
1.0
0.67
0.0
0.0
close fridge
0.67
0.33
1.0
0.67
0.33
0.33
banana stove
1.0
0.33
0.33
0.0
0.0
0.0
banana oven
0.0
0.0
0.0
0.0
0.0
0.0
pot in sink
1.0
0.67
1.0
0.0
1.0
0.0
Table 10: Relative performance for task groundings and successful task completions. We report the average performance of all tested tasks, where the first number depicts the relative number of correctly grounded tasks, and the second the relative number of successful completions.
Real Robot Policy, For our experiments, we use the BESO policy architecture [57]. The model consists of transformer architecture and uses a continuous time diffusion generative model to gen- erate sequence of future actions. A pretrained CLIP text encoder encodes the text instructions, while images encoded with FiLM-conditioned ResNet-18s. We train the resulting policy our real-robot dataset for approx. 400 epochs with batch size of 512. Our policy learns to predict sequence of joint state positions.

---

Results. Next, we train Eunuuae--cnditioon policy [57] on the kitchen dataset and compare its performance using labels provided by NILS and human annotations. The policies are evaluated on their instruction understanding and success rate in solving natural language tasks (Table F.3). Figure shows that the model trained with NILS labels performs competitively against the model trained with human annotations across 12 tasks, further demonstrating the quality of the labels generated by NILS.
G Limitations
Perception. NILS demonstrates the ability of off-the-shelf specialist models to annotate challeng- ing long-horizon data. The major limitations of our framework are induced by these off-the-shelf models. Commonly used robotic environments and their contained objects are still very challeng- ing for state-of-the-art models as most of them have not been trained on robot domain data. For instance, common evaluation environments in robotics are toy kitchens. Open-vocabulary detectors often struggle with grounding in such environments. For instance, NILS frequently detects the ba- as sponge in our toy kitchen setup. While there models specifically applicable to the robotic domain, such Spatial-VLM [20], RoboVQA [17] PGBlip [58], these models are either not open-source or too specific for broader grounding applications. Lastly, the initial object retrieval might fail for abstract environments. However, this can be assessed through minimal human in- tervention. NILS allows users to specify objects they expect to appear in the scene, which would improve performance in these scenarios. Given the modularity of NILS, better models also result in a higher grounding accuracy.
Runtime. Using multiple different models to generate scene representations introduces substantial computational cost. The inference time of our framework is significantly higher compared to tested baselines. NILS requires minutes to label one long-horizon trajectory consisting of 50 tasks on a single 3090 RTX GPU. The framework is designed to be applied offline to prerecorded play data, thus the higher runtime should be manageable.
Accuracy. While our framework shows good performance for different scenes and objects, it some- times produces wrong labels. Although the heuristics used to detect keystates can filter noise effec- tively, they are sometimes correlated. In such cases, noise triggers all keystate heuristics, resulting in high confidence keystate. Furthermore, NILS is designed for long-horizon demonstrations. To generate robust scene annotations, NILS relies on temporal consistency and diversity, which is not always given in short horizon demonstrations.
Cbject-Centric Trade-off NILS has high objectness assumptions for accurate labeling. This makes it challenging to label tasks with granular objects such as rice corns. In future work, we want to explore how to better integrate the VLM in such scenarios. Additionally, due to the state of object detectors, we can not distinguish between objects that are very similar appearance, such as similar blocks.
H Extended Related Work
H.1 Learning from Play
Robots usually learn from short-term demonstrations of single task, but this approach has draw- backs such lack of diversity, inefficient data collection [17], and poor generalization. As an alternative, Learning from Play [59] proposes collecting unconstrained operator interactions in the environment, resulting in more diverse data and better generalization. Extracting goal states from these long demonstrations is necessary for goal-conditioned imitation learning. While sampling random windows or using robot proprietary information can provide image goals [59, 60, 11], ex- tending language goals is challenging they must align with observations. Most methods use joint goal spaces to embed multiple modalities [2, 61, 49, 62, 57, 63], but still require some lan- guage annotations. Our approach densely predicts actions performed in long-horize trajectories to
25

---

address this issue. Long-horizon datasets with annotated languages, like CALVIN [2] or TACO Play [40, 64), are searce due to these challenges.
I Keystate Visualizations
Figure 13: Visualization of keystates extracted by NILS for the CALVIN environment. Keystates are extracted only via RGB Images, without incorporating gripper-close signals. the second sequence, some intermediate keystates not detected. Nevertheless, the predicted keystates precise.
Open
Open fridge
Close nicrowave
Place sponge in
Move right
Close
left
Close fridge
Open oven
Figure 14: Example trajectory labeled with NILS. NILS successfully detected the keystates in the long-horizon trajectory and correctly grounds the performed action for tasks.
26

---

Foundation Model Prompts
<Frame 1>... <Frame 8>
Given the image sequence, what task did the robot perform? Output the task the robot performed short instruction with three paraphrases, delimi ted by comma. The task should include the interacted object other objects.
object's resul ting relation to
Example tasks: "Place the pot to the left of the fork", "Move the dishrag to the bottom of the table next to the blue towel", "Pick up the cucumber and place it to the right of the bowl", "Turn stove", "Open the microwave", "Put the knife inside the sink". Output valid json in the following format:
"tasks": "the predicted tasks. The tasks should always include the final object positions if possible.")
Figure 15: VLM Baseline Prompt
<Image
Create json list where each entry is an object in the image. The entry should have the keys "name" with unique, concise, up to six-word description of the object and "color", containing the object color. Always output the surface the objects are placed the first entry. If the surface is unknown, output "None". An example:
[["name": pot with handle, "color": "silver"]] ...
Figure 16: Stage Object Retrieval Prompt.
You will be provided with list of objects observed by robot. Based the objects, give possible instructions to robot. Infer type of environment from the provided objects. Follow these guidelines:
Keep the instructions simple. Focus tasks that only require single step.
Include like placing an object inside another object or moving the object.
Only for movable objects.
Dont assume the presence of any objects not listed. Output at least 20 possible instructions delimited by comma.
Here are a few examples: "Place the tin can to the left of the pot.". "Move the dishrag to the bottom of the table next to the towel", "owel","Put "Put the pot to the right of the fruit", "Turn stove", "Open the microwave"
The following objects are the environment:
[OBJECT LIST]
Figure 17: Task generation prompt
27

---

You will be provided with observations of robot interaction with an environment delimited by triple quotes.
Determine the task the robot could have solved. The robot only solve task. If the observations indicate that the robot interacted with multiple objects, focus on the most frequent and precise observations.
Follow these guidelines: Step Answer what objects appear the observation. List all objects. Then, determine the object for which observations align the best.
Step 2: Determine the object movement and the resulting object relations. Think about where the object and its relational objects are located in the scene on global scale. Think step by step and list the locations and relations of all objects. Explain
object movements.
Step 3: Determine what tasks result in the object relations from Step 2.
Step 4: Output tasks that that accomplish the observations as short instructions. Focus simple, single-step tasks that only require interaction with the determined ned object from Step Focus tasks that include changing the object relation and moving the object.
Example tasks: "Place the pot to the left of the fruit"; Slide the dishrag to the bottom of the table next to the towel"; Pick up the spoon and place it at the bottom left of the table; the pot the right of the fruit"; "Move the pot forward and to the left"; "Turn stove"; "Open the microwave"; "Relocate the knife inside the sink". Follow the steps above. Explain your reasoning. Output the reasoning delimited by ***.
After, produce your output as JSON. The format should be:
''. "tasks": "The determined tasks, delimited by semicolons. Output different, diverse task instructions. The instructions should cover all observations and each include different observations. Example: Place the pot to the left of the fruit; Move the pot backward and to the right; Relocate the pot at the left of the table the center of the table; Lift up the pot and place it next the spoon;", "confidence": "A confidence score for each task between and 10, delimited by commas. Be pessimistic." }
Observations:
[OBSERVATIONS] ..
Figure 18: Main action retrieval prompt
28

---

K Play Kitchen Dataset Description
Top camera
Front camera
Leader robot
Human plays
Follower robot
Figure 19: Overview of the teleoperation setup in the real kitchen environment. The human operates on the leader robot. The follower robot imitates the actions the leader. The top and front cameras record the play trajectory at 30Hz.
open aethggerator
Open microwave
roftigerator
melrooaane
Move right
Move pot to the left
Opan the oven
Place banana the oven
Place banana on the right
Close the oven
Place banana on the left
Figure 20: Overview of the 12 tasks recorded during play from the preprocessed front camera per- spective.
L Comparison to Gemini Vision Pro on Bridge V2
29

---

Instructions generated by NILS: "Move the pepper from inside the strainer to in front of the strainer", "Take the pepper out of the strainer and place it forward", "Relocate the pepper position in front of the strainer"
Instructions generated by Gemini Vision Pro: "Move the yellow bell pepper the left", "Place the yellow bell pepper in the pot". "Move the pot to the right."
Instructions generated by NILS: "Place the saucepan top of the dishrag", "Move the saucepan to the right of the soap". "Position the saucepan behind the ladle."
Instructions generated by Geminí Vision Pro: "Move the cheese the right", "Move the bowl to the right", "Move the spoon to the right", "Move the dishrag to the right."
Instructions generated by NILS: "Move the sushi from top of the dishrag to a new location away from the saucepan", "Relocate the sushi clear the area on top of the dishrag", "Shift the sushi to organize the workspace, ensuring it is no longer next to the saucepan."
Instructions generated by Gemini Vision Pro: "The robot moved the green spatula from the left of the cutting board to the right of the cutting board", "The robot moved the yellow cloth from the right of the cutting board to the left of the cutting board".
Instructions generated by NILS: "Put the reamer (juicer) inside the plate", "Move the reamer (juicer) from next the plate inside it". "Place the reamer into the plate for storage or preparation" Instructions generated by Gemini Vision Pro: "The robot picked up carrot that was resting on green plate and placed it in the sink." "The robot moved carrot from green plate the sink."
Figure 21:
Comparison of generated labels for samples form the bridge V2 dataset. We show the labels generated by NILS and Gemini Pro for several tasks.
30


Reviews:
[{"review_id": "kwc3wPMJT7", "reviewer": "Reviewer_p3J4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to address the challenge of labeling natural language instructions in robot datasets. To do so, it proposes NILS (Natural language Instruction Labeling for Scalability), which uses pre-trained VLMs to identify objects, detect changes, and segment tasks. The authors evaluate on BridgeV2 and a kitchen play dataset and finds that the annotations lead to comparable policy performance compared with human annotations.", "summary_of_recommendation": "The current state of the paper requires some additional experiments and analyses to be convincing. If my concerns are addressed, I will be happy to bump up my recommendation.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See weaknesses above. Also:\n- For NILS vs NILS Noisy, how do you determine what is an ambiguous label?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2tdQxYMV0j", "reviewer": "Reviewer_PVKm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "With this paper the authors are introducing NILS, a system that can be used to label long-horizon robot videos with diverse language annotations. They achieve this by sequentially using an ensable of VLMs that processes the videos in an object centric way. At a high-level this process is divided into 3 stages. The first stage consists in detecting the objects in a scene in a temporally consistent way, filtering redundancies and classifying the objects' properties. The second stage tracks each object's movement and relative relations among other things and defines these as templated language descriptions. Finally, stage 3 identifies the keyframes in the video and provides the templated information for the relevant frames to an LLM which combines and merges these to create a set of diverse language descriptions of variable granularity. Additionally, they evaluate this system on a couple of annotated datasets as well as testing the quality of the labels by training a real world policy with them.", "summary_of_recommendation": "This work can potentially help the community to scale data labelling without the need for humans. The information needed to understand the underlying system is all there. They present enough experiments that can provide insights on the performance and usefulness of the method. However, in order to have a strong enough paper I advise working on the clarity of the manuscript as well as focusing more on the experiments.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I want to be clear that I believe all the necessary material for the paper is definitely there. I only think it would be wise to address the overall clarity of the paper.\n\n**Weakly Suggest**:\n- In the method section I think it would be good to filter out some of the things you discuss and move them to the appendix, where you can discuss them in as much detail as possible. In the actual method section I would discuss the system on a higher level. The sections that would benefit the most are \"Stage 1: Identifying Objects in the Scene\", \"Object Annotations and Segmentations\" and \"Object Relations and Object Movement\". It is perfectly fine in my opinion if the method section becomes shorter, it would just free up some space for more discussion of your findings or moving an additional experiment up from the appendix.\n- Try discussing more insightful ideas or conclusions in the experiments section.\n\n**Strongly suggest**:\n\nAddress the few things mentioned in the Review above.\n\n**Minor Corrections**:\n- Figures 3 and 4 should be flipped\n- Figure 12 is before Figure 11\n- Appendix K is empty because it starts after the relevant figure.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1WSuuxLM3r", "reviewer": "Reviewer_B6Tb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces NILS, a framework for automatically labeling long-horizon robot demonstrations without human annotations. NILS uses an ensemble of pre-trained vision-language models to detect objects, segment tasks, and generate natural language labels for uncurated robot data. The key novelty is leveraging multiple frozen foundation models to enable zero-shot labeling of diverse robot behaviors. The authors evaluate NILS on a subset of BridgeV2 and a custom kitchen dataset, showing it can generate high-quality labels and train language-conditioned policies competitive with human-annotated data.", "summary_of_recommendation": "I recommend a weak accept for this paper. The proposed NILS framework presents a novel and potentially impactful approach to zero-shot labeling of robot demonstrations using pre-trained foundation models. While the preliminary results are promising, the framework's complexity, limited evaluation, and lack of thorough ablations are significant weaknesses. However, the core idea is valuable and addresses an important challenge in scaling language-conditioned robot learning. With revisions to address the identified weaknesses, particularly simplifying the framework and expanding the evaluation, this work could make a strong contribution to the field.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. Can the framework be simplified while maintaining performance? What is the justification for each component?\n\n2. How does NILS perform on larger, more diverse datasets?\n\n3. How does NILS compare to simpler baselines or individual foundation models for this task?\n\n4. Can you provide results on label quality?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work proposes a language annotation strategy for unlabelled video demonstrations using foundation models. They propose a factored approach to generate annotations taking into consideration objects in the scene, their relationships etc. Authors are encouraged to improve the technical exposition in relation to the effectiveness of the annotation strategy in facilitating downstream policy learning, scalability of the method and include comparison with other models.
Decision: Accept