RoboPoint: A Vision-Language Model for Spatial Affordance Prediction in Robotics

Authors: wentao  yuan
jiafei  duan
valts  blukis
wilbert  pumacay
ranjay  krishna
adithyavairavan  murali
arsalan  mousavian
dieter  fox
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-01 04:23:05

Abstract: From rearranging objects on a table to putting groceries into shelves, robots must plan precise action points to perform tasks accurately and reliably. In spite of the recent adoption of vision language models (VLMs) to control robot behavior, VLMs struggle to precisely articulate robot actions using language. We introduce an automatic synthetic data generation pipeline that instruction-tunes VLMs to robotic domains and needs. Using the pipeline, we train RoboPoint, a VLM that predicts image keypoint affordances given language instructions. Compared to alternative approaches, our method requires no real-world data collection or human demonstration, making it much more scalable to diverse environments and viewpoints. In addition, RoboPoint is a general model that enables several downstream applications such as robot navigation, manipulation, and augmented reality (AR) assistance. Our experiments demonstrate that RoboPoint outperforms state-of-the-art VLMs (GPT-4o) and visual prompting techniques (PIVOT) by 21.8% in the accuracy of predicting spatial affordance and by 30.5% in the success rate of downstream tasks. Anonymous project page: https://robopoint.github.io.

Main paper: 
 introduction: 
 Spatial reasoning is fundamental to all intellectual processes [1] . Beyond its prominence in understanding geometry, science, and architecture [2] , spatial reasoning significantly impacts our everyday lives. Even mundane tasks like purchasing groceries require us to identify the vacant space in our shopping carts to load more items. One critical mechanism through which we communicate plans that involve navigation and manipulation is by pointing. Studies in developmental psychology demonstrate that infants and adults alike point to share information about their environment [3] . In robotics, pointing has been operationalized through waypoints for navigation and task execution. Roboticists have found that when robots use waypoints effectively, it mimics human pointing, leading to more intuitive plans [4] . Recent explorations have cast aside pointing in favor of language instructions with the advent of large VLMs [5, 6, 7] . Trained on large datasets of images and texts, VLMs can provide visual semantic understanding and guidance to robotic tasks, such as which object a manipulator should pick up or which goal a mobile robot should reach [8, 9, 10] . However, language is not precise enough to successfully guide robot behavior. Even the most powerful VLMs, such as GPT-4o [11] , have limited accuracy in real robot execution, especially when language commands contain spatial relations to identify objects or refer to object-free locations, such as "place the cup next to the plate". In this work, we introduce ROBOPOINT, an open-source VLM instruction-tuned to point. Specifically, we fine-tune a pre-trained language model to perform spatial affordance prediction, the task of pointing at where to act according to language instructions. Two key features differentiate ROBO-POINT from other VLMs for robotics: a point-based action space and a scalable data pipeline. First, inspired by prior works [12, 13, 14] , the actions are specified via points in the RGB image, Figure 1 : ROBOPOINT is a Vision-Language Model that predicts affordance points based on language instructions. It is able to generate precise actions (red crosses in the image) which satisfy spatial relations in the instruction. ROBOPOINT is a generic VLM that can be applied to many domains such as manipulation, augmented reality and navigation. and then transformed to 3D using depth information, removing the need for pre-defined action primitives [9, 10] , external object detectors [15, 16] , or iterative visual prompting [17] . Second, we design a fully autonomous pipeline generating a large, diverse dataset of ground truth action points by computing spatial relations from the camera's perspective and sampling points within object masks and object-surface intersections. Compared to approaches that require expensive human demonstration data [18, 19, 20] , our pipeline is much easier to scale. Even though we only added data containing simulated images along with templated language, the resulting model's performance improves on real images with natural language commands. Our results show that ROBOPOINT significant outperforms various powerful VLMs such as GPT-4o [11] , LLaVA-NeXT [21] , Qwen-VL [6] and SpatialVLM [22] on relational object reference, free space reference and object rearrangement in cluttered, real-world environments, without losing accuracy on standard visual question answering (VQA) benchmarks. To evaluate relational free space reference, we collect WHERE2PLACE, a manually annotated, challenging real-world benchmark. We also show very promising results beyond robotic applications in an interactive augmented reality (AR) setting, where ROBOPOINT provides visual action suggestions, effectively guiding users through tasks by predicting target points based on common sense. 
related work: 
 Inspired by prior works on spatial reasoning and affordance prediction, ROBOPOINT takes a distinct approach to build VLMs for robotics in contrast to recent methods using zero-shot language models. Spatial Reasoning Many VQA benchmarks [23, 24, 25, 26, 27, 28] have included problems about spatial relations as indicator for a model's ability to understand 3D. These problems can be solved using state estimation [29] plus symbolic reasoning [30] , but these methods have poor generalization to novel objects. More recently, SORNet [31] shows that a transformer model conditioned on object prompts can generalize zero-shot to unseen objects on spatial reasoning tasks, similar in spirit to modern VLMs. However, existing works on spatial reasoning mostly focused on coarse-grained relations. SpatialVLM [22] took a step forward to predict spatial relations in metric space, but we show that ROBOPOINT can achieve better performance on real-world spatial reasoning tasks by locating affordances as points.Affordance Prediction Affordance is defined as the functions of an object, i.e. in what ways it can be manipulated. It goes beyond the visual properties and ties observations to actions. The efficacy of affordance prediction has been shown by many learning-based manipulation methods for 6-DoF grasping [32, 33, 34] and stable object placement [35, 36, 37] . Affordance can be represented in many ways such as part segmentation [38] , dense image feature descriptors [39] and keypoints [40, 41, 12] . We use the 2D keypoint representation to train ROBOPOINT since it can be readily converted into language format. Zero-shot Language Models for Robotics Several works [8, 9, 10] have shown that language model are capable planners for robotic tasks. Using in-context learning [42] , these methods generate reasonable plans in structured language, but require pre-defined action primitives to execute. More recent works leverage VLMs to generate more fine-grained outputs. VoxPoser [15] generates 3D value maps. PIVOT [17] iteratively samples and evaluates possible actions in image space. MOKA [16] predicts keypoints specific to an action type. Unlike ROBOPOINT, all of these approaches still rely on external models for detecting objects relevant for the task. 
method: 
 ROBOPOINT is instruction-tuned from Vicuna-v1.5-13B [43] with a mix of synthetic and real-world data on spatial affordance prediction. This section will cover 3 critical aspects of the tuning pipeline: 1) the problem formulation 2) the instruction tuning procedure and 3) the curation of the data mix.We formulate the problem of spatial affordance prediction as predicting a set of target point coordinates {(x 0 , y 0 ), (x 1 , y 1 ), ..., (x n , y n )} in image space that satisfy the relations indicated by a language prompt. This formulation has several advantages. First, compared to fuzzy language actions such as "place the apple in the drawer", which require detection of apple and drawer before execution, a point prediction is much more precise and can be directly converted to actions. Most VLMs are trained to predict bounding boxes. However, from Fig. 3 , we can see that bounding boxes often include a lot of undesirable clutter due to camera perspective and are not as specific as point outputs. Second, our formulation is general enough to enable various robotic tasks. For example, the predicted points can be interpreted as waypoints for navigation, contact points for grasping or region proposals for placement. This not only allows the model to perform multiple tasks but also means it can be trained with multi-task data.Instruction Fine-tuning Min et al. [44] has shown that rather than learning new tasks, in-context learning [42] works by activating patterns from the training data. Thus, instead of mining patterns from the non-public training dataset, we opt to build our own dataset (see Sec. 4) and fine-tune the language model's parameters. Specifically, we follow the instruction tuning pipeline in Liu et al. [7] . As shown in Fig. 2 , the model consists of an image encoder, a MLP projector, a language tokenizer and a transformer language model. The image encoder processes the image into a set of tokens which are then projected by a 2-layer MLP into the same space as the language tokens. The multimodal tokens are concatenated and passed through the language transformer. All modules are initialized with pre-trained weights. The projector and the transformer weights are allowed to update while the vision encoder and tokenizer weights are frozen. The model is autoregressive and the objective is to predict the response tokens and a special token delineating the boundary between instruction and response. Our results (Table 2 , Fig. 5 ) show that our instruction-tuned model achieves much higher precision than baselines using in-context learning [17, 22] .Co-finetuning with Synthetic Data We find that providing the appropriate mix of data is crucial to the model's performance on downstream tasks. As observed by Brohan et al. [19] , co-training with a mix of robotic data and internet data ensures the model does not forget the knowledge it has learned during pre-training. Our dataset for fine-tuning consists of 4 different sources, as illustrated in Table . 1. The VQA data is a mix of 665K conversations from [45] where the model is asked to answer questions in natural language based on the input image. This ensures the model can reason in language. The LVIS data is converted from [46] , where the model is asked to predict bounding box Figure 2 : Overview of ROBOPOINT pipeline. An RGB image is rendered from a procedurally generated 3D scene. We compute spatial relations from the camera's perspective and generate affordances by sampling points within object masks and object-surface intersections. These instruction-point pairs fine-tune the language model. During deployment, ROBOPOINT predicts 2D action points from an image and instruction, which are projected into 3D using a depth map. The robot then navigates to these 3D targets with a motion planner.center and dimensions for all instances of a certain category. This teaches the model how to ground language to image regions. The last two data sources, object reference and free space reference, are from our synthetic data pipeline (Sec. 4), where the object is to identify points on an object or a vacant region, satisfying certain spatial relations. These data enable the VLM to generate precise action points. We formulate different data sources into the same format and co-train with all of them. Table 4 evaluates the importance of each component in our data mix. 
dataset: 
 We generate a diverse dataset in simulation by procedurally randomizing scene layouts, objects, and camera viewpoints. A novel aspect of our pipeline is generating affordance in free space, allowing the model to detect regions without distinct visual cues.To train ROBOPOINT, we generate a large photorealistic dataset in simulation annotated with affordance points. Most existing robotics datasets [47, 48, 49, 50] only have a handful of fixed artist-designed scene layouts which limits the types of relations that can be generated. We create a diverse dataset by procedurally randomizing several aspects of the scene: the 3D layouts, objects and camera view points. The scene is represented as a articulated body, including revolute (e.g. fridge, dishwasher doors) as well as prismatic joints (e.g. cabinet drawers). Objects are sampled from a large repository [51] with over 8K instances and 262 categories. The objects can be placed on any support surface. This allows our model to learn relations in a truly 3D environment. Once the 3D scene is created, we compute spatial relations among the objects and render an image for each relation from a diverse set of viewpoints in parallel. The relations include left, right, in front, behind, above, below, next to, and in between objects; on, on the left, right, front, and back parts of a surface; and inside a container. The diverse view distribution allow ROBOPOINT to maintain a consistent prediction across different viewpoints (Fig. 4 ). Around 660K (image, relation) pairs are generated from 10K scenes. Some examples from the dataset are shown in Table 1 . More details can be found in Sec. B in the supplementary.Free Space Reference VQA [45] LVIS [46] Quantity 347K 320K 665K 100KLocate several points on an item situated beneath the bordered item.Find some spots within the vacant space situated between the marked items.What is the person feeding the cat?Find all instances of cushions. Table 1 : Our dataset for instruction-tuning combines object and space reference data with VQA and object detection data. ROBOPOINT leverages spatial reasoning, object detection, and affordance prediction from these diverse sources, enabling it to generalize combinatorially. Some answers are not shown in full due to space.Generating Affordance in Free Space A key novelty in our data pipeline is the generation of affordance in free space. This allows ROBOPOINT to detect regions without distinct visual cues, e.g. "the left part of pizza box" in Fig. 5 , which an off-the-shelf object detector will not be able to detect. We employ a simple yet effective strategy. Namely, we first compute relations between a target object and another object or surface. Then, we remove the target object, re-render the image, and sample points inside the intersection of the target object mesh and the surface supporting it. This creates affordance labels in free space in relation to other entities in the scene. 
experimental results: 
 We demonstrate that ROBOPOINT achieves superior accuracy in spatial affordance prediction and real-world language-conditioned manipulation than state-of-the-art VLMs [21, 11] and visual prompting methods [17, 22] . Its view-point consistent prediction and conversational ability also enables application to navigation and augmented reality.ROBOPOINT significantly outperforms baselines in terms of accuracy on pointing to objects and free space referred by language. In addition, it generalizes to novel relation types, respects physical constraints, maintains common sense knowledge and produces view-consistent predictions.Benchmarks We evaluate spatial affordance prediction on two problems: object reference and free space reference. The object reference data is a 750-image subset of RoboRefIt [52] . Unlike human-centered dataset such as RefCoco [23] , RoboRefIt features cluttered images with similarlooking objects that can only be distinguished by relational references. Unlike object reference, no existing dataset addresses identifying free space. Therefore, we collect WHERE2PLACE, a dataset of 100 real-world images from homes and offices in the wild. To minimize bias, we ask one group to label each image with an instruction describing a vacant region relative to other entities, and a different group to label masks according to the instruction. As shown in Fig. 3 , WHERE2PLACE features diverse and challenging scenes with clutter. A subset of 30 examples (WHERE2PLACE (h)) contain relation types not in our synthetic data. More details can be found in Sec. D in the supplementary.Baselines We compare ROBOPOINT against 3 state-of-the-art VLMs, Qwen-VL [6] , LLaVA-NeXT [21] , GPT-4o [11] as well as SpaceLLaVa [53] , a community implementation of SpatialVLM [22] . We employ a zero-shot visual prompting strategy effective for pretrained VLMs. We label the input image with axes indicating its dimensions and ask the model to output a bounding box (top-left and bottom-right corners) of the target object/region, then sample evenly within the bounding box.For GPT-4o, we also tested in-context learning (GPT-4o-ICL) by providing 14 input-output pairs from our synthetic dataset as context before the query. In-context learning achieved zero accuracy for Qwen-VL and LLaVA-Next, likely because point outputs were not part of their training data. Qwen-VL [6] LLaVA-NeXT-34B [21] SpaceLLaVA [53] GPT-4o [11] GPT-4o-ICL [11] ROBOPOINT Table 3 : Quantitative evaluation on standard VQA benchmarks. ROBOPOINT performs on par with state-of-the-art VLM, maintaining the common sense knowledge learned from pretraining.No VQA [45] No LVIS [46] In Table 2 , we report the average prediction accuracy for ROBOPOINT and the baselines along with standard deviation computed from 3 different runs. The accuracy is calculated as the percentage of predicted points within the ground truth target mask. We can see that ROBOPOINT achieves significantly higher accuracy than all baselines, demonstrating the power of ROBOPOINT in spatial reasoning and precise target generation. Some results are visualized in Fig. 3 .The synthetic dataset we constructed in Sec. 4 contains templated language and a fixed set of relations. Nevertheless, ROBOPOINT is able to produce accurate predictions for combinations of seen relations (Fig. 3a ) and novel relation types such as in the middle, rightmost etc. that are not in the fine-tuning dataset (Fig. 3b ). It is also able to maintain its advantage over baselines on these novel relations (Table 2 ).Does ROBOPOINT respect physical constraints? ROBOPOINT's outputs not only satisfy the spatial relations but also respect physical constraints. The target points generated by ROBOPOINT avoid obstacles such as the the bowl in Fig. 3c , whereas the baselines fail to do so.We evaluate ROBOPOINT's performance on VQA benchmarks and summarize the results in Table 3 . ROBOPOINT performs on-par with LLaVA-v1.5-13B [45] , a VLM trained on the same pre-trained weights as ROBOPOINT on VQA data. This shown that ROBOPOINT serves a generic VLM rather than a domain-specific model.How important is each component in the data mix? In Table 4 , we evaluated the importance of each data component on the WHERE2PLACE benchmark. Each data component -VQA on real images, object detection from LVIS, object and free space reference on synthetic images -significantly contributes to overall accuracy. This highlights the value of a general problem formulation that incorporates diverse data sources. Additionally, data quantity is crucial, as the model's performance drops significantly when fine-tuned on only 10% of the data.Are RoboPoint's predictions consistent across views? As shown in Fig. 4 , ROBOPOINT maintains consistent predictions with camera movement. This makes it particularly suitable for mobile platforms and AR, where ROBOPOINT provides consistent action suggestions with moving cameras. Videos can be found on the project page robo-point.github.io.To assess ROBOPOINT's capabilities on downstream robotics and vision tasks, we curated various scenarios for manipulation, navigation and AR assistance. We demonstrate ROBOPOINT's superior performance against state-of-the-art baselines on these tasks. Recordings of robot executions can be found on the project page robo-point.github.io.We set up 3 manipulation environments with 7 tasks (Fig. 5 ). The robot processes image observations and language commands through ROBOPOINT, which returns 2D point targets. These targets are converted to 3D points using a depth map (Fig. 2 ). The robot's end-effector pose is computed from these 3D points plus an offset. A motion planner then executes the trajectory to the target pose. Success is determined by collision-free execution and accurate placement of the target object as per the language instruction. We conducted 10 trials per task and compared ROBOPOINT against zero-shot VLMs like Qwen-VL [6] and GPT-4V [5] , as well as iterative prompting methods such as PIVOT [17] . ROBOPOINT surpasses GPT-4V, the best-performing baseline, by a margin of 39.5% on average success rate. It also enables new capabilities. For instance, in the packing scene, ROBOPOINT's relational reasoning allowed the robot to differentiate regions within a pizza box, fitting multiple objects accurately. guage response using common sense and then provide visual guidance using spatial affordance prediction, which the user can execute with greater ease than language guidance.Navigation To evaluate ROBOPOINT's spatial affordance predictions beyond tabletop scenarios, we created 3 room scenes using the YouBot mobile manipulation platform in CoppeliaSim [60] ,where the robot is tasked to navigate to a target region with respect to certain entities in the scene. Fig. 6 shows the distribution of affordance generated by ROBOPOINT, PIVOT [17] and GPT-4V [5] and the success rate of navigating to the correct region using the predicted points with a simple path planner. ROBOPOINT outperforms PIVOT and GPT-4V in 2 out of 3 scenarios, demonstrating its effectiveness in large-scale room environments for navigation.Augmented Reality ROBOPOINT, which is co-trained with VQA data, retains conversational capabilities in natural language. As demonstrated in Fig. 1 , users can interact with ROBOPOINT through language and receive action suggestions visually with the predicted affordance. In addition to the set a formal dining table task in Fig. 1 . We demonstrate two more real-world scenarios-win tic-tac-toe and get to carpool lane-in Fig. 7 , where ROBOPOINT gives visual guidance to solve the tasks by predicting the correct spatial affordance points. 
conclusion: 
 We propose ROBOPOINT, a novel VLM designed to predict spatial affordances in images based on relational language instructions. By integrating real-world VQA data with automatically generated synthetic data, ROBOPOINT is able to generate precise action points that adhere to spatial and physical constraints, overcoming the limitations of current VLMs in robotics, which often rely on pre-defined motion primitives or large-scale expert demonstrations. Experimental results show ROBOPOINT's superior performance in complex tasks, such as relational free space reference and object rearrangement in cluttered environments, compared to state-of-the-art methods. Additionally, ROBOPOINT's versatility extends its applicability to augmented reality and robot navigation, showcasing its potential for broader applications in robotics.Limitation: ROBOPOINT does not provide confidence estimates for the point predictions. The number of output points are also not controllable. The points also do not provide a direction for action, which can be useful in manipulating articulated objects [61, 62] . All of these are valuable directions to explore in future work.Relation Above Behind PromptThe image features an item encased in a red rectangular border. Locate several spots within the vacant space situated above the bordered item.In the image, an object is framed by a red rectangle. Locate a few points on an object that is situated behind the framed object. 
 
 Appendix: In the image, there is an item framed by a red rectangle and another item encased within a green rectangle. Locate several points upon the item situated between the two highlighted items.The image depicts a container delineated by a red rectangular border. Pinpoint several spots within the vacant area enclosed by the outlined container.The image features an object outlined by a red rectangle. Locate several points on an item that is situated on the right side of the marked item.The image showcases an area demarcated by a red rectangle. Locate a few points within a vacant area on the left side of the marked surface. The process of collecting and annotating WHERE2PLACE consists of three steps, each conducted by different groups of people:1. Image Collection: Images of random scene with multiple objects are taken. Each environment typically has only one image taken, with a maximum of five images captured per environment.2. Language Description: Another group uses language to describe interesting object-region relationships. For example, "Find some free space in between the mouse and the keyboard."3. Mask Annotation: The final group annotates the ground truth mask of the region corresponding to the language description using CVAT, an online annotation tool.For both stage 2 (language description) and stage 3 (mask annotation), two people independently annotate each image and verify each other's annotations to ensure quality and consistency.The scenes encompass a wide variety of environments, including homes, offices, and labs. Images are taken in different areas such as dining rooms, bathrooms, staircases, closets, with viewpoints ranging from top-down, sideways, close-up, to pan-out. The instructions include 3D relations like "on top of" and "below," as well as novel types of relations not present in our fine-tuning data, such as "leftmost" and "in the middle". More examples of WHERE2PLACE can be found in Fig. A .ROBOPOINT can also be applied zero-shot to locate object parts based on their function (affordance).  


Decision: This paper proposes to train a VLM that predicts image keypoint affordances given language instructions to solve robot manipulation problems.

The paper is well motivated, timely, tackles a relevant problem for the robotics community, contains sufficient novelty, is well written, and sufficient experiments are provided.

The authors addressed the main weakness of the paper, namely missing details, in the rebuttal phase. The authors should make sure to include these changes in the final version of the paper.

I recommend acceptance.

Reviews: review: title: title: Good novel and versatile approach for robotic control
paper_summary: paper_summary: This paper introduces Robopoint, a novel VLM trained specifically for spatial affordance prediction in robotics. The key idea is to frame the problem as predicting 2D image coordinates such that they specify spatial relations described by a given language instruction.  Main contributions: New approach to robotic control; using point coordinates as output rather than high-level language plans or pre-determined primitives; Data generation pipeline; procedural method to generate large-scale synthetic data for training affordance models; Performance: Robopoint outperforms existing VLMs and other visual prompting techniques on spatial affordance prediction tasks, including the Where2Place benchmark introduced by the authors; Versatility: authors demonstrate that the approach is effective across a variety of robotic applications such as manipulation, navigation, AR, without domain-specific training
main_review: main_review: Quality: I think the paper is high-quality research with a well-designed methodology, comprehensive experiments, and great analysis. The authors have clearly put significant effort into developing a novel approach and evaluating it rigorously.

Clarity: The paper is generally well-structured and clearly written. The figures and tables effectively support the text.

Originality: Robopoint presents a novel approach to spatial affordance prediction for robotics, combining vision-language models with a point-based action space. The demonstrated effectiveness of the synthetic data generation pipeline is also innovative.

Significance: This work demonstrates significant potential impact, potentially bridging the gap between high-level language instructions and low-level robotic control in a more flexible and generalizable way than previous approaches.

Strengths:
- Novel approach: The point-based action space is a good solution to the precision problem in language-guided robotics.
- Scalable data generation: The procedural synthetic data generation pipeline allows for diverse and large-scale training data.
- Versatility: Robopoint demonstrates effectiveness across various robotic applications without domain-specific training.
- Strong performance: The model outperforms state-of-the-art baselines on multiple tasks.
- Generalization: Shows ability to handle unseen relation types and maintain common-sense knowledge.
- Practical implications: The approach has clear potential for real-world robotic applications.
Weaknesses:
- Limited exploration of failure cases: could benefit from a more in-depth analysis of scenarios where ROBOPOINT struggles or fails.
- Lack of confidence estimates: As acknowledged by the authors, the model doesn't provide confidence measures for its predictions, which could be crucial for real-world applications. Also the number of output points is not controllable, which could be a limitation in some scenarios.
- Relatively small real-world dataset: While the synthetic dataset is large, the real-world WHERE2PLACE dataset is relatively small (100 images).

Overall, despite these weaknesses, the paper presents a good contribution to the field with a novel, well-executed approach that is promising for language-guided robotics.

POST-REBUTTAL

The authors addressed my concerns, and I recommend accepting the paper. Thank you for the information about the dataset, and overall I think that this is a creative approach that has good potential and versatility for robotic tasks.
questions: questions: Despite the fact that performance on the object and free space reference tasks is less than 50%, there is very limited discussion on failure modes or arrow analysis with regards to the proposed approach. Could you provide specific examples of failure cases, and an analysis of common failure types?
What kinds of scenarios does the approach find challenging, and what are potential explanations for observed failure modes?
In regards to the proposed dataset, can you provide more comprehensive information about the WHERE2PLACE dataset?
What was the process for collecting and annotating this dataset?
How diverse are the scenes and instructions in the dataset?
How did you ensure the quality and consistency of the annotations?
Could you give more details about what the annotators were told to do?
Can you provide more details and examples of the types of instructions and scenes included in the dataset?
Also, what are the computational and time requirements for training or deploying the approach?
Could you also provide similar details about the time and resources used to generate the WHERE2PLACE benchmark?
limitations: limitations: 3
review_summary: review_summary: Robopoint presents a novel and promising approach to spatial affordance prediction in robotics. Its strengths lie in the innovative point-based action space, scalable synthetic data generation, versatility across applications, and strong performance compared to baselines. The work shows significant potential for advancing language-guided robotics. However, there are areas for improvement. The authors should address: More in-depth analysis of failure cases and challenging scenarios; Limited size of the real-world dataset (WHERE2PLACE). Despite these limitations, the overall quality, originality, and potential impact of the work are high. The authors should provide more details on the WHERE2PLACE dataset creation process, computational requirements, and expand on failure mode analysis. Addressing these points would further strengthen it.
title: title: Review of Paper 138
paper_summary: paper_summary: The paper proposes RoboPoint - A VLM-basd model that takes language queries that include spatial reasoning and an image, and outputs keypoints that respond to the query.
main_review: main_review: *Strengths*:
 - Within the now relatively cluttered space of VLMs for robot perception, this work finds a nice niche. Predicting keypoints in images seems like a useful objective. 
 - The model seems very adept at reasonable about spatial relationships from single images. 

*Weaknesses*:
 - This field is growing rapidly, yet there seem to be some recent references that seem very relevant but missing. [Spatial VLM](https://spatial-vlm.github.io/) in particular seems relevant but there are also a host of other methods that combine VLMs with semantic segmentation that it would seem could do something quite similar to what is proposed here with slight modifications. 

 - One of the crucial aspects of the paper is the creation of the new dataset. Yet I have trouble to understand exactly how this was done in general. Some examples are given, such as the object removal heuristic, but is this trick always used? What are the set of spatial relations that are included in the dataset (perhaps this is in an appendix but I feel it deserves to be in the main manuscript)

- If I understand correctly, only the RoboPoint method is fine-tuned on the Object Reference and Free Space Reference datasets. Would it not be possible to fine-tune some of the baseline methods using these datasets aswell? 


*Other more minor questions/comments*:

 - Why does the model output several points and how is the number of points chosen? In the figures, how do we interpret that red box, was that added manually?

 - I find the repeated use of the term "affordance" somewhat questionable. Certainly the model demonstrates the ability to perform some level of spatial reasoning, but the bar for "affordance" is much higher and I don't believe that it is really met here. 



*Minor*:

 - L23: no comma needed here: "psychology, demonstrate". There are other places where commas could be removed. 
 - You should define what you mean by "instruction-tuned" before using it. 
 - Something wrong here: "pretrained primitives skills or external detection modules" (either need commas or not plural)
 - is the acronym VQA ever defined?
 - I'm confused by this statement: "Compared to ap- proaches that generate entire trajectories [14, 15, 16], data for spatial affordance prediction can be generated independently for each time-step" - do mean indepenence in the statistical sense? If so, this seems highly questionable.
 - Something wrong here: "points to target regions conditioned language commands" or hard to parse
 - L69: " a object" -> "an object"
 - Several places: "requires" -> "require"
 - Fig. 6 caption error: "0.0.81"


-----------
Post-rebuttal. 

The authors have done an excellent job at addressing my concerns and I believe that the updated manuscript can be presented at CoRL. I have updated my rating accordingly.
questions: questions: I have 2 main points of clarification for the rebuttal (details above):

1. I would like more details about how the dataset was generated
2. I would like to know exactly which data was used to train / fine-tune each of the baselines
3. I would like further justification that this is the right set of baselines to compare against.
limitations: limitations: 2
review_summary: review_summary: I find the work interesting, however there are some important open questions about the dataset and experimental evaluation which I feel need to be addressed.
title: title: Reviews on the RoboPoint
paper_summary: paper_summary: The paper proposes RoboPoint, a VLM that predicts image keypoint affordances given language instructions.
main_review: main_review: ## Strengths:

- The training data is mostly from the available open-sourced dataset and synthetic dataset, which greatly reduce the data collection cost.

- The author proposed to use point as a unified representation format to bridge the image space and action space. And this representation is generalized across different robotic tasks. Interesting designs.

- The method has been validated in different tasks.

## Weakness:

- The output is still limited in 2D image space, and the usage in the 3D world is still limited (still need the depth camera).

- Most of the demoed manipulation tasks are more on the pick-place ones, can this method be applied in the harder scenarios, e.g. articulated ones, as done in A3VLM[1] and GeneralFlow[2].

- The author mentioned the affordance, although the main focus is on the spatial affordance, but some experiments on the fine-grained semantic affordance as done in AffordanceLLM and ManipVQA, even the zero-shot one would be very interesting. Thought with the aid of LLM, such zero-shot ability could probably be also satisfying. 


[1] Huang, Siyuan, et al. "A3VLM: Actionable Articulation-Aware Vision Language Model." arXiv preprint arXiv:2406.07549 (2024).
[2] Yuan, Chengbo, et al. "General flow as foundation affordance for scalable robot learning." arXiv preprint arXiv:2401.11439 (2024).
questions: questions: - See the Review Section. 
- The author mentioned the number of points is not controlled, but how is the statistics? 
- And can you provide the distribution of key points, would high variance exist?
limitations: limitations: 3
review_summary: review_summary: Very good paper overall, with some modifications needed to improve.