Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: Paper introduces VLMbench, a new benchmark for vision-based robot manipulation. Compared to the prior counterparts, the tasks in this benchmark are compositional in terms of action, objects, and their attributes. The benchmark also allows two types of goal definition: goal pose constraint and path pose constraint. To generate demonstrations, the authors propose AMSolver but do not provide many details about it. Finally, some experiments on this benchmark are conducted with a model modified from CLIPort.
strengths: - I believe the problem studied here is relevant and important. Vision-based agents executing tasks specified by natural language is of interest to the NeurIPS community and could possibly help with other research including multi-task learning, multi-modal learning, etc
- The paper is overall illustrative, some write-ups could have been clearer though. 
- Extending CLIPort to 6 DoF robotic control is interesting, however, the proposed approach does not seem to be working at least on this benchmark.
weaknesses: Having said those above, I still have some major concerns about the methodology and experiment of this manuscript. I hope the authors could help clarify them in a rebuttal. 

- As stated at the end of the introduction, AMSolver is one of the major contributions made by this paper. However, I cannot find enough details about it, thereby making it hard to justify the novelty. Sec. 3 seems to be talking about the grammatical structure of their tasks, which I suppose, should be part of Sec. 4 instead? It is not clear how these task definitions connect to a demonstration generator or a robotic trajectory solver. If my understanding is correct, the authors seem to apply an off-the-shelf solver provided by RLBench to their tasks. If this is the case, I doubt if this can be viewed as a "contribution". The authors are encouraged to elaborate more on the comparison between their proposed AMSolver are other prior arts, ex. the solver in RLBench, BEHAVIOUR, Meta-World, etc. 

- In table 1, the authors claim that their benchmark is the only one that includes "automatic 6 DoF grasping". Specifically, they pre-generate some plausible two-fingered grasps on the objects in their benchmark using GPD. However, it's very unclear how could models benefit from these grasping samples when solving their tasks. The authors should at least demonstrate a prototype system that could leverage this data to justify the rationale for making it part of their benchmark.

- VLMBench is claimed to be a "compositional" benchmark. I may be biased but indeed I found it necessary to also include some compositional or OOD generalization tests to further verify the compositional structure of their tasks. This has been a common practice for other compositional benchmarks, including CLEVR, ALFRED, etc. Please kindly remind me if there are already (and make it clearer in the paper).

- The ablations in the experiments indicate that providing ground truth position and orientation could significantly lift the overall performances. However, more details on these model variants should be provided. I can see a total of 6 outputs of the proposed 6DOF-CLIPort. Does "pos" mean x/y/z are already given while the other three will remain predictable, and 'ori' suggest the contrary? It could be hard to parse the results without them. The existing results also could have been more informative other than some numbers, ex. why do ground truth orientation useless than position in most tasks even 6 DOF tasks (Door)? Why does the model still fail even with this privileged information? Some failure mode analysis seems to be necessary to help understand the challenge imposed by this benchmark.
correctness: I found the claims made by the authors supported mainly by their results well, except for the contribution on AMSolver, which I've elaborated on above.

The experiment setting seems riguous. Although I do expect authors could make compositional generalization part of their evaluation protocol.
clarity: Unfortunately, the authors fail to provide enough details on many of their claimed contributions, ex. the trajectory solver and the experiments. The write-ups could have been less glitchy with better proofreading.
relation to prior work: I found the discussion of prior works comprehensive. 
additional feedback: See [Weaknesses].
review_title: Review of paper "VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation"

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a framework for vision and language manipulation aimed at being a benchmark. VLMBench is a benchmark for vision and language manipulation which contains a manipulator and a tabletop environment. VLMBench provides access to several sensors such as RGB, Depth, segmentation as well as other agent-specific observations such as end effector pose. The agent can be commanded to perform a variety of tasks such as pick and place, stack, drop etc., each task with a potential for varying several parameters such as object type, qualifying the task using constraints such as direction, size, extent etc. Given this information, VLMBench solves the task using a rule based task solver called AMSolver. AMSolver can compute solutions that can adhere to goal/path constraints. Finally, the paper also describes a manipulation agent called 6D-CLIPort, which takes inspiration from CLIPort, and given RGBD observations and a language instruction set for a task, outputs relevant goal positions for the end effector. The authors apply 6D-CLIPort to environments in VLMBench and show that the performance of 6D-CLIPort surpasses that of simple unimodal baselines.  
strengths: - The paper presents a benchmark for a domain that is of significant interest to the community. 
- The simulator/dataset allows for specifying constraints of several kinds - such as position/orientation constraints, constraints related to the trajectory executed by the agent. 
- Similarly, the tasks have a good amount of variations possible in sizes, shapes, directions of motion etc. Linking such high fidelity task specification with language has the potential to enable more complex architectures and solution methodologies for vision and language manipulation. 
weaknesses: - The paper identifies the compositional nature of task specification as a key highlighting feature of the benchmark, but it is unclear how this is better/different than the composition seen in existing benchmarks/datasets like ALFRED [1] or CALVIN [2]. 

- The purpose of the discussion regarding the 6D-CLIPort agent is unclear. If the intent was to showcase how the environments/tasks from VLMBench could be solved by existing algorithms, or pose a challenge to such techniques, it would have been useful to implement a set of representative baseline algorithms for vision-language manipulation and show the performance on VLMBench. If the intent was to demonstrate a new algorithm for vision language manipulation in general, it would have been useful to compare with other baselines. Fundamentally, the unimodal baseline seems to always be at a disadvantage, as it is intuitive that an RGB+D+language algorithm would perform better. RGB + Depth + Language is a multimodal combination that is not very uncommon in the manipulation field, so several other datasets can also provide this data, and hence it is not really necessary to prove that multimodality is useful and important. It is also not clear how 6DCLIPort performs relative to other best performing VLM techniques in the literature . I think this section needs reframing to really understand the purpose of 6DCLIPort.

- It is also not unclear why the performance of the unimodal baseline (and to some extent, 6D CLIPort without GT) is so extremely low on the benchmark tasks. Also, considering one of the strengths of VLMBench is its ability to generate variations and constraints in the task space, the fact that 6D CLIPort fails so badly at tasks that have high variations (see section 6.3) raises questions about the relevance of this technique. 

[1] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox, "ALFRED A Benchmark for Interpreting Grounded Instructions for Everyday Tasks"
[2] Oier Mees, Lukas Hermann, Erick Rosete-Beas, Wolfram Burgard, "CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks"
correctness: The evaluation could have been better (through the usage of well known baseline algorithms - currently there seem to be none, and the unimodal baseline is always at a disadvantage making it an unfair comparison), and the associated discussion can be improved greatly.
clarity: The paper is generally well written. The sections regarding AMSolver and VLMBench are easy enough to follow, but as mentioned above, it is hard to gauge the real purpose of 6D CLIPort, how it fits in the existing body of work on vision language manipulation, and why it (and the baselines) perform the way they do.

A couple things that seems to be missing from the paper and would be useful to add: 

1. How many classes of objects can VLMBench instantiate and simulate? 
2. Does the underlying simulator simulate the physics for the objects as well? i.e., would it be possible to have dynamic objects in the simulation? 
relation to prior work: The paper does discuss several kinds of manipulation benchmarks from recent years such as RLBench, MetaWorld, CausalWorld. One relevant benchmark/simulator missing from the comparison is CALVIN [1], especially given the compositional focus of CALVIN.

[1] Oier Mees, Lukas Hermann, Erick Rosete-Beas, Wolfram Burgard, "CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks"


additional feedback: As discussed in the Weaknesses section, it would be great to clarify 1) the extent of compositionality in VLMBench and whether/how it differs from such properties in ALFRED/CALVIN, and 2) the performance of 6D CLIPort and how it can be localized within the existing body of work for vision language manipulation. That said, I think having a dataset/benchmark simulation that allows for specification of constraints and a good degree of variation in the task/environment is a good contribution. 

One more question that might be interesting to discuss: is it possible to extend the types of constraints? For instance, would it be possible to add a constraint that has to do with vicinity to a certain type of object, for example, if the manipulation workspace contained a glass, one natural constraint instruction to specify would be to keep sufficient distance from the glass. Similarly, one might want the robot to speed up or slow down depending on the surroundings. Would it be possible to add such new constraints?
review_title: Review of VLMBench

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The presented paper proposes a robotics manipulation benchmark containing language instructions and corresponding manipulation tasks (e.g. pick and place, wipe a table, open a drawer, etc.). 
The main contributions of the paper are:
1) Automatic generator of individual demonstrations with corresponding linguistic instructions. The demonstrations might be composed of simple tasks. There are 8 basic task types (pick & place objects, stack objects, drop pencil, put into shape sorter, pour water, wipe table, open drawer, use door) that can be varied by color, shapes, size, relative positions, etc. 
2) VLMbench - benchmark including a set of manipulation tasks that compose of visual observations accompanied by linguistic instructions. The manipulation tasks have several variations and can be automatically evaluated.
3) Baseline algorithm evaluated on VLM tasks.
strengths: I fully agree with the authors that such benchmark is filling an important blank spot to measure quality of embodied agents who should follow language instructions to perform object manipulation tasks. While there are several benchmarks and datasets for language descriptions to images, there are not that many for the embodied domain, especially not those that would enable measuring the quality in an organized structured way. 
The paper is well organised and describes clearly the presented benchmark. The benchmark itself is from my perspective very well designed. It enables evaluation of the quality of individual subtasks and more complex tasks can be created by joining the individual subtasks. 
I appreciate that the evaluation is enabled by the design of the dataset. All the tasks are specified by the change in position and orientation and therefore the ground truth waypoints are known. The evaluation is done by checking if individual waypoints have been reached.
The installation guide, datasheet file as well as appendix information are very helpful and detailed.

weaknesses: It is claimed in the paper that more complex tasks can be built as a composition of the simple tasks. However from the text itself it is not clear how they are built and evaluated.
The actions are restricted to the manipulation with an object - start of the action is determined by grasping an object and the end by releasing. Therfore actions such as push/kick/pull/hammer/etc. are not included. It is also not clear how these could be included (is there a way how to add a new task?) and evaluated. I would appreciate if the authors could discuss on this.
I am also not sure how the difficulty of the tasks might be compared or evaluated. Do the authors consider that all the tasks are on the same level? Is there any way to evaluate the difficulty of the individual tasks? (e.g., number of the waypoints needed to pass?)
The dataset itself is quite big to download (aprox. 10 GB). It might be good to include also smaller version for users who want to just test it and see samples from the dataset.

I was able to run the pretrained models, generate new data and test configurations as described in the readme. However, it is not clear how to turn on the simulator - is there any argument to do so? The idea/process is described well in the paper, but it is hard to navigate in the code as there is not any detailed documentation. I would appreciate more detailed documentation for the code itself.
correctness: The described methods are sound, the dataset is designed well. The benchmark evaluation is clear. I think that as a benchmark, the authors should present on the webpage or github the baseline and provide a table of the results of other methods (encourage their upload, etc.). Maybe in the future the authors might consider if it would be possible to provide an automated evaluation of any uploaded method. 

One question considering the described methods is that I am not sure how the object detector is working. It is mentioned that the object location in the end is evaluated, but I am not sure about the details. Maybe this could be also described in the appendix.

clarity: The paper is very clearly written, easy to follow and well structured.
There are some grammar mistakes and typos (e.g., wiper table instead of wipe table (in Table 2); M2[a4,c2] on line 142 - I think should be M2[a4,b2];the agent receive -> receives (line 171); "consist of language instruction" -> "consists of language instruction"; "shape Sorter," -> "Shape Sorter", (line 273); "we can find out the position" -> "...that the position" on line 287; rotations(e.g....) -> rotations (...) - missing space (line 288))
- x, y, z - should be in italics font $x$...

relation to prior work: The presented paper is very well put in the context with the prior work. 

additional feedback: I really appreciate the creation of such a benchmark. I would like to see if the benchmark is extended for more tasks and if the authors provide a comparison of current algorithms/methods on the website as people will start to use the benchmark. Further questions/concerns are mentioned in the sections above.
review_title: The presented paper provides a needed benchmark for evaluating embodied agents that should perform various manipulation tasks based on the language instructions. The tasks are restricted to the tasks where an object is manipulated.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper considers the problem of 6 degrees of freedom (DoF) robot manipulation following language instructions, which is an interesting and promising direction, and there are several works emerged recently. In particular, the authors proposed a simulator AMSolver based on the robot learning framework RLBench. The authors added the language instruction part in RLBench and proposed modular rule-based task templates to classify and construct tasks, including control constraints, moving the target object with goal pose constraints (M1), and moving the target object along a trajectory while satisfying the motion constraints (M2). An object-centric representation is used to add variations when generating data, including class, color, size, and geometry shape. Based on AMSolver, the authors constructed the dataset VLMBench for 6DoF language-guided robot manipulation research. Based on CLIPort, the authors further proposed a 6DoF keypoint-based model as a baseline for VLMBench.

In summary, this paper considers an important problem, language-guided robot manipulation, and the contributions consist of a simulator AMSolver, a dataset VLMBench, and a baseline model 6D-CLIPort.

strengths: 1. I think the dataset is important for the language and robotics/vision research community and makes a step towards language-guided 6DoF robot manipulation. Existing 6DoF datasets, such as [ManipulaTHOR](https://github.com/allenai/manipulathor), [CALVIN](https://github.com/mees/calvin), etc., usually focus on long horizontal task planning. While [Ravens](https://github.com/google-research/ravens) is suitable for language-guided manipulation, it only operates on a 2D plane. Therefore, I think VLMBech fills the gap of language-guided 6DoF robot manipulation well.

2. The authors propose and build a complete pipeline for language-guided 6DoF robot manipulation research, including a simulator  AMSolver, a dataset VLMBench, and a baseline model 6D-CLIPort, which can help novices get to know this field quickly and researchers develop and compare algorithms conveniently. 

weaknesses: 1. Following the GitHub repository of this paper, it is hard to reproduce the results (Table 3 and Table 6 in the appendix). For instance, I used the pre-trained model for the 'stack' task, and always got a 0 success rate.

2. The process of generating the dataset is not described very clearly. How many episodes for each task in training? What is the difference between seen and unseen? How is the training and validation split? Following the GitHub repository of this paper, I think that running 'python tools/dataset_generator_NLP.py' only gives the validation dataset?

3. Some parts of the paper are not written very well. For the AMSolver, how do the three modular rule-based constraints correspond to the three general household task categories? 
Some literature on language-guided robot manipulation, such as [1], is missing. The conclusion is missing. 

[1] Mees et al. What Matters in Language Conditioned Imitation Learning. IROS 2022.
correctness: The authors first build a simulator AMSolver and use it to construct the dataset VLMBech. The AMSolver adopts modular rule-based task templates to classify and construct tasks, including control constraints, moving the target object with goal pose constraints (M1), and moving the target object along a trajectory while satisfying the motion constraints (M2). The design of variations adopts an object-centric representation, including class, color, size, and geometry shape. The rationale for constructing the dataset is sound. However, the details of how to generate the dataset, including the number of episodes, and the training and validation split, are missing. 

The paper also proposed a baseline 6D-CLIPort and conducted experiments on it. The experiment design is appropriate. However, following the Github repository, it is hard to reproduce the main results (Table 3 and Table 6 in the appendix), and the paper also misses experimental details for reproducing the results. Additionally, for Tables 3 and 4, what does the unimodal mean?

clarity: The paper is generally easy to follow and well written. But for the AMSolver, the correspondence of the three modular rule-based constraints and the three general household task categories are not written very clearly. Some literature on language-guided robot manipulation, such as [1], is missing. The conclusion is missing. 

Some typos:

1. Line 25, Additional -> Additionally

2. Line 26, the biases -> biases

3. Line 60, DOF -> DoF. Please check other DOF or DoF, both are fine, but please be consistent.

4. Line 67, demonstration -> demonstrations

5. Line 98, object -> objects

6. Line 100, considered -> considered as

7. Line 141, with -> while

8. Line 171, receive -> receives, generate -> generates

9. Line 179, tasks -> task

10. Line 185, Tasks -> Task

11. Line 191, the format of double quotes not right

12. Line 217, entire image -> the entire image

13. Line 231, extra space after the word concatenation

14. Line 245, we -> We

15. Line 284, given -> giving


[1] Mees et al. What Matters in Language Conditioned Imitation Learning. IROS 2022.

relation to prior work: In general, the paper clearly discusses the difference between this work and previous contributions. The literature [1] should be added. In addition, the authors may consider distinguishing between 6DOF robot manipulation and long horizontal task planning, which in my opinion is very different, and long horizontal task planning may introduce additional complexity to the task.


[1] Mees et al. What Matters in Language Conditioned Imitation Learning. IROS 2022.
additional feedback: In my opinion, 6DOF robot manipulation and long horizontal task planning are very different. Adding long horizontal planning may introduce additional complexity to the task. The authors may consider it when designing tasks. 

I like the paper and appreciate the efforts made by the authors. Please add the missing details and proofread the paper more carefully. Thanks.
review_title: I cannot reproduce the authors' results, and I am conservative in accepting this paper

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors of VLMbench propose a new benchmark for robotic manipulation based on visual input and a language query. A benchmark assesses the completion of the task based on task-related final state constraints. Additionally, the work proposes a new dataset associated with the benchmark. The authors propose 8 various manipulation tasks. For each task, a set of randomised 
template-based scenarios is defined. Additionally, the paper introduces AMSolver which is an automatic tool for demonstration generation.
strengths: I believe the authors did a good job in formalising various manipulation tasks. The work proposes a systematic way to describe new tasks in robotic manipulation problems (Sec. 3.1). I think the modular design suggested in the paper is a good idea. Object-centric representations along with unit task builder allow for the scalability of data generation. 

AMSolver seems to provide a flexible way to generate demonstrations for new tasks after having defined proper goal constraints. I believe providing automatic 6DoF grasping is a big benefit of the tool.

The dataset provided with the benchmark offers a good variety of manipulation scenarios.

A baseline and ablation suggested by the authors are sufficient to emphasise the challenges of the proposed benchmark.
weaknesses: Given the proposition of the dataset, I would expect more details on the collected data to be provided. Some of the details are covered in the appendix. However, it would be good to discuss what set of objects is used in the tasks (list of objects, do they come from the existing dataset) and how many different objects can be used per task (e.g. how many can be opened).

Some more details on how the initial scenes were randomised would be also useful (how many objects, were placed randomly/with constraints). Additionally, what is the process of creating the instructions from templates? Is it similar to CLEVR [1]?

Additionally, for the dataset description, what is the distribution of demonstrations over the tasks?

There are 5 different views generated in the tool, and a description of what views are incorporated, and why such views were chosen, would be a nice addition. Also, what is the exact set of observations provided in the system output?

For the benchmarking, it would be good to discuss the performance with respect to FPS that can be achieved. Parsing 5 images per 50ms may provide a noticeable delay.

Sec. 6.1. suggests that multi-step tasks are provided with ground truth sub-goals. I think it would be worth showing the model dealing with multi-step tasks, and then maybe showing how compositioning of simple tasks affects the performance, and probably provides a significant challenge to the proposed model.

I believe that the issues I mentioned are of a rather minor nature, and are easily correctable.

[1] Justin Johnson, Li Fei-Fei, Bharath Hariharan, C Lawrence Zitnick, Laurens Van Der Maaten, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
correctness: The dataset is constructed in a sound way.  The 8 manipulation tasks chosen, are of different nature and provide various challenges for robotic systems.
clarity: Overall, the paper is written clearly. I believe some details mentioned in the Weaknesses section, could be moved to the main article from the appendix, and some details could be explained a bit more.
relation to prior work: Relation to prior work is well covered.
additional feedback: Questions: 
- How is the grasp pose chosen in a particular scenario? Are grasp randomly chosen from available ones, or is there an optimisation w.r.t. grasp choice?
- How are [target object description]s generated? How are the variations of the descriptors chosen (e.g. is it always all of *colour, size, shape*, or a random subset is used)?
review_title: Benchmark and dataset for vision and language manipulation

Meta-review: metareview: The paper presents a benchmark for a domain that is of significant interest to the community, and a complete pipeline is proposed and demonstrated. The work proposes a systematic way to describe tasks in robotic manipulation problems suitable for the dataset/benchmark presented. While reviewers were able to explore the tools presented, clarifications on how to run the tools provided and reproduce results should be clearly present in the documentation for the toolkit itself. The authors are encouraged to consider the strengths and weaknesses identified by reviewers for further clarifying and strengthening the manuscript.

Strengths:
- The problem studied here is relevant and important. Vision-based agents executing tasks specified by natural language is of interest to the NeurIPS community.
- The authors demonstrate a complete pipeline for language-guided robot manipulation research, suitable for getting novel research tasks in this area up and running quickly. 
- The paper is well organized and clearly describes the presented benchmark, especially with revisions.
- The presented system allows for specifying a variety of kinds of constraints, making it overall fairly widely applicable.
- Providing automatic 6DoF grasping is a significant improvement of the tool over existing related systems.

Weaknesses:
- Stronger differentiation from existing datasets/benchmarks in the body of the paper would help clarify the contributions of the work.
- The proposed approach does not seem to be working well on CLIPort, which raises questions about the overall relevance of the technique that should be addressed.
- It is not clear from the paper itself how new tasks can be included and evaluated.
- Some newer literature on language-guided robot manipulation should be included.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept