On Evaluation of Generative Robotic Simulations

Authors: Feng Chen, Botian Xu, Pu Hua, Peiqi Duan, Yanchao Yang, Yi Ma, Huazhe Xu
Primary area: applications to robotics, autonomy, planning
Keywords: Robotics; Embodied AI; Foundation Models; Generative Simulations;

Abstract:
Due to the difficulty of acquiring extensive real-world data, robot simulation has become crucial for parallel training and sim-to-real transfer, highlighting the importance of scalable simulated robotic tasks. 
Foundation models have demonstrated impressive capacities in autonomously generating feasible robotic tasks. However, this new paradigm underscores the challenge of adequately evaluating these autonomously generated tasks. 
To address this, we propose a comprehensive evaluation framework tailored to generative simulations. 
Our framework segments evaluation into three core aspects: ***quality***, ***diversity***, and ***generalization***.
For single-task quality, we evaluate the realism of the generated task and the completeness of the generated trajectories using large language models and vision-language models. In terms of diversity, we measure both task and data diversity through text similarity of task descriptions and world model loss trained on collected task trajectories. For task-level generalization, we assess the zero-shot generalization ability on unseen tasks of a policy trained with multiple generated tasks.
Experiments conducted on three representative task generation pipelines demonstrate that the results from our framework are highly consistent with human evaluations, confirming the feasibility and validity of our approach. 
The findings reveal that while metrics of quality and diversity can be achieved through certain methods, no single approach excels across all metrics, suggesting a need for greater focus on balancing these different metrics. Additionally, our analysis further highlights the common challenge of low generalization capability faced by current works.
Our anonymous website: https://sites.google.com/view/evaltasks.

Decision: reject
Reviewer scores: 3, 8, 3, 5

Reviews and discussion:
<thread>
creation date:
1730325461914

thread_type:
official_review

<thread_object>
creator: reviewer_xfsc

<content>
summary:
This paper addresses the problem of evaluating autonomously generated tasks by VLM in simulation. The authors propose an evaluation strategy focusing on three aspects: the quality of individual generated tasks, the diversity of multiple generated tasks, and the generalization ability of a diffusion policy trained on the generated tasks. The approach is applied to tasks generated by RoboGen, GenSim, and BBSEA, and the evaluation scores are compared with human evaluation scores.

strengths:
* The paper tackles an important and under-explored problem: evaluating the quality of generated tasks in a systematic way.  
* The criteria used for evaluation—task quality, task diversity, and generalization ability—are all crucial for assessing whether a generated task set is useful.

weaknesses:
* **Clarity of contribution**: The contribution is not entirely clear. Is this the first evaluation strategy for task sets? If not, how does it compare to existing evaluation methods, such as Self-BLEU or Embedding similarity scores from RoboGen? More clarification is needed on how this approach improves upon or differs from existing evaluation techniques.  
* **Realism of generated tasks**: The claim regarding "evaluating the realism of generated tasks" seems overreaching. The approach measures scene alignment score, which reflects consistency between task descriptions and scene images. However, it's not clear why this metric necessarily reflects the "realism" of the generated tasks. A more thorough justification or evidence supporting this claim would strengthen the paper.  
* **Human evaluation details**: The paper lacks specifics on the human evaluation process. Human evaluations were conducted on 10 generated tasks, but key details are missing. How many human evaluators were involved? Were they experts in robotic tasks or general participants? What guidelines were provided to ensure consistency in ratings, particularly for scene alignment and task completion scores?  
* **Writing and presentation**: The paper could benefit from improved clarity in writing, particularly in explaining key concepts and the methodology used.

questions:
1. **Pearson correlation in Figure 3**: Figure 3 shows Pearson correlation between evaluation methods and human evaluations. However, it does not quantify how much the proposed evaluation method deviates from human evaluation. Why are the scores in Figure 3 (left) generally better than those in Figure 4 (right)? Does this imply that the proposed method is more reliable for scene alignment scoring than for task completion scoring?  
2. **Figure 4 method clarification**: The paper mentions various LLM/VLM options in Section 4.1.1, but it’s unclear which one is used in Section 4.1.2 Figure 4\. Do all VLM/LLM choices yield similar results, or do specific models shows the result trend in Figure 4? For example, does RoboGen consistently outperform in task completion scores while GenSim excels in scene alignment, regardless the choice of evaluation VLM/LLM?  
3. **Line 354 clarification**: The paper attributes the shortfall in performance to the vision-language model’s “lack of access to top-view rendered data”. Is this limitation specific to the VLM used in GenSim or to the VLM used in the proposed evaluation method?  
4. **Task set generalization (Section 4.3)**: In this section, different task groups are used to measure the generalization ability (as shown in the Appendix Table 6). Given that the task groups are not identical across task sets, is it fair to conclude that GenSim exhibits better generalization while the others perform poorly? How does this variability in the definition of task groups affect the generalization conclusions?  
5. **Definition of text embedding (Line 264\)**: What is the formal definition of the text embedding e\_i​ used for task descriptions? More clarity here would help.  
6. **Image sampling (Line 246\)**: The paper mentions extracting 8 images from the trajectory video. Are these images randomly sampled, or are they evenly spaced throughout the trajectory? Does the selection method include the final frame? Could different sampling strategies affect the task completion score?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730552068453

thread_type:
official_review

<thread_object>
creator: reviewer_2pam

<content>
summary:
This paper presents an evaluation framework for simulations generated by the generative foundational model, focusing on three key metrics: (i) the realism of tasks and generated trajectories, (ii) task diversity, and (iii) generalization to new tasks. This framework indicates a correlation between expert human evaluations.

strengths:
- This paper seeks to tackle significant challenges in robotics, especially the high costs associated with evaluating generative simulations. The proposed framework leverages the capabilities of LLMs/VLMs to automate this process.
- Certain VLM/LLM choices demonstrate a correlation with human expert evaluations of task quality, suggesting the potential effectiveness of these methods. However, some VLM/LLM show less alignment with human assessments.

weaknesses:
- **Clarification on Experiment Design**
  + Below are some points for clarifying the experiment setup:
  + Could the authors explain why the three selected metrics are considered essential for generative simulation? While single-task quality and task diversity are understandable, it's unclear how generalization fits within this context. How does it differ from simply having diverse tasks?
  + Could the authors clarify how scene alignment is evaluated? From my understanding of lines 348-350, it seems to rely on the robot's trajectory. If that’s the case, shouldn’t the quality of the generated trajectory also play a role? Additionally, I’m curious why scene alignment is based on video rather than image.
  + The connection between high prediction loss and task diversity is unclear. For instance, a single task could involve varied dynamics, like table rearrangement. Could the authors provide a justification for this?

- **Fairness of Evaluation**
  + Each baseline uses a different task for evaluation, but certain tasks or simulator setups (as mentioned in lines 318-319) might inherently pose challenges for LLM/VLM-based score evaluation, potentially affecting fair comparison.

questions:
- How do humans evaluate the task? (e.g., Do they use a 1-5 scoring system, rank tasks by preference, or something else?)
- Related to the point above (in the Weaknesses section, high prediction loss), lines 274-277 weren’t clear about the exploration aspect. Could the authors clarify how diverse dynamics are connected to task diversity?

### Minor Suggestions
- Section 3.3 on Scene Alignment Score could reference prompt (A.1).
- Section 4.1.1 could refer to the detailed experiment setup in (A.2).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730610352365

thread_type:
official_review

<thread_object>
creator: reviewer_mwfy

<content>
summary:
This work presents an evaluation framework for "generative" robot simulators. Here, generative simulations are a simulator, task set, and possibly set of demonstrations for each task. The authors propose evaluating these simulations in three criteria: quality, diversity, and generalization. Task quality is evaluated by scene alignment between the task description and the scene, and also scoring the task completion video with an mLLM to see whether the associated data or policy completes the task. For task diversity, the authors evaluate the description embedding, and also a trajectory prediction error to understand the data/trajectory diversity. Finally, for generalization, the authors train policy on the training data and evaluate – seeing how well the performance is on evaluation set.

strengths:
1. This paper is pretty timely and addresses an important recent development in simulation based robotics – generative simulations. While there are multiple such works coming out in recent months, there is no easy way to understand which one someone should use for their applications. A quantitative evaluation like the one proposed in the paper goes a long way to address this concern.

2. The three axes proposed by the authors are also reasonable – and the way to evaluate them also seems reasonable. Splitting the three criteria into the smaller sub-criteria is also pretty pertinent. 

3. The fact that the authors also do human evaluations to find alighment between the mLLM evaluations and human evaluations make their evaluation more trustworthy.

4. The task diversity evaluations are also interesting – although maybe less meaningful than the other axes since the type of tasks i.e the task class proposed by each simulator may vary a lot.

5. Finally, the authors expose interesting issues with the generalization ability in RoboGen and BBSEA – that the data generated by them is not high quality, and thus they need to improve their data quality before they can be useful.

weaknesses:
1. There doesn't seem to be a consistent correlation with human evaluation and mLLM proposed results in Figure 3.
2. The alignment score given by the mLLMs is not calibrated – some calibration would make the results more reasonable.
3. Generalization evaluation would be more convincing if it was trained on some kind of task-conditioned multi-task policy like BAKU [1] rather than an unconditional policy like diffusion policy.

[1] Haldar, Siddhant, Zhuoran Peng, and Lerrel Pinto. "BAKU: An Efficient Transformer for Multi-Task Policy Learning." arXiv preprint arXiv:2406.07539 (2024).

questions:
1. Do the authors think the task diversity captures the overall size of the possible task sets in a generative simulation?
2. Would the authors be open sourcing their evaluation suite so developers of generative simulations can develop with this benchmark in mind?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709485728

thread_type:
official_review

<thread_object>
creator: reviewer_myjx

<content>
summary:
This paper proposes a framework to analyze existing simulation benchmarks for embodied AI. The authors consider 3 metrics: quality, diversity, and generalization. For quality, authors analyze the realism of the generated task and the completeness of the generated trajectories. For diversity, they measure instruction diversity and trajectory diversity. For generalization, they train a diffusion policy and check success rate on unseen tasks. The results reveal that while metrics of quality and diversity can be achieved through certain methods, no single approach excels across all metrics.

strengths:
1. The authors proposes 3 aspects (quality, diversity, and generalization) to assess current simulation benchmarks.
2. Experiments and analysis are interesting.

weaknesses:
1. I think in addition to quality, diversity, and generalization, sim-to-real gap is a big issue that needs to be addressed/discussed when analyzing simulation benchmark. Have you considered metrics to assess how well the simulated tasks translate to real-world scenarios? Or maybe include a section on potential sim-to-real challenges for each evaluated benchmark.
2. It seems the diversity score only considers the text and trajectory. However, I think it should also consider at least visual input.
3. It seems this paper did a great job of analysis of existing benchmarks, but did not propose a new benchmark to resolve these issues.
4. It seems generalization and diversity has some connection. The diversity is to measure how out-of-distribution tasks are from each other while generalization is to measure how in-distribution tasks are with each other. I wonder whether we can use the same metric to measure these 2 things. Is it possible to investigate correlation between diversity and generalization scores? Or is it possible to potentially unify these two metrics?

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732167110818

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all reviewers and ACs for their time and effort in reviewing the paper. We are glad that the reviewers generally recognized the following contributions. All reviewers have recognized that our work proposes a novel evaluation framework for Generative Simulation. Some reviewers **found our study on the consistency with human evaluation behavior to be convincing** (`MwfY`,`2PAM`,`XFSC`). Additionally, regarding the three aspects we proposed, some reviewers expressed agreement, **acknowledging our contributions in defining and evaluating these different axes**. (`MwfY`,`XFSC`)

### Key Answers
Several reviewers have raised similar concerns. We address them below.

***How is 'Generalization' different from 'Diversity':*** Our work focuses on proposing an evaluation method for generative simulation that reflects the overall quality of the generated data. **For generative simulation, enabling data to serve a broad range of robotic tasks is a fundamental objective, making generalization a critical evaluation criterion**. Unlike diversity, which merely indicates the range of potential tasks (text diversity) or the complexity and variety of trajectories for a single task (trajectory diversity), generalization emphasizes the need for generated data to support imitation learning. In other words, diversity measures how diverse the generated data are, while generalization reflects how generally the trained policy performs. (`2PAM`) Though diversity and generalization can be related, they are neither identical nor unified. As shown in Tables 1, 2, and 3 of the paper, although BBSEA and RoboGen outperform GenSim in diversity, their generalization capabilities are not as strong as those of GenSim. From a qualitative perspective,both extremely low and high levels of diversity can undermine generalization capabilities. Considering a scenario where a dataset consists of a single task trajectory, representing minimal diversity, the policy will inevitably overfit, resulting in no generalization. Conversely, if data diversity is excessively high to the extent that the model or algorithm cannot effectively capture and learn from it (e.g. the case in RoboGen), generalization will also suffer due to a decline in overall training performance. This highlights the need for a balanced approach to diversity in training data to optimize generalization. (`MyJX`)

***How to evaluate 'Scene Alignment':*** For complex scenes, we used multi-view images for evaluation, instead of robot trajectories, as we believe that assessing a scene from a single view does not accurately reflect the complexity of such scenarios. (`2PAM`) Additionally, we observed a degree of consistency between the model's and human performance in scene alignment. (`MwfY`) Regarding "the realism of generated tasks", it includes whether the scene is aligned with the text, as well as the “semantics” of real scenarios. Failure examples of the former case include simulation penetration, or putting something supposed to be in kitchen into a bathroom. We have updated the paper for clarification. (`XFSC`)

***Task Diversity:*** Visual diversity is important for training policies in imitation learning, but it provides limited insight into the intrinsic quality of the data. The relevance of visual diversity as a metric does not closely align with the aspects we aim to evaluate. Achieving visual diversity in a simulator can be straightforward; for example, adding complex but irrelevant textures to the background of a simulator might artificially increase visual diversity without actually improving data quality or diversity. (`MyJX`) Additionally, in trajectory diversity, the dynamics model often captures some basic visual diversity; tasks with relatively high visual diversity tend to exhibit greater trajectory diversity. Similarly, if a single task includes sufficiently varied policies, it also indicates good trajectory diversity. (`2PAM`)

***Open Source:*** We will open source our evaluation pipeline in the camera-ready version.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors of this paper take on a very challenging task: analyzing the quality, diversity, and generalization of various robotics simulators. They evaluate these based on a mixture of different methods, using large language models for quality, world-model predictions for diversity, and zero-shot generalization of trained policies for generalization. They compare to human evaluations.

Strengths:
Given the importance of robotics simulators, it's important to study their qualities and come up with metrics to assess them. This paper seems like a good and interesting first step in that direction.

Weaknesses:
No characterization of sim-to-real or physical realism of a simulation
Characterization of visual diversity is lacking
Only two simulations, GenSim and RoboGen - why not Robocasa, for example?
Generalization metric is not task-conditioned -- which actually would punish simulators for creating "hard" environments where multiple tasks are possible.
Some of the language model 

While the authors raised very important points about the quality of the simulators, I think there are a large number of clarifications still necessary, and the writing of the paper could be improved, especially if they want others to use the metrics they have proposed.