UnrealCV Zoo: Enriching Photo-realistic Virtual Worlds for Embodied AI Agents

Authors: Fangwei Zhong, Kui Wu, Churan Wang, Hao Chen, Hai Ci, Zhoujun Li, Yizhou Wang
Primary area: datasets and benchmarks
Keywords: Virtual worlds; Embodied AI; Embodied Tracking and Navigation; Visual RL;

Abstract:
The embodied artificial intelligence agents should be capable of sensing, reasoning, planning, and acting in complex open worlds, which are unstructured, high-dynamic, and uncertain. To apply agents in the real world, the realism of the simulated worlds is important for training and evaluating the built agents. This paper introduces UnrealZoo, a rich collection of photo-realistic 3D environments that mimic the complexity and variability of the real world based on Unreal Engine. For embodied AI, we provide a diverse array of playable entities in the environments and a suite of tools, based on UnrealCV, for data collection, reinforcement learning, and evaluation. In the experiments, we benchmark the agent on visual navigation and tracking, two fundamental tasks for embodied vision agents, in complex open worlds. The results provide valuable insights into the strengths of enriching the diversity of the training environments and the challenges to current embodied vision agents in the open worlds, e.g., the latency in the closed-loop control to interact with the dynamic objects, reasoning the accordance of the spatial structure in the complex scenes.

Decision: reject
Reviewer scores: 6, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1729587010228

thread_type:
official_review

<thread_object>
creator: reviewer_u5ud

<content>
summary:
The paper proposes a photo-realistic simulation environment for embodied training and a scene dataset of 100 diverse scenes.
The simulator is based on UnrealCV develops interfaces for RL training, and supports multi-agent.
The simulator highlights its simulation speed and diversity of scenes and entities. Necessary experiments are conducted to show its usability.

strengths:
- The environment is photo-relaisitic. 
- Scene quality an is good. Entities are diverse. Simulation efficiency is improved.
- There are a lot of experiments.

weaknesses:
- There are only 100 scenes and they are not scalable. 
- There is no comparison with more recent related works like [1][2].
- The motivation is not clear in introduction. What's the disadvantages of previous simulators? How does UNREALZOO settle them?
- The interaction between agents and the world is not discussed, which is important for embodied training.
- There are no language-modal input/communication among agents. The multi-agent feature is not highlighted.


[1] Cheng, Zhili et al. “LEGENT: Open Platform for Embodied Agents.” ArXiv abs/2404.18243 (2024): n. pag.

[2] Yang, J., Ding, R., Brown, E., Qi, X., & Xie, S. (2024). V-irl: Grounding virtual intelligence in real life. arXiv preprint arXiv:2402.03310.

questions:
- About the choice of base engine, is UE the only engine that can render photo-realisitic images?
- There is a lot of Agent body mentioned in table 1. Are they driven by pre-defined policy? What are their action spaces?
- Cannot find Table 3.3 in line 308 and Table 4.1 in line 368. `GPT4-o `in table 5 should be `GPT-4o`.
- Why GPT-4o performs so poorly? Do you do the error breakdown analysis? Can you use other information like position or third-person view to increase the success rate?
- In social tracking experiment, as the training set grows, the growth of SR is intuitive. Can you conduct the experiment on same scale training (compared to the less diverse data) to give a fair evaluation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730301652406

thread_type:
official_review

<thread_object>
creator: reviewer_dbro

<content>
summary:
This paper introduces UnrealZoo, a collection of photorealistic environments based on the Unreal game engine. These environments serve as a platform for training and analyzing embodied agents in highly realistic virtual environments. Moreover, unlike previous works that limit to small scenarios (e.g., a kitchen or a room), UnrealZoo provides vast virtual environments with carefully designed assets that resemble the challenges that agents operating in the real world would face. UnrealZoo is based on UnrealCV, which authors have modified to suit the needs of embodied agent research, improving its usability and performance (FPS). Experiments demonstrate the usability of UnrealZoo for embodied agent research, showcasing applications such as visual navigation, social tracking, and more. Results highlight the importance of diversity in the training environments for offline RL methods, how these fail to generalize across embodiments, and show the limitations of VLM models in highly complex realistic environments.

strengths:
- The paper introduces realistic, highly complex open-world scenarios that resemble real-life scenarios embodied agents would face in the real world. UnrealZoo fills the gap between open-world environments that are far from the complexity of real-life scenarios (e.g., MineDojo, NetHack Learning Environment, or Craftax) and realistic environments that are usually far from the vast scenarios typically encountered in real life (e.g., 
ThreeDWorld or Habitat 3).

- UnrealZoo provides 100 scenes of very different natures and many playable entities (humans, animals, cars, robots, etc.). Although realistic virtual environments exist for embodied AI research, these often focus on some specific domain (e.g., CARLA and autonomous driving). The variety of the UnrealZoo environments makes the proposed framework a very versatile tool for embodied AI research.

- Although tools exist for creating environments in the Unreal game engine and libraries for computer vision (UnrealCV), the authors modify UnrealCV to suit the needs of embodied agent research. Moreover, they make some modifications to the agent-environment communication and the rendering pipeline improving the performance (frames per second) of the environments significantly.  Finally, the authors provide an easy-to-use but versatile API based on OpenAI Gym.

weaknesses:
Although I think that the main contribution of this work (UnrealZoo) could be of great relevance to the embodied AI field, I have two major concerns that I strongly believe should be addressed before:

**Concern 1:**  In lines 185-187 authors state that the environments are sourced (and paid) from the Unreal Engine Marketplace (now renamed as Fab, see https://www.fab.com/), and in lines 236-238 they mention: *"[...] we will package the projects and release binaries for the community."*. However, the standard license of Fab states that (verbatim from section 5.a of https://www.fab.com/eula): *"Under a Standard License, you may not Distribute Content on a standalone basis to third parties [...]"*. Please **make sure** that distributing the binaries as mentioned in the paper is legal under the terms in which the assets were purchased. 

Moreover, if the binary distribution of the environments is legal, the fact that this has to be shipped in a binary package greatly limits the open-source nature of the project. For example, if a contributor or developer wants to modify existing UnrealZoo environments, they would be greatly limited by the binary format of the Unreal environment. I'm open to discussion and willing to read the responses of the authors in this regard. 

**Concern 2:**  I strongly believe that the overall presentation and writing quality of the paper should be improved. Examples:

- Lines 123 to 142 discuss previous literature on virtual environments. This text uses the "virtual environments" term which is very generic, but only discusses environments based on realistic simulators. Please modify the text to explicitly refer to **only** realistic simulator-based environments, or include the vast literature on environments that don't simulate the real world (e.g., NetHack, MineDojo, MineRL, Craftax, Atari, VizDoom,...).   

- Missing details in the experimentation (Section 4). The authors do not provide any detail (or reference) on the specific methods used for the experiments. For example, the online and offline RL methods are unknown, there is no mention of the NN architectures employed. Moreover, encourage the authors to include the training curves of the methods (at least in the appendix). Currently, there is no evidence of the level of convergence of the methods. Furthermore, there is no information on how the human benchmark was collected: how many humans have been used? Were the humans informed on the task to solve or only had access to reward values? Were the humans experienced in video games?   

- In L402 authors define the "Average Accumulated Reward" metric. I believe the authors refer to the average episodic return typically employed in RL. If this is the case, I think that common terms are much preferable instead of introducing new terminology.

- Section in the appendix should be listed with letters, not with numbers. For example, the first section of the appendix should not be "7 Data", but "A Data". I believe this could be fixed by including the "\appendix" command in LaTeX. Moreover, the appendix sections include many of the minor issues from the main text (some are pointed out below). Please consider also proofreading the appendix sections. 

- The references section should be carefully checked and improved. These are some of the issues, but please check all the references:
    + Some papers referenced as preprints have been published in conferences or journals. For example, the CARLA paper was published in CoRL 2017 but is listed as an arXiv preprint.
    + Incorrect capitalization in many titles. Example: CARLA in Dosovitskiy et al. 2017 should be all uppercase. 
    + Some names have been abbreviated, for example: in Gaidon et al. 2016. The same reference is missing the full name of the conference. 
    + Gupta et al. 2017 and many other references have inconsistent use of capitalization.
    +  Some references include the acronym of the conference and others don't.
    + Inconsistent format of the proceedings name, for example: some include the year, some don't.
    + Some references are included as @article when should be @inproceedings. Example: Yuan et al. 2020. 

**Minor issues:**

I found many minor writing and presentation issues in the paper, in the following I list some of them. The issues are individually minor, but many minor minor issues add a major issue. Please consider an exhaustive revision of the full paper.  

- When listing the contributions (last part of the intro) the numbers jump from 1) to 3), missing 2).
- The caption of Table 1 is missing the final dot. Moreover, the employed icons should be explained in detail somewhere in the main text or the appendix.
-  L197: "[...]Project Website [...]" should be lowercase.
- L309: "To be user-friendly [...]"; L311: "In this way, the beginner can easily use and customize [...]", reformulate to maintain formalism. 
- L349: "visual Navigation tasks [...]" fix capitalization.
- L350 and L351 have missing whitespace before and after parenthesis.
- L368: "In Table. 4.1". Extra dot after Table and there's no Table 4.1, I believe the authors refer to Table 3.
- Table 3 does not define what the numbers on the table represent. I believe these are EL and SR, but these should be explicitly mentioned.
- L413: missing whitespace before parenthesis.
- L414: extra dot inserted after "Figure".
- L428: "[...] are provided in the appendix.",  please specify the appendix section.
- L431: extra dot and whitespace inserted after "Figure". 

I want to emphasize that I believe that the contribution of the work is relevant to the field, thus, I'm willing to update my score if the authors address the mentioned concerns and issues.

questions:
- **Q1:** The paper includes experiments on the cross-embodiment generalization capabilities of some offline RL, however, it does not motivate cross-embodiment generalization per se, which I find non-trivial. What is the motivation behind cross-embodiment? Why is cross-embodiment generalization an interesting capability for embodied agents? Is there any practical or real-life scenario where cross-embodiment generalization is relevant?  

- **Q2:** What happens when the agent's implementation can not handle the control frequency specified by the environment? Are the actions repeated or no action is taken (something like no-op in some RL environments)? Is it possible to freeze the environment to wait for the action of the agent?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662564624

thread_type:
official_review

<thread_object>
creator: reviewer_bmxe

<content>
summary:
This paper proposed UnrealZoo, a photorealistic and large-scale environment for embodied AI agents. In UnrealZoo, agents can perform much more complicated actions than just traditional navigation, such as jumping and climbing. Further, agents can control vehicles, humanoids, animals, etc, allowing experimentation with different embodiments.

The authors propose and instantiate various tasks in their proposed simulator. They evaluate both VLMs and RL-trained agents on a subset of their proposed tasks.

strengths:
The visuals from the proposed simulator look great.

There are a variety of task and environments. Environments also have a wide variety of scales.

The authors improved the performance of the rendering pipeline.

The environment supports multiple agents.

weaknesses:
My biggest concern with this paper is that the proposed tasks and simulator lack a defined direction.

There has been a considerable amount of interest centered around indoor navigation with the goal of sim2real transfer. Indoor navigation has been the subject of focus due to real robots that can serve as a deployment target existing, albeit it far from perfect hardware. The reviewer is unaware of any existing hardware platforms that would be sensible deployment targets.

There is also considerable interest in environments to evaluate new reinforcement learning methods and algorithms. While there are multiple criteria for these environments, one key one is speed -- the environment itself must be very performant as the ability to quickly iterate on new ideas is key. UnrealZoo is unfortunately very slow by these standards.

While I do find the proposed simulator, tasks, and environments interesting, I am concerned about the value of the contribution.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683831126

thread_type:
official_review

<thread_object>
creator: reviewer_7keb

<content>
summary:
This paper introduces UnrealZoo, a collection of 100 3D environments based on top of the UnrealCV engine. This collection of environments is novel since it spans a variety of scales, agent types, has a rich navigation system and is multi agent.

strengths:
Originality: The paper demonstrates originality by contributing large scale diverse environments that can't be found for training RL agents today. Performance of RL agents on such environments from different playable entities hasn't been studied in detail before and having such environments would allow for that.
Quality: Inclusion of images, algorithmic benchmarks, and comparison tables make the paper comprehensive 
Clarity: Paper is well written and clear. Details make it clear that the environment will be easy for the community to leverage for training and benchmarking.
Significance: With the integration of OpenAI Gym and ability to use this without expertise in Unreal, this work could allow for significant downstream RL benchmarking and explorations that could lead to interesting insights.

weaknesses:
While the types of environments included in UnrealZoo are diverse, there is little diversity in the types of interactions and actions the agents can have in these environments. This limits how complicated these environments can be and how much novelty these environments will drive in terms of RL algorithms explored for learning and becoming experts. 

It is a little hard to understand what the "ceiling" for each task is in terms of performance. How hard are the given tasks for current RL algorithms? How long does it take for them to learn and become experts? 

Inclusion of results for only DowntownWest feels lacking. Would like to see what performance looks like for other environments too.

questions:
1. How hard are the given tasks for current RL algorithms? With some limited compute, is it possible for RL algorithms to achieve the same performance as a human expert today? These environments can drive downstream RL innovation by just simply being impossible to become experts in today.

2. How long does it take for an RL agent to become an expert in these environments? Another interesting problem to explore in RL is becoming an expert fast. If it takes unreasonably long to become an expert in an environment, then these environments become interesting to share with the community.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733158680270

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response to All Reviewers

comment:
Dear Reviewers,

We sincerely thank you for your thorough and constructive reviews of our manuscript. We appreciate the time and effort you have invested in providing valuable feedback, which has greatly contributed to improving the quality of our work. 

We have carefully considered your comments and have made several revisions to address the concerns raised. Specifically, we have:

- **Enhanced Clarity and Presentation:** We have revised sections of the paper to improve clarity and coherence, ensuring that our contributions and the motivations behind our work are clearly articulated. We have reorganized the appendix and added more details about the environments and experiments. We have also corrected minor issues in the writing and presentation as suggested.  We will conduct a thorough proofreading to enhance the presentation of the next version.

- **Expanded Details of Experiments:** We have included additional qualitative and quantitative results to provide a more comprehensive view of the performance of our agents, including the [demos of agents in various environments](https://unrealzoo.notion.site/task-evt), the [failure case analysis](https://unrealzoo.notion.site/vlm-agent), and more results in Appendix D. This includes clarifications on the [diversity of interactions](https://unrealzoo.notion.site/playable-entities), the implementation details of each agent (Appendix B&C), the time required for training RL agents, and the weakness of current VLM(GPT-4o) agents.

- **Addressed Technical Concerns:** We have provided detailed analyses of simulation speed and the physical capabilities of our platform, emphasizing UnrealZoo's flexibility and potential for exploring emerging topics in embodied AI, including multi-modal reasoning, planning in dynamic environments, and social interactions. The sim2real transferability has already been demonstrated by previous works that also use unreal engines to train agents.

- **Clarified Licensing and Open-Source Strategy**: We have clarified your compliance with licensing agreements and the rationale behind our distribution strategy, which aims to balance customization ability with the need for high-quality content. We will open-source UnrealCV+ and the gym interface we built for the community. We also noted the source of each environment in the [scene gallery](https://unrealzoo.notion.site/scene-gallery) to help potential developers access the source of the content. 

We believe these revisions improve our submission and address your concerns. We appreciate your guidance in refining our work and remain committed to contributing to the community. 

Thank you once again for your insights. **As the deadline for the open discussion approaches**, we look forward to your feedback on our response. If you have any further concerns, we are open to discussing them with you **before the deadline (Dec 2nd at midnight AoE)**.

Best regards

Authors of UnrealZoo
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces UnrealZoo, a photorealistic and large-scale environment designed for training and evaluating embodied AI agents. Built on the Unreal Engine, the proposed collection includes 100 high-quality 3D scenes and offers diverse playable entities to facilitate embodied AI research. The authors enhance the UnrealCV APIs for better communication efficiency and provide easy-to-use Gym interfaces to accommodate diverse research needs. Experiments conducted on UnrealZoo benchmark tasks such as visual navigation and tracking underscore the value of diverse training environments, while also analyzing the limitations of current RL- and VLM-based agents when working in dynamic and unstructured open worlds.

The reviewers appreciated the variety and realism in UnrealZoo, highlighting its 100 diverse scenes and a range of playable entities such as humans, animals, cars, and robots. They also acknowledged the improvements to the rendering pipeline and enhancements to UnrealCV and agent-environment communication, emphasizing how these could make UnrealZoo a useful tool for embodied AI research.

Reviewers also raised key concerns: the paper misses a clear focus or defined direction, with UnrealZoo not aligning well with sim2real transfer goals or the needs of highly performant RL environments. They mentioned UnrealZoo’s limited practical relevance to real-world deployment platforms and its slow performance. Additionally, binary-only distribution of UnrealZoo was noted as limiting flexibility for third-party modifications. Reviews inquired about missing technical and experimental details. Persistent issues with writing and presentation were also raised. The AC acknowledges that the authors made notable improvements to the writing through multiple rounds of responses.

This submission received scores of 5, 5, 5, and 3 in the initial review phase, and 6, 6, 5, and 3 after the rebuttal phase, with two reviewers (7kEB and u5UD) increasing their ratings. During the post-rebuttal discussion phase, R-u5UD (score 6) noted limited experience in the area, explicitly stated they were "not sure about", and marked their review confidence as 2. R-7kEB (score 6) was asked about potentially championing this paper, but they did not support this. After careful examination and post-rebuttal discussion with reviewers, the AC finds arguments from R-BMxe (score 5) and some from R-dbRo (score 3) most significant for this submission’s evaluation. R-BMxe maintained their "primary concern still stands" regarding the "lack of a clear contribution". 

In post-rebuttal discussion, while the AC and the committee did acknowledge writing improvements and some clarifications, they concluded that paper needs more efforts to refine its focus and communicate a clear and compelling contribution to meet ICLR standards.