Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper describes the ThreeDWorld platform for interactive multi-modal physical simulation which allows users to simulate sensory data and mechanical interactions between robot agents and objects in a variety of 3D environments. The paper provides a detailed system overview and comparison of the provided capabilities to the previous physical simulators. It also provides an example downstream application of the ThreeDWorld system, and evaluation of an agent trained on the synthetic image and sound data collected directly using the physical simulation system.  

Edit: score updated on July 19th.
strengths: The paper compiles a great effort in implementing a virtual simulation framework which leverage from existing art. It would likely facilitate progress in the field. The authors collect several datasets and open source them. The work combines several existing simulators as well, so it subsumes those efforts to a certain extent.
weaknesses: Simulated data are widely used in sciences (e.g. particle physics) and embodied AI. A primary challenge with simulated data is to understand and quantify the potential sources of systematic differences between simulated and real data, and provide ways to correct for these differences to match the state-of-the-art. The authors state the quality of the systems trained on the TWD data have begun to approach the quality of those learned from ImageNet. I think the adoption of this platform depends on whether the agents would be as good as those trained on real data, otherwise it can only be used for experimentation purposes. For instance, would adding more TWD generated data allow to surpass the ImageNet-trained model?  
correctness: Yes, to the best of my knowledge, but I am not an expert in robotics and embodied AI.
clarity: Yes
relation to prior work: Builds on the prior work.
additional feedback: Additional quantitative examination of how well feature representations learned using TDW-generated images and audio data transfer to real world scenarios would strengthen the paper.
review_title: A virtual simulation framework and a platform for interactive simulation

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This work proposes ThreeDWorld, a 3D virtual world simulation platform being able to support multi-modal interactions between various agents and diverse objects. TDW could be used to learn visual features, produce synthetic datasets, training intelligent agents, etc.
This general-purpose world simulator has various potential applications and paves the way for diverse tasks.

strengths: 1.	TDW has several unique properties that distinguish it from previous works, including the real-time rendering of both indoor and outdoor scenes, comprehensive physics simulation, diverse user interaction, etc.
2.	This work is in a promising direction, which could largely promote future research towards many tasks, including but not limited to scene understanding, action prediction, multi-agent interactions, robot controlling, etc. All these are significant for building more intelligent AI systems that could better perceive and understand the world.

weaknesses: The usage seems a little complicated. More demo code and sample cases are encouraged to be included in their repo.
correctness: The construction process is conducted in a sound way. 
clarity: Yes, it is well written and easy to follow.
relation to prior work: Yes. TDW is not limited to specific domains, it’s a general-purpose platform.

additional feedback: Hope this could be an ongoing effort, including more objects, action types, etc, which could better simulate real-world situations.
review_title: It’s really an impressive work which could significantly facilitate future research related to a variety of tasks.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work presents a holistic simulation platform to evaluate various embodied tasks. It covers the most important embodied tasks that constitute recent research in embodied learning: navigation, manipulation, and interaction. Besides, it also allows for the rendering of realistic multimodal inputs, i.e. both audio and visual. Additionally, the interactions between the agent and surrounding object and between surrounding objects are made to obey laws of physics, thus adding another dimension of realism to the simulation and allowing for training systems that can reason about and infer such laws of interaction. This is very important for building embodied agents that we can deploy in the real world. 


Edit: score updated on July 13th
strengths: 1. Holisticity: AFAIK, this is the only simulation platform that has provisions for solving tasks in 3 main embodied learning paradigms together -- navigation, manipulation and interaction. Although, the extent to which it currently allows manipulation and interaction is kind of limited in comparison to other state-of-the-art simulators (improvements in such directions have been promised as part of future work), such integration could allow for research in building systems through multi-task learning, transfer learning, etc. It also serves as a common platform for comparing different visual or multi-modal tasks and analyze their complexities. 
2. Simulation Realism: the level of photo-realism is pretty good and at par with other state-of-the-art simulators that I know of, if not better. The level of realism in physical interactions is also impressive and it tries to cover a wide range of scenarios (hard object vs. hard object, soft material vs. hard object, liquid vs hard object). The realism is also evaluated by visual and audio recognition experiments, where the training is done on simulated data and testing is done on real-world data.
3. User interaction: the simulation also allows for interaction with a human user through VR. This, in my opinion, is an important feature as it allows for the collection of data containing fine-grained human control and manipulation, that's often hard to script. Such data could help in the modeling of human behavior and learning from human demonstrations. 
weaknesses: 1. Missing comparison in terms of simulation speed: a very vital feature of a simulation platform is the speed at which the simulator renders input data and allows for feedback control. The comparison of ThreeDWorld with other simulators in terms of rendering speed is missing. 

2. Missing comparison with other simulators in terms of realism: although the paper compares different aspects of its simulation with other simulators in a qualitative fashion in table 1, it would be nice to see some numbers. For example, one could perform similar visual and audio recognition experiments in other state-of-the-art simulators (Habitat + SoundSpaces is one example as it provides both audio and visual simulation) and compare the results. It would also be nice to compare the physics simulation with some other state-of-the-art simulator that allows for realistic physical interactions.  

P.S.: I am leaning towards acceptance but would like to see the above-listed concerns addressed. 
correctness: The claims made in the paper and the supporting experiments/evaluation methods look correct to me.
clarity: The paper is written very clearly.
relation to prior work: The work extensively compares with prior work and tries to address their shortcomings. However, some more evaluation would make the work even stronger. I have listed such possible comparisons as part of 'Weakness'.
additional feedback: None
review_title: A one-in-all kind of simulator for embodied learning

Meta-review: The proposed platform for Interactive Multi-Modal Physical Simulation shows to be very useful and flexible. It provides a large set of features, and experiments can be performed in several domains of interest for the NeurIPS community. All reviewers have a positive opinion regarding the contribution, and the discussion period between authors and reviewers was shown to be useful.
Decision: Accept