Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduced a large-scale dataset of 3D Environments for Embodied AI. The proposed dataset is with a larger scale, more complete of reconstructed scenes, and higher visual fidelity over previous datasets. The paper also benchmarks the PointGoal navigation task on the proposed dataset and existing ones and shows the better transferring capability of agents trained on the proposed HM3D over others.
strengths: - The proposed dataset is on a larger scale and the reconstruction quality is quite better than previous datasets.
- The comparison with previous datasets is clear with lots of statistical and experimental support.
- The transferring experiment of the PointGoal navigation task shows the better quality of the proposed dataset over the previous.
- The dataset is publically accessible with good documentation. The previewing website is cool and we can easilly have a brief view of each scene.
weaknesses: - The proposed dataset seems to be simple as the algorithms trained and tested on it achieve almost 100% (99%Depth & 98%RGB) performance in Table 2. That limits its application and promotion to the community as it didn't reveal any new challenging problems that will guide and help to boost the development of the field. For the example PointNav task shown in the paper, the proposed dataset is easier than the previous MP3D, which limits the upper bound of algorithms developed and selected from it. 
- The paper simply manually removed those scenes that failed to be reconstructed, which may hurt the distribution between the selected dataset and the real world and lead to a simpler dataset. What caused the reconstruction failure? The depth missing because of out of range of depth sensors or objects with reflective, transparent, or dark surfaces that cannot be captured by depth sensors? And are these challenging cases what agents with RGBD sensors confront in the real world? It's thankful to have an analysis and give some insight into the field.
- The paper didn't compare their reconstruction procedure with previous datasets, which is also helpful to evaluate the dataset quality.
- If the dataset is labeled with object semantics and physical attributes, that will enable more application and get more impact.
correctness: Yes.
clarity: Yes, the paper is well written and easy to understand.
relation to prior work: Yes.
additional feedback: None.
review_title: A high-quality dataset but seems to be simple

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper introduces HM3D, which is a large scale 3D scan dataset for embodied AI. It demonstrates the scale and quality of the dataset, both visually and numerically. Extensive experiments about PointGoal navigation are conducted in the dataset, including comparison with Gibson and MP3D. 
strengths: 1. HM3D emphasizes both the visual fidelity and the reconstruction completeness. As a real 3D scan dataset, this is very important to a lot of downstream tasks. For example, bad visual fidelity will limit the generalization from COCO pretrained model to the target dataset, since the model has never "seen" these images. Replica has good visual fidelity but it is a small dataset. The great visual and reconstrution quailty of HM3D will definitely advance the progressive of embodies AI.

2. The experiments are extensive. The navigation agent is trained and cross-validated on Gibson, MP3D and HM3D. It demonstrates the effectiveness of HM3D.

3. The data collection is diversified in terms of geographic regions. As stated in the paper, scenes come from 38 countries and 181 geographic regions. This will help models generalize better.
weaknesses: 1. The dataset is only validated in pointgoal navigation. However, HM3D is actually a very useful dataset and many tasks can be trained and evaluated on HM3D. Real-world 3D datasets are very useful and important for 3D computer vision systems such as single view depth estimation. Besides that, embodies ai also has other tasks besides pointgoal navigation. By looking at table 2, I feel it's actually an easy task, especially when the input has depth. An agent trained on MP3D has already had 0.90 SPL on Gibson dataset. Given that the improvement is mainly on hard episodes, evaluating on a more challenging task makes more sense to me. 

2. The dataset does not specify all the available ground truth. From the experiments I believe 3D mesh, depth and rgb are available. However, how about other annotations? For example, floors and rooms are very useful and supported in AI Habitat. Gibson does not have it while MP3D has it. Is it available in HM3D? Also, does it have semantic annotations (most datasets in Table 1 have and it's useful)?
correctness: The dataset is constructed in a sound way.
clarity: Yes
relation to prior work: Yes. Extensive comparison is conducted in the paper including MP3D, Gibson and ScanNet.
additional feedback: > L140 This is computed as the ratio between the raw scene mesh area within 0.5m of the navigable regions and the navigable space

According to my understanding, it is the ratio between all the raw scene mesh area and navigable space. all mesh area = navigable space + obstacles. Is it correct? Why is 0.5m necessary here?

> Fig 5(a) 

The camera pose of real and rendered images do not look the same. Why?

>  Finally, the visual fidelity of images rendered from HM3D is 20 - 85% higher than prior large-scale datasets,

How do you compute 20 - 85%? Is it based on FID? I cannot compute it from Fig 5(b).
review_title: official review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a new large-scale 3D building-level indoor dataset captured and reconstructed with Matterport's pipeline, mainly used in simulation environment for embodied AI, such as RL-based robot navigation in the room. The dataset is shown to be the largest scale (navigation area), less reconstruction artifacts (e.g. more complete mesh) and high perceptual quality (of rendered images). They also show agents trained in the environment simulated with HM3D achieves better PointNav performance than previous datasets. 
strengths: - a larger dataset from improved Matterport pipeline (since Matterport3D, 2017), largest building-level 3D datasets, mainly for robot navigation task
- show better performance for point navigation task when trained with this dataset due to its diversity, scale, and perceptual quality
weaknesses: - only provide the reconstructed mesh and texture, material, but not the raw images / depth, semantic labels, potentially useful for 3d reconstruction and understanding tasks
- only evaluated the datasets for pointnav, but not evaluated for other embodied AI tasks e.g. interaction, manipulation, e.g. compared to iGibson (2.0)
correctness: Put relevant comments in `Clarity` section
clarity: paragraph around L166
- render images from different views: only mentioned the view angle and (x,y) location (1m x 1m) , what is the height? e.g. human or robot's height? does it matter? 
- are the rendering process the same for each dataset? which renderer do you use? (I assume Habitat) 
- do you need to reconstruct the mesh and textures for other datasets before rendering novel views, or extract from panoramas like the caption in Fig 5?
- can you render with new lighting conditions? from the navigation video, the shadows are not realistically changing as robot moving around the room
	
Fig 3
- x labels not quite clear

Fig4 Right
 - is it true there are scenes in ScanNet with 100% defects?
 - maybe better show the y axis in log scale to tell the datasets apart in terms of % defect?
 - How does HM3D compare to Matterport3D for this metric? is the improvement mainly from the improved reconstruction from Matterport's pipeline over the last 4 years?
 - Confused that there are 9 curves shown in the figure but only 7 legends if I look at the left most column? 
 - How much does the missing roof contribute to this metric?

Fig5
- Left figure: it is not clear what are the 4 images shown on the left. 
- How many real images in Gibson real and MP3D real to compute FID against? 
- what is your opinion on the visual quality of rendered images? e.g. compared to GAN literature, are the FIDs score still high, is it good enough for embodied AI tasks? (potential more advanced task than pointnav)

how to do count the area of stairs in multi-floor buildings? how are they used in nav tasks? are the robots capable of going up and down the stairs?

L217 - when evaluate agents (trained on different datasets) on a test environment, does the agents start from the same `random` locations to the same target goal?

L224 - what is SPL?

L230 - the training episodes are 4.11M for Gibson and 8M for HM3D. Is the comparison fair?

Fig 8 - top left subfigure - how do you explain for depth input, when tested on Gibson (test): agents trained on Gibson is better than trained on HM3D even with less training episodes / training dataset size?
relation to prior work: - how does this dataset compared to ARKitScenes? both claim to be the largest 3D dataset. It seems ARKitScenes provide all captured images / HR and LR depths, annotations which are missing in this dataset
- missing reference on OpenRooms (https://arxiv.org/abs/2007.12868v2)
additional feedback: see clarity section
review_title: large building-scale and high-quality 3D datasets for indoor navigation, but NO raw images / depths, camera poses / intrinsic and semantic annotation

Meta-review: All the reviewers agree that this is a high-quality dataset, easily accessible, and larger-scale compared to existing datasets. Several reviewers noted that the PointNav task showcased is close to saturation, hence this benchmark will not drive much progress on this task. However, other tasks can be set up, as the authors noted, hence this is still a very useful resource for the Computer Vision and Robotics communities.
Decision: Accept