Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an approach for the visual room rearrangement challenge that creates a voxelized map of the scene during the walkthrough and unshuffle phases, finds the differences between the maps, and trains an agent to rearrange the objects based on the differences. The approach improves substantially over the previous state-of-the-art for the 2-phase variant of the task.
strength_and_weaknesses: Strengths:
- The improvement of the work over the previous state-of-the-art is very impressive.
- The idea is very simple and creative. It is also intuitively easy to visualize and understand failure cases since the internal representations are just voxel-based.
- The component ablation section is very interesting and well thought out.

Weaknesses:
- W1. The setup is a bit overengineered to the visual room rearrangement challenge, and doesn't seem all that robust to practical real-world rearrangement scenarios. Suppose there were small changes in the scenes that might occur when humans actually live in these rooms (e.g., background objects unexpectedly moved or the rest of the scene wasn't perfectly stable). It seems like this approach would have a pretty hard time.
- W2. Using voxel-based representations will scale poorly from a memory perspective to much larger scenes. For this task, it seems sufficiently workable, since all the scenes are 1-room scenes, but it seems like it might become very inefficient to scale beyond simple rooms.
clarity,_quality,_novelty_and_reproducibility: The work is sufficiently novel and well-written. I do not expect any issues with reproducibility.
summary_of_the_review: In general, I'm positive about the paper. It is fairly creative and sufficiently novel. However, I get the sense that it's a bit over-engineered to solve the visual room rearrangement challenge. Regardless, I recommend an acceptance of the work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper targets the room rearrangement task, and proposes a method outperforming the current state-of-the-art end-to-end method. The approach introduced in this work is about building two voxel-based 3D semantic maps of the scene before and after (specified goal)  rearrangement to then rearrange objects with different location in the two maps. First, a trained policy navigates both scenes to build the maps with as many semantic objects as possible. Once the maps are built, disagreements are searched and corresponding objects are rearranged.  The method is pretty simple, but efficient as showcased in the experimental study. It can become a strong baseline for the task.
strength_and_weaknesses: Strengths:
1. The improvement in performance brought from the method is significant. This is particularly important to note as the targeted task is hard.
2. The method is clearly presented in the paper, and the experiments are thorough.
3. Simplicity of the approach can be considered as a strength as it allows easier reproducibility.


Weaknesses:
1. Training policies to use and build semantic voxel maps of 3D scenes has already been done in prior work in Embodied AI. This is properly acknowledged by authors but a lack of novelty is still a major concern.
2. I am not convinced by the necessity to learn the semantic priors in the first phase (i.e. scene exploration from the policy). Wouldn’t a simple exploration (e.g. maximising coverage) of the full environment be the best thing to do as objects could be placed anywhere in the environment ?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. In terms of originality, most components in the paper were already proposed in previous work. However, the right combination of these different pieces to be applied to the rearrangement task can be considered as novel.
summary_of_the_review: The major weakness of this work is the lack of novelty in the proposed method. However, as already mentioned, a proper combination of known methods to achieve a strong performance on a challenging task is still interesting for the community. Moreover, authors propose a detailed experiment section to evaluate the performance of their method, and provide promising future directions to build on top of their work by considering the upper bound performance when using ground-truth perception. I would thus tend to consider this work as marginally above the acceptance threshold, and am waiting for answers from the authors about my remarks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes establishing high-fidelity semantic 3d maps of a scene for the visual room rearrangement task. The method generates two maps, including one for the scene at the goal configuration in the walkthrough stage and the other one for the scene at unshuffle stage. A post-processing is further used to connect the same object instances between two maps. Further, the method introduces a high-level planner, namely the search-based policy, to decide which object needs to be rearranged so that a low-level policy could be used to find the shortest path to the target object or location on the built map. The proposed method achieves state-of-the-art in the 2-phase RoomR dataset, and various ablation studies demonstrate its effectiveness.
strength_and_weaknesses: +) The idea of the proposed method is self-motivated and interesting. The method first reconstructs the screen into a 3d semantic map in the walkthrough stage to store the necessary information. Later, the method reconstructs the scene into another 3d semantic map in the execution stage and compares the disagreement between the two maps. In this way, the agent can not only explicitly record the misplaced object information in the scene but explore the environment and understand its traversable space.

+) The performance improvements on the challenging task over a competing baseline are significant. An ablation study in Table 2 further shows that this two-stage semantic map reconstruction idea has more room to improve over the previous graph-based approach. Moreover, various studies (e.g., different detection confidence thresholds, different numbers of navigation goals, different object sizes, different initial distances to the goal, and different nearest same object category distances) confirm the method’s effectiveness.

+) The proposed method only needs 2 million samples, thanks to offline training. It is very sampling efficient compared to the online RL training counterpart, which requires 75 million environment steps.

-) The idea of voxel-based semantic map reconstruction is highly similar to several works of literature (Min et al., 2021; Chaplot et al., 2021), although the authors point out that the previous works do not reconstruct multiple maps to determine the disagreement between them. However, previous works do not produce multiple maps because the interested tasks do not require it.

-) Based on Table 2, we can find that the proposed method highly relies on the quality of the semantic map. The fidelity of such a 3d semantic map depends on the perception model (MaskRCNN) and the agent pose (ground truth pose in this work). It is unclear whether the model can maintain effectiveness when facing unseen objects or a noisy environment (noisy pose estimation).
clarity,_quality,_novelty_and_reproducibility: Although the idea of a voxel-based semantic map is highly similar to previous works, the authors proposed to employ such an idea in the two-stage room rearrangement task and demonstrate the effectiveness of the proposed method. As a result, the proposed method is self-motivated and reasonable. Moreover, the paper is well written and easy to follow, except for some missing details and confusing parts:

-) Missing details:

(a) Is the MaskRCNN used in the proposed method fine-tuned in the environment? If so, what are the details about the training/validation/test scene splits, and how much data is used to fine-tune?

(b) Is the mask v_t^{mask} also generated by the segmentation predictions (e.g., registered as an unoccupied voxel by the background class and occupied voxels otherwise)?

(c) It is unclear how many steps are used in the “navigate to goal” operation in Algorithm 1. If the agent can use many steps to reach the goal location based on the estimated shortest path, it results in an unfair comparison to baselines. On the other hand, at the end of the algorithm (the while-loop), the agent seems allowed to take unlimited steps for a single goal. Again, it would pose an unfair comparison to the baselines.

(d) Some implementation details are missing, such as the action space used in the environment, the number of allowed steps for an episode, etc.
	
-) Confusing parts:

(a) The notation in Algorithm 1 is confusing. The notation “g” for the segmentation model has been used for the goal specification defined at the beginning of Section 3. The notation “D” sometimes refers to depth observation and sometimes refers to the size of the reconstructed map.

(b) In the paragraph “Inferring the rearrangement goal from the maps”, the authors mentioned considering voxels with “nonzero probability of class c” to contain an instance of that class. However, it is unclear how to determine a voxel’s class when the voxel carries many nonzero probabilities over many classes.

(c) In the last paragraph in section 4.2, the authors mentioned that the proposed method with both ground-truth segmentation and ground-truth search locations achieves a 35.35 absolute percentage improvement in Success over the existing work given by the same experts. However, is not the 35.35 improvement from “Ours+GT Segmentation - CSE + GT BT”, which does not use ground-truth search locations?
summary_of_the_review: This work demonstrates that high-fidelity semantic voxel-based maps could significantly improve the visual room arrangement task performance. The idea is self-motivated, and the results are encouraging and exciting. Moreover, the ablation study also shows that the proposed method has more room for improvement by having a better perception model. I would recommend this work to the conference with more clarification of the missing details and confusing parts mentioned above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach to the Visual Room Rearrangement problem (Weihs et al.). First, a 3D semantic map of the scene is created. Using the map, the agent is able to identify the differences between the initial configuration and the goal configuration (that was observed during the walkthrough stage).

Strengths:
- The paper establishes a new state-of-the-art for the task of room rearrangement, and there is a large gap between the performance of this method and the previous approaches. 
- The method is clearly described, and the experiments are well thought out.

Weaknesses:
- The mapping approach is almost identical to Blukis et al. The citation should be where the mapping approach is explained and not hidden in the related work section. 
- The effectiveness of the approach depends on the perfect pose of the agent. (New results using noisy pose addressed this concern)

The reviewers found the contributions valuable for the community. The AC follows the recommendation of the reviewers assuming that proper citations will be provided for the Blukis et al. work. 

justification_for_why_not_higher_score: The 3D mapping component, which is claimed as a main contribution of the paper is almost identical to previous work (e.g., Blukis et al.). 
justification_for_why_not_lower_score: The paper includes valuable contributions so I vote for acceptance.
Decision: Accept: poster