A Simple Approach for Visual Room Rearrangement: 3D Mapping and Semantic Search
Authors: Brandon Trabucco
Gunnar A Sigurdsson
Robinson Piramuthu
Gaurav S. Sukhatme
Ruslan Salakhutdinov
Venue: ICLR 2023 poster (2023)
Keywords: Embodied AI
Deep Learning
Object Rearrangement

Abstract: Physically rearranging objects is an important capability for embodied agents. Visual room rearrangement evaluates an agent's ability to rearrange objects in a room to a desired goal based solely on visual input. We propose a simple yet effective method for this problem: (1) search for and map which objects need to be rearranged, and (2) rearrange each object until the task is complete. Our approach consists of an off-the-shelf semantic segmentation model, voxel-based semantic map, and semantic search policy to efficiently find objects that need to be rearranged. Our method was the winning submission to the AI2-THOR Rearrangement Challenge in the 2022 Embodied AI Workshop at CVPR 2022, and improves on current state-of-the-art end-to-end reinforcement learning-based methods that learn visual room rearrangement policies from 0.53% correct rearrangement to 16.56%, using only 2.7% as many samples from the environment.

Published as a conference paper at ICLR 2023
A SIMPLE APPROACH FOR VISUAL ROOM REARRANGE- MENT: 3D MAPPING AND SEMANTIC SEARCH
Brandon Trabucco
Gunnar A. Sigurdsson
Robinson Piramuthu
Cameeiie Mellon University btrabucce cs cmu. edu
Amazon Alexa Al
Amazon Alexa Al
robinpir@amazooom com
Gaurav S. Sukhatme
Ruslan Salakhutdinov
University of Southern California and Amazon Alexa Al gauravêusc.cm
Carnegie Mellon University
rsalakhu@cs cmu.
ABSTRACT
Physically rearranging objects is an important capability for embodied agents. Visual room rearrangement evaluates an agent's ability to rearrange objects a room to desired goal based solely on visual input. We propose a simple yet effective method for this problem: (I) search for and map which objects need to be rearranged, and (2) rearrange each object until the task is complete. Our approach consists of an off-the-shelf semantic segmentation model, voxel- based semantic map, and semantic search policy to efficiently find objects that need to be rearranged. Our method was the winning submission to the A12- THOR Rearrangement Challenge in the 2022 Embodied AI Workshop at CVPR 2022, and improves on current state-of-the-are end-to-end reinforcement learning- based methods that learn visual room rearrangement policies from 0.53% correct rearrangement to 16.56% using only 2.7% as many samples from the environment.
INTRODUCTION
Physically rearranging objects an everyday skill for humans. but remains a core challenge for embodied agents that assist humans in realistic environments. Natural environments for humans are complex and require generalization a combinatorially large number of object configurations (Batra et al.] 2020a). Generalization in complex realistic environments remains an immense practical challenge for embodied agents, and the rearrangement setting provides a rich test bed for embodied generalization in these environments. The rearrangement setting combines two challenging perception and control tasks: (1) understanding the state of a dynamic 3D environment, and (2) acting over a long horizon to reach goal. These problems have traditionally been studied independently by the vision and reinforcement learning communities (Chaplot et al.) [02T]. but the advent of large models and challenging benchmarks is showing that both components are important for embodied agents.
Reinforcement learning (RL) can excel at embodied tasks, especially if lot of experience can be leveraged (Weihs al.) 2021 Chaplot al. [2020b] Ye al.) 2021) for training. In a simulated environment with unlimited retries, this experience is cheap to obtain, and agents can explore randomly until good solution is discovered by the agent. This pipeline works well for tasks like point-goal navigation (Wijmans et al.) 2020). but in some cases this strategy is not enough. As the difficulty of embodied learning tasks increases, the agent must generalize an increasing number of environment configurations, and broadly scaled experience can become insufficient.
In the rearrangement setting, a perfect understanding of the environment simplifies the problem: an object is here, it should go there, and the rest can be solved with grasping and planning routines. Representing the information about the locations and states of objects in an accessible format is therefore important contribution for the rearrangement setting. Our initial experiments suggest that accurate 3D semantic maps of the environment are one such accessible format for visual rearrangement. With accurate 3D semantic maps, our method rearranges 15.11% of objects correctly, and requires significantly less experience from the environment to do so. While end-to-end RL requires up 75 million environment steps in Weihs et al. (202T), method only requires 2 million

---

Published as a conference paper at ICLR 2023
Rearrange
Search
11 Locating
(2) Moving To Goal
Map
(3) Placing Object At Goal
Figure I: Our method incrementally builds voxel-based Semantic Maps from visual observations and efficiently finds objects using a Semantic Search Policy. We visualize an example rearrangement on the right with the initial position of the pink object (laptop the bed), followed by the agent holding the object (laptop). and finally the
destination position of the object (laptop on desk).
samples and trains offline. Our results suggest end-to-end RL without an accurate representation of the scene may be missing out fundamental aspect of understanding of the environment.
We demonstrate how semantic maps help agents effectively understand dynamic 3D environments and perform visual rearrangement. These dynamic environments have elements that move (like furniture), and objects with changing states (like the door of a cabinet). We present a method that builds accurate semantic maps these dynamic environments, and reasons about what has changed. Deviating from prior work that leverages end-to-end RL, we propose a simple approach for visual rearrangement: (1) search for and map which objects need to be rearranged, and (2) procedurally rearrange objects until a desired goal configuration is reached. We evaluate our approach on the A12-THOR Rearrangement Challenge (Weihs al. 202T) and establish new satt--of-the-are
We propose an architecture for visual rearrangement that builds voxel-based semantic maps of the environment and rapidly finds objects using a search-based policy. Our method shows an improvement of 14.72 absolute percentage points over current work in visual rearrangement, and is robust to the accuracy of the perception model, the budget for exploration, and the size of objects being rearranged. We conduct ablations to diagnose where the bottlenecks are for visual rearrangement, and find that accurate scene understanding is the most crucial. As an upper bound, when provided with a perfect semantic map, method solves 38.33% of tasks, potential for significant out-of-the--be gains as better perception models are developed. Our results show the importance of building effective scene representations for embodied agents in complex and dynamic visual environments.
RELATED WORK
Embodied 3D Scene Understanding. Knowledge of the 3D environment is at the heart of various tasks for embodied agents, such as point navigation (Anderson et al.) (2018)), image navigation (Batra)
et al. 2020b Yang al. 2019), vision language navigation (Anderson al. 2018b Shridhar et al. 2020 embodied question answering (Gordon et al. [018 Das et al. 2018). and more. These tasks require an agent reason about its 3D environment. For example, vision language navigation (An- derson et al. 2018b; Shridhar et al. 2020) requires grounding language in an environment goal, and reasoning about where navigate and what modify in the environment reach that goal. Reasoning about the 3D environment is especially important for the rearrangement setting, and has rich interdisciplinary history in the robotics, vision, and reinforcement learning communities.
Visual Room Rearrangement. Rearrangement has long been one of the fundamental tasks in robotics research (Ben-Shahat & Rivlin| 1996| Stilman et al. 2007 King et al. 2016 Krontiris & Bekris 2016 Yuan et al. 2018 Correll et al. 2018 Labbé al. 2020). Typically, these methods address the challenge in the context of the state of the objects being fully observed (Cosgun et al. 2011 King al. 2016). which allows for efficient and accurate planning based solutions. In contrast,
there has been recent interest in visual rearrangement (Batra et al. 2020aa] Weihs et al., 202T (02)(0 Qureshi
al. 2021 Goyal et al. 2022 Gadre al. 2022) where the states of objects and the rearrangement
2

---

Published as a conference paper at ICLR 2023
goal are not directly observed. In these cases, the agent is provided direct visual input, and the environment is relatively complex and realistic. This latest iteration of rearrangement shares similarity with various other challenging embodied AI tasks such embodied navigation (Anderson et al.
2018a
Batra et al., 2020b] Chaplot et al. 2020a (Shidhhrr et al. 2020) Francis et al.) 2021 Min et al.
Min
2021 Pashevich et al.
2021 Singh et al. 2021 and embodied question answering (Gordon et al.
2018
Das et al.
2018 which require finding objects and reasoning about their state.
A12-THOR Rearrangement Challenge. Our work builds on the latest rearrangement methods and demonstrates how building accurate voxel-based semantic maps can produce significant gains. We focus on the A12-THOP Rearrangement Challenge (Weihs al,, 202Tb, which uses A12-THOR an open-source and high-fidelity simulator used in many prior works (Gadre al. 2022 Weihs cct al. 2021 Shridhar et al., 2020, Gordon et al. 2018). Prior works on this challenge have studied a variety of approaches, including end-to-end RL in Weihs et al. (202T), and planning- -band approach in Gadre et al.] (2022). Our approach is the first to use voxel-based semantic maps to infer what to rearrange from experience goal as described by |Batra et al. (2020a). Though both Gadre et al. (2022) and our method use planning. Gadre et al. (2022) use a graph-based continuous scene representation, and we use voxel-based semantic maps instead, which we show is more effective.
3D Mapping& & Search. Agents that interact with an embodied world through navigation and ma- nipulation must keep track of the world (mapping) (Thrun] (2002) and themselves (localization) (Thrun extensively studied in robotics by processing low-level information (Engel et al. 2014), building semantic maps (Kuipers & Byun 1 and more recently, via techniques specifically developed to handle dynamic and general aspects of the environment (Rünz & Agapito 2017 Rosinol ict al. 2021;; [Wong et al.) 2021). When semantics are more important than precision, such as for embodied learning tasks, recent methods have looked at neural network- -aaeed maps (Gupta et al.
et al.
2017 Chen et al. 2019 Wu et al. 2019a; Chaplot et al., 2020b Blukis et af.) 2021 Chaplot et al.
2021p. Our method builds on these and adopts the use of voxel- baaeed semantic map and pretrained
semantic segmentation model-a similar methodological setup to CCaplot et al. (2021) ; Min et al.
(2021 Bluuis et al. [2021). However, our method diverges from these prior works by using multiple
voxel -based semantic maps to infer what to rearrange from an experience goal as described by Batra
ct al. (2020a). These prior works have instead considered geometric goals in Chaplot et al. (2021) and
language goals in Min et al. (202T); Blukis et al. (202T). and ours is the first to consider an experience
goal (Batra et al., 2020a). Furthermore, while search-based policy is used in Min et al. (2021), ) we are the first to use search with an unspecified destination (the target object is not known priori).
3 METHODOLOGGY
In this section, we present simple approach for solving visual rearrangement problems. We begin the section by discussing the visual rearrangement problem statement and metrics we use for evaluation. We then discuss our methodological contributions. First, we propose to build multiple voxel- based semantic maps representing the environment in different configurations. Second, we propose a policy that efficiently finds objects that need to be rearranged. Third, we propose method for inferring the rearrangement goal from two semantic maps to efficiently solve visual rearrangement tasks.
Visual rearrangement definition and evaluation metrics. Consider the rearrangement setting defined by Batra et al. (2020a), which is a special case of a Markov Decision Process (MDP) augmented with goal specification g = ((sa, S*)). This goal specification encodes the set of states S" for which the rearrangement task is considered solved from initial state 86- The agent typically does not directly observe the set of goal states S*, and this is reflected by the goal specification function o : S x 2 - G. We consider setting where the rearrangement goal is specified visually and the agent initially observes the environment in its goal configuration. This setting is especially challenging because the agent must remember what the environment initially looked like to infer the set of goal states. Once the goal has been understood and rearrangement has been attempted, we evaluate agents using metrics introduced by Weihs et al. (202I). We consider Success metric that measures the proportion of tasks for which the agent has correctly rearranged all objects and misplaced none during rearrangement. This metric is strict in the sense that agent receives success of 0.0 if at least one object is misplaced even if all others are correctly rearranged. We consider additional % Fixed Striet metric that measures the proportion of objects per task correctly
3

---

Published as a conference paper at ICLR 2023
Semuntic Maps
Map Mlagrrement
Laptop
Mapping
D
Unsbuffle Phase
Book
:
Mapping
D
Walkthrough Phuse
Pencil
Layers
Deptbwtse Max
Softmax
GroupNorm
Search
Relu
Convd
Rearrange
Figure 2: Our method builds voxel-based Semantic Maps from visual observations. Our Semantic Search Policy helps build accurate maps by selecting navigation goals to efficiently find objects that need to rearranged. Once accurate maps are built, our method compares the Semantic Maps to identify disagreements between the maps, and rearranges objects resolve those disagrcements using deterministic rearrangement policy.
rearranged, equal to 0.0 per task if any were misplaced. This second metric is more informative regarding how close the agent was to solving each task. Effective agents will correctly rearrange all objects in the scene their goal configurations, maximizing their Success and %Fixed Striet.
Building two semantic maps. Our approach builds off recent work that uses voxel-based semantic maps in embodied settings (Blukis et al.) [02T]] Min et al. [02T Chaplot et al. 202T). Our work differs from these that we use multiple basel semantic maps to encode both the goal state and current state of the environment. In particular, we build two semantic maps mo, m2 E R ">x Dxc that represent 3D grids with H W x D voxels. Each voxel is represented with a categorical distribution on C' classes encoding which class is likely to occupy each voxel. Empty voxels are assigned the zero vector. In an initial observation phase, our agent navigates the scene and builds mo. semantic map encoding the goal configurations for objects in the scene. Likewise, in a second interaction phase, our agent navigates the scene and builds m., a semantic map encoding the current state of objects in the scene. At every timestep during each phase, pose, RGB, and depth images observed, and either my or m, is updated depending on which phase the agent is currently observing.
Incorporating semantic predictions in the maps. Each semantic map is initialized to all zeros and, at every timestep semantic predictions from Mask R-CNN (He al. 2017) are added the map. Given the RGB image observation I,, we generate semantic predictions from Mask R-CNN consisting of the probability of each pixel belonging to remove those with detection confidence lower than 0.9 and conduct an ablation in Section 4.3 We follow Blukis et al. (202T): Min et al.) (202T): Chaplot et al. (20TT) and generate an egocentric point cloud c,ah using the depth observation D, Each point in this point cloud is associated with pixel in the image 4 and a vector of class probabilities from Mask R-CNN. Given the current pose It, we then transform the egocentric point cloud e from the agent's coordinate system to world coordinate system. This transformation results in a geocentric point cloud e:eo that converted to geocentric voxel representation uge0 xxxxxxx of the same cardinality the maps. We generate a voxelized mask umnosk E Hx Dx1 that equals one for every occupied voxel in eg and zero otherwise. New semantic predictions are added to the maps with moving average.
particular class. We filter these predictions
my t + 1] = m4 @ @ - 2 mask (1 c)) + 0000 c)
(1)
The update in Equation I allows voxels to be updated at different rates depending on how frequently they are observed. The hyperparameter E (0, 1) controls how quickly the semantic maps are updated to account for new semantic predictions, and is set to 0.5 in experiments. An overview of how our two semantic maps are built shown in Figure 2] We've detailed how the semantic maps constructed from observations, and we will next describe how navigation goals selected.
4

---

Published as a conference paper at ICLR 2023
Algorithm 1 3D Mapping and Semantic Search For Visual Rearrangement
Require: visual rearrangement environment e, initial voxel-based semantic maps mo, mi R"xWxDXC search-based policy To To (x|m), pre-trained semantic segmentation model for each phase i E {0. 1} do
for each It. Dr, 2y observed do
ut vi < project( g(4), Dr, Jt mi[1] f- mi[t - 1]
project to voxels D update map
oi uf (1 +)) + 21 -
goal is reached or goal does not exist then goal ~ (x))
emit semantic search goal
end if
navigate to goal end for end for
while disagreement d between mi and mi is detected do navigate in my and rearrange match mo
end while
Locating objects with a search-based policy. Building accurate maps requires locating and observing every object in the scene they can be added to the maps. This requires intelligently selecting navigation goals based on where objects are likely to be. We learn high-level policy (x(()m)) that builds off recent work in Min et al. (2021): ) Chaplot al. 2021 and parameterizes distribution over 3D search locations in the environment. The input to the policy is a 3D semantic map m, from whichever phase currently active. The policy a 5-layer 2D convolutional neural network that processes 3D semantic map m, and outputs a categorical distribution over voxels in m,, corresponding to 3D search locations. The policy is trained using maximum likelihood training with an expert distribution p" (x) that captures the locations of the K objects the agent should rearrange in the current scene. This expert distribution in Equation 2 is a Gaussian mixture model with mode centered at the location μa of each object, and variance hyperparameter o² for each mode.
K
(x α
1 K
Σ
N(x;
(2)
Once a policy (x(m,, is trained that captures a semantic prior for object locations, we use planning to reach goals sampled from the policy. We build a planar graph that represents traversable space derived from voxel occupancy in the semantic map, and use Dijkstra's 's algorithm (Dijkstra 1959) to find the shortest path from the agent's current location to the goal. We filter navigation goals to ensure only feasible goals are sampled, and then allow sufficient time for each navigation goal to be reached. Once the current goal is reached, we sample another goal and call the planner again.
Inferring the rearrangement goal from the maps. Once two semantic maps are built, we compare them extract differences object locations, which we refer as map disagreements. These disagreements represent objects that need to be rearranged by the agent. To locate disagreements, we first use OpenCV (Bradski) (2000) to label connected voxels of the same class as object instances. We consider voxels with nonzero probability of class to contain an instance of that class. Object instances are then matched between phases by taking the assignment object instances that minimizes the difference in appearance between instances of the same class. We leverage the Hungarian algorithm (uhh & Yaw 1955), and represent appearance by the average color an object instance in the map. Once objects matched, we label pairs separated by > 0.05 meters disagreements. Given set of map disagreements {(x), 27); (22, ..)...., (I xxy) represented by the current pose or and goal pose x. for each object, we leverage a planning-
rearrangement policy to solve the task. Our rearrangement policy navigates to each object succession and transports them to their goal location. By accurately mapping with search-based policy, inferring the rearrangement goal, and planning towards the goal, our method in Algorithm efficiently solves visual rearrangement,
Code for reproducing our method is available http:////gttthb...o////aaartrrbd
5

---

Published as a conference paper at ICLR 2023
Table I: Evaluation on the 2022 A12-THOR 2-Phase Rearrangement Challenge. Our method attains tfte-of-thheart performance on this challenge, outperforming prior work by 875% %Fixed Striet. Results are averaged over 1000 rearrangement tasks in each of the 2022 validation set and 2022 test set. Higher is better. A Success of 100.0 indicates all objects are successfully rearranged if none are misplaced and 0.0 otherwise. The metric %Fixed Striet is more lenient, equal the percent of objects that are successfully rearranged if none are newly misplaced, and 0.0 otherwise.
Validation
Test
Method
% Fixed Strict
Success
Fixed Strict
Success
VRR + Map (Weihs et al. 2021
1.18
0.40
0.53
0.00
CSR (Gadre al. 2022)
3.30
1.20
1.90
0.40
Ours w/o Semantic Search Ours
15.77
4.30
+795%
15.
+900%
3.60
17.47
6.30
+871%
16.56
+1175% 4.70
4 EXPERIMENTS
In this section we evaluate our approach and show its effectiveness. We first evaluate our approach on the A12-THOR Rearrangement Challenge Weihs et al. (20TT) and show our approach leads an improvement of 14.72 absolute percentage points over current work, detailed in Subsection 4.1 This benchmark tests an agent's ability to rearrange rooms to a desired object goal configuration, and is suitable choice for measuring visual rearrangement performance. Next, we show the importance of each proposed component, and demonstrate Subsection [4. 4.2ourr our voxel-based map and search- based policy exhibit large potential gains more performant models for perception and search are developed in the future. Finally, we show in Subsectior 4.3 our approach is robust to the quality of object detections and budget for exploration.
Description of the benchmark. In this benchmark, the goal is to rearrange up to five objects to a desired state, defined in terms of object locations and openness. The challenge based on the RoomR (Weihs et al.) (02T) dataset that consists of training set with 80 rooms and 4000 tasks, validation set with 20 rooms and 1000 tasks, and a test set with 20 rooms and 1000 tasks. We consider two-phase setting where agent observes the goal configuration of the scene during an initial Walkthrough Phase. The scene is then shuffled, and the agent is tasked with rearranging objects back to their goal configuration during second Unshufffe Phase. This woo-phase rearrangement setting is challenging because it requires the agent to remember the scene layout from the Walkthrough Phase, to identify the rearrangement goal. Goals are internally represented by set of valid object poses S C ER² x (3)))) x (R' x (O())) ... x (R3 x (3)). but the agent does not observe S* directly. At every time step & during either phase, the agent observes geocentric pose Tt. an egocentric RGB image 4T and an egocentric depth image D, The rearrangement goal is specified indirectly via observations of the scene layout during the Walkthrough Phase. During training, additional metadata is available such as ground-truth semantic labels, but during evaluation only the allowed observations Te, I, and D, can be used. Once both the Walkthrough Phase and Unshuffle Phase are complete, we measure performance using the FFxied Strict and Success metrics described in Section 3
4.1 EFFECTIVENESS AT VISUAL REARRANGEMENT
We report performance in Table T and show an improvement in &Fixed Strier from 1.9 to 15.11 over the current state-of- thttherrt art method, namely Continuous Scene Representations (CSR) (Gadre et al.) 2022). These results show our method is more effective than prior work at visual rearrangement. Our success of 4.63% on the test set indicates our method solves 46 / 1000 tasks, whereas the best existing approach, CSR, solves 4/ 1000 tasks. Furthermore, method correctly rearranges 499 3004 objects in the test set, while CSR rearranges only 57 /3004 objects in the test set.
The results in Table I support two conclusions. First, 3D Mapping is helpful inductive bias. Ours is currently the only method on the challenge to leverage 3D Mapping for identifying rearrangement goals. The next best approach, CSR Gadre et al. (2022), represents the scene with graph, where nodes encode objects, and edges encode spatial relationships. We speculate determining which objects need to be rearranged benefits from knowing their fine-grain 3D) position, which our method

---

Published as a conference paper at ICLR 2023
Table 2: Ablation of the importance of each component of method. Our method produces significant gains perception and search models become more accurate. Results are averaged over 1000 rearrangement tasks in each of the 2022 validation and 2022 test set. As Table [ higher is better, and Success of 100.0 indicates all objects are successfully rearranged if none are misplaced and 0.0 otherwise. Our results show that as perception and search models continue to improve with future research, we have an out-of-the-box improvement of 34.73 Success the test set.
Validation
Test
Method
% Fixed Strict
Success
% Fixed Striet
Success
CSR GTT
3,80
1.30
2.10
0.70
CSR GT BT
7.90
3.00
5.90
2.20
CSR GT MBT
26.00
8.80
27.00
10.00
Ours GT Semantic Search
21.24
7.60
19.79
6.10
Ours + GT Segmentation
66.66
45.60
+1004%
59.29
+1707%
37.55
Ours GT Both
68.46
48.60
+1008%
59.50
+1742% 38.33
directly represents via semantic maps. Second, these results also suggest that our method more successfully rearranges small objects. This is important (see additional results in Subsection 4.5) because many common objects humans use are small-cutlery, plates, cups, etc.
4.2 COMPONENT ABLATION
The goal of this experiment
determine the importance of each component to our method. We consider series of ablations in Table that replace different components of our method with ground truth predictions. We first consider Ours + GT' Semantic Search, where we substitute the predictions of our search-based policy To with the ground truth locations of objects that need to be rearranged. We also consider Ours + GT Segmentation, where we substitute the predictions of Mask R-CNN (He et al. 2017) with ground truth semantic segmentation labels. The final ablation in the table Ours + GT Both includes both substitutions at once. In addition to reporting our performance, we reference the performance of CSR (Gadre et al. 2022) in similar set of ablations. We consider CSR GT T which uses expert trajectories that observe all objects needing to be rearranged, CSR + GT BT which also uses ground truth object detection labels, and CSR GT MBT which additionally truth object instance pairs between the Walkthrough Phase and the Unshuffle Phase. Table shows our method produces better out-of-the-box improvement in all metries as the perception and search components become more accurate, suggesting both components are important.
ground
Table demonstrates our method produces significant gains when paired with accurate semantic search and accurate semantic segmentation. When using ground-tru semantic segmentation labels and ground-truth search locations, our method attains an improvement of 35.35 absolute percentage points in Success compared to existing work given access to the same experts. CSR GT BT makes the same assumptions as our method with both components replaced with ground-truth and used to compute this improvement margin. When prior work is given the additional accommodation of ground-truth object instance pairs between the two environment phases, CSR + GT MBT, our method maintains improvement of 27.55 absolute Success points without the accommodation.. These results show our method has greater room for improvement than prior work, with a &Fixed Striet 32.50 absolute percentage points higher than current work. Our method's room for improvement with more accurate perception and search models is appealing because accurate 3D vision models are an active area of research, and our method directly benefits from innovations in these models.
4.3 STABILITY VERSUS PERCEPTION QUALITY
In the previous sections, we evaluated method's effectiveness rearrangement, and room for growth as better perception and search models are developed. This direct benefit from improvements in perception quality resulting from better models is desireable, but effective method should also be robust when perception quality is poor. In this section, we evaluate our method's 's performance stability function of the quality of object detections. We simulate changes in object detection
7

---

Published as a conference paper at ICLR 2023
Num Newty Misplaced
0.5
RXoeed Striet
1:3
0.85
0:15
Sucvess
0.-
0.82
022
04
66
63
0.
04
10
Detection Confidence Thre
Detection Confidence Threshold
Detection Confidence Threshold
Validation
Test
Figure 3: Rearrangement performance versus perception quality. Dark colored lines represent the average metric across 1000 tasks, and shaded regions correspond to 68% confidence interval. Lower Num Newly Misplaced (left plot) is better, higher %Fixed Strict (center plot) and Success (right plot) are better. Our method improves smoothly as perception quality increases, simulated by varying the detection confidence threshold used to filter Mask R-CNN predictions detailed in Section 3
quality by varying the detection confidence threshold of Mask R-CNN (He et al. 2017) described in Section A low threshold permits accepting detections where Mask R-CNN makes high-variance predictions, reducing the quality of detections overall. In the following experiment, we vary the detection confidence threshold on the validation and test sets of the rearrangement challenge.
Figure 3 shows our method is robust to small changes in perception quality. As the detection confidence increases, simulating an improvement in object detection fidelity, performance of our method smoothly increases. Peak performance with our method on the validation set is attained with a detection confidence threshold close to 0.9, which is the value we employ throughout the paper. Error bars in this experiment are computed using a 68% confidence interval with 1000 sample points, corresponding to 1000 tasks in each of the validation and test sets. The small width error bars indicates the observed relationship between perception quality and performance most likely holds for tasks individually (not just on average), supporting the conclusion our method robust to small changes in perception quality. We make final observation that perception quality increases, fewer objects are misplaced as our method more accurately infers the rearrangement goal. These results suggest our method produces consistent gains in rearrangement as perception models improve.
4.4 STABILITY VERSUS EXPLORATION BUDGET
We conduct an ablation in this section evaluate how the exploration budget affects our method. This is important because the conditions an agent faces in the real world vary, and an effective agent must be robust when the budget for exploring the scene is small. We simulate a limited exploration budget by varying the amount of navigation goals used by the agent when building the semantic maps. lower budget results in fewer time steps spent building the semantic maps, and fewer updates to voxels described in Section With fewer updates, sampling goals intelligently is crucial to ensure the agent has the information necessary to infer the task rearrangement goal.
Figure shows method is robust when the exploration budget is small. Performance is stable when less than navigation goals are proposed by our semantic search module, where no penalty in %FFxed Strict and Success can be observed. This result confirms the effectiveness of semantic search: sampled goals correspond to the locations of objects likely to need rearrangement, so even when the budget is small, these objects are already observed. The experiment also shows that as the budget decreases, fewer objects are misplaced. This is intuitive because when the budget is small, fewer objects in the environment are observed and added the map. reducing the chance of incorrect map disagreements being proposed. Additionally, when the budget is large, the agent spends the majority of the episode in navigation, and may not have enough time left to correct map disagreements, resulting in slightly lower overall performance. These results suggest our method is effective for variety of exploration budgets, and is robust when the budget small.
4.5 FAILURE MODES
Our previous experiments showed instances where our method effective, but an understanding of its limitations is equally important. The goal of this subsection is to identify how and why
8

---

Published as a conference paper at ICLR 2023
Num Newty Misplaced
0:23
0.90
PiFoeed Strict
0.95
0:20
0.90
0.00
8:1e
0:90
25
12.5
1566
5.6
16.0
12.5
150
25
50
75
125
156
Num Navigation Gouls
Num Navigution Goals
Num Navigation Goals
Semantic Search Validation
Semantic Search Test
Figure 4: Rearrangement performance versus navigation budget. Dark colored lines represent the average metric across 1000 tasks, and shaded regions correspond to a 68% confidence interval. Lower Num Newly Misplaced (left plot) is better, higher %Fixed Striet (center plot) and Success (right plot) is better. Our method performs well even when the number of navigation goals is small and maintains performance gain over prior work for all sizes of the budget, from navigation goal (x-axis left) to navigation goals (x-axis right).
method can fail. To accomplish this, we conduct an ablation to study how three indicators-obije size, distance to the goal position, and amount of nearby clutter-affect our method. These capture different aspects what makes rearrangement hard. For example, small objects can be ignored, objects distant their goal can be easier to misplace, and objects too close one another can be mis-detected, We measure the performance of our method with respect to these indicators in Figure in Appendix B and analyze the experimental conditions when method is less effective.
Our results illuminate what kinds of tasks are difficult for our method. We find experimentally that objects further from the rearrangement goal are harder for our method successfully rearrange. Objects within 0.326 meters of the goal are correctly rearranged >30% of the time, whereas objects further than 4.157 meters from the goal only correctly rearranged <20% of the time. One explanation for this disparity in performance could be matching object instances between phases more difficult when those instances are further apart. Better perception models can mitigate this explanation by providing more information about object appearance that may be used to accurately pair instances. While this first observation intuitive, our second is more surprising. We find that our method rearranges small objects as effectively as large objects, suggesting our method is robust to the size of objects rearranges. This quality is desireable because realistic environments contain objects in variety of sizes. Effective agents should generalize to a variety of object sizes.
CONCLUSION
We presented simple modular approach for rearranging objects to desired visual goals. Our approach leverages voxel- based semantic map containing objects detected by perception model, and semantic-search policy for efficiently locating the objects to rearrange. Our approach generalizes to rearrangement goals of varying difficulties, including objects that small in size, far from the goal, and in cluttered spaces. Furthermore, our approach is efficient, performing well even with a small exploration budget. Our experimental evaluation shows our approach improves over current work in rearrangement by 14.7 absolute percentage points, and continues to improve smoothly as better models developed and the quality of object detections increases. Our results confirm the efficacy of active perceptual mapping for the rearrangement setting, and motivate several future directions that can expand the flexibility and generalization of the method.
One limitation of the rearrangement setting in this work is that objects only have simple states: position, orientation, and openness. Real objects are complex and have states that may change over time, potentially from interactions not involving the agent. Investigating tasks that require modelling these dynamic objects the map is an emerging topic that can benefit from new benchmarks and methods. Another promising future direction is using an agent's experience to improve its perception. Feedback from the environment, including instructions, rewards, and transition dynamics, provides rich information about how to improve perception when true labels may be difficult to acquire. Investigating how to leverage all sources of feedback available to an agent is a useful research topic that may unlock better generalization for embodied agents in dynamic environments.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Amazon for supporting this work financially, providing access to computational resources, and feedback intermediate drafts of this manuscript. In addition, thank the reviewers for their suggestions and critique in the review process, which improved this paper. We thank Ben Eysenbach, Devendra Chaplot, Minji Yoon, Theophile Gervet, Murtaza Dalal, and So Yeon Min for their discussion and feedback on the paper. Finally, we thank the teams at the Allen Institute for AI that developed the AI2-THOR Rearrangement Challenge and provided benchmarking code, which has been crucial in this work.
REFERENCES
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. On evaluation of embodied navigation agents. 1vXivv/807..6333 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-languaae navigation: Interpreting visually- grounded navigation instructions in real environments. In CVPR, 2018b.
Dhruv Batra, Angel X Chang. Sonia Chernova, Andrew Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement A challenge for embodied ai. wrXvv:0011.01955. 2020a.
Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, A. Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating objects. arXiv:2006. 13171, 2020b.
Ohad Ben-Shahar and Ehud Rivlin. Practical pushing planning for rearrangement tasks. ICRA, 1996.
Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In CoRI,, 2021.
G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In NeurIPS, 2020a.
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological slam for visual navigation. In CVPR, 2020b.
Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, and Ruslan Salakhutdi- nov. SEAL: self-supervised embodied active learning using exploration and 3d consistency. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), NeurlPS, 2021.
Kevin Chen, Juan Pablo de Vicente, Gabriel Sepulveda, Fei Xia, Alvaro Soto, Marynel Vazquez, and Silvio Savarese. behavioral approach visual navigation with graph localization networks, In RSS, 2019.
Nikolaus Correll, Kostas E. Bekris, Dmitry Berenson, Oliver Brock, Albert Causo, Kris Hauser, Kei Okada, Alberto Rodriguez, Joseph M. Romano, and Peter R. Wurman. Analysis and observa- tions from the first amazon picking challenge. IEEE Transactions Automation Science and Engineering, 2018
Akansel Cosgun, Tucker Hermans, Victor Emeli, and Mike Stilman. Push planning for object placement on cluttered table surfaces. In IROS, 2011.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Question Answering. CVPR, 2018.
Edsger W Dijkstra. note on two problems connexion with graphs. Numerische mathematik, 1959.
10

---

Published as a conference paper at ICLR 2023
Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In ECCV, 2014.
Jonathan Francis, Nariaki Kitamura, Felix Labelle, Xiaopeng Lu, Ingrid Navarro, and Jean Oh. Core challenges in embodied vision-language planning. arXiv:2106. 13940, 2021.
Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song, and Roozbeh Mottaghi. Continuous scene representations for embodied AI. S.ti::2200.1225:// 2022.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. IQA: Visual question answering in interactive environments. In CVPR, 2018.
Ankit Goyal, Arsalan Mousavian, Chris Paxton, Yu-Wei Chao, Brian Okorn, Jia Deng, and Dieter Fox. IFOR: iterative flow minimization for robotic object rearrangement. CoRR, abs/2202.00332. 2022. URL ARLL
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik, Cognitive mapping and planning for visual navigation. In CVPR, 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEFE Computer Society, 2016. doi: 10.1109/CVPR.2016 9.00. Utt.https:
.rgg 1109/CVPR. 2016. 90
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, 2017.
Jennifer King, Marco Cognetti, and Siddhartha S Srinivasa. Rearrangement planning using object-centric and robot-centric action spaces. In ICRA, 2016.
A. Krontiris and K. E. Bekris. Efficiently solving general rearrangement tasks: A fast extension primitive for incremental compling-based planner. In ICRA, 2016.
H. W. Kuhn and Bryn Yaw. The hungarian method for the assignment problem. Naval Res. Logist. Quart, 1955.
Benjamin Kuipers and Yung- Byun. robot exploration and mapping strategy based on a semantic hierarchy of spatial representations. Robotics and autonomous systems, 1991.
Yann Labbé, Sergey Zagoruyko, Igor Kalevatykh, Ivan Laptev, Justin Carpentier, Mathieu Aubry, and Josef Sivie. Monte-carlo tree search for efficient visually guided rearrangement planning. IEEE Robotics and Automation Letters, 2020.
Tsung- Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 936-944. IEEE doi
Computer Society, 2017. doi: 10. 00.1109/CVP..2017.110 CVPR. 2VPR.2017.105.
So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. FILM: following instructions language with modular methods. arXiv:2110007342. 2021.
Adithyavairavan Murali, Tao Chen, Kalyan Vasudev Alwala, Dhiraj Gandhi, Lerrel Pinto, Saurabh Gupta, and Abhinav Gupta. Pyrobot: An open-source robotics framework for research and bench- marking. CoRR, abs//9066 082366 2019, URL BtLphtt:://arat
org/ /bs/////////// 1906. 08236
Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and-language navigation. In ICCV, 2021.
Ahmed Hussain Qureshi, Arsalan Mousavian, Chris Paxton, Michael C. Yip, and Dieter Fox. Nerp:
Neural rearrangement planning for unknown objects. In Dylan A. Shell, Mare Toussaint, and M. Ani Hsieh (eds.). Robotics: Science and Systems XVII. Virnall Event, July 16, 2021, 2021. doi: 10. 15607/RSS.002 XVIL.070. URL WtRl/httss:///
org 1.010.15555 15671 /SSS 2021 XVTI 072
11

---

Published as a conference paper at ICLR 2023
A. Rosinol, A. Violette. M. Abate, N. Hughes, Y. Chang, J. Shi, A. Gupta, and L. Carlone. Kimera: from SLAM to spatial perception with 3D dynamic scene graphs. Intl. J. of Robotics Research, 2021.
Martin Rünz and Lourdes Agapito. Co-fusion: Real-time segmentation, tracking and fusion of multiple objects. In ICRA, 2017.
Gabriel Sarch, Zhaoyuan Fang, Adam W. Harley, Paul Schydlo, Michael J. Tarr, Saurabh Gupta, and Katerina Fragkiadaki. TIDEE: tidying up novel rooms using visuo-semantic commonsense priors. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Hassner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXXIX. volume 13699 of Lecture Notes in Computer Science, pp. 480 496. Springer, 2022. doi: 7/doi 0. 1007/078-3-00111 0011188422-22
URL https
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In CVPR, 2020.
Kunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. Factorizing perception and policy for interactive instruction following. In ICCV. IEEE, 2021.
Mike Stilman, JaanJannnllnenn Ean-Uh Schamburek, James Kuffner, and Tamim Asfour. Manipulation planning among movable obstacles. In ICRA, 2007.
Sebastian Thrun. Robotic mapping: A survey, Exploring artificial intelligence in the new millennium, 2002.
Sebastian Thrun, Dieter Fox, Wolfram Burgard, and Frank Dellaert. Robust monte carlo localization for mobile robots. Artificial Intelligence, 2001.
Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In CVPR, 2021.
Erik Wijmans, Abhishek Kadian, Ari S. Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. In ICLR, 2020.
Yu-Shiang Wong, Changjian Li, Matthias Niessner, and Niloy J. Mitra. Rigidfusion: Rgb-d scene reconstruction with rigidly-moving objects. Computer Graphics Forum, 2021.
Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian. Bayesian relational memory for semantic visual navigation. In ICCV, 2019a.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan Lo, and Ross Girshick. Detectron2. 2019b.
Wei Yang. Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. ICLR, 2019.
Joel Ye, Dhruv Batra, Abhishek Das, and Erik Wijmans. Auxiliary tasks and exploration enable objectnav. arXiv:2104.04111 2021.
Weihao Yuan, Johannes A Stork, Danica Kragic, Michael Y Wang, and Kaiyu Hang. Rearrangement with nonprehensile manipulation using deep reinforcement learning. In ICRA, 2018
12

---

Published as a conference paper at ICLR 2023
APPENDIX
In this appendix we include the following supporting experiments and visualizations:
A. We begin this appendix by presenting the performance of our map disagreement detec- tion module for each object category. We find that method effectively detects map disagreements for both small and large objects, and is therefore robust to object size.
B. We then present performance breakdown of our method for object size, distance goal, and amount of clutter, and find that our method is less effective when objects are further from the goal when nearby objects are closer together.
C. We report confidence intervals for our method's performance on the rearrangement challenge.
D. Finally, we outline the compute infrastructure needed to reproduce our experiments.
E. We list the hyperparameters used in our paper.
F. We categorize why our method can fail and provide a qualitative example.
The official code for our method will be released at publication.
A OBJECT TYPE VERSUS DETECTION ACCURACY
In this section, we visualize the relationship between the performance of our map disagreement detection module, detailed in Section 3 and the category of objects to be rearranged. For each of 1000 tasks in the validation set and test set of RoomR (Weihs et al. 2021), we record which object categories are detected as needing to be rearranged, and log the ground truth list of object categories that need to be rearranged. For each object, we calculate precision as the proportion of objects per category that were correctly identified as map disagreements out of all predicted map disagreements. Similarly, we calculate recall the proportion of correctly identified map disagreements out of all ground-truth map disagreements. Each bar in 3 precision and recall over 1000 tasks per dataset split. The experiment shows that our method is robust to the size of objects that it rearranges because small objects such as the SoapBar, CellPhone, CreditCard, and DishSponge have comparable accuracy to large objects in Figure
a 68% confidence interval of
B PERFORMANCE ANALYSIS
This section extends Section 4.5 with an experiment to show potential failure modes. We consider three failure modes: (1) object size, (2) object distance the goal, and (3) closest object in the same class. These indicators are visualized in Figure
against aggainst %Fxxed. Our experiment suggests our method robust to the size of objects, shown by the lack of global trend in the left plot in Figure [ and confirmed by Appendix AA] Additionally, the experiment shows that objects further from the rearrangement goal are solved less frequently (middle plot), which intuitive. Instances that have been shuffled to faraway locations in the scene may require longer exploration to find, and may be more difficult for our map disagreement detection module to match. final conclusion we can draw from this experiment is that our method can fail when object instances are too close together. This is shown in the right plot in Figure 6 by the steep drop in performance when objects in the same category are < 1 meter apart. In this situation, our semantic mapping module incorrectly detect two nearby objects as a single object, which prevents their successful rearrangement. For each of these potential failure modes, better perception and mapping approaches that more accurately describe object locations and appearance can improve fidelity of our method and reduce failure.
C PERFORMANCE CONFIDENCE INTERVALS
We report 68% confidence intervals in Table 3 to supplement our evaluation in Section 4.1 and Sec- tion 4.2 We calculate intervals using 1000 tasks from the validation and test sets of the RoomR (Weihs) et al. 1 dataset, and report the mean followed by + interval width. Note that the official rearrange- ment challenge leaderboard does not expose confidence intervals, nor the sample-wire wise performance needed to calculate them. Due to this, we unable to compute confidence intervals of the baselines VRR (Weihs et al., 202T) and CSR (Gadre et al. 2022) at this time. These additional results show that our improvements over prior work significantly exceed the 68% confidence interval.
13

---

Published as a conference paper at ICLR 2023
Which Map Differences (Small Large)
Validation Set?
1.00
Recall
035
CD
SA
Bewt
Van
Micsenne
Doose
Which Detections (Small Lurge) Are Correct In The Validution Set?
Precision
Tead
CD
Curck
Telehee (p
Saptto
Wasteel
Band
Kelk
Momeed
Ca
Shreciber
Which Map Differences (Small Large) Are Detected In The Test Set?
Recall
233
Wese
Spook
Clalo
Telathos
Popertaa
Mur
Nowgge
Baok
Cup
Lade
Straa
Doc
Ste
Trias
Tale
Wisk
Dosc
Cale
Which Detections (Small Large) Correct In The Text Set?
Preckton
00
Sootha
Wab
Diatpea
Cade
Spuzzs
Kelk
Pir
Bosk
Wandeel
Lahe
Momerr
Sawet woa
Dave
Figure 5: Performance breakdown on the validation and test sets for various types of objects. The height of bars corresponds the sample mean of precision or recall for our map disagreement detection module. Error bars show 68% confidence interval for each kind of object. The top two plots correspond to precision and recall on the validation set, while the bottom two plots correspond to precision and recall the test set. Object categories are shown the x-axis, and are ordered in ascending order of size. The experiment shows our method is robust to size, with small objects at the left end of the plots having comparable accuracy to large objects at the right end of the plots.
D REQUIRED COMPUTE
The goal of this section is to outline the amount of compute required to replicate our experiments. We will describe the amount of compute required for (1) training Mask R-CNN, (2) training semantic search policy no (x|m;), and (3) benchmarking the agent on the rearrangement challenge. For training Mask CNNN a dataset of 2 million images with instance segmentation labels were collected from the THOR simulator using the training split of the RoomR (Weihs et al. 2021) dataset. then used Detectron2 (Wu et al. 2019b) with default hyperparameters to train Mask R-CNN with ResNet50 (He et al. 2016) Feature Pyramid Network backbone (Lin et al. 2017). We trained our Mask R-CNN for five epochs using a DGX with eight Nvidia 32GB v100 GPUS for 48 hours. Our semantic search policy requires significantly less compute: completing 15 epochs
a dataset of 8000 semantic maps annotated with an expert search distribution nine hours on a single Nvidia 12GB 3080ti GPU. Evaluating method on the A12-THOR rearrangement challenge requires 40
14

---

Published as a conference paper at ICLR 2023
0.50
65
%4Fltd
1.5
16"
0.0
(Meterx")
Distance To Goal (Meters)
Neurest Same Object (Meters)
Validation
Test
Figure 6: Performance of various ablations for different Size (Meters"), Distance To Goal (Meters), and Nearest Same Object (Meters). These indicators measure properties of objects that make rearrangement hard. Colored lines represent the average performance over 1000 tasks in each dataset split. Error bars represent a 68% confidence interval over those same 1000 sample points. The experiment shows our method can fail when objects the same class are too close together (right plot), and when objects are too far from the goal location, typically >4.157 meters (center plot).
Table Confidence intervals for method the A12-THOR rearrangement challenge. Intervals are calculated from 1000 sample points from RoomR (Weihs et al. 2021) validation and test sets. We report pertformance starting with the sample mean, followed by 68% confidence interval width. Our improvements over prior work significantly exceed the 68% confidence interval, which suggests that our improvements are significant and our method performs consistently well.
Validation
Test
Method
% Fixed Strict
Success
% Fixed Strict
Success
Ours w/o Semantic Search
15.77 0.85
4.30 + 0.63
15.11 + 0.84
3.60 + 0.58
Ours
17.47 0.92
6.30 + 0.76
16.56 + 0.89
4.70 = 0.67
Ours GT Semantic Search
21.24 = 0.99
7.60 + 0.83
19.79 = 0.96
6.10 = 0.75
Ours + GT Segmentation
66.66 1.21
45,60 + 1.57
59.29 + 1.26
37.55 = 1.53
Ours GT Both
68.46 1.20
48.60 + 1.57
59.50 = 1.31
38.33 1.57
GPU-hours with a 2080ti GPU or equivalent. In practice, parallelize evaluation across 32 GPUs, which results in an evaluation time of 1.25 hours for each of the validation and test sets.
E HYPERPARAMETERS
We provide list of hyperparameters their values in Table 4 These hyperparameters held constant throughout the paper, except in ablations that study the sensitivity of our method to them, such Section 4.3
Our ablations show our method is robust these hyperparameters.
F REASONS FOR TASK FAILURES
This section explores the reasons why certain tasks in the validation and test sets are not solved by our method. We consider four reasons for task failures that cover all possible outcomes: (1) the agent correctly predicts which objects need to be moved where, but fails to rearrange at least one object, (2) the agent incorrectly predicts an object needs to be rearranged that doesn't, 't, (3) the agent runs out of time, and (4) the agent misses at least one object that needs to be rearranged. We visualize the proportion of failed tasks for each category Figure 7 We find that our method with ground truth perception and search (Ours GT Both) tends to fail rearrange objects after correctly identifying which objects need to rearranged. contrast, the largest reason for failure for our method (Ours) is the agent rumning out of time. followed by rearranging incorrect objects. This suggests the largest potential gains for our method arise from improving the speed and fidelity of map building, whereas, the optimality of the rearrangement policy becomes the bottleneck perfect map is available.
15

---

Published as a conference paper at ICLR 2023
Table 4: Hyperparameters used by our approach for all rearrangement tasks.
Hyperparameter
Value
voxel size
0.05 meters
map height H
384
map width W
384
map depth
96
classes C
54
detection confidence threshold
0.9
rearrangement distance threshold
0.05 meters
expert search distribution a
0.75 meters
He convolution hidden size
64
Ty convolution kernel size
3 x
Ty layers
activation function
ReLU
m optimizer
Adam
wo learning rate
0.0003
Te batch size
Te epochs
15
To dataset size
8000
Task Outcomes (Ours GT Both)
Task Outonmes (Ours)
Proportion Of Thelss
Proportion Of Tasiss
Reasons For Task Fallures
Task Solved
Falled To Rearrange Disagreement
Incurrect Object Rearranged
241
Exereded
Falled Detect Disagroement
validation
validation
Figure 7: Categorization of the reasons why our method fails to solve tasks. The proportion of tasks that are solved (shown in blue) fail due to one of four reasons (orange, green, red, purple) is shown for different ablations of our method. The height per bar corresponds to the proportion of tasks in the validation test set in each category, and error bars indicate a 68% confidence interval. This experiment shows the largest reason for failure is result of mapping errors. In the right plot, the agent fails most frequently by rearranging the wrong object, and by running out of time, which can result from imperfect semantic maps. In contrast, once perfect maps are available in the left plot, the largest source errors are due to an imperfect planning-base based rearrangement policy instead.
G IMAGE FEATURES FOR MATCHING OBJECT INSTANCES
We conduct an experiment where we use image features for matching instances of objects between phases instead of their average color. We use ResNet50 (He al. 2016) pretrained ImageNet, and compute mean image feature h2 E 256 for each object. We process images with the ResNet50 backbone, extract a spatial feature map after the first residual block of ResNet50, and back-project the features to voxel grid. In this fashion, we introduce 256 additional channels at each voxel to store the image features. We then compute h; by averaging the voxel features corresponding occupied voxels for object in the map. Results in 5 show that matching object instances using image features improves the performance our method by 6.97 Fixxe Strict on the test set.
H EFFECT OF SEMANTIC SEARCH ON FOUND OBJECTS
To understand how our Semantic Search policy leads to an improvement in downstream performance, hypothesis is that Semantic Search leads the agent to find more objects during episodes. We
16

---

Published as a conference paper at ICLR 2023
Locating ToeletPaper
Moving To Goal
Attempt Placing Goal
Placed Wrong Location
Figure 8: Qualitative example for why rearranging the correct object can fail. In this task, the agent correctly predicts the ToiletPaper needs to be rearranged, but fails to place the ToiletPaper in the correct location. The rightmost image shows the goal is located on the floor, but the agent mistakenly places the ToiletPaper the bathtub instead, shown in the second image from the right.
Table 5: Matching object instances using image features. Results show that our method can be further improved by matching object instances between the walkthrough and unshuffle phases to maximize similarity of corresponding features from ResNet50 (He al. 2016) pretrained on ImageNet.
Validation
Test
Method
% Fixed Striet
Success
% Fixed Strict
Success
Ours
17.47 + 0.92
6.30 + 0.76
16.56 + 0.89
4.70 + 0.67
Ours + Feature Matching
23.06 + 1.04
7.81 + 0.88
23.59 + 1.03
6.79 0.82
test this hypothesis by measuring the percent of objects found during episodes of the official 2022 Rearrangement Challenge. We consider an object found once the agent navigates within I meter of the object. We track the cumulative percent of objects found, and report in Figure 9 the mean and 68% confidence interval of this metric across 1000 test set tasks. The results confirm our hypothesis: in both phases, Semantic Search leads the agent find more objects faster.
particular, 250 episode timesteps during the Walkthrough phase, the agent has 9.13 + 1.23 higher percent than a uniform baseline. During the Unshuffle phase, the agent has 6.23 + 1.20 higher percent than uniform baseline at 100 timesteps. This improvement becomes less significant as time increases during the Unshuffle phase, suggesting Semantic Search is most helpful with a small time budget.
1 NOISY POSE ESTIMATION
We simulate noisy pose estimation due to imperfect localization and mapping using the sensor noise model introduced by Sarch al. (2022). which builds earlier work from Chaplot et al. (2020b). and constructs a Gaussian distribution based on real robot localization data collected from a real LocoBot robot (Murali et al. 2019), and applies this noise model in simulation. This model adds Gaussian noise with o = 0.005 meters to positional observations, and adds Gaussian noise with 0.5 degrees yaw observations at each timestep. Results with noisy pose are given in Table 6 Our method is robust to this sensor noise model. On both the validation and test sets of the 2022 A12-THOR Rearrangement Challenge, sensor noise leads slight improvement performance. which we attribute to noise producing smoother semantic map.
17

---

Published as a conference paper at ICLR 2023
Wulkthrough
Unshuffle
% Ohjects Found (Test)
% Objects
101
300
252
100
300
300
410
36
Episode Timestep
Episode Timestop
Semantic Search
Uniform Baseline
Figure 9: Impact of Semantic Search on the percent of shuffled objects found during either phase. In both cases, observe increase in the percent of shuffled objects found when using Semantic Search. The improvement is most significant during the walkthrough phase, where Semantic Search leads to improvement of +9.13 percent of objects found at 250 episode timesteps. The improve- ment during the unshuffle phase is smaller, with +6.23 at 100 timesteps, and +1.67 at 500 timesteps.
Table 6: Impact of imperfect localization due to sensor noise on our method.
Validation
Test
Method
Fixed Strict
Success
% Fixed Striet
Success
Ours
17.47
6.30
16.56
4.70
Ours Noisy Pose
19.84
6.80
17.33
4.90
18


Reviews:
[{"review_id": "cv1_TKTSyXY", "reviewer": "Reviewer_jZAh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach for the visual room rearrangement challenge that creates a voxelized map of the scene during the walkthrough and unshuffle phases, finds the differences between the maps, and trains an agent to rearrange the objects based on the differences. The approach improves substantially over the previous state-of-the-art for the 2-phase variant of the task.", "summary_of_the_review": "In general, I'm positive about the paper. It is fairly creative and sufficiently novel. However, I get the sense that it's a bit over-engineered to solve the visual room rearrangement challenge. Regardless, I recommend an acceptance of the work.", "strength_and_weaknesses": "Strengths:\n- The improvement of the work over the previous state-of-the-art is very impressive.\n- The idea is very simple and creative. It is also intuitively easy to visualize and understand failure cases since the internal representations are just voxel-based.\n- The component ablation section is very interesting and well thought out.\n\nWeaknesses:\n- W1. The setup is a bit overengineered to the visual room rearrangement challenge, and doesn't seem all that robust to practical real-world rearrangement scenarios. Suppose there were small changes in the scenes that might occur when humans actually live in these rooms (e.g., background objects unexpectedly moved or the rest of the scene wasn't perfectly stable). It seems like this approach would have a pretty hard time.\n- W2. Using voxel-based representations will scale poorly from a memory perspective to much larger scenes. For this task, it seems sufficiently workable, since all the scenes are 1-room scenes, but it seems like it might become very inefficient to scale beyond simple rooms.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cMEYOL_h-D", "reviewer": "Reviewer_wg2G", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets the room rearrangement task, and proposes a method outperforming the current state-of-the-art end-to-end method. The approach introduced in this work is about building two voxel-based 3D semantic maps of the scene before and after (specified goal)  rearrangement to then rearrange objects with different location in the two maps. First, a trained policy navigates both scenes to build the maps with as many semantic objects as possible. Once the maps are built, disagreements are searched and corresponding objects are rearranged.  The method is pretty simple, but efficient as showcased in the experimental study. It can become a strong baseline for the task.", "summary_of_the_review": "The major weakness of this work is the lack of novelty in the proposed method. However, as already mentioned, a proper combination of known methods to achieve a strong performance on a challenging task is still interesting for the community. Moreover, authors propose a detailed experiment section to evaluate the performance of their method, and provide promising future directions to build on top of their work by considering the upper bound performance when using ground-truth perception. I would thus tend to consider this work as marginally above the acceptance threshold, and am waiting for answers from the authors about my remarks.", "strength_and_weaknesses": "Strengths:\n1. The improvement in performance brought from the method is significant. This is particularly important to note as the targeted task is hard.\n2. The method is clearly presented in the paper, and the experiments are thorough.\n3. Simplicity of the approach can be considered as a strength as it allows easier reproducibility.\n\n\nWeaknesses:\n1. Training policies to use and build semantic voxel maps of 3D scenes has already been done in prior work in Embodied AI. This is properly acknowledged by authors but a lack of novelty is still a major concern.\n2. I am not convinced by the necessity to learn the semantic priors in the first phase (i.e. scene exploration from the policy). Wouldn’t a simple exploration (e.g. maximising coverage) of the full environment be the best thing to do as objects could be placed anywhere in the environment ?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "M9NK1BjMgw", "reviewer": "Reviewer_Vjug", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes establishing high-fidelity semantic 3d maps of a scene for the visual room rearrangement task. The method generates two maps, including one for the scene at the goal configuration in the walkthrough stage and the other one for the scene at unshuffle stage. A post-processing is further used to connect the same object instances between two maps. Further, the method introduces a high-level planner, namely the search-based policy, to decide which object needs to be rearranged so that a low-level policy could be used to find the shortest path to the target object or location on the built map. The proposed method achieves state-of-the-art in the 2-phase RoomR dataset, and various ablation studies demonstrate its effectiveness.", "summary_of_the_review": "This work demonstrates that high-fidelity semantic voxel-based maps could significantly improve the visual room arrangement task performance. The idea is self-motivated, and the results are encouraging and exciting. Moreover, the ablation study also shows that the proposed method has more room for improvement by having a better perception model. I would recommend this work to the conference with more clarification of the missing details and confusing parts mentioned above.", "strength_and_weaknesses": "+) The idea of the proposed method is self-motivated and interesting. The method first reconstructs the screen into a 3d semantic map in the walkthrough stage to store the necessary information. Later, the method reconstructs the scene into another 3d semantic map in the execution stage and compares the disagreement between the two maps. In this way, the agent can not only explicitly record the misplaced object information in the scene but explore the environment and understand its traversable space.\n\n+) The performance improvements on the challenging task over a competing baseline are significant. An ablation study in Table 2 further shows that this two-stage semantic map reconstruction idea has more room to improve over the previous graph-based approach. Moreover, various studies (e.g., different detection confidence thresholds, different numbers of navigation goals, different object sizes, different initial distances to the goal, and different nearest same object category distances) confirm the method’s effectiveness.\n\n+) The proposed method only needs 2 million samples, thanks to offline training. It is very sampling efficient compared to the online RL training counterpart, which requires 75 million environment steps.\n\n-) The idea of voxel-based semantic map reconstruction is highly similar to several works of literature (Min et al., 2021; Chaplot et al., 2021), although the authors point out that the previous works do not reconstruct multiple maps to determine the disagreement between them. However, previous works do not produce multiple maps because the interested tasks do not require it.\n\n-) Based on Table 2, we can find that the proposed method highly relies on the quality of the semantic map. The fidelity of such a 3d semantic map depends on the perception model (MaskRCNN) and the agent pose (ground truth pose in this work). It is unclear whether the model can maintain effectiveness when facing unseen objects or a noisy environment (noisy pose estimation).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster