Task Planning for Visual Room Rearrangement under Partial Observability
Authors: Karan Mirakhor
Sourav Ghosh
Dipanjan Das
Brojeshwar Bhowmick
Venue: ICLR 2024 poster (2024)
Keywords: Task Planning
Object Search
Deep-RL
Robotics
Primary area: applications to robotics, autonomy, planning

Abstract: This paper presents a novel hierarchical task planner under partial observability
that empowers an embodied agent to use visual input to efficiently plan a sequence
of actions for simultaneous object search and rearrangement in an untidy room,
to achieve a desired tidy state. The paper introduces (i) a novel Search Network
that utilizes commonsense knowledge from large language models to find unseen
objects, (ii) a Deep RL network trained with proxy reward, along with (iii) a novel
graph-based state representation to produce a scalable and effective planner that
interleaves object search and rearrangement to minimize the number of steps taken
and overall traversal of the agent, as well as to resolve blocked goal and swap
cases, and (iv) a sample-efficient cluster-biased sampling for simultaneous training
of the proxy reward network along with the Deep RL network. Furthermore,
the paper presents new metrics and a benchmark dataset - RoPOR, to measure
the effectiveness of rearrangement planning. Experimental results show that our
method significantly outperforms the state-of-the-art rearrangement methods Weihs
et al. (2021a); Gadre et al. (2022); Sarch et al. (2022); Ghosh et al. (2022).

Published as a conference paper at ICLR 2024
TASK PLANNING FOR VISUAL ROOM REARRANGE- MENT UNDER PARTIAL OBSERVABILITY
Karan Mirakhor", Sourav Ghosh", Dipanjan Das & Brojeshwar Bhowmick Visual Computing and Embodied Intelligence Lab TCS Research, Kolkata, India (karan. mi rakhor g.sourav10, di jpanjanndoe b.bhowmi ck @tcc com.
ABSTRACT
This paper presents a novel modular task planner under partial observability that empowers an embodied agent to use visual input to efficiently plan a sequence of actions for simultancous object search and rearrangement in an untidy room, to achieve a desired tidy state. The paper introduces (i) a novel Search Network that utilizes commonsense knowledge from large language models to find unseen objects, (ii) a Deep RL network trained with proxy reward, along with (iii) a novel graph-based state representation to produce scalable and effective planner that interleaves object search and rearrangement to minimize the number of steps taken and overall traversal of the agent, well as to resolve blocked goal and swap cases, and (iv) a sample-efficient cluster-biased sampling for simultaneous training of the proxy reward network along with the Deep RL network. Furthermore, the paper presents new metrics and a benchmark dataset ROPOR, to measure the effectiveness of rearrangement planning. Experimental results show that our method significantly outperforms the state-of-the-art rearrangement methods Weihs
et 2021) Gadre et al. (2022) ) Sarch al. (2022) : Ghosh et al. (2022).
INTRODUCTION
Tidying a disordered room based on user specifications is challenging task as it involves addressing issues related to perception, planning, navigation, and manipulation |Bttra et al. (022)) An agent performing an embodied room rearrangement must use the sensor observations and a prior knowledge to produce a long horizon plan for generating sequence of object movements to achieve the tidy goal state. This goal state is specified through geometry, images, language, etc. Batra et al. (2020).
Majority of the existing research room rearrangement emphasizes on perception and commonsense reasoning while assuming navigation and manipulation abilities, without incorporating efficient planning. Based on the goal state definition, they broadly fall into two categories; (i) Commonsense based reasoning without a predefined goal state: The methods Kant et al. (2022): [arch et al.) (2022)
in this category utilize image language-based commonsense reasoning to identify if an object is misplaced from the correct receptacles in their egoview followed by rearranging them using a sub-optimal heuristic planner. Moreover, utilizing text or semantic relation-based anomaly detectors to identify misplaced objects does not resolve blocked goal or swap cases, where an object's ' goal position is obstructed by another misplaced object or vice versa. (ii) User-specific room rearrangement with pre-defined tidy goul state: In this setting, the rearrangement is done based on explicit user specification. Methods like Weihs al. (202)) Gadre et al.) (2022) focus on egocentric perception and use image image feature- based scene representation identify misplaced objects and greedy planner to sequence actions for rearrangement. |Sarch et al. (002)) also performs a user-specific room rearrangement by using semantic relations to identify misplaced objects in agent's egoview, and then rearrange them as they appear without planning. Methods such as Kant et al. (022)); [aachh et al. (2022); Gadre et al. (2022) explicitly explore the room to find objects that initially outside the agent's egoview, since it only provides partial information about the room. However, these approaches incur significant traversal cost due to exploration. Additionally, these methods employ non-optimal planning that does not optimize the number of steps or overall traversal.
In contrast, efficient planning makes rearrangement more effective by optimizing the sequence of actions and minimizing the time and effort required to achieve the goal state. Ghosh et al. (2022).
"Theee authors contributed equally.

---

Published as a conference paper at ICLR 2024
a
(b)
Figure (a) shows the top down view of our Rearrangement task and (b) is the agent's initial egocentric view in the untidy current state for the same setup. The solid 2D bounding boxes indicate the desired goal state for all objects, while the dashed ones show the initial positions of visible objects in the untidy current state. The dotted 2D bounding boxes represent initial positions of unseen objects in the untidy current state. The sponge (magenta), an unseen object, is in a drawer near the stove, while the tomato (green), another unseen object, is stool behind the countertop. There are two scenarios: a blocked goal case with the lettuce (blue) and kettle (yellow) and a swap case between the bread (dark magenta) and pot (dark cyan).
addresses the rearrangement task planning problem by assuming the complete visibility of the room, through the bird's eye view. Their method addresses some planning problems, such as the combinatorial expansion of rearrangement sequencing, and blocked goal and swap cases without explicit buffer. However, the approach does not minimize overall agent traversal during the planning, and its state representation is not scalable to large numbers of objects. Moreover, their reliance the ground truth object positions in both the current and goal states is impractical in real-life. Our aim directed towards novel and more practical aspect of the room rearrangement problem through efficient task planning under partial observability of room using agent's 's egocentric camera view.
The major challenges associated with efficient task planning for room rearrangement under partial observability, as shown in Fig. [ are (i) uncertainty over the location of unseen objects due to partial observability (objects outside the agent's field of view presently which are visible from a different perspective, objects placed within a closed receptacle e.g. spoon in drawer), (ii) scalability to a large number of objects, (iii) combinatorial expansion of sequencing due to simultaneous object search (for unseen objects) and rearrangement, (iv) minimizing the overall traversal during simultaneous object search and rearrangement, (v) blocked goal and swap cases without explicit buffer.
In this paper, we propose novel modular method for task planner to address the aforementioned challenges. At the beginning, our agent captures the goal state by exploring the room to record the semantic and the geometric configuration Batra et al. (2020) of objects and receptacles through egocentric perception. Once the goal state is captured, the objects in the room are shuffled. In the untidy current state, method partitions the task planning problem into two parts; object search and planning, with the aim of minimizing the overall agent traversal during simultaneous object search and rearrangement. First, we propose novel commonsense knowledge based Search Network using large language models (LLMs) et al. (2019) 9 Kant et al. (2022) that leverages the object-receptacle semantics to predict the most probable receptacle for an unseen object in the egoview. Second, we use Deep RL network with hybrid action space Ghosh et al. (222)) to plan our action sequence for simultaneous object search and rearrangement by resolving blocked goal and the swap cases. To this extent, we define the Deep RL state space with a novel graph-based state representation for the current and the goal state that incorporates geometric information about objects. This representation compactly encodes the scene geometry that aids in rearrangement planning and makes the Deep RL state space scalable to a large number of objects and scene invariant. In addition, we present a novel, sample-efficient cluster-biased sampling for simultancous training of the proxy reward network Ren et al. (2022) and Deep RL to get better estimate of the problem's 's true objective from the episodic reward than the dense reward in Ghosh et al. 2022). The judicious combination of all the
2

---

Published as a conference paper at ICLR 2024
Graph based Stuta Representation
based
R
Z
Exploration, Napping and object detection
Z
Recepticc
Goal State
GEEN
D-DETR Delottor
Plzee Object
or
0º w
ra(s, a,
P
8,0,
Current (Egocentric Image)
Search Network
Position Fromit
Mast Probullie Class
W
SRIM
SCN
updatte (Of.). W
RoBerta
W
Dass
X
0
busbbbeee Class
Figure 2: Overall pipeline of our proposed method.
aforementioned components effectively tackle the challenging combinatorial optimization problem in rearrangement as detailed in Sec. 3.6
The major contributions of this paper are
To the best of our knowledge, this is the first end-to-end method to address the task planning problem for nommrearrangemeet from an egocentric view under partial observability, using user-defined goal state.
2. A novel Search Network that leverages object-receptacle semanties using the commonsense knowledge from LLMs to predict the most probable receptacle for an unseen object.
3. Use of Deep RL based planner trained with proxy reward overcome combinatorial expansion in rearrangement sequencing and, to optimize the overall traversal and the number of steps taken. new Graph-based state representation for the current and goal state include geometric information about objects, making the Deep RL. state space scalable to large numbers of objects and inveriiiv.ri.nt.
5. Introduction of novel, sample-efficient cluster-biased sampling for simultaneous training of the proxy reward network and the Deep RL network.
6. We introduce new set of metrics in Sec.3344Joo obtain thorough assessment of the rearrangement planner's 's effectiveness by not only evaluating the success of the rearrangement, but also taking into account the number of steps taken and the overall agent traversal.
/eiiarksssehhsseeeee To address the inadequacies in existing benchmarks Weihs et al. (2021) for evaluating task planning under partial observability, release the dataset to enable further research in this domain.
introduce the RoPOR Benchmark Dataset. We plan openly
METHODOLOGY
In our noom-rcarrangement setup, the agent explores the room to capture the tidy user-specified goal state. During this exploration, the agent creates a 2D occupancy map 2d for the agent's navigation while, 3D map M is utilized to augment
detection of 3D object and receptacle centroids to fixed global reference frame (R3). Additionally, we generate an object list O { Pt). 1, 2., N) and receptacle list R {[W p 1, 2, .., NR). Here, N, W and P E R23 are the total numbers of objects, their semantic labels, and 3D object centroids, respectively. While Nu. H and R E R are the total numbers of receptacles, their semantic labels including name from Ai2Thor Kolve et al. (017)). and the 3D receptacle centroids respectively. Then, randomly shuffle a few objects from the goal state to make the room untidy and fork the agent at random location in the room. In this untidy current state, the agent's knowledge is limited the visible part the room in its egocentric view. In the agent's 's egocentric perception, only set of objects O = {[W P ] = 1,2, Nv are visible. Ny, w and p' E R3 are the number of visible objects, their semantic labels, and their 3D object centroids respectively in the current state. Comparing O in the goal state with O V in the current state allows for determining only the semantics
room
3

---

Published as a conference paper at ICLR 2024
of unseen objects O (W 1,2, Np), where Np is the number of unseen objects and w their semantic labels. To plan efficiently and achieve the goal state, the agent must know the positions of all objects in the current state. This involves optimizing the search for unseen objects based on the object-receptale semantics and simultancously rearranging visible objects based on their positions in the current and goal state. To this end, we present a modular approach for task planner, as shown in Fig. 2 with : (i) Search network, (ii) Graph-based state representation, (iii) Deep RL network trained with praxy reward. The objective of our task planner is to minimize the number of steps and the agent's 's overall traversal by simultaneously sequencing high-level actions to either pick-place misplaced objects or search for unseen objects predicted receptacles.
2.1
BACKGROUND
The agent maps the room in the goal state using an exploration strategy Sarch al. (2022) and receives RGB-D images and egomotion information at each step from Ai2Thor Kolve et al. (2017).
The agent constructs 2 and MDD of the environment using the RGB-D input and egomotion. A
d-DETR Zhu et al. T202T) detector used on the RGB images obtain 2D bounding boxes and semantic labels for objects and receptacles, and the corresponding 3D centroids are obtained using depth input, camera intrinsic and extrinsic. Finally, the agent has O, R. M and M.D 3D from the goal state. In the current state, the agent uses d-DETR detector Zhu et al. (2021) along with M to obtain Or
2D
The agent uses the Djikstra path planner on M to navigate and execute high-level actions by assuming perfect motion and manipulation capabilities.
2.2 SEARCH NETWORK
We present novel LLM- L-Ma-ddddd Search Network to reliably predict the receptacles for O In case the predicted receptacle articulated, the agent opens it and looks for the object. The agent uses the predicted receptacle's 's position from the goal state to be the probable location for O in the current state, since receptacles are static in the room. To this end, we finetune the RoBERTa embeddings
V
to exploit the commonsense knowledge in LLM and learn the semantic relationship between 0 % and R. Fine-tuning LLM embeddings is essential because L.Ms,, being trained on large data corpus, may not necessarily produce human-commonsensse compliant predictions for untidy scenes (see the Appendix for more details). Our Search Network (SN) consists of two parts: the Sorting Network (SRTN) and the Scoring Network (SCN). We use RoBERTa- Large model Liu et (2019)
to generate pairwise embeddings (E.R) R for (W\) Ji=1,2...,N and Ji=.,,,...... in the current state. Therefore, there are NE = Np x Nn number of embeddings for all the ahject-room-recetaadee (ORR) pairs. Each ORR embedding is classified into of the 3 classes, based on the probability {px 1=,,23 from the Sorting Network. The ground truth class labels {=1.3 for each ORR in the dataset (Sec. 3.I is based the probability to find object at that room-receptacle, where {i = 1: Most Probable Class, 2 : Less Probable Class 3 Implausible Class). SRTN filters out the room-receptacles, where there is negligible chance of finding the misplaced object. For instance, even in an untidy room, it is nearly impossible to find a cup in the bathtub of a bathroom. This sorting step reduces the scoring network's computation and minimizes the chances of erroneous scoring of an implausible ORR. We train fully connected MLP in SRTN using the Cross-Entropy Loss (LCE) as shown Eq. (.. The Scoring Network estimates probability scores {x of higher probability classes, with Nsa representing the total number of such embeddings. SCN provides probability score metric, choose the most probable receptacle for O%, For training the fully connected MLP in SCN, we calculate the MSE Loss (LMsB) of probability scores, as in Eq. (2. with respect to the ground truth probability scores {xi)==1,..Na. Finally, we get the position ((P) (==),.. of the unseen objects the position of their most probable receptacle.
for embeddings
NE
1
LCE
Y, log
NE
NyR
1
LMSE
(x+ x
(2)
NSR
To prevent fruitless searches, implement simple strategies. If the agent cannot find the unseen object at the predicted receptacle, the Search Network identifies the next most probable room- receptacle, and the prior prediction is discarded before re-planning new sequence. Additionally, if
4

---

Published as a conference paper at ICLR 2024
the agent encounters receptacle its path that does not contain any unseen objects, it is removed from future searches. The agent updates O% whenever it detects an unseen object in its egoview. If the agent locates the unseen object it is searching for before arriving at the predicted receptacle, updates O and re-plans new sequence. Refer appendix for more details on the re-planning strategy.
2.3 GRAPH-BASED STATE REPRESENTATION
For our task planning algorithm, we create spatial graph (G (V, E}) representation of the current and the goal state namely G. {Ve, Eej and G3 (Vg: F.g) respectively. The nodes V. {OK) and V = {O}. The fully connected edges of the graph contain the path length as edge features, where E. = (D(P." .P))) and E, = (D(P,, P2) F)),, The path length D D(A,, Az)iz is the length of the shortest collision free path, computed using Djikstra, between the 2D projections of
A,, Aj E R on MDD For unseen objects in the current state, the object nodes and edges in G. are augmented with pVR
R from the search network as V. VoutOr, pv R) and E. = (D(P,, Py } where P = pV U ppR This graph representation helps the Deep RL state space to understand the semantic and geometric information of the current and the goal state. We use a novel Graph Representation Network (GRN) with an encoder-decoder to generate meaningful embeddings from Ge and Gg for Deep RL state space to incorporate the residual relative path length notion between every pair of current and goal state nodes. GRN consists of two major blocks, the Graph Siamese Encoder Network (GSEN) and the Residual Geodesic Distance Network (RGDN). GSEN uses a Graph Convolution Network (Gao et al.) 2020) to encode the graphs G. and Gg and produce the graph embeddings Ze and Zg respectively. These graph embeddings are concatenated to get the final embeddings Z, = Zo U Zn RGDN acts as a decoder and predicts the residual relative path length Tр between the two graphs. This network is trained in supervised way as in Eq. (3) using the Graph Dataset (Sec. (.T). which contains the ground truth relative path length (r) between the two graphs. This graph embedding makes the Deep RL state space invariant to large number of objects and the scene. This compact representation concisely encodes the pairwise distance between the source and target nodes which aids in the reduction of the combinatorial expansion of rearrangement sequencing.
Tp GR. Ggy
(3)
LGRN = ||| - Tppl
2.4 DEEP RL BASED PLANNER
Our task planner needs to select the objects or the probable receptacles for the unseen objects in an efficient manner, to minimize the overall traversal of the agent to simultancously search the unseen objects and rearrange the visible ones. Moreover, the planner needs to identify free locations, when selecting objects with swap cases.
2.4.1 PARAMETERIZED DEEP- NETWORK
In order to achieve the aforementioned goals, we implement a Parameterized Deep-Q Network with hybrid action space, similar to Ghosh et al. (2022). We define a binary Collision vector (CNx 1). that signifies the objects with a blocked goal or swap case. The Deep RL state space defined as Zp UC. Each action {a. (k. p in our sequence of actions {a:):=1......f of length K is made up of discrete action k. denoting the index of the selected object or the probable receptacle, followed by a continuous parameter Pk: which signifies the location for object placement or receptacle search. We use a Parameter network (b)) and the Q-network (QQ) to generate a continuous parameter Pk and a discrete action k respectively, similar to Ghosh et al.. According to a Markov Decision Process (MDP), our method receives a reward r(s, a) at each time step t, for choosing an action a, that advances the agent from the current state to the next state 5. Inspired by the work in Ghosh et al. (202)); Bester et al. (2019), we define the Q-values as a function of the joint continuous action parameter p- [pkjk= 1,2,,, K instead of updating the Q-values with its corresponding continuous parameter sample ps. The modified Bellman equation is shown in Eq. (4). This prevents our method from producing degenerate solutions by incorporating the effect of other parameters for updating the Q-values.
(s Eir + mar Q(5, (8,
(4)
KEK
5

---

Published as a conference paper at ICLR 2024
The loss function Lp(pp) and LQ( for the parameter network((p)) and the Q network @+)). is given by Eq. (5)
Rg K
Lp(pp) =
Σ
Qs, k. (P()):
(5)
12(00 ks $2))
(2( (e.k.p.,,))-aa
E
Here, y = r + +maa (8, k p(;; ;; p) Φ2) is the updated target from Eq. (4) and R is the replay
KEK
buffer. Lp( indicates how the p must be updated to increase the Q-values. Here &Q works as critic to p.
For Long Horizon planning, the sparse reward is not sampling efficient for training the Deep RL Gehring et al. (202T). Hence, we use step-wise environmental feedback based on the hierarchical dense reward similar to Ghosh er al.. The detailed reward structure is explained in the Appendix. This reward structure provides per-step feedback, but we need episodic reward-basee feedback to improve RL policy generalization Amodei al. (2016); Dewey (0014). Thus, for every episode (A), we calculate the episodic reward (Rep) using the step-wise hierarchical dense reward (r) and overall episodic path length (L) in Eq. (6). and save the reward and each step (s, a, 3) of the episode into the replay buffer (R As this episodic reward is sparse, we use a proxy reward network to generate per-step dense Markovian reward with an episodic notion.
2.4.2 PROXY REWARD NETWORK
Our proxy reward network trained on the sampled experience data from the replay buffer, give our agent a notion of the overall objective of the episode. The random return de- composition (RRD) method used in Ren et al. (202)), trains a proxy reward network by ran- domly sampling steps from episode. This training method is sample efficient be- cause it uniformly samples the steps without considering the reward distribution in the episode. To this end, we propose novel cluster- biased return reward decomposition (CB-
RD) to train our proxy reward network. We cluster the per-step reward for the episode into 3 clusters each of size Tj. where j {1, 3), using the c-means clustering. These clusters represent the reward distribu- tion in episode. This information helps us to efficiently sample N₂ number steps from the episode. We randomly sample U, {(sig Ubjy )
Sparse Reward Reward
RRD
CB-RD
Average Return
N3 from each clus- ter j. such that N2 = =N, T, Us- ing {U,}==1,2,
-20
we estimate the learned episodic reward (Rep.0) from the proxy ward network (ro(s, a, 8)), where 0 is the learned weight.
20000
40000
60000
S0000
100000
Number Of Steps
Nap
Figure The study compares various reward methods (6) used to train our Deep RL network and finds that our proposed method, CB-RD, achieves the highest average return in shorter training time, hence improving sam- pling efficiency.
Nep
Rep =
r
L
i=1
N,
T
Rep.0 0=
Pj
ro ro( aijs aij.j si.)
N
1=1
(7)
LCBRD 1 M
(Rep. Rep.
(8)
Here, M is the number of episodes sampled, Nep is the number of steps in episode and p5 Ty/Nep is the uniform probability of choosing sample from the episode that belongs to cluster j. We simultaneously train Deep RL using Eq. (5) and proxy reward network using Eq. (8) as shown in Algorithm]] T Fig. 3 shows that CB-RD provides effective feedback to our Deep RL method to achieve higher average return lesser number of steps during training. Hence, CB-RD makes

---

Published as a conference paper at ICLR 2024
our Deep RL method more sample efficient compared to RRD, hierarchical dense reward and sparse reward.
We an off-policy method with replay buffer to train Deep RL method with diverse set of rearrangement configurations, similar to the work proposed by Kalashnikov et al. (2018). We use the c ggredy method Kalashnikov et al. (2188) to strike balance between exploration and exploitation. We stabilize our Deep RL. training using target networks for & and p,. and update the weights of target networks using polyak Lillicrap al. (2015) averaging similar to [Bester et al. (2019); Ghosh et al. (2022). Our ablation study in Appendix, shows that the selection of has a significant impact on the solution.
EXPERIMENTS
In this section, we describe the datasets, metrics, and detailed re- sults of our proposed method and its modules, in addressing the room- rearrangement problem.
Algorithm 1: Training Proxy Reward Network
Initialize Rg + ]
for 0, 1, do
3 Using 6-greedy we rollout episode A;; Calculate RB + RB U (A,, Rep.
reward Rep, using Eq. (6)
3.1 DATASET
Graph Dataset We generate this
5
for 1 + 0, 2, do.
Sample M episodes {A₃ RB RBiis: Group steps from each episode into
dataset to train GRN using Ai2Thor
Kolve et al. (2017). by randomly plac-
7
jE [1, 2, 3) clusters based on r;
ing objects for two types of rearrange- 8
Cluster Biased Sampling to get Uí from AG;
ment scenarios: (i) 40% without goal
Compute Rep.e using Eq. (7)
occupied rearrangement: by placing the objects in free spaces and (ii) goal occupied rearrangement: by placing
Calculate LCBRD as in E'g. (8):
0 aVoLCBRD with learning rate o;
the object in another object's target.
Optimise the policy & using (s, a) and Eq.
Search Network Dataset The
AMT dataset in Kant et al. (2022) contains 268 object categories in 12 different rooms and 32 receptacle types. Each object-room-rreeptaad (ORR) pair is ranked by 10 annotators in 3 classes: correct (positively ranked), misplaced (negatively ranked), and implausible (not ranked). For our problem statement, the misplaced class is utmost importance. Hence, we rename the classes as (i) misplaced class most probable class, (ii) correct class less probable class, and (iii) implausible class remains the same. We find the ground truth score values for each ORR as the mean inverse of the ranks.
3.2 BENCHMARK DATASET FOR TESTING
The existing benchmark dataset, RoomR Weihs et al. (2021). has limitations only allows up objects, no object placement within another receptacle, and no blocked goal or swap cases. Thus, cannot fully evaluate planning aspects such the number of steps taken, agent traversal, blocked goal, or swap cases. To address this, we introduce RoPOR, a new benchmark dataset for testing task planners in Ai2Thor. It includes diverse range of rooms (120) and object-reeeptacle pairs (118), allowing for wide variety of rearrangement scenarios with up to 20 objects and random partial observability cases, object placement within receptacles in the current state, and blocked goal and swap cases. Moreover, object placement configurations in RoPOR affect sub-optimal planning policies in terms of agent traversal. The mean room dimensions along x-axis and y-axis are 3. mm and 5.80m, respectively. Refer Appendix for details on the distribution of objects, rooms and receptacles.
3.3 TRAINING
The training details of our Search network, Graph- based state Representation Network, Deep RL planner, and proxy reward network are available the Appendix.
3.4
METRICS
Metrics in Weihs et al. 2021 do not highlight the efficacy of a task planner to judge efficient sequencing reduce the number of steps taken the agent traversal during rearrangement. For fair evaluation of our method, and comparison against the existing methods and ablations, we define new metrics:
SNS Success measured by the inverse Number Steps uses a binary success rate (S) to evaluate the successful completion of rearrangement episode along with the number of steps (Nr) taken by
7

---

Published as a conference paper at ICLR 2024
Visitee offes Obrezs
Namter
Unacr Offechs
Swup Cusc
Oun-GT
Ours
Wiliotaal
Clure of
DOF
OPR
ATChui SNS1
ENRT
SNST
ENRT
ATCCa))
ENR:
ATC)m)
ATCina SNST
ENR
ATCina
0.74
NC
18.13
NC
20.15
0.08%
1:52
NC
13.5%
0.53
NC
NC
NC
NC
D66
NC
16.73
0.60
0.45
0.000
0.3T
19.45
0.005
0.99
20.39
0.066
NC
NC
13.99
0.58
0.47
14,9
0
NC
NC
NC
NC
NC
2211V
11.5
1000
NC
3035
0008
NC
16.59
150
NC
2738
0.70
34,3
0.52
NC
21:33
0
NC
NC
NC
NC
MC
NC
065
MC
50.65
0.86
8.69
23.78
0.64
0:53
25.56
0.00
0,20
1623
0.506
0.12
17.01
0.001
0.25
15.58
NC
NC
0683
8,67
24.15
0.62
0.52
25.97
NC
NC
NC
NC
NC
NC
0
NC
NC
1L55
18.3
44315
NC
NC.
NC.
NC.
NC.
MC
103
NC
W179
20
20
11770
NC
45.32
0.52
NC
45.32
NC
NC
NC
NC
NC
MC
062
NC
$2.56
1157
0.75
41.29
0.67
0.58
45.24
NC
NC
NC
NC
NC
MC
NC
MC
0.57
0.74
42.13
0.66
0.57
45.78
NC
NC
NC
NC
NC
NC
NC
Table 1: (OOF: Objects outside agent's field of view initially, which are visible from a different perspective, OPR Objects placed inside closed receptacles. NC: ; Not computable). When there are no unseen objects. the ENR is NC. Similarly, when SNS is zero, ENR and ATC are NC. Weihs al., Gadre al., and Sarch et al. do handle 20 objects and cannot resolve swap cases without explicit buffer OPR cases (SNS = 0). Ghosh et al. shows a slight decline performance as the number of objects increase under complete visibility and swap cases, but fails to account for unseen objects. In comparison, Ours significantly outperforms Weihs et al., Gadre et al. and Sarch et al. in terms of SNS, ENR, and ATC for visible objects, unseen objects, and swap cases without explicit buffer. Similarly, ours-GT performs better than Ghosh et al. in terms of SNS and ATC under complete visibility and swap cases without explicit buffer.
an agent to rearrange a given number of objects (N). S is 1 if all object positions in the current and goal state are approximately equal. Higher the SNS implies lower Ny for given N indicating more efficient and successful rearrangement episode. (SNS = S x N/NT)
ENR: Efficiency in Number of Re-plans during object search by taking the ratio of the number of
unseen objects initially (Np) with respect to the number of attempts to search (N st)) ). A higher
ENR shows a lower N Ngv for a given No indicating a more efficient search to find unseen objects.
(ENR = NN (////)s)
Absolute Traversal Cust(ATC): The metric shows the overall distance traversed by the agent during the successful completion of rearrangement episode. In identical test configuration, lower ATC indicates a more efficient rearrangement sequencing.
3.5 ABLATION
We ablate our task planner against ground-truth perception, various methods for object search and dense reward structure. To study the effect of erroncous perception on our task planner, we assume the availability of Ground-Truth object detection labelling and 3D centroid localisation from Ai2Thor (Ours-G7) G7). To understand the importance of our Search Network in planning, we replace it by a (i) Random Search policy (Ours-RS), which predicts probable receptacles for unseen objects with uniform probability and a (ii) Greedy Exploration strategy (Ours-GE) Chaplot et al. (2020) that optimizes for map coverage to discover all the unseen objects. To highlight the generalisation of proxy reward network to the overall objective of the rearrangement episode, we replace it with hierarchical Dense Reward structure Ghosh et al. (2022) (Ours-DR) .Please refer to the appendix to find the results for the ablations, along with the analysis for the choice of hyper-pun parameters for each of our learning based modules.
3.6 QUANTITATIVE RESULTS
We evaluate our approach along with the existing methods on RoPOR Benchmark Datset in Ai2Thor. Tab. [ indicates that our method is scalable to large number of objects, as demonstrated by the consistent value of SNS despite the increasing number of objects across complete visibility, partial observability, and swap cases without an explicit buffer. The gradual increase in ENR with the increase in number of objects can be attributed to the fact that rearrangement of visible objects and the search for some unseen objects, indirectly aids in finding other unseen objects.
Comparing our method against Housekeep Kant et al. (022) would be unfair because does not perform a user-specific room room-rearrangee rearrangement with a pre-delined goal state. Instead, we have compared our method to previous works such as Weihs es al. |Weihs et al. (2021). Gadre et al. [adre et al. (2022), Sarch et al. [Sarch et al. (2022) and Ghosh et al. [Ghosh et al. (2022). all of which have demonstrated results for user-specific noom-rearrangementt For a fair comparison with Weihs et al., we have used their best performing model- RNI8+ANNI PPO+IL Weihs al. (202T). Since, Ghosh et al., uses groundtruth object positions in the current and the goal state, we compare with our ablation method Ours-GT. Without erroneous perception. Ours-GT demonstrates efficient planning, by performing significantly better than all the existing methods Weihs et al. (02T)); Gadre et al. (2022); (aarh et al. (2022): Ghosh al. (2022), including Ours, in terms of SNR, ENR and ATC.
8

---

Published as a conference paper at ICLR 2024
Under complete visibility, ours significantly outperforms Weihs et al., Gadre et al. and Sarch et al. in terms of SNS and ATC. Similarly, Ours-GT significantly outperforms Ghosh et al. in terms of ATC. The improvement over Weihs et al., Gadre et al. and Sarch et al. shows their heuristic planner is neither scalable nor does it optimize the overall agent traversal or the number of rearrangement steps. In contrast, our method leverages compact graph-based scene geometry capable of addressing large numbers of objects, and robust Deep RL makes our planner efficient in reducing the redundant traversal of the agent. Our method uses path length cost and proxy reward with the episodic notion, which helps to improve the overall traversal of the agent to produce lower ATC. In comparison, Ghosh et al. uses greedy Euclidean distance based reward without having an episodic notion, thus failing to optimize overall traversal. Moreover, Ghosh el al. shows drop in performance on the RoPOR dataset as compared to their results evaluated on RoomR Weihs et al. (202T), due to the variations in the testing scenarios in RoPOR that significantly impact agent traversal for sub-optimal rearrangement policies.
Under partial observability, there are two cases (i) OOF: Objects located outside the field of view initially which are visible from a different perspective and (ii) OPR: Objects placed inside closed receptacles. In the case of OOF, our method substantially outperforms Weihs et al., Gadre al. and Sarch et al. in terms of SNS, ENR and ATC. All these above methods use greedy sub-optimal planners and employ explicit scene exploration find objects outside the field of view, incurring huge traversal cost as indicated by their ATC. To gauge the performance of the exploration strategy for object search in terms of ENR, we consider each newly generated location or a set of navigational steps from the exploration policy as a search attempt. Our approach's 's significantly higher ENR shows that the Search Network outperforms the exploration policies of Weihs et al. (202T): Gadre et al. (2022); |arch et al. (2022) in terms of the number of attempts to find unseen objects. Ghosh et al. does not address any case of partial observability. While Weihs er al., Gadre et al. and Sarch el al. do not solve the case of OPR, which involves object placement inside receptacles (SNS = 0). However, our approach performs equally well in both cases of partial observability due to our search network's 's ability to comprehend commonsense based semantic relationship between an object and any type of receptacle rigid or articulated.
Swap cases without explicit buffer are not handled by Weihs et al., Gadre et al. and Sarch et al., which is evident from SNS = 0. Ours, Ours-GT and Ghosh et al. can effectively resolve an increasing number of swap cases without explicit buffer using the hybrid action space Ghosh et al. (2022) in the Deep RL network. However, Ours-GT performs better than Ghosh et al. in terms of ATC due to novel collsion resolution reward that optimizes the agent's traversal.
To ground the values of our RoPOR dataset, we show the results for Ours, the ablation methods and the SOTA in the test of RoomR in the Appendix. Moreover, additional results for individual methods in our pipeline can be found in the Appendix.
3.7 QUALITATIVE RESULTS
To show the results of our method in wom-rearrangemmntt we have created videos in a number of test scenarios to highlight the robustness of our method. We also test our method in new environment Habitat, as demonstrated in our supplementary video. This transfer does not require any additional training for our Search Network, Graph-basee based State Representation Deep RL planner. This shows the capability of our method for seamless sim-to-sim transfer, further emphasizing its suitability for real-world deployment. Please refer the supplementary video.
4 LIMITATIONS
Our approach is not capable of identifying unseen objects that are occluded due to clutter on receptacles (for c.g. spoon may become occluded, if bread, box, lettuce etc. is placed before it). Our method also assumes the availability of perfect motion planning and manipulation capabilities.
CONCLUSION
This paper presents an innovative task planner designed for organizing rooms under conditions of partial observability. Our approach minimizes agent traversal and step count during both object search and rearrangement by leveraging Search Network followed by Deep RL-based planner. By utilizing graph-based state representation and episodic proxy reward, our method exhibits versatility and applicability across range of scenarios. The RoPOR benchmark dataset facilitates additional research in the realm of Embodied AI- l-based rearrangement. Future endeavors will concentrate on deploying our approach in real-worle settings.

---

Published as a conference paper at ICLR 2024
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016. httpppss//aa.w...or/////a////e/e /00ob///166..... 06565
Dhruv Batra, Angel X Chang, Sonia Chernova, Andrew Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement A challenge for embodied ai. arXiv preprint arXiv:201 1.01975, 2020.
Craig J Bester, Steven D James, and George D Konidaris. Multi-pass q-networks for deep reinforce- ment learning with parameterised action spaces. arXiv preprint arXiv:
vrXXvv/905.044 1995.0438, 2019.
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdi-- nov. Learning to explore using active neural slam. In International Conference Learning Representations, 2020. URL /ettps::/openreviiiie
Dan Dewey. Reinforcement learning and the reward engineering principle. In AAAI Spring Symposia, www.eettttnnnoaa..eeeeeeeaa wrg/CorpusII 51991163
Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song. and Roozbeh Mottaghi. Continuous scene representations for embodied ai. 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 14829-14839, 2022. URL https: //api. wwwantentitt.m icschol ar
Stg/CoozpusID:24783300
Xiang Gao, Wei Hu, and Guo-Jun Qi. Graphter: Unsupervised learning of graph transformation equivariant representations via auto-encoding node-wise transformations. In 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7161-7170, 2020. doi: 10.1 (0.1199/CVP44260.222..0777
Clement Gehring. Masataro Asai, Rohan Chitnis, Tom Silver, Leslie Pack Kaelbling, Shirin Sohrabi, and Michael Katz. Reinforcement learning for classical planning: Viewing heuristics as dense reward generators. CoRR, 0oos22199.14800 2021. URL httph:t///// 14830
Sourav Ghosh, Dipanjan Das, Abhishek Chakraborty, Marichi Agarwal, and Brojeshwar Bhowmick. Planning large-scale object rearrangement using deep reinforcement learning. In 2022 International Joint Conference on Neural Networks (IJCNN) pp. 1-8, 2022. doi: 10. 1,.1000CN55550002222222 9889793.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Iharz, Alexander Herzog, Fric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 651-673. PMLR, 29-31 Oct 2018. URL kalashnikov Ba. m1ml
Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, An- drew Szot, and Harsh Agrawal. Housekeep: Tidying virtual houscholds using commonsense reasoning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 355-373, 2022. doi:
21. Utt/httss:///i..rg///2
3 031-19842-77 2_
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, 509.02999 2015. ht.ps:tt::://// stw.t//t.p.....n.//.m///mm www.nti i cscho arr wtw.olaarrorrr//oouaaseeee 16326763
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 16692, 2019. /tt.
10

---

Published as a conference paper at ICLR 2024
Zhizhou Ren, Ruihan Guo, Yuan Zhou, and Jian Peng. Learning long-term reward redistribution via randomized return decomposition. In International Conference on Learning Representations, 2022.
Gabriel Sarch, Zhaoyuan Fang, Adam W. Harley, Paul Schydlo, Michael Tarr, Saurabh Gupta, and Katerina Fragkiadaki. Tidee: Tidying novel rooms using visuo-semantic commonsense priors. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 480 0900496, 2022.
Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 5922-5931, 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL http://pp.e/ogeeeevee.eeeeeeeeee http://wwww...etooooooeo.eeaaeee
A APPENDIX
This supplementary document first shows our results on RoomR Weihs et al. (2021) dataset. Subse- quently, we discuss our RoPOR dataset statistics and configurations. Detailed information about our reward structure and rearrangement algorithm. Additionally, we provide our network details and training schedule. Finally, we show our ablation study and discuss our failure scenarios.
B RESULTS ON ROOMR DATASET (WEIHS ET AL. (2011)
Tab. shows the results for Ours, the ablation methods and the SOTA in the test set of RoomR. The similarity in the results of our ablation methods in RoomR reflects the redundancy the dataset and its metrics to gauge the planning efficacy. Both Ghosh et al. (2022) and Ours-GT use ground-truth perception, hence comparing them with other methods would be unfair. Ours outperforms all the existing methods in RoomR, showing the generalizability and robustness of our method.
C RoPOR BENCHMARK DATASET
The existing datasets and benehmarks for room rearrangement, such RoomR Weihs al, (202T). contain most five objects in a scene. Further, this dataset and benchmark does not explicitly mention the inclusion of different configurations required to test the efficacy rearrangement sequencing. To this end, propose benchmark dataset RoPOR, which contains various configurations for swap and object placements. Moreover, to cater to the needs of object rearrangement under partial observability, we take inspiration from Housekeep Kant al. (2022) define the distribution of objects, rooms, and receptacles based on the ground-truth human annotations from AMT Kant et al. (2022).
C.1 DISTRIBUTION OF OBJECT- SHRECT-ROOM-REEE CO0OM-RECEPPTCLEES
To create realistic untidy scenarios in the RoPOR dataset, object placement on the room-receptacles is not entirely random but follows certain semantic prior to reflect the underlying human preferences for untidiness. This semantic prior is induced using approximately 34,000 annotations, where each annotation comprises of rankings from 10 individuals, for each pair of 128 room- receppecess and 269 objects the AMT human preference dataset (Kant et al., 2022). A plausible room- receptccee for given object is randomly selected based on the annotations. In each room of the RoPOR dataset, we ensure to keep a higher number of objects with a misplaced class of room-receptacles than with a correct or implausible class of room-receptacles. In Fig. 4 we show the average percentage of objects placed in each room-recs receppacee class for any given room with a fixed number receptacles in the RoPOR dataset. As the room is untidy, there is a greater likelihood of finding objects in misplaced
Owr
Owr-R
Owr-GE
OwrsDR
Wells ctal.
30211
Gadre ct
0023
Sarch
0022
Ows--G7
Glosh et al
202
Success Rate
0,43
0.42
0.42
0.40
0024
0.9
Fited Serce
0.519
048
0.46
0.41
0.04
2A9
0.116
0.9
Energy Remaining
0.3
066
068
0.71
1.10
1.77
0331
0023
Table 2: Result of our method, the ablations and the existing methods on the Test set of RoomR and its metries.
11

---

Published as a conference paper at ICLR 2024
misplaced
60
correct implausible
Percentage of objects
EHEL (10,15)
40
30
20
(0,5)
(5,10)
(15,20)
(20,25)
Number of room-receptacles
Figure 4: Distribution of objects in room of RoPOR dataset for each class of room-receptacle. i.e. misplaced, correct, and implausible as AMT dataset.
room-receptacles compared finding them in their correct nom-rrceptacles. Additionally, we place small percentage of objects in the implausible class of room-receptacles to simulate complete disorderliness.
Moreover, it is logical to have things in more disordered configurations for an untidy room, hence higher representation of object-room receptacles from the misplaced and implausible class. The graph in Fig. shows the distribution of the high-level object categories in the misplaced and the correct class of room-receptacles for the RoPOR dataset. The high-level categories in the dataset are shown in Tab. g It is evident from Fig. that the number of misplaced room-rece nem-reccetacle is greater than the number of correct room-receptaclee for most of the high-level object categories in the RoPOR dataset. As have borrowed the ground truth classes from AMT, results show similar distribution AMT Housekeep.
C.2 OBJECT CONFIGURATIONS
C.2.1 INCLUSION SUFFICIENT SWAP CASES
In every testing scenario, we have two types of swap configuration (i) With distance of more than Im, which minimising the traversal distance is more important than minimising the number of pick-place actions and (ii) With distance of less than Im, in which minimising the traversal distance insignificant compared to minimising the number pick-place actions. This testing scenario is depicted in Fig. 9
C.2.2 IMPORTANCE OF PATII LENGTHI DISTANCE
To emphasize the importance of path length distance during rearrangement, ensure to have sufficient representation object configurations each room so that the objects are closer in terms of euclidean distance than the path length distance and vice versa. An example of this configuration can be seen in Fig. 77, where lettuce (violet) is farther from the goal position (blue) of tomato (red) than the kettle (pink) in terms of path length distance due the table. Also, this ensures that the lettuce is closer than the kettle to the goal position of the tomato in terms of Euclidean distance.
C.2.3 IMPORTANCE OF NON-GREEDY POLICY
In order to account for greedy policies during rearrangement, we have included scenarios in the RoPOR dataset which lead to poor performance in terms of agent traversal for policies that choose only the nearest object. For instance, see Fig. 6 where the policy which chooses the immediate nearest object after every move leads to longer traversal path for the agent. After similar step 1, the greedy policy chooses to pick-place lettuce (violet) which closest in terms of the Euclidean distance. In contrast, optimal policy chooses the kettle (pink), which farther in terms of Fuclidean distance.
12

---

Published as a conference paper at ICLR 2024
70
misplaced correct
Percentage of room-reed rene-taaes
60
50
40
30
20
10
cankting accensores
kitchen Accessories
cuttery
crockery
edibee
sports accerseries
statbonery
rooo moon-dacor
bathroon accesiories
uniiiies
consumet Boms
High-level object categories
Figure 5: Distribution of room-receptacles for the high-level object categories (Tab. 8 in the RoPOR dataset. This graph shows the percentage room-receptacles of the misplaced and correct class (as KMT et al. (2022)) in RoPOR.
MARMARE SI LITIT M in HI-0
US MARMARRRRtnnnn M
Path length notion based optimal policy
Euclidean notion based greedy policy
Figure 6: A sample run of a Fuclidean distance-hased greedy policy and Path length distance-based optimal policy on the configuration shown in Fig. 7 Both policies traverse the same step (shown in sky blue) pick and place the red object. For step 2, the greedy policy based euclidean distance chooses the violet object as it is closer (in terms of Euclidean distance). However, the path length settance-based optimal policy chooses the pink object (as it is closer in terms of the path length distance and more optimal considering the overall rearrangement scenario). Hence, the path traversed in step (shown in violet for greedy euclidean and pink for optimal-path length) is much greater for Euclidean-grees -geeeyygeeyyy policy than the optimal-pathe length one. Moreover, due to repeated greedy actions without the notion of an overall rearrangement scenario, the greedy policy traverses much longer path than the optimal one.
The optimal policy-based based method chooses step taking into account the whole rearrangement episode, which leads to a shorter traversal path overall. After taking greedy step 2. the greedy policy leads to much longer path traversed.
13

---

Published as a conference paper at ICLR 2024
Finally, have shown sample configuration for 5 and 10 objects, including all the scenarios mentioned above Fig. g
Source configuration
Goal configuration
Figure 7: An example configuration of a source and goal with three objects in RoPOR dataset signifying the importance of path length distance over euclidean distance. The source location for the object in the violet box is closer to the goal position (blue box) for the object in the red box in terms of the Fuclidean distance. In contrast, the object in the pink box closer to the blue box terms of the path length distance.
D REWARD STRUCTURE
use a dense reward structure to compute the per-step reward and store it the replay buffer. The computed rewards train our proxy network, and the step-wise proxy reward from the proxy reward network is used to train our Deep RL method. The proxy reward imparts episodic awareness and improves the sampling efficiency during our Deep RL training. Inspired by the work in Ghosh et al. (2022), we use a similar reward structure so that our Deep RL method minimizes the number of moves and reduces the overall traversal of the agent. We take special care for swap instances strike a balance between (i) optimizing the number of moves and (ii) minimizing the overall traversal, unlike Ghosh el al., which focuses only on (i). Further, describe in detail the dense reward structure in Fig. TO which enables our RL to produce an effective plan to move the correct objects and efficiently handle the blocked goals, specifically swap instances.
Infeasible action reward R1: The objective of this reward is to restrain the agent from producing
infeasible actions that can not be realized in environment. Fig. 10 R1 block shows this reward structure.
This reward tries to strike balance between (i) optimizing the number of moves and (ii) reducing the overall traversal of the agent. Fig. shows example scenario where objects (pink) and (blue)in the right image are far apart (more than 1m in the path length distance), whereas object D (pink) and C. (blue) in the left image are close to each other. This reward ensures that the traversal is optimized for the case of A and B by slightly compromising the number of moves (one extra pick-place step). In contrast, in the case of D and E, we slightly comprise on the traversal cost to optimize the number of moves. It prioritizes the goal-occupied objects (objects which occupy the goal location of other objects) to move first instead of the coal-blocked objects. For example, moving B and C first helps free the space for the goal-blocked objects (A, D) before they move to resolve the collision. It also ensures that the free location of goal occupied object will be nearest to the coal-occupied object's 's target location instead placing any random location. Instead of moving A to B's source location, this reward moves A to its nearby location and frees up B's goal location. This enforces that the traversal of the agent is minimized. In Fig. 10 R2 block describes this reward.
Collision resolution reward R2
Nearest neighbour reward R3 We use this reward (defined in R3 block of Fig. 10 to ensure that the agent's 's traversal should be minimal by first arranging the nearest objects from the previously placed object rather than arranging the objects randomly.
14

---

Published as a conference paper at ICLR 2024
Source
Goal
5 objects room rearrangement
Source
Target
10 objects room rearrangement
Figure 8: Rearrangement configuration under partial observability and swap cases for and 10 objects. This is sample configuration of objects in dataset, with various configurations for swap, path length distance, and non-greed non-greedy policies.
Goal reaching reward for object and receptacle R4: We use this reward (depicted in R4 block of Fig. [0) eliminate erroneous redundant actions. This reward ensures that our method can place the object in its goal location in single move whose goal position is free. It penalizes negative residual Euclidean distance it fails to place the object in its goal location.
Episodic Traversal Cost We novel episodic reward to impart the notion of path length traversed along with the number of steps taken in an episode as shown in Eq. (6)
E ALGORITHM FOR OUR METHOD
We present the pseudo-code of our method in Algorithm 2 From the goal state room G. capture the object list O {[,, P. P:].,=1,2.... for all the detected objects as shown in Line Moreover, we also get the room-receptacle list R = VWW.PP/ii = 1,2,.. ..,....e NR)- Also, we generate the 2D occupancy grid map 2D and 3D map MaD from the room in goal state. Using this information, we create a goal state graph (G₂) using O as described in Line 2 and Line 3 From the image I; in the current state at time ([W) J 1, Nv Comparing O in the goal state with 0% in the current state allows for determining only the semantics of unseen objects O [W? i= 1, Np Our goal is to solve the rearrangement sequence for the visible objects (OV) and search the unseen ones (OP) The Search network takes the input the pairwise concatenated names of W and W generate
get the semantic labels and positions of the visible objects as
15

---

Published as a conference paper at ICLR 2024
MARM
ARMIEWARE
MARA
Figure 9: An example of two types of swap cases. The left image shows the swap between the pink and blue objects shortest path length distance of 0.55m, whereas the right image shows swap instance with object separation of 2.5m in terms of the shortest path length distance. For the swap instance in the left image, if we reduce the traversal to 2 x 0.55m, we end up taking 1 more pick- pick-place move, whereas if we compromise only 0.55m more. For the swap instance on the right, if we reduce the number of pick-place place moves by traverse 2.5m more, whereas if we compromise the number of moves, complete the swap by just traversing 2 2.5m.
the traversal,
end up with less move and traverse
the position of probable receptacles (P²R). We generate the current state graph as shown in Line 10 and Line 11 using OV for visible objects and, o and p for the unseen ones. The Graph Siamese Encoder Network(GSENN) is used to calculate the embedding Zp from the graph representation Ge and Gg. We find the binary collision vector (C) by computing the similarity between PV and P. Using the state space 8, as computed Line 14, the Deep RL network, we generate a sequence of actions(a) such that
fa, (ks, Pk. ) >...2.... This rearrangement continues unless the becomes empty (Ø) or the maximum number of steps are exhausted (TM).
Each action in the sequence consists of individual action o, = Pk,)), where k; is the discrete action which specifies the objects that needs to be moved or searched and Pk, is the continuous parameter to specify the location of the pick-place or the search. If k; points towards object, say o² E 0 as shown Line 21 then a search action is performed on the most probable receptacle whose position is p" pf and name is w" E W The agent looks for unseen object o¹ at p" and along the path from the agent's current position to p", unless the maximum number of attempts exhausted, shown in Line 24 In case the predicted receptacle is articulated, the agent opens and looks for the object. After the search loop ends, we remove the searched receptacle (w") from W /2 it no longer contains any information about other unseen objects which is depicted in Line 29 k; points towards a visible object whose position P; in the goal state is known, then we perform pick-place action to place k; close possible to its target P. in Line 32
unseen
E. RE-PLANNING STRATEGY
The agent needs to re-plan a new sequence of actions if, (i) the agent fails to find the unseen object the predicted receptacle in Line 29 In this case, the agent discards the predicted receptacle (W \ w²) from further search attempts and also predicts new receptacle for the unseen object from the Search Network.
(ii) The agent finds the unseen object, it searching for, before arriving at the predicted receptacle as shown in Line 27 then the agent updates O and O These updates reflected in G. and using the state representation embeddings from the graph, the Deep RL understands the objects which need to be searched for or rearranged and accordingly generates a new sequence.
F EXPLORATION AND PATH PLANNING
Our method uses similar exploration strategy as in Sarch al aarch et al. 2022). We explore the scene using a classical mapping method. We take the initial pose of the agent to be the fixed coordinate frame in the map. We rotate the agent in-place and use the observations to instantiate
16

---

Published as a conference paper at ICLR 2024
k- selected object, pk placement parameter. previously selected object, pk: previously selected object's placement parameter, Os Source lecations all objects, Geal location of objetts, C- collision steps, Geodesic and b, dia.yy- Daclidean distance between and
points
RI
Vo
(0.0ghph)
x kauuma that aeal sccuped abject
Y6
digkla
hGGalDrogiodd OhjeetCal
ko-28em
Ves
Return
Rd
ditrkki kjr 1ilm
Retere 20
B
laccktial hand geodeti ditanee produndly placd abjeet's
R4
Algiitipp <5on
Return
Figure 10: Algorithm of our hierarchical dense reward structure.
17

---

Published as a conference paper at ICLR 2024
Algorithm 2: Task planner
Input: It TM
Data: RobertaModee odel, RobertaTokenizer
Result: Sequence of actions {ao, .., ok)
O (:W., R ([W P I
NR MaD MD +- 9;
< D(Pi, Pj)izz on MDD;
00;
while S T' and (W)) \ WV to pV + P) do OF {[W) PY + L6
O OOOO;
pvR + SearchNetwork(WW x WR) P pV UPPR,
Nodes(G,, (G.) (Ov, o u pppn:
10
Edges (G.) + D(P.,Pa)ie on M
11
7p GSEN(G)) GS
13
C CollisionDettcttoon( nP);
14
s + Zp UC:
a {a, = (ki, Pk..P))l==.. t Dsep
15
Oº or:
16
W R WR:
17
18
t;
while l2 S Tas&((a + &&O" == Ov OV &&&R = WR) do
19
20
(ke, Pk.) t a;
if E 8 then
{w", 8 .P'} +- kü;
22
+ 0;
while t3 S (no found o in Search (w", ,²)) do Update(OV ,08 Ttt1;
24
if of EOV then break;
1u +1; WR WR w":
else
31
(4 + 0: while ta S TTs&PPcckPlcce(kk) & do
Update {OV, 0 } < It11
33
14 + 44 1
35
0; 0i41;
36
l2 - l2 + 1;
37
4 1 + 1;
18

---

Published as a conference paper at ICLR 2024
an initial map. Second, the agent incrementally completes the maps by randomly sampling an unexplored, traversible location based on the 2D occupancy map built so far and then navigates to the sampled location, accumulating the new information into the maps each time step. The number of observations collected at each point in the 2D occupancy map is thresholded to determine whether given map location is explored not. Unexplored positions sampled until the environment has been fully explored, meaning that the number of unexplored points is fewer than a predefined threshold.
To navigate to a goal location, we compute the shortest path length to the goal from the current position using the Djikstra algorithm, given the 2D occupancy map M
G NETWORKS
G.1 SEARCH NETWORK
Generate RoBERTa embedding Each object and roce-receptacle label is concatenated together and encoded into an embedding( R) of size 1024 using the average of the output from the second last hidden layer of pre-trained RoBERTa large model.
Sorting Network(SRTN); The combined embedding((F E ( of the object-rooee room-receptacle is passed through SRTN consisting of 4 fully connected(FC) layers followed by ReLU and softmax, which outputs the softmax probability of an object being at that room receptacle represented by the three classes: 1) Most probable class, 2) Less probable class and 3) Implausible class. These FC
layers contain a dropout of 0.2, i.e. (F g::p2:512 512) - ReLU + FC(512, 256) ReLU FC(256,, 64) - ReLU FC 3) - softmar.
Scoring Network(SCN): The embeddings of the most probable class(E)) R) and the less probable class(E)) VR
is passed through the SCN consisting of 3 fully connected(FC) layers followed by ReLU, which outputs score(X) denoting the probability of finding an object that room-receptacle. These FC layers contain dropout of 0.2, i.e. FC(E 256) - ReLU + FC(256, 64) The argmoz of the probabilities decides the class to which the combined
ReLU + FC(64, 1). embedding belongs. For the objects in the implausible class the X 0 is used.
G.2 GRN
GSEN: It takes Ge and Gg as input and uses Siamese network to encode Ge and Gg. Each encoder of the Siamese network consists of 2 fully connected GraphConv followed by RELU, i.e. GraphConui(Ge. 256) - RELU - GraphComo(256, 128) and GraphConw (Gg. 256)
RELU >> StraphConn1255) 128).
RGDN: It takes the combined embedding of Ge and G, Zp which is passed through 2 fully connected (FC) layers followed by RELU and produces Tp. These FC layers contain a dropout of 0.25. i.e (C 64) + RELU + Dropout(0.25) + FC((
G.3 PROXY REWARD NETWORK
It takes input Zp UCUaU U Z, UČ where Zp U is the graph embedding and collision vector to represent the state space of the next state. The network consists of 4 fully connected networks followed by RELU. These FC layers contain dropout of 0.25, i.e. FC UCUU Z, UC, 512) - RELU + Dropout( (0.25) + FC(512, 128) + RELU + Dropout (0.25) + FC 64) + RELU Dropout(0.255 + 1))
G.4 DEEP RI.
(oo) It takes input state as Zp U C and all action parameters p. and uses 3 fully connected (FC) layers followed by RELU, which outputs discrete actions K. These FC layers contain a dropout of 0.5. i.e. FC(Zp U C Cp 512) + RELU (0o5 + FC(512, 128) + RELU - (0.2out(0.25) + FC(128, K).
(#p): It takes input as Zp U C. and uses 3 fully connected (FC) layers followed by RELU, which outputs all action parameter p. These FC layers contain a dropout of 0.5, ie. FC U C, 512) + R.LU (0.5) + FC(512, 128) R.E.U. (0.25))(0225) - FC(128, p).
19

---

Published as a conference paper at ICLR 2024
epclion-v.uux
Aerrge
200000
40000
60800
Of Steps
Figure 11: Comparison of the effect of e on average return with increasing steps. c 0.025 gives the best average return.
Number clusters
Numaer of clusters
20
Avenge
-20
40008
60000
80000
Figure 12: Comparison between the different number of clusters for the training of proxy reward using CB-RD. Number of clusters = gives the best average return indicating better sampling efficiency.
H TRAINING METHOD
H.1 SEARCH NETWORK
For training the Search network, we use the Search Network dataset, which contains the modified classes of the AMT dataset with the ground truth class labels and the probability score calculated from the mean inverse of Human annotated ranks. The entire dataset is split into train, val, and test sets with a ratio of 15 30.
Training SRTN:
Adam optimizer with learning rate(ası 0.001) and a weight decay rate(Aşı = 0.0001). Further, we use a randomized weighted data sampler to equalize the classes in each batch size of 512. The argmaz of softmax probabilities for each ORR in the three classes decides the class to which the combined embedding belongs. This is used to entropy loss(Fq.(1)) using the ground truth class labels in the Search Network dataset. This loss is backpropagated during each epoch to train the network.
To train SRTN,
calculate the
Training SCN: To train SCN, we use the ORR belonging to the Most probable and the Less probable class in the Search Network dataset. Further, we use an Adam optimizer with learning rate(asc = 0.001) and weight decay rate(Asc 0.01). Further, we use randomized weighted data sampler to equalise the classes in each batch size of 512. The output is used to calculate the mean square error (2))(Eq.(2) using the ground truth probability score in the Search Network dataset. This loss is backpropagated during each epoch to train the network.
H.2 GRN
The two encoders of GSEN use share weights to produce embeddings Z, and Z, for Ge and G, respectively... To train our GRN, Adam optimizer with learning rate(OrGıN 0.01).
20

---

Published as a conference paper at ICLR 2024
Average Return
Without pretraining
with pretraining
0.0
0.2
0.4
0.6
0.8
10
Number Steps
Figure 13: The plot demonstrates the effectiveness of a pre-trained GRN encoding relative distance notion well.
Further, we use a randomized data sampler with a batch size of 512. The output is used to calculate the mean square error loss (B). using the ground truth r.
H.3 DEEP RL AND PROXY
Weights of target network of q... 92
Current Weights $q
Step 3: Update dargel (Palyak averaging)
Step 1: roll out trajectories
Policy (Deep RL, e greedy)
4 000 |00
MAS
S. 8.8
Store (8. a. I. S. Rep
Training Prosy network
as, 3, B
Conram Clastering
Sampled For Training Prasy Newwrk
For training Deep RI. a,
Step 2: Update 47 p
Update 4q 4
[(s, 3, R
8, a,
Presy Reward Network
rg
8, a. 3. .ro
The diagram shows the training procedures of our Proxy Network and Deep RL method. We train the Deep RL in a 3-step method by fetching the samples from the replay buffer and using the predicted reward from the Proxy Network. We train the Proxy Network using the reward coming from the replay buffer.
Figure 14:
Fig. 14 shows our overall training method to train our Deep RL and Proxy Network. We train the Proxy Network using the loss shown in Eq. 88. We use off-policy method to train our RL. Our training method consists of the following three steps
Step 1: We use e-greedy exploration to generate trajectories {s, a, r, 3} and store into replay buffer.
Here the size of the replay buffer is 1000000000.
21

---

Published as a conference paper at ICLR 2024
Figure 15: Occlusion of fork(blue) by bread(pink) on the kitchen countertop.
Figure 16: Occluded fork(blue) becomes visible when the bread(pink) is picked from the kitchen countertop.
Step 2: We sample batch {s,a, 3} from replay buffer and query the proxy network to produce Tpredicted- We use {s, a, prredicted: 3) to update the Q-value by minimizing the Bellman error. We set the value of the learning rate for &Q and Φ as 0.0001 and 0.000001 respectively. We use Adam optimizer for both networks.
Step 3: We use polyak averaging to update the target networks of & andæ, The value of the rate of averaging for target networks &g and & P is 0.0075 and 0.00085 respectively. use an eviction policy similar to Kalashnikov et al. (2018) for our replay buffer.
We use PyTorch to train our models.
I QUALITATIVE RESULTS
We show the results of our method to solve the room-rearrangement problem under partial observabil- ity and swap cases with 5, and 10 objects in the supplementary video.
J ABLATION
We ablate our task planner against ground-truth perception (Ours-GT), various methods for object search (Ours-GE and Ours-RS) and a dense reward structure (Ours-DR). To study the effect of erroneous perception on our task planner, we assume the availability of Ground-Truth object detection labelling and 3D centroid localization from Ai2Thor Kolve et al. (017). To understand the importance of Search Network in planning, we replace it by (i) Random Search policy, which predicts probable receptacles for unseen objects with uniform probability and a (ii) Greedy Exploration strategy Chaplot cet al. (2020) that optimizes for map coverage to discover all the unseen objects. To highlight the generalization of proxy reward network to the overall objective of the rearrangement episode,
22

---

Published as a conference paper at ICLR 2024
of Offects
Nunker
Yabk Objects
Oljaas
Skap Case
OarsGT
Ours
Ouss-8
Ours-G
Dans-IR
OOF
OPK
SNSt
EXR:
ATCim)
SNS
ENR:
ATCmm)
SNS1
ENK1
ATCCnn)
SNS:
ENR
ATCmm)
5N8T
ENR1
ATCon)
0
0
0.98
NC
10.57
1.74
NC
11.95
0.74
NC.
11.96
6.74
N:
11.98
1L34
NC
12.75
0
0
0.70
NC
12.56
0.53
NC
13.46
0.53
MC
13.66
0.51
NC
13:46
1L55
NC
15.35
0.36
0.20
0.60
2
0.79
0:60
13.39
0.58
0,47
14.89
0.1
0.23
23,86
000
NC
NC
0LE8
0.47
16.91
0
0
0.37
2219
0.75
NC
24.31
0.73
SC
3451
6:23
NC
2151
1013
NC
1531
0.52
0.52
NC
27.32
IL52
NC
19.45
0
0.54
0.69
23.78
6.64
1L53
25.56
0.48
0.54
36.72
0.41
0.23
4018
R664
0.53
26.13
34.15
642
0.45
0.25
0
NC
Ra2
0.5
1659
0
0.355
0116
1.73
NC
41.8
0.73
4016
6.73
NC
40B
IC33
NC
4K227
20
31
01
0.70
NC
45.52
0.52
NC
46.31
0.52
MC
48.32
0.52
NC
45,32
IL52
MC
51.55
0:46
0.28
0667
0.58
0.57
0.74
42.13
0.66
0.57
45.78
0.47
0.28
54.68
0
NC
NC 0.66
0.57
47.67
Table 3: (OOF: Objects outside agent's field of view initially, which are visible from different perspective, OPR: : Objects placed inside closed receptacles, NC: Not computable). Ours- GE fails to handle the partial observability due to OPR (SNS = 0). Whereas, Ours-RS addresses both the cases partial observability but incurs a huge traversal cost (higher ATC) and takes more number of steps to search object (lower ENR) compared to Ours due to a random search policy. Ours-GE shows a higher ATC and lower ENR for OOF compared to both Ours and Ours-RS due to explicit exploration policy for search. Ours-DR shows drop in ATC as compared Ours due the greedy planning policy based on the nearest neighbour reward.
Number of Objects
Unseen Objects
With SKIN
Without SRTN
OOF
OPR
SNS1
ENR
ATC(m)
SNST
ENRI
ATC(m)
2
0
0.60
0,48
14.33
0.58
0.44
17.27
0.58
0.54
0.41
18.65
4
0.64
0,53
25.56
0.60
0.47
29.35
10
0.62
0.56
0.45
30.94
Table 4: Performance of our Search Network with and without the Sorting Network (SRTN)
replace with hierarchical Dense Reward structure as in Ghosh et al.. Tab. shows the results of the aforementioned ablation methods.
Without erroneous perception, Ours-GT demonstrates efficient planning, by performing significantly better than all the ablation methods in terms of SNR, ENR and ATC.
Under complete visibility and swap cases Ours, Ours RS and Ours - GE show similar results, since no partial observability cases exist. However, Ours-DR has higher traversal (ATC) cost due to its greedy action selection based on the nearest neighbour reward. In contrast, Ours uses the episodic notion based proxy reward that considers the overall notion of the episode train the Deep RL, which minimizes the agent's 's traversal.
Under partial observability, Ours performs significantly better than Ours-GE, Ours-RS and Ours- DR in terms SNS, ENR and ATC. This is due to the efficacy of the Search Network and the efficient planning of the Deep RL for simultaneous object search and rearrangement, which is trained with episodic notion based proxy reward. Whereas, Ours-GR GEE incurs a high traversal cost in terms of ATC because it explicitly explores the entire room to find the OOF objects. Moreover, Ours-Gr fails to address the OPR cases (SNS 0) because the greedy exploration policy Chaplot et al. (2020) in terms of map coverage does not include opening and closing receptacles to find OPR. However, Ours-RS randomly visits receptacles to discover OOF or OPR cases, which again increases ATC. More number of attempts to search in Ours-GE and Ours-RS leads to a lower ENR as well as SNS. We observe that Ours-RS performs slightly better than Ours-GE in terms of ENR and ATC for OOF, because Ours-RS interleaves object search and rearrangement, rather than doing an explicit exploration strategy for finding objects. This is due to the fact that rearrangement of visible objects and search for some unseen objects, aids in the discovery of other unseen objects. Ours- DR shows a slightly higher ATC compared to Ours for both OOF and OPR cases in partial observability due to the greedy planning policy based on the nearest neighbour reward.
Further ablation study highlights the efficacy of different modules and our choice of hyper- sapee-pprameters..
23

---

Published as a conference paper at ICLR 2024
Number of Objects
Unseen Objects
With SCN
Without SCN
OOF
OPR
SNS1
ENRT
ATC(m)
SNS1
ENRI
ATC(m)
2
0
0.60
0.48
14.33
0.52
0.41
19.57
5
0
0.58
0.47
14.89
0.49
0.38
20.83
4
0.64
0.53
25.56
0.54
0.45
31.17
10
0
0.62
0.52
25.97
0.52
0.39
32.11
Table 5: Performance of our Search Network with and without the Scoring Network (SCN)
J.1 SEARCH NETWORK
J.1.1 IMPORTANCE OF SORTING NETWORK
In the two-stage approach, Sorting Network (SRTN) reduces the likelihood of ranking implausible class room-receptacle in Scoring Network (SCN), thereby preventing inefficiencies in the search process. Tab. shows the result of our search with and without SRTN. The slight degradation in performace in terms of SNS, ENR and ATC shows that without SRTN the method requires slightly more steps to search objects due to incorrectly scoring implausible class receptacles.
J.1.2 IMPORTANCE OF SCORING NETWORK
The sorting network just classifies room-receptacles for objects into three groups based on the likelihood of finding the object at each location. However, it does not assign scores rank them, consequently failing to prioritize the search in the most probable receptacle first. Without the scoring network, the search strategy must randomly select room-receptacle from the higher probability class. This random selection degrades efficiency in terms of ENR and ATC, increasing the number of search attempts, indicated in Tab.
J.1.3 REASON FOR ROBERTA FINETUNING
Large Language Models (LLMs) such as RoBERTa Liu al. (2019) are trained on a huge corpus of available online text, enabling them to comprehend and make predictions based on general language understanding. Moreover, their large training data corpus enhances their ability to generalize to diverse range of input data. However, it does not necessarily translate to optimal performance for specific tasks, thereby emphasizing the importance of fine-tuning. Fine-tuning serves as the key to harness full potential of these models, allowing it to proficiently grasp domain-speciffe terminology and generate tailored predictions. To illustrate, consider the task of predicting probable room-receptacles for unseen objects. In this context, possessing knowledge of the specific names of objects and room-receptacles within the environment becomes essential. Fine-tuning empowers the model incorporate this domain-specific knowledge, allowing it to make more precise predictions and enhance its suitability for practical applications.
Suppose we need find spoon, unseen object, in an untidy living-room. For a pre-trained RoBERTa, consider the following masked prompts : (i) In an untidy living-room, a spoon is usually placed in the <mask> and (ii) In an untidy living-room, a spoon is usually placed on the <mask>. The output obtained by unmasking these prompts in decreasing order of score is middle, floor, table, corner, center, counter, carpet. Whereas, the output from our Search Network module based on fine-tuned RoBERTa embeddings in decreasing order of score is : table, carpet, coffee-table, console-table, bottom-cabires stool, sofa-chair. As evident, the output produced by the pre-trained RoBERTa model is linguistically sound but fails to effectively address the specific problem at hand and lacks the domain knowledge room receptacles within a household. In our evaluation on the test set of Search Network Dataset (Sec 3.1.. the Search Network module based fine-tuned RoBERTa embeddings exhibits approximately a 40% improvement in performance compared to using just the pre-trained RoBERTa model.
J.1.4 DIFFERENT HIDDEN-LAYED ROBERTA EMBEDDINGS
We present quantitative results obtained by utilizing different embeddings from the RoBERTa- large model for the Filter and Ranking networks in Tab These findings indicate that leveraging embeddings from the second-to-last hidden layer offers improved performance in capturing the underlying embedded commonsense within our Search Network framework.
24

---

Published as a conference paper at ICLR 2024
Input Embedding
% N.Accurayy
MSE J
train
val
test
train
val
test
Average-22 2dd
Jast last hidden layer
92.87%
81.67%
79.08%
1.001
0.046
0.031
Average-last last hidden layer
88 41%
59.19%
37.93%
0.01
0.04
0.09
Table 6: This table shows ablation study against different input embeddings from Roberta-large for the Search Network. We tried the last and the second last hidden layer from RoBERTa and our results show that the second-last hidden layer better captured the commonsense knowledge.
Edge feature graph
SNS 1
ENR
ATC(m)
Fuclidean distance
0.73
0.71
29.78
path length distance
0.73
0.71
23.21
Table 7: Results to show the effect of using the path length distance based edge feature over the euclidean distance one.
J.2 GRAPH REPRESENTATION NETWORK
We try to understand the effect of pre-trained GRN features on our state space. Fig. shows that our pretrained GRN produces good features which capture the relative pairwise distance between the source and goal graph well. Further, we also show the effect of using the edge features of pairwise Euclidean distance v/s pairwise path length distance between the objects. The results in Tab. [7sho that the performance of the task planner is impacted significantly by using the edge feature path length distance over the Euclidean one because the path length metric better captures the real scene geometry compared to the Euclidean.
J.3 DEEP RL
We the ablation study decide the effective value of e. Fig. shows that Deep RL method works best for e 0.025. Moreover, we have shown the effect of number of clusters for Cluster Biased Return Decomposition (CB-RD) in terms of sampling efficiency. Fig. s2oosws that the cluster size of three gives the best performance in terms of sampling efficiency.
J.4 REWARD
We show the ablations for all the reward components mentioned in Appendix D in our supplementary video. The study highlights the significance of each reward component and also the improvement in the distance traversed due to the change in the Collision Resolution reward.
K FAILURE CASE
As depicted in Fig. 15 there instances where object the receptacle is hidden from the agent's egocentric view of the current state due to occlusion by another object. Our method can not resolve the search in such scenarios. To deal with these cases, we need a method that generates set of manipulation actions such as pick-place and moves the objects which are causing the occlusion as shown in Fig.
High-leve objecct categories
Objects
Cooking
spatila, pol, kattle pon
Kitchen Arcossorias
auminum foil, botll. dish sponge, salt shake: pepper shaker. lodie, soap botic, scrub brush botterKnite, knis,, foet, spoon, plate
Curlery
Crockery
bool, mug
Edible
bread, bread-slsced apple. apple-tlood 4gg, cracet, lonzin, potaoo lenuce, wine, wan-bbolle tomato-sliod,, polstosllod letuxe-slicd
Spors Accessories
bescball bet basktball, boois, tenni nacket, dumbbell
Statinery
book, pencil
Rocur-occe
watch, statue, vase, toddy -bea, lable topo--tp- posier, pillon, painting desk lang, lourlamp weel, tolle--pperr soup-botlc, scop-bir, scrub-brus,, pluneer jogd--lwee-rol, handowwd walkring-can, sacuum-llanee, lisur box, spmy botle. cundle, muror, alann clock, box coll-pione. CD, remotb-control, newsgaper, lapoop. crodit-cand, keyohain
Bathromm sccessoties
Utilises
Consumer limms
Table 8: Table showing example of objects in each of the high-level object categories.
25


Reviews:
[{"review_id": "6jJwMQzmsZ", "reviewer": "Reviewer_YNy7", "summary": "This paper tackles the task of visual room rearrangement where an agent must first explore a 3D scene autonomously to map its content before objects in the scene are moved around. The agent must then, in a second phase, re-organize objects whose position has been changed compared with the initial scene. The authors propose a series of contributions to both evaluate methods, with a new benchmark and metrics, and improve autonomous agents’ overall performance and efficiency, with different architectural and training contributions.\n\nFirst, the paper introduces a new Search Network whose goal is to predict the position of unseen objects after the room has been untidied. The claim is that such a search process could leverage prior common sense to find candidate receptacles more efficiently. Thus, the Search Network is composed of a large language model (LLM) that incorporates prior knowledge about the relationships between objects and receptacles.\n\nAnother contribution is a hybrid action space Deep RL agent that tackles both object search and rearrangement. The state space of this Deep RL agent is a graph representation of both the initial scene (known as goal state) and the current scene (known as current state) to provide information about the position of objects. Finally, the paper proposes a proxy reward network predicting a dense reward signal to facilitate the RL training of the policy.\n\nThe method is compared with different baselines on a new benchmark, RoPOR, and additional metrics, evaluating the efficiency of taken paths, are considered.", "questions": "All questions and suggestions are already mentioned in the “Weaknesses” section as a list of numbered points.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* **S1**: The paper tackles an important and challenging task, i.e. visual room rearrangement, and motivates its different contributions based on important considerations about the task.\n\n* **S2**: This work proposes many different contributions, both from an evaluation point of view (benchmark, metrics) and from an architectural and training points of view.\n\n* **S3**: The method is compared against different relevant baselines and shows a promising gain in performance.\n\n* **S4**: Many qualitative videos are presented in the Supplementary Material, helping to better compare methods by visualizing their behavior.", "weaknesses": "* **W1**: **[Major]** How can the Search Network leverage prior knowledge as the room is untied? More specifically, if object shuffling/placement is done randomly, shouldn’t there be no prior remaining? Indeed, in phase 2, objects are placed in locations were there shouldn’t be (so that the agent can re-organize them). It is hard for me to understand how, in this case, the Search Network can learn anything meaningful. A result in the ablation study (Table 2) seems to confirm this intuition: the performance of Ours-RS, where the Search Network is replaced with a uniform random sampling of the next receptacle to visit, is very close (1p Success Rate) to the performance of Ours. The difference does not seem to be significant enough to claim the Search Network does more than random search. In order to claim a gain, authors should report the mean and standard deviation over a few training runs (random seeds). The following could also be done by authors:\n    * **W1.1**: Reporting the performance of Ours-RS on the introduced RoPOR benchmark.\n    * **W1.2**: Comparison of the Search Network with another simple baseline: selection of the closest receptacle (with reported performance on both RoPOR and RoomR).\n    * **W1.3**: Provide more details about how objects are moved in phase 2: there should not be any prior remaining, and if there is, it might mean that the comparison is unfair with other methods because the authors’ search model might have been trained to learn those “shuffling priors” while it is probably not the case of previous work.\n\n* **W2**: **[Major]** This comment is quite related to the previous one: as mentioned in the paper, the Search Network is finetuned to incorporate prior knowledge about object-receptacle relationships (see W1 about why I am not convinced any such relationship can be learned in the untidy scenario). Authors should still show the pre-training of the LLM brings a performance gain. What about the same LLM architecture initialized with random weights and trained as done in the paper?\n\n* **W3**: **[Major]** It is not clear to me how the ground-truth data to train the Sorting Network (SRTN) is generated. Could authors elaborate on this?\n\n* **W4**: **[Major]** One might argue that the Sorting Network only could be enough to predict the most likely object-receptacle pairs. Authors should provide an ablation study showing the impact of the additional Scoring Network.\n\n* **W5**: **[Major]** I would like authors to clarify the following points regarding the Proxy Reward network:\n    * **W5.1**: What is the interest of this Proxy Reward network? The paper mentions it is a way to predict a dense reward to train the RL agent. However, given a simulator, couldn’t we simply compute a dense reward from privileged simulator information at training time?\n    * **W5.2**: What is the training ground truth for the Proxy Reward network?\n    * **W5.3**: What is the average return on the y-axis of Figure 3? Such return is indeed associated with a specific reward function: what are the terms of this reward function? Moreover, I would like the authors to provide more details about the comparison done in Figure 3 and thus the conclusions we can draw from it.\n\n* **W6**: **[Major]** The paper mentions the introduced method “assumes the availability of perfect motion planning and manipulation capabilities”. While this is a strong assumption, it does not outweigh the contributions in this work. However, an important question is: Are all the baselines this method is compared against also benefiting from the same assumption? Otherwise, this could be considered as an unfair comparison.\n\n* **W7**: **[Minor]** When introducing their SNS metrics, authors should cite *Anderson et al., On Evaluation of Embodied Navigation Agents* that introduced quite similar metrics such as SPL.\n\n* **W8**: **[Minor]** Paper citations are not properly inserted in the text. Authors should use parentheses (\\citep{} in Latex) when needed, and remove double citations (e.g. “Sarch et al. Sarch et al. (2022)”).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XMZD1oGcQ4", "reviewer": "Reviewer_JNsv", "summary": "The authors tackle the problem of object re-arrangement. To define the goal state, the agent is able to explore the room in its goal state and build up its own (graph-based) internal representation of the room. Then, at test time, the objects in the room are shuffled around such that they can be occluded or hidden in receptacles. The job of the agent is to put the room in the desired goal state as efficiently as possible. To do this, the proposed method keeps track of which objects have been seen and which haven't. A search network predicts probable locations of unseen objects. Both the current scene and goal scene are encoded as graphs, with objects as nodes and geodesic distances between objects as edges, and this scene is encoded with graph networks. Lastly, they use Q-learning with a proxy reward network to train the planner. To this end, the authors also build a dataset to train the search network and the graph network, in addition to contributing a new benchmark in Ai2Thor (RoPOR). The proposed approach significantly outperforms the baselines.", "questions": "* What are the failure cases? Does the agent ever get stuck in a loop?\n* Are duplicate objects handled (e.g. multiple sponges in a scene)? Are these included in the test data, and how does the agent perform under these conditions?\n* SRTN requires manually defined rules and heuristics about where objects are unlikely to be found (e.g. cup in bathtub). Is the network unable to learn these probabilities automatically?\n* Does the SRTN need to be learned via the MLP? It seems like the probabilities can be pulled directly from the data without learning. We could simply build a hardcoded table of probabilities. For example, the probablity of finding a sponge in the sink can be hardcoded to 0.80. Is there a reason why that number would change? What is the benefit of learning here?\n    * It is odd that we are trying to learn where objects are likely to be, but the training dataset is designed such that the authors \"ensure a random distribution of object placements\" (Appendix C.1).\n* How does the agent go from the output of the planner to discrete motions (rotate, move forward, pick up, place)?\n    * How is navigation performed when going from point A to point B? Is it assumed that this problem is solved?\n* There is already an existing dataset (and associated metrics) for the same setup in Sarch et al. (Sec. 4.5 of their paper). While there are some deviations between the proposed dataset and the existing dataset, it still makes sense to benchmark the proposed approach against the already existing dataset that has baselines already benchmarked against it. Is there a reason this wasn't done? The setup of the existing seems like a subset of the proposed dataset's setup, so it seems doable.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The authors provide a new benchmark for object re-arrangement in AI2Thor. The new dataset supports swap cases (objects' positions that are swapped compared to the goal scenario) and blocked goals (ie. there is an object occupying the goal position of another object).\n* Extensive supplementary video and supplementary material\n* Modified RL training approach seems to outperform other approaches in that it converges substantially faster.\n* The approach significantly outperforms the baselines.", "weaknesses": "* Convoluted approach that is a bit hard to follow, especially because many details are omitted in the main paper. It would also be helpful to more clearly define the task setup (inputs and outputs and their explicit representations).\n* Not obvious why the SRTN needs to be learned instead of manually defined.\n* Manual heuristics (where objects are unlikely to be found) and GT information (e.g. odometry, collision vector, etc.) is required.\n* More details on the reward design in the main paper would be appreciated\n* Critical ablations are not in the main paper.\n* There is already an existing dataset (and associated metrics) for the same setup in Sarch et al. (Sec. 4.5 of their paper). While there are some deviations between the proposed dataset and the existing dataset, it still makes sense to benchmark the proposed approach against the already existing dataset that has baselines already benchmarked against it. Is there a reason this wasn't done? The setup of the existing seems like a subset of the proposed dataset's setup, so it seems doable.\n* Somewhat similar to Sarch et al.\n    * Both use 2D and 3D representations of the environment\n    * Both use object detector + a search network to guide exploration\n    * Both represent the scene as a graph on top of which they perform inference\n\nMinor comments:\n* Some strange wording such as contributing to the \"research fraternity\", misspelling such as \"detetector, \"Paramater\", etc.\n* Assumes perfect motion and manipulation\n* Goal definition of having the goal scene already set up and utilizing an exploration stage is a bit impractical, as it requires setting up the goal scene every time it is changed. Also the robot needing to explore the goal state takes time compared to, say, language defined goals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tOIAP0Xck5", "reviewer": "Reviewer_eRZf", "summary": "The paper proposes a hierarchical task planner equipped with several proposed components for room rearrangement for user-defined goal states.\nSearch Network exploits LLMs to query possible receptacles where unseen objects may be present.\nThe graph-based state representation encodes the objects' spatial relationships and their distances for the current and goal states in the form of graphs.\nThis is later used for Deep RL based Planner trained by the proposed cluster-biased return reward decomposition.\nFor the evaluation, the paper introduces a new benchmark, RoPOR, for room rearrangement that addresses blocked goals and swap cases, with newly introduced metrics that mainly measure agents' efficiency.\nThe proposed method outperforms the baselines in their empirical validations by noticeable margins.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Tackling the blocked goals and swap cases is well-motivated and sounds sensible. Addressing them seems to be an important problem.\n- Exploiting prior knowledge encoded in LLMs for possible target receptacles looks reasonable.\n- Exploring an end-to-end framework for room rearrangement for user-defined goal states is intriguing.", "weaknesses": "- In Search Network, it is unclear why we need the \"two-staged\" approach: 1) filter out some implausible receptacles and 2) obtain the most plausible one. Why not just use SCN alone to get the most plausible receptacle, as the implausible receptacles should result in low scores and thereby be not chosen, consequently?\n- The graph-based state representation requires shortest-path computation for all fully connected edges, but this seems quite computationally heavy, especially when we have a large number of nodes, leading to a drastically increasing number of edges.\n- The comparison with some baselines seems unfair. For example, Weihs et al. and Gadre et al. do not use depth maps as input while the proposed method does, but they are compared in a single table. In addition, the authors utilize additional training datasets (Sec. 3.1).\n- Some new metrics are introduced to measure agents' efficiency but they look a bit similar to SPL in navigation literature, which basically penalizes an agent's success rate by the length of trajectories it took so far. Similarly to the introduced metrics, as agents take more steps for rearrangement or search, SPL penalizes the success rates more, accordingly.\n- The environments used in the proposed benchmark look quite \"clean.\" It seems that we have objects only related to rearrangement tasks, as illustrated in Figure 16 and the supplementary video. This looks quite far from practical scenarios as we usually have many objects inside rooms.\n\n\\* Minor\n - It might be better to divide a result table (e.g., Table 1) into one for the main results and the other for the ablation study for better readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)