TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specification
Authors: Mingdong Wu
fangwei zhong
Yulong Xia
Hao Dong
Venue: NeurIPS 2022 Accept (2022)
Keywords: Object Rearrangement
Reward Learning
Score-matching
Reinforcement Learning

Abstract: Object Rearrangement is to move objects from an initial state to a goal state. Here, we focus on a more practical setting in object rearrangement, i.e., rearranging objects from shuffled layouts to a normative target distribution without explicit goal specification. However, it remains challenging for AI agents, as it is hard to describe the target distribution (goal specification) for reward engineering or collect expert trajectories as demonstrations. Hence, it is infeasible to directly employ reinforcement learning or imitation learning algorithms to address the task. This paper aims to search for a policy only with a set of examples from a target distribution instead of a handcrafted reward function. We employ the score-matching objective to train a Target Gradient Field (TarGF), indicating a direction on each object to increase the likelihood of the target distribution. For object rearrangement, the TarGF can be used in two ways: 1) For model-based planning, we can cast the target gradient into a reference control and output actions with a distributed path planner; 2) For model-free reinforcement learning, the TarGF is not only used for estimating the likelihood-change as a reward but also provides suggested actions in residual policy learning. Experimental results in ball and room rearrangement demonstrate that our method significantly outperforms the state-of-the-art methods in the quality of the terminal state, the efficiency of the control process, and scalability.

TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specification
Mingdong 1, 1.3 Fangwei Zhong" 2.3, 2.33, Yulong Xia', Hao Dongl
Center on Frontiers of Computing Studies, School of Computer Science, Peking University
2 School of Intelligence Science and Technology, Peking University Beijing Institute for General Artificial Intelligence (BIGAI) Peng Cheng Laboratory {wming, zfw, hao don.) edu. cn
Abstract
Object Rearrangement is to move objects from an initial state to a goal state. Here, we focus on a more practical setting in object rearrangement, iee, rearranging objects from shuffled layouts to normative target distribution without explicit goal specification. However, it remains challenging for AI agents, it is hard to describe the target distribution (goal specification) for reward engineering collect expert trajectories as demonstrations. Hence, is infeasible to directly employ reinforcement learning or imitation learning algorithms to address the task. This paper aims to search for a policy only with a set of examples from target distribution instead of handerafted reward function. We employ the acore-matching objective to train a Target Gradient Field (TarGF), indicating a direction each object to increase the likelihood of the target distribution. For object rearrangement, the TarGF can be used in two ways: 1) For model-based planning, cast the target gradient into reference control and output actions with a distributed path planner; 2) For model-free reinforcement learning, the TarGF is only used for estimating the likelihood-change provides suggested actions in residual policy learning. Experimental results in ball rearrangement and room rearrangement demonstrate that our method significantly outperforms the state-of-the-ar art methods in the quality of the terminal state, the efficiency of the control process, and scalability. The code and demo videos are on bttps://sitee .oogle. com/ com/view/ee view/targf.
reward but also
Introduction
As shown in Fig. 1, we consider object rearrange- ment without explicit goal specification [1, 2], where the agent is required to manipulate set of objects from an initial layout to a normative distribution. This task is taken for granted by hu- mans [3] and widely exists in our daily life, such as tidying up a table [4], placing furniture [5], and sorting parcels [6].
Initial State
However, compared with conventional robot tasks [7, 9, 10, 11], there are two critical chal- lenges in our setting: 1) Hard to clearly define the target state (goal), as the target distribution diverse and the patterns are difficult to describe in the program, e.g., how to evaluate the tidiness
Tanget Examples
St
Terminal State Sy
Figure 1: Our goal: Learning to rearrange ob- jects without explicit goal specification.
* indicates equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

quantitatively? As a result, it is difficult to manually design reward/objective function, which is essential to modern control methods, e.g., deep reinforcement learning (DRL) [12]. 2) In the physical world, the agents also need to find an executable path to reach the target state efficiently and safely, ie., finding a short yet collision-free path to reach the target distribution. To this end, the agents are required to jointly learn to evaluate the quality of the state (reward) and move objects to increase the quality efficiently. In other words, the agents should answer the two questions: What kind of layouts does the user prefer, and How to manipulate objects to meet the target distribution?
Thus, searching for a policy without dependence on the explicit goal state and shaped reward is necessary. Imitation learning (IL) can learn a reward function [13, 14, 15] or learn policy [16, 17, 18, 19] from the expert's 's supervision, e.g., demonstrating expected trajectories. However, collecting a large number of expert trajectories is expensive. Recently, example- basel reinforcement learning (RI.) [20, 21] has tried to reduce the reliance on expert trajectories, i.e., learning a policy only with examples of successful outcomes. However, it is still difficult to learn classifier to provide accurate reward signals for learning in the case of nigh-dimensional state and action space, as there are many objects to control in object rearrangements.
In this work, we separate the task into two stages: 1) learning a score function estimate the target- likelihood-change between adjacent states, and 2) learning/building a policy to rearrange objects with the score function. Inspired by the recent advances in score-based generative models [22, 23, 24, 25, 26], we employ the denoising score- score-matching objective [27] to train target score network for estimating the target gradient field, i.e., the gradient of the log density of the target distribution. The target gradient field can be viewed as object-wise guidance that provides pseudo velocity for each object, pointing to regions with higher density of the target distribution, e.g., tidier configurations. To rearrange objects in the physical world, we demonstrate two ways to leverage the target gradient field in control. 1) We integrate the gradient- -based guidance with a model-based path planner to rearrange objects without collision. 2) We further derive a reward function for reinforcement learning by estimating the likelihood change via the target gradient. With the TarGF-based reward and a TarGF-guided residual policy network [28]. we can train effective RL-based policy.
In experiments, we evaluate the effectiveness of our methods on two typical rearrangement tasks: Ball Rearrangement and Room Rearrangement. First, we introduce three metrics to analyse the quality of the terminal state and the efficiency of the control process. In Ball Rearrangement we demonstrate that our control methods can efficiently reach high-quality and diverse terminal configuration. We also observe that our methods can find path with fewer collisions and shorter lengths to converge compared with tearning-based baselines [14, 21] and planning- based baselines [29]. In Room Rearrungement, we further evaluate our methods' feasibility in complex scenes with multiple heterogeneous objects.
Our contributions are summarised as follows:
We introduce novel bosre-based framework that estimates a target gradient field (TarGF) via denoising score matching for object rearrangement.
We point out two usages of the target gradient field in object rearrangement: 1) providing guidance in moving direction; 2) deriving a reward for learning.
We conduct experiments to demonstrate that both traditional planning and RL methods can benefit from the target gradient field and significantly outperform the state-of-the-art methods in sample diversity, realism and safety.
2 Related Works
2.1 Object and Scene Arrangeeettt
The object (re-)arrangement is a long-studied problem in robotics community [2, 30, 31] and the graphics community [32, 33, 34, 35, 36]. However, most of them focus on manually designing rules/ energy functions to find a goal [2, 30, 31] or synthesising a scene configuration [33, 34, 35, 36] that satisfies the preference of the user. The most recent work [4] tries to learn a GNN to output an arrangement tailored to human preferences. However, they neglect the physical process of rearrangement. Hence, the accessibility and transition costs from the initial and target states are not guaranteed. Considering the physical interaction, the recent works on scene rearrangement [37, 38] mitigate the goal specification problem by providing a goal scene to the agent, making the reward shaping feasible. Concurrent works [39, 40] also notice the necessity of automatic goal inference for
2

---

tying rooms and exploit the commonsense knowledge from Large Language Model (LLM) or memex graph infer rearrangements goals when the goal unspecified. Another concurrent work [41] aims at discovering generalisable spatial goal representations via graph-based based active reward learning for 2D object rearrangement. this paper, we focus on a more practical setting of rearrangement: how to estimate the similarity (i.e., the target likelihood) between the current state and the example sets and manipulate objects to maximise it. Here, only a set of positive target examples are required to learn the gradient field rather than prior knowledge about specific scenarios.
2.2 Learning without Reward Engineering
Without hand-engineered reward, previous works explore algorithms to learn a control policy from expert demonstrations via behavioural cloning [16, 17, 18, 13] or inverse reinforcement learning [42, 15, 43, 44]. Imitation learning (IL) [16, 17, 18] aims to directly learn policy by cloning the behaviour from the expert trajectories. The inverse reinforcement learning (IRL) [42, 15, 43, 44] tries to learn a reward function from data for subsequent reinforcement learning. Considering the difficulty in collecting expert trajectories, some example-bast based methods are proposed to learn a policy only with set of successful examples [20, 14, 21]. The most recent work is RCE [21] which directly learns a recursive classifier from successful outcomes and interactions for policy optimisation. We also try to develop example-based control method for rearrangement with set of examples. Our method can learn the target gradient field from the examples without any additional effort in reward engineering. The target gradient field can provide meaningful reward signals and action guidance for reinforcement learning and traditional path planners. To the best of our knowledge, such usage of gradient field is not explored in previous works.
3 Problem Statement
We aim to learn policy that moves objects to reach states close to a target distribution without relying on reward engineering. Similar to the example-based control [21], the grounded target distribution Pear (s) is unknown to the agent, and the agent is only given a set of target examples S {s*} where s ~ Ptar (s). In practice, these examples can be provided by the user without accessing the dynamics of the robot. The agent starts from initial state so (so),, where N objects are randomly placed in the environment. At time step t, the agent takes action π T(a (s:) imposed on the objects to reach the next state with dynamics p(s (st, az). Here, the goal of the agent is to search for a policy x (ay(st) that maximises the discounted bog-targer-likeiheee of the future states:
= arg max E E((50) TI/
Σ ET
72t log Ptar (st)
where 7 E (0, 1] denotes the discount factor. Notably, we assume that Ptor (s) > everywhere since we can perturb the original data with tiny noise (e.g., (0, 0.00001)) to ensure the perturbed density is always positive. [22] also used this trick to tackle the manifold hypothesis issue. This objective reveals three challenges: 1) Inaccessibility problem: The grounded target distribution ptar in Eq. 1 inaccessible. Thus, we need to learn a function to approximate the rog-target-likelibee for policy search. 2) Sparsity problem: The tatttttteekkeelheeeeee
is sparse in the state space due to the Manifold Hypothesis mentioned in [22]. Hence, even we have access to the target distribution Pter, it is still difficult to explore the high-density region. 3) Adaptation problem: The dynamics p(s++1|s, |St, a;) is also inaccessible. To maximise the cumulative sum in Eq. 1, the agent is required to adapt the dynamics to efficiently increase the target likelihood.
To address these problems. we partition the task into learning to estimate the log-target- likellood
(reward) of the state and 2) learning/building policy to rearrange objects to adapt to the dynamics.
4 Method
4.1 Motivation
Estimating the gradient of the log-target-likellhore V, Ptar (s) can tackle the first two problems mentioned in Sec. 3: For the inaccessibility problem, we can approximate the likelihood increment
3

---

(a) Learning Target Gradient Field (TarGF)
(b) Control Objects with TarGF
Pertubbed
Graph State
= My (G. ((a))
Targel Score Netwods
Pulky To
8r
(5
Peeturtation o')
Score Matching (2(6.
Target Score Network
9t
GradlAction
Denotse
Stt
Groph Statne
Target Stattos
Gragh Slate
of
Sample
ORCA Planner +
Sg-1
Tarpet Etample Set (aa Pearr)))))
RL-based Control
Model- Planning
Figure 2: An overview of our method. (a) We train the target score network via score matching. The target examples are first perturbed by Gaussian noise. The target score network is forced match the denoising direction of the perturbation. (b) Our framework is based on the trained target score network. The target score network provides exploration guidance and reward estimation for RL in the model-free setting. In the model-based setting, the TarGF provides reference velocities for model-based planner based on ORCA. The planner then outputs collision-free velocities for objects.
between two adjacent states by the first-order Taylor expansion log Ptar (Sc-1) log Pter (s:) Rs (V, Ptar(s)). St+1 - -s)). This helps to derive a surrogate objective (i.e., Eq. 4) of Eq. 1.
For the sparsity problem, we can leverage the gradient field to help with exploration since the gradient Vx lolooPto Ptar (8) indicates the fastest direction to increase the target likelihood. To estimate Vg Poao (s). employ the score-matching [23], a novel generative model that achieved impressive results in many research areas recently[24, 26, 25], to train target score network Pior S + S.
To address the adaptation problem, we further demonstrate two approaches to incorporate the trained target gradient field & Ptar with control algorithms for object rearrangement: 1) For the model-based setting, our framework casts the target gradient into reference control and outputs action with a distributed path planner. 2) For the model-free setting (reinforcement learning), we leverage the target gradient to estimate reward and provide suggested action in residual policy learning [28].
In the following, we will first introduce how to train the target gradient field Ptar via score-matching in Sec. 4.2 and then introduce our rearrangement framework under the model-based and model-free settings in Sec. 4.3 and Sec. 4.4 respectively.
4.2 Learning the Target Gradient Field from Examples
The target score network Pear is the core module of our framework, which aims at estimating the score function ti.e.the gradient of log-density) of the target distribution V, log (tar (s).
To train the target score network, we adopt the Denoising Matrchna (DSM) objective proposed by [27], which can guarantee reasonable estimation of the V, log Ptar (s). In training, pre- specify a noise distribution (B]ss) = (s; I²1). Then, DSM matches the output of the target
score network with the score of the perturbed target distribution go (5) 9 (5(s)) (s)darr(ss:::)
2 E (8ss).)) P tar
Vg log qo (S]s)
E Bgg
| (8)
(2)
2
The DSM objective guarantees the optimal score network satisfies T far (s) V sol (s) almost surely. When a is small enough, we have Vego (s) x V, log (s), so that Piar (s) R V. log (to.
In practice, adopt variant of DSM proposed by [23]. which conducts DSM under different noise scales simultaneously. This way, we can efficiently try different noise scales after only one training. The target score network is implemented as graph-based network Ptar (s, t), which is conditioned on a noise level t E (0, 1]. The input states = [s esa ss,,... N is constructed as a cully-connected graph that takes each object's state (e.g.posing position, orientation) s² RJ and static properties (e.g category, bounding-boxx p' e f the node feature [s', p²] E R I. The input graph is pre-processed

---

by linear feature extraction layers and then passed through several Saage-Convolution layers. After message-passing the output on the i-th node serves as the component of the target gradient on the i-th object's state D (s) E R. We defer structure details to Appendix. B.
4.3 Model-based Planning with the Target Gradient Field
The target gradient g = 2 Ptar (s) can be translated into a gradient- bassed action a = (gg) where denotes the pradient-to-action translation. A gradient over the state space be viewed as velocities imposed on each object. For instance, if the state of each object is a 2-dimensional position [2r, y). then the target gradient component the i-th object can be viewed linear velocities on two axes 4 (s)' = Miz vy]- When the action space is also velocities, we can construct a by simply projecting the target gradient into the action space a G(g) = Vmoz
llglla g, where the Umaz denotes the speed limit. Following the gradient- baasee action, objects will move towards directions that may increase the likelihood of the next state, which meets the need for the rearrangement task.
However, the gradient-based action will potentially lead objects collide with each other since the target gradient cannot infer the environment constraints from only target examples.
To adapt the gradient-based action to the environment dynamics, incorporate the target gradient field with a model-based off-the-shelf collision-avoidante algorithm ORCA. The ORCA planner takes the gradient- based action a reference velocities and then outputs the collision-fred velocities a ó(a") with the least modification to the reference velocities. In this way, the adapted action can lead objects move towards higher likelihood region efficiently and safely. For more details of the ORCA planner, we defer to Appendix B.2.
4.4 Learning Policy with the Target Gradient Field
The model-based approach 'correct' the target gradient assumes the objects are of circular shape. This limits the scope of this method when objects are of non-ciree non-circular shape (e.g., furniture). To this end, we propose model-free approach based on reinforcement learning (RL) for object rearrange- ment, where the agent needs to adapt the dynamics via online interactions.
To tackle the inaccessibility problem mentioned in Sec. 3, first derive equivalent J*(r) of the original objective in Eq. by subtracting a constant from the original objective:
E Ep(so) log Ptar (5o)
J(x) = E ((u))
Σ
2d
log Ptor (s,)]
S,ET
constant C
= =Ep(so) rMol
Σ 15tT
r'
Σ 15kSt
[log Ptar (tar((s) - log Ptar (Sk- 1)]] def J*(T)
Further, derive surrogate objective (iT) to approximate the (T)).. We notice that in most physical simulated tasks, the distance between two adjacent states ||se 8 ll2 is quite small, which inspires approximate the log- Nog-likelihood--hhaee
of the adjacent states using the Taylor expansion:
J (²) RJ (ppso) TOO
Σ
( (Se- 1), - Sk -k-1)] def ((T)
ISST
IskSt
(4)
In this way, we can optimise the surrogate objective ((T) by assigning
Σ (orrr
-
1skSt
as the immediate reward. In practice, we simply assign the last term of the cumulative summation as the immediate reward r, = (fera St 1) ) and this version shown to be the most effective.
To further tackle the sparsity problem, our idea is to build residual policy тa upon the gradient-basse action ad mentioned in Sec. 4.3, as aï helps to explore the regions with a high target likelihood:
a, (St, 8:) ag.
a = G(g)
Unar
ge:
g: tar (8t)
(5)
1gll2
output action
target gradient
gradient- based action
5

---

This approach is similar to the residual policy learning proposed by [28]. The af serves as an initial policy, and the residual policy network Ty takes the target gradient gi and the current state as input and outputs i (S;, g:) to 'correct' the a. The agent finally outputs 'corrected' action a, = (8z, ge) + a.. this way, the agent can benefit from the efficient exploration and have less training burden as it 'stands the giant's shoulder' (i.e.the mradient-based action a.).
We employ the multi-agent Noft-Actor-Criti(SAAC. [45] as the reinforcement learning algorithm backbone. Each object is regarded as an agent and can communicate with all the other agents. At each time step the reward for the i-th agent is rt raar -Arri. where rlar
Σ
( tar (St), Sk Sk-11
1sk5
is centralised target likelihood reward, decentralised collision penalty where Cij is a collision penalty. Specifically, when a collision is detected between object and j. Ci.j -1, otherwise c; = 0.
0 is hyperparameter and r Ejze C.j is
5
Experiment Setups
5.1
Environment
We design two object rearrangement tasks without explicit goal specification for evaluation: Ball Rearrangement and Room Rearrangement, where the former controlled environments for better numerical analysis and the latter is built on a real dataset with implicit target distribution.
Ball Rearrangement includes three environments with increasing difficulty: Circling, Clustering. and the hybrid of the first two, Circling Clustering, as shown in Fig. There are balls of the same geometry in each task. The balls are divided equally into three colours in Clustering and Cireling+ Clustering. The goals of the three tasks are as follows: Circling requires balls to form a circle. The circle's centre can be anywhere in the environment; Clustering requires all balls to form into three clusters by colour. The joint centre of each cluster has two choices (i.e., red-green-bee blue or bed-blue-gee green clockwise); Circling Clustering requires all balls to form into circle where balls of the same type are adjacent. To slightly increase the complexity the physical dynamics, we enlarge the lateral friction coefficient of each ball and make each ball with different physical properties, e.g., different friction coefficients and masses. The target examples of each task are sampled from an define 'pseudo likelihood" process to measure the similarity between the state and the target distribution.
explicit process. Thus,
given state according to the sampling
Target Examples
Initial
GAIL
RCB
SQIL
Goal(SAC) QurGSACCGGolKORRAAAOOeee
Cireling
Clustering
Cireling + Clustering
Figure 3: Starting from the same initial states on the far left, we demonstrate the rearrangement results of different methods. On the far right, we demonstrate some target examples of each task.

---

GAIL
RCE
SQIL
GSAl(SAC)
Ours(SAC)
Target Examples
Figure 4: Qualitative results on room rearrangement. We obtain eight rearrangement results of each method starting from the same of eight initial states. Then we demonstrate the rearrangement result closest to each method's 's target example.
realistic dataset, 3D-FRONT [46]. After cleaning the data, we get dataset with 839 rooms, 756 for training, e.g., target examples, and 83 for testing. In each episode, sample room from the split and shuffle the room via Brownian Process the beginning to get an initial state. The agent can assign angular and linear velocities for each object in the room at each time step. To tidy up the room, the agent must learn implicit knowledge from the target examples, as shown in Fig. 4. More details are in the Appendix A.
Room Rearrangement is built
5.2 Evaluation
For baseline comparison and ablation studies, collect fixed number of trajectories T {T²} starting from the same initial states for each of the random seeds. For each random seed, we collect 100 and 83 x 8 (8 initial states for each room) trajectories for ball and room rearrangement, respectively. Then we calculate the following metrics on the trajectories, where the PI. only reported in ball rearrangement. More details of evaluation in the Appendix D).
(PL) measures the efficiency the rearrangement process and the sample quality of the results. We specify pseudo-likelihond function F prozy S - I for each environment that measures the similarity between given state and the target examples. For each time step we report the mean PL over trajectories Exwr Proooyy (rozz((:)) and the confidence interval over random seeds. We do not report PL on room rearrangement as hard to program human preferences.
Coverage Score (CS) measures the diversity and fidelity of the rearrangement results (i.e.the terminal states ST (sT) of the trajectories) by calculating the
Seaa--MMaaiin--istaaee (MMD) [47] between S7 and a fixed set of examples Sgt from Ptar: Σ min
||s sr.l If the rearrangement
Bet Er
results of method miss some modes in the ground truth example set, the coverage will increase to hurt the performance.
Averaged Collision Number (ACN) reflects the safety and efficiency of the rearrangement pro- cess since object collision will lead to object blocking and deceleration. ACN is calculated as Eter Exsey Ct, where c denotes the total collision number time step t.
5.3 Baselines
compare our framework, iee. Ours (ORCA) and Ours (SAC), with planning- baselines and learning- based baselines. Besides, we design heuristic baselines as the upper bounds for ball
7

---

Oracle
Figure 5: Quantitative results on ball rearrangement. From top to down, we report pscudo-likelibood (PL), coverage score (CS), and averaged collision number (ACN). The PL curves and CS bars of each task are normalised by the oracle's performance of this task. For CS and ACN, we report the mean Ours (ORCA)
Circling
Clustering
Circling + Clustering
Goal (ORCA)
1.0
Ours (SAC)
Goal (SAC)
0.5
PL
GAIL (SAC)
0.0
SQIL
-0.5
Ours Residual
25
50
75
100
25
50
75
100
0
100
200
300
Time Step
Room Arrangement
2.0
CS
1
0.0
Ours (ORCA)
Goa (ORCA)
ours (SWC)
Goal (SAC)
Our Residual
GAIL (SAC)
SQL
RCE
Ours (ORCA)
Goal (ORCA)
Goal (SAC)
Durs Residus
Ours (SAS (SAG)
CAll (SAC)
SOL
ACE
Curs (ORCA)
Gea (SWC)
Gos (ONCA)
(SAC)
Ours wis nu
(SAC)
SQu
Dur we Residaal
(SAC)
GALL (SAC)
Goal (SAC)
ICE
SQL
was
4.0
ACN
Our wio Reabuul
Ours (ORCA
Gea (SAC)
Goal LORCAL
GAL (SAG
Duss w/s
RCE
SQL
Ours TORCH
Our whs Resiiull
Oues (SaC)
Goal (SAC)
Goal FORCAI
Durs wis
GAL (SAC)
RCE
sou
Ours tOnCc
our wil Resiual
Goal FORCAL
Goal (SAC)
Ours RL
GAL (SAC)
SoL
RCL
Gurs (SAC)
Goal (SAO
GAL (SA
50L
RCE
and confidence interval over five random seeds.
rearrangement tasks. All the reinforcement dearning-based methods share the same network design, capacity and hyperparameters. More details of baselines are in Appendix. C.
Leaning-basse based Baselines: GAIL: A classical inverse RL method that trains discriminator the reward function. RCE: A strong example-base reinforcement learning method proposed recently. SQII.: A competitive example-based imitation learning method. Goal-RI.: An 'open-loop' method that first generates goal at the beginning of the episode via a VAE trained from the target examples and then trains goal-conditioned policy to reach the proposed goal via RL.
Planning- Baselines: Goal (ORCA): This method first generates goal via the same VAE as in Goal (SAC). Then the agent assigns action pointing to the goal as reference control and adjusts the preference velocity using ORCA, similar to Sec. 4.3.
Heuristic-based Baseline: Oracle: We slightly perturb the samples from the target distribution and take these samples as the rearrangement results. This method's rearrangement results will be close to the target distribution yet different from the target examples. Hence, we use this method to normalise the PI. curve and CS bars.
Result Analysis
6.1 Baseline Comparison
Ball Rearrangement: present the quantitative results on ball rearrangement tasks in the first three columns of Fig. 5. As shown the top row of Fig. 5, the PL curves of Ours (SAC) and

---

(ORCA)
Circling
Clustering
Circling + Clustering
Goal (ORCA)
(SAC)
Goal (SAC)
PL
(SAC)
RCE
0
SQIL
Ours w/o Residual
Ours w/o RL
21
24
27
21
27
21
27
Number of Objects
Figure 6: Scalability Anaslysis: Each policy trained on environments with 3 x 7 balls and is zero-shot transferred to environments with different ball numbers.
Ours (ORCA) are significantly higher and steeper than all the baselines in all three tasks, which shows the effectiveness of our method in arranging objects towards the target distribution efficiently. In the middle row of Fig. 5, Ours (ORCA) achieves the best performance in CS across all three tasks. Goal (SAC) outperforms Ours (SAC) on Clustering and Circlin++Clustering since the goal is proposed by a generative model that can naturally generate diverse and realistic goals. From the qualitative results shown in Fig. 3, our methods, especially Ours (ORCA), also outperform the baselines in realism. Our methods also achieve the lowest ACN in most cases, except that Goal (SAC) is slightly better than Ours (SAC) in Circling.
The Goal (SAC) is the strongest baseline yet loses to our framework in efficiency since the generated goals are far from the initial state. We validate this problem by comparing the averaged path length of the Goal (SAC) with ours in Appendix. E.3. Results show that our methods converge in a shorter path than goal-based baselines across all three tasks. Besides, the Goal (SAC) suffers from invalid goal proposals. As shown in Fig. 3, the Goal (SAC) and Goal (ORCA) can rearrange objects in the right trend, but the objects may block each other's way. This is because the goals generated by the VAE are not guaranteed to be valid (e.g. the balls may overlap with each other), so the agent cannot reach the invalid goal. We demonstrate this problem Appendix. E.4.
The classifier- based reward learning method. GAIL, RCE, and SQIL, basically fails in most tasks, as shown in Fig. 3 and Fig. This due to the over-exploitation problem that commonly exists in classifier- based reward learning methods. We validate this in Appendix E.2.
Room Rearrangement: We only compare Ours (SAC) with caseeng-based baselines because it is infeasible to employ the ORCA planner in room rearrangement since all the furniture is non-circular. As shown in Fig. 4, our method can obtain more realistic rearrangement results than baselines. As shown in the last column of Fig. 5, the coverage score of Ours (SAC) is the lowest and almost half of that of Goal (SAC). Meanwhile, Ours (SAC) achieves the lowest ACN compared with baselines. These results prove our method can work on more complex task and generalise well to unseen conditions.
6.2 Ablation Studies and Analysis
We conduct ablation studies on ball rearrangement tasks to investigate: 1) The effectiveness of the exploration guidance provided from the target gradient field and 2) The necessity of combining the target gradient field with control algorithms to adapt to the environment dynamics. To this end, we construct an ablated version named Ours w/o Residual that drops the residual policy learning and another named Ours w/o RL that only takes gradient-based action a as policy.
The CS bars in Fig. 5 shows that Ours (SAC) significantly outperforms the Ours w/o Residual in CS, which means the agent is easy to over-exploit few modes with high density. We also show qualitative results in Appendix E that the objects are usually arranged into a single pattern without the residual policy learning.
Ours w/o Residual:
Ours w/o RL: The PL curves in Fig. 5 show that without the control algorithm serving as the backbone, the rearrangement efficiency of Ours w/o RL is significantly lower than Ours (SAC) and Ours (ORCA). Meanwhile, the ACN of Ours w/o RL is significantly larger than Ours (SAC) and Ours (ORCA), which indicates the gradient- based action may cause severe collisions without adapting to the environment dynamics.

---

Scalability: We further evaluate the scalability of our methods by zero-shot transferring the learned model to rearrange different numbers of objects. Specifically, we train all the policies in the ball environment with 3 x 7 21 balls. During the testing phase, we directly transfer each policy to environments with increased ball numbers, i.e., 3 8,
and 10, without any fine-tuning. In Fig. 6.2, we report the averaged PL. increment from each method's s initial to target states. Results show that our method still outperforms baselines under different numbers of balls.
7 Conclusion
In this study, we first analyse the challenges of object rearrangement without explicit goal specification and then incorporate target gradient field with either model-based planning or model-frec learning approaches to achieve this task. Experiments demonstrate our effectiveness by comparisons with learning-based and planning- based baselines.
Limitation and Future Works. This work only considers the planar simplified environments where the agent can directly control the velocities of objects. In future works, it is necessary to extend our framework to more realistic scenarios, egg., manipulating objects by robots in realistic 3D environment [11, 48]. We summarise the future directions in the following:
rearrangement with
Visual Observation: We may explore how to conduct efficient and accurate score-based reward estimation from image-based examples and leverage the recent progress on visual state representation learning [49, 50, 51, 52] to develop the visual gradient- based action.
object time, may explore hierarchical methods where high-level policy chooses which object to move and low-level policy leverages the gradient-based action for motion planning. When forces on objects instead of velocities, we may explore a hierarchical policy where a high- level policy outputs target velocities and a low-level employ velocity controller, such as PID controller, to mitigate the velocity errors.
Complex Dynamics: When the agent only
only impose
Multi-Agent Scenarios: The framework can also extended to multi-agent scenarios [19, 53,54, 55), where the decentralised agents need to estimate its gradient field and take actions according to the local observation for multi-agent cooperation or competition.
Ethics Statement. Our method has the potential to build home-assistant robots. We evaluate our method in simulated environments, which may introduce data bias. However, similar studies also have such general concerns. We do not see any possible major harm in our study.
Acknowledgement
This work supported by the National Natural Science Foundation of China Youthh Science Fund (60662000066: Learning Visual Prediction of Interactive Physical Scenes using Unlabelled Videos and China National Post- doctoral Program for Innovative Talents (Grant No. BX2021008). We also thank Yizhou Wang, Tianhao Wu and Yali Du for their insightful discussions.
References
[1] Yun Jiang, Marcus Lim, and Ashutosh Saxena. Learning object arrangements in 3d scenes using human context. In ICML, 2012.
[2] Nichola Abdo, Cyrill Stachniss, Luciano Spinello, and Wolfram Burgard. Robot, organize my shelves! tidying up objects by predicting user preferences. In 2015 IEEE International Conference Robotics and Automation (ICRA), pages 1557-1564. IEEE, 2015.
[3] Henry N Ricciuti. Object grouping and selective ordering behavior in infants 12 to 24 months old. Merrill-Palmer Quarterly of Behavior and Development, 11(2):129-148, 1965.
[4] Ivan Kapelyukh and Edward Johns. My house, my rules: Learning tidying preferences with graph neural networks. 5th Annual Conference on Robot Learning. 2021.
[5] Hanqing Wang, Wei Liang, and Lap-Fai Yu. Scene mover: Automatic move planning for scene arrangement by deep reinforcement learning. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.
10

---

[6]
Ju-Bong Kim, Ho-Bin Choi, Young Hwang, Kwihoon Kim, Yong- Beng-Geun Hong, and Youn- Hee Han. Sortation control using multi-agent deep reinforcement learning in n n-grid sortation system. Sensors, 20(12):340,, 2020.
[7] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages (094-1100. PMLR, 2020.
[8] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3357-3364. IEEE, 2017.
[9] Wenhan Luo, Peng Sun, Fangwei Zhong, Wei Liu, Tong Zhang, and Yizhou Wang. End-to- end active object tracking and its real-world deployment via reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(6):1317-1333 2019.
[10] B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions on Intelligent Transportation Systems, 2021.
[11] Yiming Zuo, Weichao Qiu, Lingxi Xie, Fangwei Zhong, Yizhou Wang, and Alan L Yuille. Craves: Controlling robotic with vision-based economic system. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4214 4223, 2019.
[12] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533. 2015.
[13] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 29:4565-453 2016.
[14] Siddharth Reddy, Anca D. Dragan, and Sergey Levine. (SQIL}: Imitation learning via reinforce- ment learning with sparse rewards. In International Conference on Learning Representations, 2020.
[15] Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning. arXiv preprini arXi::2004.09399, 2020.
[16] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88-97, 1991.
[17] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668. JMLR Workshop and Conference Proceedings, 2010.
[18] Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011.
[19] Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. Towards distraction- robust active visual tracking. In International Conference on Machine Learning, pages 12782- 12792. PMLR, 2021.
[20] Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for Information Processing Systems, 31, 2018.
reward definition. Advances in Neural
[21] Ben Eysenbach, Sergey Levine, and Russ R Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification. Advances in Neural Information Processing Systems, 34: 14:11554 115521 2021.
[22] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances Neural Information Processing Systems, 32, 2019.
[23] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[24] Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In International Conference on Machine Learning, pages 9558-95680 9568. PMLR, 2021.
11

---

[25] Shitong Luo and Wei Hu. Score-based point cloud denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4583-4592, 2021.
[26] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[27] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):661-16644 2011.
[28] Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual policy learning. arXiv preprint 1rxh::18/..06298, 2018.
[29] Jur van den Berg. Stephen J Guy. Ming Lin, and Dinesh Manocha. Reciprocal n-body collision avoidance. In Robotics Research, pages 3-19. Springer, 2011.
[30] Martin Schuster, Dominik Jain, Moritz Tenorth, and Michael Beetz. Learning organizational principles human environments. 2012 IEEE International Conference on Robotics and Automation, pages 3867-3874. IEEE, 2012.
[31] Nichola Abdo, Cyrill Stachniss, Luciano Spinello, and Wolfram Burgard. Organizing objects by predicting user preferences through collaborative filtering. The International Journal of Robotics Research, 35(13): (5(11))1187-111 1587-16888 2016
[32] Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan. Example-based synthesis of 3d object arrangements. ACM Transactions on Graphics (TOG), 31 31(6):1-11, 2012.
[33] Lap Fai Yu, Sai Kit Yeung, Chi Keung Tang, Demetri Terzopoulos, Tony F Chan, and Stanley J Osher. Make it home: automatic optimization of furniture arrangement. ACM Transactions Graphics (TOG)-Proceeddins of ACM SIGGRAPH 2011, v. 30,(4), July 2011, article no. 86, 30(4), 2011.
[34] Yi-Ting Yeh, Lingfeng Yang, Matthew Watson, Noah D Goodman, and Pat Hanrahan. Syn- thesizing open worlds with constraints using locally annealed reversible jump meme. ACM Transactions on Graphics (TOG), 3144::1-11, 2012.
[35] Ankur Handa, Viorica Păträucean, Simon Stent, and Roberto Cipolla. Scenenet: An annotated model generator for indoor scene understanding. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 5737-5743. IEEE, 2016.
[36] Siyuan Qi, Yixin Zhu, Siyuan Huang, Chenfanfu Jiang, and Song-Chun Zhu. Human-centric indoor scene synthesis using stochastic grammar. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5899-5908, 2018
[37] Dhruv Batra, Angel X Chang. Sonia Chernova, Andrew J Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv preprini arXiv:2011.01975, 2020.
[38] Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rear- rangement. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1992259311,000 2021.
[39] Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, An- drew Szot, and Harsh Agrawal. Housekeep: Tidying virtual households using commonsense reasoning, arXiv preprint arXhv:2205.107122 2022.
[40] Gabriel Sarch, Zhaoyuan Fang, Adam W Harley, Paul Schydlo, Michael J Tarr, Saurabh Gupta, and Katerina Fragkiadaki. Tidee: Tidying up novel rooms using visuo-semantic commonsense 2022.
priors. arXiv preprint 22Xi
[41] Aviv Netanyahu. Tianmin Shu. Joshua Tenenbaum, and Pulkit Agrawal. Discovering gener- alizable spatial goal representations via graph- based active reward learning. In International Conference on Machine Learning, pages 16480-16495. PMLR, 2022.
[42] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[43] Iyya Kostrikoy, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actorrritttc Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019.
12

---

[44] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pages 1438.1338. Chicago, IL. USA, 2008.
[45] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 1861-1870, PMLR, 10-15 Jul 2018.
[46] Huan Fu, Bowen Cai, Lin Gao, Ling- Ling-Xiao Zhang, Jiaming Wang. Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10933-
10942, 2021.
[47] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning repre- sentations and generative models for 3d point clouds. In International Conference on Machine Learning, pages 40-49. PMLR, 2018.
[48] Weichao Qiu, Fangwei Zhong, Yi Zhang. Siyuan Qiao, Zihao Xiao, Tae Soo Kim, Yizhou Wang, and Alan Yuille. Unrealev: Virtual worlds for computer vision. In ACM International Conference on Multimedia, pages 1221 1221-1224, 2017.
[49] Drew Linsley, Girik Malik, Junkyung Kim, Lakshmi Narasimhan Govindarajan, Ennio Mingolla, and Thomas Serre. Tracking without re-recognition in humans and machines. Advances in Neural Information Processing Systems, 34:19443-19486, 2021.
[50] Thomas Kipf, Elise der Pol, and Max Welling. Contrastive learning of structured world models. arXiv preprint 19I.i::911..2220 2019.
[51] Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn, Improving generative imagination in object-centric world models. In International Conference on Machine Learning, pages 61 6140-6149. PMLR, 2020.
[52] Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. Ad-vat+: An asymmetric dueling mechanism for learning and understanding visual active tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1467-14822 2019.
[53] Yuanfei Wang, Fangwei Zhong, Jing Xu, and Yizhou Wang. Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind. In International Conference on Learning Representations, 2022.
[54] Jing Xu, Fangwei Zhong, and Yizhou Wang. Learning multi-agent coordination for enhancing target coverage in directional sensor networks. Advances in Neural Information Processing Systems, 33:100533 -00044, 2020.
[55] Xuehai Pan, Mickel Liu, Fangwei Zhong, Yaodong Yang, Song- Chun Zhu, and Yizhou Wang. MATE: Benchmarking multi-agent reinforcement learning in distributed target coverage control. In shitt Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[56] Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin, Linxi Fan, Guanzhi Wang, Claudia Pérez-D' Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent Vainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. igibson 1.0: simulation environment for interactive tasks in large realistic scenes. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), page accepted. IEEE, 2021.
[57] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybuulet. org, 2016-2021
[58] Yang Song and Stefano Ermon. Improved techniques for training basee generative models. Advances in Neural Information Processing Systems, 13:32888-2 2020.
[59] Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https: //gi thub. cww/deni htam///enssarratspttorhhh. 2020.
2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] see Sec.
(b) Did you describe the limitations of your work? [Yes] see Sec. 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see
supplementary
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] see Sec. 7 and supplementary
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] please see the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] more details in the supplemental material.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] more details in the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] more details in the supplemental material.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're rou're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "c7-lJcUukr3", "reviewer": "Reviewer_ZnF9", "summary": "## Post Rebuttal\n\nI thank the authors for their response. I have increased my score.\n\n## Pre Rebuttal\n\nThis paper attempts to tackle the problem of learning to rearrange a set of objects to a \"sensible\" state without a pre-defined reward function. Towards this end, they propose Target Gradient Fields (TarGF). TarGF estimates the gradient of the likelihood that the current environment state is a goal (or \"sensible\") state for the objects.\n\nTarGF is learned by randomly permuting correct arrangements.\n\nThe output of TarGF is used to estimate the best action to take at every timestep and/or used to estimate the reward.\n\nThe method evaluated in two scenarios. In the first a set of balls must be rearranged into either a circle, clusters by color, or a combination. In the second the furniture in rooms must be rearranged back to its initial states.", "questions": "Is (1) a well-defined optimization problem? For any non-trivial starting state, won't p_tar(s_0) always be negative infinity, so the discounted sum will always be negative infinity\n\nComments:\nRearrangement should be used instead of arrangement to be inline with what the community is calling this task. See Batra et. al. 2020.", "limitations": "Limitations are addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe proposed method is able to learn solely from examples and does not require any additional data other than correct examples (negatives are generated by adding noise to the correct examples).\n\nIn the circle scenario, the proposed method outperforms various baselines across a series of metrics -- namely, collision count and success.\n\n# Weaknesses\n\nThe evaluation is very simple. There isn't an agent that rearranges these objects, instead the objects all rearrange themselves. This is a good initial evaluation that shows the method provides a reasonable reward signal, but it leaves a lot of questions since this reduces the time-horizon, isn't representative of how the objects would be moved by a single agent (one object moves at a time), and removes the initial exploration to find the objects. An evaluation that is closer to a real scenario would be helpful.\n\nIdeally, the authors would evaluate on something like HAB (Szot et al 2021), or AI2Thor rearrangement.\n\nThe residual policy derived from the gradient-based action seems quite important to overall performance of the model free approach. However, this residual policy requires an oracle entity-based representation of the state -- requires all object positions, object categories, and bounding boxes. While it is fine to assume this at training time, this is also required at evaluation time. I am concerned by the assumption that such a representation is available as the task complexity increases.\n\nOn this topic, it is unclear if the baselines receive the same information. The supplement says they operate on state, but state is defined just as object position, not position + category + bounding box, in the paper.\n\nFinally, in the most realistic scenario, Room Arrangement, the proposed method likely doesn't outperform GAIL + SAC by a statistically significant margin. Can the authors report confidence intervals instead of variance?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9hBcUj-YNF", "reviewer": "Reviewer_eUMp", "summary": "This work introduces a novel approach to tackling object rearrangement. The main idea is to learn a scoring function in the form of a gradient field. The learned gradient field could be used by a model-based planning algorithm to find collision-free arrangement plans or as a reward function to train RL agents. Experiment results show promising results of the proposed method for rearranging a room and sample efficient policy learning.", "questions": "1. When talking about collision-free room arrangement, do we consider the 2d rotation of the furniture during the movements? Basically, what’s the action space, and why it is hard compared to the ball arrangement?\n2. Did you test the generalization of the proposed method? e.g. Testing in an unseen room with novel furniture.", "limitations": "1. Might fail to handle multi-model goal distributions.\n2. Missing some details of the experimental setups in the main paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is well-written with a clear introduction of the implementation details.\n2. I really enjoy reading the paper following the motivation of how the learned gradient field could be used to tackle three major challenges of object arrangement, which is easy to follow and makes a lot of sense.\n3. The velocity-based action space of the designed task showcases that the proposed method could be used in complex continuous control scenarios. The discussion of collision avoidance planning with ORCA further strengthens the practicality of the proposed approach in real-world tasks.\n4. It is impressive to see such a gradient-based scoring function could be used as a reward function for training the RL agent. Also, the gradient-based action could further alleviate the burden of the action generation network so that only a small residual policy network needs to be trained, which largely improves the sample efficiency of the policy learning process. \n\nConcerns:\n1. How does the RL residual policy learn to avoid collision? As mentioned in the model-based planning method, the gradient field does not have environment information so the gradient cannot avoid collision. Since the residual policy is based on gradient-based action, does it need an additional collision penalty reward during training? In the paper, you mentioned the centralized and decentralized reward but more details here would be better.\n2. The gradient field might struggle to handle multi-model distributions. What if there are multiple arrangement goals for the same environment. Will the gradient become a mean of different distributions which will mislead the planner or the policy learning? I feel a probabilistic model would make more sense for this situation.\n3. The authors mentioned that the room-arrangement task is too complex for planning-based methods. I would suggest trying with bi-level planning which first plans high-level arrangements plans with the learned gradient and then use a collision-avoidance trajectory planner to optimize the path to avoid the collision.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3CZzxXMp7vG", "reviewer": "Reviewer_NBX4", "summary": "This paper learns a reward function / score function for objective rearrangement tasks, making use of recent advancements in score-based generative modeling. The authors term this score function a “target gradient field”, and show that it can be used with either a path planner or a reinforcement learning algorithm to solve object rearrangement tasks in a toy setting (i.e. where you can directly control object velocities). The authors compare against a number of baselines and competing methods in these simulated object rearrangement domains, and show that their method performs well across several different metrics. \n", "questions": "Some questions\n\n- “learning a score function to qualify a state” → what does “qualify” mean here? Can this sentence be re-written to make it less ambiguous? \n- How do you ensure collision avoidance when using model-free RL? Or is collision avoidance not pursued in this setting? \n", "limitations": "I don't think the authors have done a good job of addressing the limitation of the work. In the limitations section, the only limitation that the authors mentioned is the fact that their work only considered planar settings, and not 3D settings. There is only a very brief mention of the fact that future work could use real robots. I think the authors should provide a more detailed limitations section (by reducing space used elsewhere). Some ideas for limitations can be seen in the weaknesses section above. A somewhat more comprehensive plan on how the method could be applied to more realistic settings would also be useful. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Learning reward functions is an interesting application of score-based generative modeling, and has not been explored before, as far as I know. \n- Proposed approach performs well against other reward learning approaches on the object-rearrangement task. \n\nWeaknesses\n- The paper is somewhat limited in scope -- it is only applied to a very specific robotics problem (that of object rearrangement), and even here some major simplifying assumptions had to be made (such as the fact you can directly control the velocity of any object. \n- The paper only shows results in low-dimensional domains (small graphs). Since score-based generative modeling also works in high-dimensional domains (such as images), it would be interesting to see if the method can be used for reward learning from scene images, for example. \n- While the paper is nicely structured, it contains a large number of typos and grammatical errors. These errors must be fixed before the paper is ready for publication. I will provide a non-exhaustive list here, but I would also suggest making useful of professional proof-reading services if possible. \n\nTypos and grammatical errors:\n- “We do also tries to find a example-based control method”\n- “Differently, our proposed approach focus on learning the target gradient fields from the examples.”\n- “Besides, both RL method and traditional planner can be supported by our target gradient fields in the object arrangement task.”\n- “even if we are accessible to the target distribution”\n- “In address these problems”\n- \"reducing the risk of objects being collided”\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept