Example-based Planning via Dual Gradient Fields
Authors: Mingdong Wu
fangwei zhong
Yizhou Wang
Hao Dong
Venue: Submitted to ICLR 2023 (2023)
Keywords: Example-based Planning
Score-Matching
Path Planning
Reinforcement Learning

Abstract: Path planning is one of the key abilities of an intelligent agent. However, both the learning-based and sample-based planners remain to require explicitly defining the task by manually designing the reward function or optimisation objectives, which limits the scope of implementation. Formulating the path planning problem from a new perspective, Example-based planning is to find the most efficient path to increase the likelihood of the target distribution by giving a set of target examples. In this work, we introduce Dual Gradient Fields (DualGFs), an offline-learning example-based planning framework built upon score matching. There are two gradient fields in DualGFs: a target gradient field that guides task completion and a support gradient field that ensures moving with environmental constraints. In the learning process, instead of interacting with the environment, the agents are trained with two offline examples, i.e., the target gradients and support gradients are trained by target examples and support examples, respectively. The support examples are randomly sampled from free space, e.g., states without collisions. DualGF is a weighted mixture of the two fields, combining the merits of the two fields together. To update the mixing ratio adaptively, we further propose a fields-balancing mechanism based on Lagrangian-Relaxation. Experimental results across four tasks (navigation, tracking, particle rearrangement, and room rearrangement) demonstrate the scalability and effectiveness of our method.

Under review as a conference paper at ICLR 2023
EXAMPLE-BASED PLANNING VIA DUAL GRADIENT FIELDS
Anonymous authors Paper under double-blind review
ABSTRACT
Path planning is one of the key abilities of an intelligent agent. However, both the learning-basee and sample-based planners remain to require explicitly defin- ing the task by manually designing the reward function or optimisation objec- tives, which limits the scope of implementation. Formulating the path planning problem from new perspective, Exumple-buse planning is to find the most ef- ficient path to increase the likelihood of the target distribution by giving set target examples. In this work, we introduce Dual Gradient Fields (DualGFs), offline-learing example-based planning framework built upon score match- ing. There are two gradient fields in DualGFs: a target gradient field that guides task completion and support gradient field that ensures moving with physi- cal constraints. In the learning process, instead of interacting with the environ- ment, the agents are trained with two offline examples, i.e., the target gradients and support gradients are trained by target examples and support examples, re- spectively. The support examples are randomly sampled from free space, i.e.. states without collisions. DualGF is weighted mixture of the two fields, com- bining the merits of the two fields together. To update the mixing ratio adap- tively, we further propose fields-balancing mechanism based on Lagrangian- Relaxation. Experimental results across four tasks (navigation, tracking, par- ticle rearrangement, and room rearrangement) demonstrate the scalability and effectiveness our method. Our codes and demonstrations can be found at attps://sites: google wawpgge..cm/vee. wwm/viewiiivooeeeeoomeoeemee.e.e dualgf.
INTRODUCTION
Planning paths to reach goal is fundamental
function of an intelligent agent (Russell, 2010)
and has wide range of real-world applica-
tions, such as navigation (Patle et al., 2019), object tracking (Zhong al., 2019), and ob- ject rearrangement (King et al., 2016). Ex- isting planning algorithms, whether sampling- based (LaValle et al., 1998a; Karaman & Fraz- zoli, 2011) learning-based (Kulathunga, 2021; Yu et al., 2020; Tamar et al., 2016), need ex- hausted test-time sampling for searching path or reward functions for learning. This severely limits the implementation scope of plamning since many real-world tasks are hard to design the ob- jectives/ reward with human priors, egg., tidying up house, rearranging desktop.
Target Examplle es
D Dual Cradisnt Fields
Support: Examples
Planning
Figure 1:
Our task setting. Left: The agent
learns task specification from target examples and physical constraints from support examples during training. Right: The agent plans path in novel conditions during the test phase.
In this paper, we consider a novel data-driven planning paradigm: Example-base planning, which the user specify the task by providing set of target examples, rather than programming task- specific objective. Benefiting from such paradigm, example-base planning can scale to various tasks, particularly tasks with implicit goals, i.e. .e..pecify the task with target distribution instead of specific goal state. Besides, the agent needs to infer the environmental constraints to safely move in physical world. Previous approaches either learn physical constraints from interacting with the

---

Under review as a conference paper at ICLR 2023
environment and collision penalty (Wu et al., 2022), offline demonstrations (Janner et al., 2022), or exhaustively sampling points at test time (LaValle et al., 1998a). However, online interaction is costly and unsafe, while offline demonstrations are expensive to collect and test-time sampling is time inefficient.
To this end, we propose fully example-based planning framework that learns two gradient fields with different purposes from examples by score-matching (Vincent, 201 1), namely DualGF. DualGF consists of two fields: A larget gradient field and a support gradient field. The target gradient field estimates the gradient of the target distribution so as to provide the fastest direction to accomplish the task. The support gradient field learns to reverse the perturbed state back to the free space so as to help avoid collisions. To combine the merits of the two fields, we further introduce a gradient mixer to adaptive balance the trade-off between the two gradients (keep safe vs. reach goal) when constructing the dual gradient field in execution. In practice, we can also incorporate the dual gradient field with a low-level controller to output primitive actions in control. The two gradients are trained from two sets of examples, respectively. For the target gradient field, we collect a set of target states sampled from the target distribution, such as a set of tidied rooms. For the support gradient field, provide the agent with another set of examples (support examples) that are uniformly sampled from the free space, i.e., states without collision. The support examples abundant and easy to obtain in real scenarios, e.g., randomly initialised objects, which largely alleviate the safety issue of learning from interactions. As illustrated in Fig. 1, the agent can learn generalisable inference of task specification and physical constraints from target and support examples and planning in unseen environment.
Our experiments validate the generalisation ability of the DualGF planning framework across a va- riety of tasks, including classical planning tasks such as navigation, tracking, and planning tasks without explicit goal specification such as object rearrangement (Wu al., 2022). Specifically, the proposed DualGF significantly outperforms the learning- based baselines in planning perfor- mance and efficiency while achieving comparable performance with reference approaches that use the ground truth model or test-time sampling. Ablation studies also demonstrate the effectiveness of the proposed support gradient field and field-balanceo mechanism.
In conclusion, our contributions are summarised as follows: a) We reformulate the path planning problem in data-driven paradigm, where we specify the tasks with examples rather than manu- ally design objectives; b) We propose novel score-based planning framework DualGF that can adaptively integrate two gradient fields trained from different example sets so as to output instruc- tions to complete task; c) We conduct experiments in four tasks to demonstrate the scalability of our method, and empirical results show that DualGF significantly outperforms the state-of-the-art methods in efficiency and safety.
2 RELATED WORK
Learning from Demonstration. Example-based planning can be viewed as a special case of Learn- ing from demonstration (LfD). LiD is long-studied studied problem aiming at learning policy from only a set of expert trajectories. There are two mainstreams of LiD: Behavioural Cloning (BC) (Pomer- leau, 1991; Ross & Bagnell, 2010; Ross et al., 201 1), which learns policy in a supervised manner; and Inverse Reinforcement Learning (IRL) (Fu et al., 2017; Liu et al., 2020; Kostrikov et al., 2018; Ziebart et al., 2008), which finds a cost function under which the expert is uniquely optimal. Differ- ent from LID, we train the agent from only two sets of examples instead of whole demonstrations.
There are some RL-based methods that consider an example-based setting: VICE (Fu et al., 2018) is similar to AIRL (Fu et al., 2017), but is designed for learning from set of success examples. SQIL (Reddy et al., 2019) is a method modified from SAC (Haarnoja et al., 2018) that labels success examples with reward of +1. RCE (Eysenbach et al., 2021), modification of the actor-critic-bast based method (Fujimoto et al., 2018; Haarnoja et al., 2018), directly learns a value function from transi- tions and successful examples. Recently, Offline- RCE (Hatch et al., 2022) extends RCE to an offline setting. Different from these methods, our method requires neither interaction with the environment nor offline demonstrations.
Path Planning. Sampling- based planning algorithms, such as the probabilistic roadmap method (PRM, (Kavraki & Latombe, 1994)) and the rapidly exploring random tree (RRT, (LaValle et al., 1998b)), are dominant in traditional path planning algorithms. PRT and its variants: (Dobson & Bekris, 2014); (You et al., 2021); (Chai et al., 2022); (Hüppi et al., 2022), intend to generate
2

---

Under review as a conference paper at ICLR 2023
roadmap and find collision-free path from starting point to goal region the roadmap. RRT and its variants: (Strub & Gammell, 2020); (Gammell et al., 2020) (Li et al., 2022); (Strub & Gammell, 2022), iteratively build tree expanding towards the random sample point instead of directly connecting to the sample point. Different from these methods, our approach enables path planning in end-to-end paradigm without test-time sampling, which is more time-efficient and scalable to high-dimensional tasks. Our method is also similar to artificial potential field Lee & Park (2003) which manually designs an attractive field guiding the agent the goal location and repulsive field pushing the agent away from the obstacles. Differently, our method learns two fields from two sets of examples that can generalise to unseen conditions. Besides, our method can plan with visual observation (See Appendix. D.4) and scale up to high-dimensional domains.
3 PROBLEM STATEMENT
The example-based planning can be described by a tuple (S, St, Ptar, P. where S denotes the state space (e.g.. configurations of objects and obstacles) while Sj S denotes the free space (the valid states without object collision or transboundary).. At each time step t, our planning agent S - AA first outputs target state within epsilon-ball of the input state, L.e,, ||s sull S Then a simple low-level controller (e.g., PID controller) outputs actions to achieve this state a, = C(s,, 81+)) During training, the agent is given of target examples Star {s* (s*,"" Ptar where Ptar S + Ft denotes target distribution. Similar to (Wu et al., 2022), the goal of the planning agent find the most efficient path to increase the likelihood of the target distribution:
arg max E p(sa)) F
Σ ET
s.t. Ass, S
To this end, the agent needs to infer the task specification, ie.. knowing where to move can probably succeed, well the free space, i.e., avoiding collisions between objects and obstacles.
In this work, we consider the offline setting of example-based planning, where the agent has to infer physics constraints from support examples instead of interacting with the environment. The support examples are uniformly sampled from the free space Szup
{s""
"s" (s)} where
Paup (s)
|Syl" 0,
S SE
18y1
f ds S,
otherwise
The rationale for 'replacing interactions with examples' is that the support examples are enough for inferring the free space S while the online interactions are unsafe and costly. Besides, the agent should learn generalisable inference of free space instead of overfitting to specific environment.
Plar (s)
Motching
(Par
Targel Grudient
Mixer
Thaget Examples
Heuristic
Paup (s) J...
Masching
St
IP
sup
Support Gradient
Dual Gradient
Enviromment
Suppori Examples
(i) Training
(b) Inference
Target Gradient Nec
Support Gradient Not
Gracleent
Target
Gradient
Support
Gradient
Mixer
Field Balancing
Mechanism
Figure 2: Method Overview. a): target and support score network are trained from examples via score-matching objective. b): DualGF computes target and support gradient at each time step, where the target gradient guides the task completion (i.e., move objects to tidy layout) and the support gradient directs the objects avoid collisions. DualGF adaptively mixes the two gradients via field-bale balanaann mechanism to output the dual gradient. Incorporated with a low-level controller, DualGF can interpret the dual gradient into action.
3

---

Under review as a conference paper at ICLR 2023
4 METHODOLOGY
Overview: To tackle example-based planning, our key idea is to learn gradient fields from examples. As illustrated in Fig. 2 (a), the targer gradient field &%, and the support gradient field & is trained from the target examples and support examples via score-matching respectively. The target gradient guides the task completion by providing the fastest direction to increase the task likelihood. The support gradient helps avoid collision by pointing to the inner of the free space. Finally, we construct a policy that plans the next state based on the mixture of the two gradients, i.e. the dual gradient. In the control scenario, a heuristic- based controller can be used to interpret the dual gradient into low-level action.
4.1 PLANNING WITH THE DUAL GRADIENT FIELDS
We tackle example-bar based planning by greedily maximising the target likelihood. Firstly, the agent aims searching for the next state that maximises the target likelihood from the free space:
argmax log Ptar s.t. log Poup (84+1) 2 log
(3)
|S
S(+1
However, this objective is problematic since both the log Psup (8++1) and log Ptar (8141) are zero in many regions, such non-target region inner of the free space. Besides, Peup (Sc4-1 is non- differentiable on the boundary of the free space aSy. As a result, we choose to 'soften' both distri- butions by adding a small Gaussian noise:
Peup (S) -
- ('Psssss Piar (S) (6)
(4)
In this way, we replace the objective in Eq. with the following:
argmax log Plar (St41), s.t. log (St+1) 2
(5)
S1-1
where e E R is threshold indicating the conservative degree of the policy. The problem Eq. 5 can be further related to an unconstrained optimisation with penalty term by the Lagrangian method:
min log Pia. (s + À (log Pa (S(+1) cc) A20
max
where À 0 is known as Lagrangian multiplier. The $ and À can be updated iteratively for multiple steps to obtain a solution. Here, we choose to update the 8 and À only one step for Eq. since the s is close to s, in our problem, iee, ||s s:ll2 e.:
8 St + Me (V log (st) + V log log A411 ReLU (A+ ya (log Poup (sc+1) c))
where M4 and Ya two step sizes. Empirically, the latter is set as a fixed scalar μュ the former properly set to ensure ||s sel12 S c. this way, our planning policy is derived as:
0.01 while
Edual
1 St
|gduat l|2
Edual
Star + Augsup
dual gradient
gtar
Vg log plar (St),
gsup
V.
log poup (st)
target gradient
support gradient
Intuitively, the state updated by the mixture of the target gradient V. (or and the support gradient V, loggr (s.) weighted by λt. As shown in Fig. 3, the target gradient is a vector field that points to the goal region, so as to guide the task completion. The support gradient fields point to the inner region of the free space. The closer state is to the boundary of the free space, the larger the magnitudes of its support gradient will 'push' it into the inner region. In Appendix. D.9, we also derive the Mosed-form of the target and support gradients and discuss the above insights.
4

---

Under review as a conference paper at ICLR 2023
Condition St
Target Gradient Field sup (st)
Support Gradient Field tar (St)
Dual Gradient Field ① taar (5,) + d sup (5t)
Figure 3: Visualisation of the learned target, support and dual gradient fields on Navigation task. The L2-norm of support gradients small the free space but grows significantly faster when close to the boundary. As result, the direction of dual gradients is consistent with the target gradient in the free space and consistent with the support gradient near the boundary.
4.2 UPDATING MIXING RATE VIA SUPPORT GRADIENT FIELD
From Eq. 7, we can infer that X is updated to maintain the 'safety level' log at a certain threshold c, since larger Å will lead to an increase of log Psup(s) on the Further states, vice versa.
However, updating A requires the exact computation log (Spp(((t))) - e which involves inte- gration over the state space as shown in Eq. This is intractable when the state space is high- dimensional. Thus, we seek to develop a practical field-balancing mechanism. We first approximate log (S441) from log (So) via first-order Taylor expansion:
log (S
log (So)) log Poup (S++1) log Poupp)))
(9)
(c log Po (sn)
log (s)) (Sx), Sk+1 8s))
k=0
We further assume that So is always initialised far away from the boundary of the free space so that Varsorsp(s) log (s)] s 0. Thus, the remaining term c- log Paup(()) (Sn) can be regarded as constant c' in the sense of expectation:
E B [c- log po ( (so)] =
Eo(s) Dlg (s) ((pl) def
(10)
4.3 INCORPORATING DUALGE WITH LOW-LEVEL CONTROLLER
Under the assumption that the system is holonomic, DualGF can also tackle the control tasks in- corporated with low-level controller C: x S + A. For instance, if the atomic action space is velocity (with speed limit Umaz and simulation duration At), we can set e Umaz At in Eq. 7 and cast the state change (st+1, 8,) into target velocity via heuristic-based controller:
a C(s St)
Sp-11
Umax
gduul
(11)
At
For the non-linear dynamics where the action space is forces imposed on the objects, we can leverage a PID controller to achieve the target velocity in multiple steps. In this work, we focus on the effectiveness of the high-level module trained under the fully example-based setting, i.e., the dual gradient fields, instead of the complexity of the dynamics. Hence, we conduct experiments on tasks with velocity-basee based action space (e.g., linear angular velocity).
5

---

Under review as a conference paper at ICLR 2023
The whole planning framework is summarised below:
Si+1 5t
+
gduail12
Edual
Sdual
logPlar de V. log Piuu Pisu((s) (5:)
log (sx). Sk+1 (4) -c)
A++1 ReLU
de MA
Σ
(12)
Umna
a, C(s(t1, ggduall22
Edual
4.4 LEARNING GRADIENT FIELDS FROM EXAMPLES
Finally, we seek to estimate the target and support gradient fields mentioned above to realise our planning framework. Thanks to score- basee generative modelling, obtain guaranteed esti- mation of these gradient fields from examples.
The Denoising ing the score function of a data distribution V. log (d)ta (s). Given sample set {s" ~ Pdtta(s) ) DSM pre-specifies a noise distribution go (5]s), e.g., (O o² I), and trains score network Do to denoise the perturbed data samples:
Preliminary:
(DSM) proposed by (Vincent, 2011) aims at estimat-
(()) Eg p(acss))
[|
(13)
This objective guarantees optimal score network satisfies $5(() (s)r(s) almost surely. When is small enough, we have Vayo (s) Re V log Paata (s), so that ()(s) R V. Pogaaatass))
Rdd and support score network $ sup Rd. -> Rd from target examples Star and support examples S respec- tively, where d, denotes the dimension of agent-state space (e.g., agent's position and orientation) and de denotes the dimension conditional-ste space (e.g., agent's category, obstacles' state).
Training:
Adopting DSM, we train target score network a
tar Rdo+de
With 8, Rda and s, Rd denoted as agent-state and conditional-stie respectively (s [S,, 8c))
the training objectives of both score networks are as follows:
(()) = Egg (Ba (so) (sar((s)
| tur (Ba, scl)
Sg
Sa
(14)
Sn S
[(6) = Eg Gr. (x.) Peup
| sup (Ba, scl)
We adopt a variant of DSM (Song et al., 2020) that conducts DSM under different noise scales simultaneously. We defer network architecture and key hyperparameters to Appendix. C.I.
Patternl
Pattern2
Pattern3
Navigation
(b) Tracking
(c) Ball Rearrangement
(d) Room Rearrangement
Figure 4: Examples of four tasks used in the experiments. Navigation and Tracking: to reach/track goal location while avoiding collision with obstacles. Ball Rearrangement: rearrange balls into certain patterns. Room rearrangement: to rearrange the furniture into a reasonable layout.

---

Under review as a conference paper at ICLR 2023
Cirelor
Choiriy
Cirlay Chiseey
1
Seatt
644
ALB
Time Step
Arraneement
40-
02
0.9
600
15.0
ACV
15.0
644
64
Recharce
040
Figure 5: Comparative Results Ball Rearrangement and Room Rearrangement. The results Ball Rearrangement are plotted in the first three columns. The CS and ACN results on Room Rear- rangement are plotted in the last column. The order of the X-axis is fixed across CS and ACN bars.
EXPERIMENT SETUPS
5.1
TASKS
We evaluate method three types of tasks with increasing difficulties. Navigation and Tracking: Classical path planning tasks with explicit task rewards. The agent needs to reach or tack a goal location while avoiding collision with the obstacles. The size and location of the obstacles changes in each episode. Ball Rearrangement: The agent needs to infer the pattern priors from different target examples and efficiently rearrange balls into different types of patterns while covering all the modes of each pattern over the trajectories. Room Rearrangement: The agent learns arrangement priors from the target examples and rearranges the furniture unseen layouts to cover the ground truth layout. The task demonstrations are illustrated in Fig. 4. We defer detailed state and action spaces, target and support examples, pseudo-likelihood function and more visualisations of the tasks to Appendix A.
5.2 EVALUATION
We collect trajectories over 5 random seeds for evaluation. For each task, we collect trajectories starting from the same initial states. We calculate the following metries on the trajectories (for more details, we defer to Appendix. B): Task Return weighted by Success (TRS) calculates the aver- aged cumulative reward of episode: E[1 is success) Ee,r re] where r; denotes the immediate reward and (( is success) = 1 only when has no collision. Success Rate (SR) calculates the per- centage of non-collision trajectories. given state and a target distribution by assigning each time step I, the PL-curve reports the averaged PL across all trajectories EE (S age Score (CS): reports the Minimal-Matching- Miii--Matciiinn--ianeeiiiiiiii Distance(MMD) (Achlioptas al., 2018) between
Sreudo-Likelihode (PL) measures the similarity between function F Frrozy SS + R+ At
(sr)). Cover-
terminal states of method Sy and fixed set of examples Syt from Ptari
Σ min || ST
Eyr SrEST
Averaged Collision Number (ACN) reports the averaged collision number E[c,] at each time step where Σ o of only if the i-th and j-th object collide and ot otherwise.

---

Under review as a conference paper at ICLR 2023
Table 1: Comparative results on Navigation and Tracking. The obstacles is always fixed in Naviga- tion (Static) while changes for each episode in Navigation (Dynamic).
Navigation (Static) SR
Navigation (Dynamic)
Tracking
TRS 1
TRS
SR t
TRS 1
SK 1
RL (SAC)
9.1 =
0.21
0.04
8.3 0.4
0.19 = 0.01
0.0 0.0
0.21 + 0.02
GAIL
1.5 1 0.4
0.35
0.01
1.6 1.0
0.20 1 0.02
0.1 0.1
0.18 1 0.04
RCE
2.2 2.2+12 1.2
0.13 + 0.01
2.2
0.22 + 0.04
0.1 0.1+000
0.17 + 0.03
Learning Based
TarGF (SAC)
7.0 0.2
0.19 + 0.02
4.4
0.19 0.01
0.1
0.18 + 0.04
DualGF (wo Sup)
0.0 1 0.0
0.07 1: 0.01
0,9 0.1
0.09 1 0.03
0.0
0.11 1 0.04
DualGF
30.0 + 1.5
0.37 + 0.02
26.2 + 6.5
0.39 = 0.05
3.1
0.56 + 0.03
Planning Based
PRM (Reference)
37.8 + 3.6
0.45 0.04
39.5 + 3.3
0.47 + 0.04
4.1 4.1++00 0.9
0.16 + 0.04
Potential Field
34.8 = 0.8
0.45 + 0.00
23.2 + 5.6
0.28 = 0.07
2.2 2 0.6
0.09 + 0.02
0.0
Lamblat0 m)
Lambatt
Lambdad
APL fo.1
AACN
Lamblat)
Lanbdac2
Lamblai2 SI
0.0
-6.0
Lumbiay
Time Step
Figure 6: Ablation results on Clustering, where AMetric Metric(DualGF) Metric(Ablated), With mixing rate Åo decreasing, the advantage of DualGF over the ablation increases significantly.
The primary metries for each task: TRS and SR for Navigation and Tracking; PL and ACN for Ball Rearrangement; CS and ACN for Room Rearrangement.
5.3 BASELINES
We compare our framework (i.e., DualGF) with the following learning-based baselines and planning-based baselines. The implementation details are deferred Appendix. C.2. Leaning- based Baselines: TarGF(SAC): is the learning-based framework of TarGF (Wu et al., 2022). GAIL; A classical inverse RL method that trains a discriminator as the reward function. This method trains two classifiers from target and support examples, respectively. RCE: The SOTA example- based reinforcement learning method. RL(SAC): The agent is trained under the ground-truth task ward and collision signals. Planning- Baselines: TORGF(ORCA): A model-based framework that requires the ground-truth model. Probabilistic Road Map (PRM): A classical sampling-basee method Kavraki & Latombe ( (1994). Potentential Field: Artificial potential field method Lee & Park
(2003). Oracle: The oracle performance is obtained by slightly perturbing the target examples with small Gaussian noise
EXPERIMENT RESULTS
6.1 BASELINE COMPARISON
As seen from the quantitative result in Tab. and Fig. 5, DualGF significantly outperforms baselines and achieves comparable performance with reference methods such TarGF(ORCA) and PRM:
In Navigation and Tracking: 1) DualGF outperforms learning-based baselines by a large margin. The classical planning algorithm PRM achieves strong results in task reward (TR) since can search for a near-optimal path without any collision to reach the goal in most cases. However, even if we have provided an exhaustive sampling budget (i.e., 2000) for PRM in each episode, PRM still fails to search for a solution in some cases. As a result, DualGF outperforms PRM in ACN on Navigation (Static) and Tracking due to the failure cases of PRM. 2) Notably, the PRM is of poor time efficiency compared with DualGF. As shown in Tab. 2, PRM takes about 6 times of DualGF in Navigation and 1 110 times in Tracking since PRM needs to search a path again when the agent reaches the goal. 3) The Navigation and Tracking is non-trivial. The vanilla reinforcement learning approach RL(SAC) ranks second in the simplest task Navigation (Static), yet it fails to keep the advantage in more challenging tasks, ie.,, Navigation (Dynamic) and Tracking.
8

---

Under review as a conference paper at ICLR 2023
In Ball Rearrangement: 1) As Fig. 5 illustrated, DualGF achieves the best perfor- mance compared with learning-bos based baselines in PL and ACN. In the hardest task Cir- cling + Chustering, DualGF even outperforms than TarGF(ORCA) that plans with the ground truth model. These demonstrate the effectiveness of our method to scale to high- dimensional tasks. 2) The CS of DualGF is comparable with other competitive base- lines such as TarGF(SAC). Hence, our method is not trivially overfitting to single mode. 3) The classifier-based methods (i.e., GAIL and RCE) fail due to the training collapse of the classifier. In practice, we observe that classifier- based baselines achieve appeal- ing results during the training process yet fail to converge at pigh-peerormmnce level due to the over-exploitation of the classifier.
Milliseconds per Step Tracking
Navigation
DualGF PRM
3.3 + 0.2
3.4 + 0.2
19.7 + 2.3
376 + 35,9
In Room Rearrangement: 1) DualGF achieves the best
The inference time per step av- eraged over 100 trajectories.
Table 2:
performance compared with baselines in both CS and ACN. This indicates that our method can generalise well to the diverse conditional attributes such as different sizes, numbers and composition of objects. 2) The planning-base based methods either suffer from scalability issues that PRM and Random Trees (RRT) Valle al., 1998a) cannot search for feasible solution in 10 minutes. Besides, TarGF(ORCA) ) assumes the agent is circular in shape which is infeasible in Room Rear- rangemens. Hence, we do not compare them Room Rearrangement.
6.2 ABLATION STUDIES AND ANALYSIS
We further conduct ablation studies to analyse the effectiveness of the key components of our method, i.e., the support gradient field and the field-balancing mechanism. Further ablations on key hyperparameters (i.e., initial and c²), size of the target and support sets and the choice of the noise level are deferred to Appendix D.I, D.2 and D.3 respectively.
Effectiveness of the Support Gradient Field: Ours wo Sup is the ablated version that fixes the mixing rate to zero À = O. 1) The task performance of DualGF wo Sup is worse than DualGF in all tasks across both Navigation and Tracking and Ball Rearrangement. Besides, the ACN of DualGF is significantly lower than DualGF wo Sup These results indicate that the support gradient field can help avoid collision effectively and further improve task performance. 2) In Room Rearrungeeent, DualGF wo Sup achieves better CS than DualGF, yet performs worse than basclines in ACN. Mean- while, DualGF outperforms the baselines in both CS and ACN. This indicates support gradient field helps to find a better trade-off between task completion and collision avoidance.
Effectiveness of baalnd-blancing Mechanism: (FixeF(Fixee) is the ablated version DualGF with mixing rate lambda is fixed at an initial level. We illustrate the advantage of DualGF over (aalGF(Fixed)) on PL and ACN on Fig. 6. With the initial lambda decrease, the advantage of DualGF increases significantly in both PL and ACN. This indicates our field balancing mechanism can moderate the sensitivity to the choice of initial mixing rate Ag-
CONCLUSION
In this study, we first reformulate the path planning problem in a novel data-driven paradigm where the agent learns to plan with two sets of examples. We further propose a novel score-based planning framework (DualGF) gradient field that guides task completion and support gradient field that ensures moving with physical constraints. Moreover, an adaptive gradient balance mechanism is introduced to combine the merits of the two fields. Experiments demonstrate the scalability of our method, and empirical results show that it significantly outperforms state-of-the-art baselines efficiency and safety.
tackle this problem. There are two gradient fields in DualGFs: a target
Limitations and Future Work: The main limitations of this work are the local minimum issue as discussed in Appendix D.6 and the holonomic assumption made See 4.3. Another minor issue is the inherent instability of the Lagrangian Relaxation which can be mitigated via small learning rate and gradient clipping, as discussed in Appendix D.7. In the future, we may incorporate our method with MCMC- approaches such Langevin Dynamics overcome the local minimum issues. Besides, we may extend our method to oon-holoooiic scenarios by integrating model predictve control (MPC).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In International conference on machine learning, pp. 40-49. PMLR, 2018.
Qisen Chai, Yujun Wang, Yong He, Caijing Xu, and Zhisong Hong. Improved prm path planning in narrow passages based pso. In 2022 IEEE International Conference Mechatronics and Automation (ICMA), pp. 41-46. IEEE, 2022.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http::/ppbullet..rgg org, 2016-2021.
Andrew Dobson and Kostas E Bekris. Sparse roadmap spanners for asymptotically near-optimal motion planning. The International Journal of Robotics Research, 33(1):18-47 2014.
Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification. arXiv preprint
2021.
Huan Fu, Bowen Cai, Lin Gao, Ling- Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and seman- tics. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10933- 10942, 2021.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv:1710.11248, 2017.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. arXiv preprint arXiv: / 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Jonathan D Gammell, Timothy Barfoot, and Siddhartha S Srinivasa. Batch informed trees (bit*): Informed asymptotically optimal anytime search. The International Journal of Robotics Research, 39(5):543 567, 2020.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.0590s 2018.
Kyle Hatch, Tianhe Yu, Rafael Rafailov, and Chelsea Finn. Example-based offline reinforcement learning without rewards. Proceedings of Machine Learning Research vol, 144:1-17, 2022.
Matthias Hüppi, Luca Bartolomei, Ruben Mascaro, and Margarita Chli. T-prm: Temporal proba- bilistic roadmap for path planning in dynamic environments. In IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS 2022), 2022.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.099) 2022.
Sertae Karaman and Emilio Frazzoli. Sampling based algorithms for optimal motion planning. The international journal of robotics reseurch, 30(7):846-894, 2011.
Lydia Kavraki and J-C Latombe. Randomized preprocessing of configuration for fast path planning. In Proceedings of the 1994 IEEE International Conference Robotics and Automation, pp. 2138-2144 2145. IEEE, 1994.
Jennifer E King, Marco Cognetti, and Siddhartha S Srinivasa. Rearrangement planning using object- centric and robot-centric action spaces. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 3940-3947. IEFF, 2016.
10

---

Under review as a conference paper at ICLR 2023
Iyy Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. aerriminator-actor-rttiic Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.0292. 2018.
Geesara Kulathunga. A reinforcement learning based path planning approach in 3d environment. arXiv preprint arXiv:2105.10342 2021.
Steven M LaValle et al. Rapidly-exploring random trees: A new tool for path planning. 1998a.
Steven M Lalll al. Repidiy-n random trees: A new tool for path planning. 1998b.
Min Cheol Lee and Min Gyu Park. Artificial potential field based path planning for mobile robots using virtual obstacle concept. In Proceedings 2003 IEEE/ASME international conference on advanced intelligent mechatronics (AIM 2003), volume 2, pp. 735-740. IEEE, 2003.
Chenming Li, Han Ma, Peng Xu, Jiankun Wang, and Max Q-H Meng. Biait Symmetrical bidirec- tional optimal path planning with adaptive heuristic. arXiv preprint arXiv:2205.0668 2022.
Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning- arXiv preprint arXiv:2004.0939) 2020.
BK Patle, Anish Pandey, DRK Parhi, A Jagadeesh, et al. A review: On path planning strategies for navigation of mobile robot. Defence Technology, 15(4):582 606, 2019.
Dean A Pomerlcau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral computation, (1)):88-97, 1991.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement
learning with sparse rewards. arXiv preprint arXiv: 108, 2019.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statisties, pp.
JMLR
Workshop and Conference Proceedings, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Stuart Russell. Artificial intelligence modern approach. Pearson Education, Inc., 2010.
Atsushi Sakai, Daniel Ingram, Joseph Dinius, Karan Chawla, Antonin Raffin, and Alexis Paques. Pythonrobotics: a python code collection of robotics algorithms, 2018
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin-Martin, Linxi Fan, Guanzhi Wang, Claudia Pérez-D' Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael F. Tchapmi, Kent Vainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. igibson 1.0: simulation environment for interactive tasks in large realistic scenes. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. accepted. IEEE, 2021.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
Marlin P Strub and Jonathan D Gammell. Adaptively informed trees (ait*): Fast asymptotically optimal path planning through adaptive heuristics. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3191-3198. IEEE, 2020.
Marlin Strub and Jonathan D Gammell. Adaptively informed trees (ait*) and effort informed trees (eit*): Asymmetric bidirectional sampling-based path planning. The International Journal of Robotics Research, 4144):390-417, 2022.
Aviy Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Picter Abbeel. Value iteration networks. Advances in neural information processing systems, 29, 2016.

---

Under review as a conference paper at ICLR 2023
Pascal Vincent. A connection between matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1674 2011.
Mingdong Wu, Fangwei Zhong, Yulong Xia, and Hao Dong. Targf: Learning target gradient field
for object rearrangement. In Thirty-sixth Conference on Neural Information Processing Systems, 2022.
Hong You, Gang Chen, Qingxuan Jia, and Zeyuan Huang. Path planning for robot in multi- dimensional environment based on dynamic prm blended potential field. 2021 IEEE Sth Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), volume 5, pp. 1157-1162. IEEE, 2021.
Jinglun Yu, Yuancheng Su, and Yifan Liao. The path planning of mobile robot by neural networks and hierarchical reinforcement learning. Frontiers in Neurorobotics, 14:63, 2020.
Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. Ad-vat+: An asymmetric ducling mechanism for learning and understanding visual active tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 146--1447-4 2019.
Brian Ziebart, Andrew I. Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008
12

---

Under review as a conference paper at ICLR 2023
A TASK DETAILS
A.1 TRACKING AND NAVIGATION
Navigation and Tracking requires the agent to reach a static location or track a dynamic location while avoiding collision with the obstacles. There are four rectangular obstacles and a particle agent. The scale, position and orientation of the obstacles are randomised before each episode in both tracking and navigation. The goal location is randomly reset at the start of the episode. Note that in tracking, the goal location will be further reset when reached by the agent. In this task, the target distribution is assigned as Gaussian distribution centred on the goal location, i.e., NVg. 0.1²I). The state space is the concatenation of the agent's position, goal location and geometry observations of obstacles. The action space is two-dimensional linear velocity.
State and Action Spaces:
= Se) where the agent state is 2-D location S, = [xx, vl E [-1, 1]2 and condition-state is the 2D goal location and obstacles' states S. [g. 01, 02, 03, 04l where E [-1, and o, denotes the state of the i-th obstacle. The obstacle state is the nor- malised position, orientation and side-length 0; [pos, ori, side) where pos E [-1, 1]2 ori (sin(yaw), cos(yaw) E [-1, side E 11.
Horizon: 100.
Initial Distribution: Before each episode, we first randomly initialise 4 box-shaped obstacles and then randomly sample the agent's state that does not collide with the obstacles.
Dynamics:
The floor and wall are all absolutely smooth planes. The arena is 0.25m x 0.25m area, with the agent's radius being 0.015m. We set the friction coefficients of all obstacles and agents to 10.0 to penalise the collision.
Target Examples: To sample a target example, we first randomly initialise the obstacles' locations and sizes and the goal location. Then, we sample an agent's 's location via repeatedly sampling from Gaussian distribution centred on the fixed goal location until the location is collision-free To ensure the target score network can work well, we sample 100,000 examples for training.
Support Examples: To sample a support example, we first randomly initialise the obstacles' loca- tions and sizes and the goal location. Then, we sample an agent's location via repeatedly sampling from the uniform distribution over the configuration space until the location is collision-fiet collect support examples, reset the environment 200 times. For each reset, we collect 1000 examples from the same condition. Thus, the number of support examples is 200,000 in total.
To
A.2 BALL REARRANGEMENT
Ball Rearrangement includes three sub-tasks: Circling, Clustering, and the hybrid of the first two, Cireling Clustering. In both sub-tasks, there are three sets of 7 balls that belong to different categories, i.e,, red, green and blue. The state and action spaces are the joint positions and linear velocities of all balls respectively. The agent required to rearrange all balls to maximise the target likelihood of the joint state while avoiding collisions. The interpretations of the target distribution of each task are as follows: Clustering requires all balls to form into three clusters by colour; Both Circling and Cireling Clustering require all balls to form into circle that can be centred anywhere in the arena, except that the latter further requires that the balls of the same type are adjacent to each other.
State and Action Spaces: There are 21 balls in the environment. The state space is the con- catenation of all sub-states of all balls: s [$1, S2,,92. .... sz1]. For each ball, the sub-state is s; = [s,, sc], S. = [x. y), s, = [c] where x, E [-1,1] denotes the two dimensional position and (0, 1, 2) is class label. The action space is also the concatenation of all sub-actions of all balls: a = [a, a.,..., 121] where a, E [-1, 1 is a 2-dimensional linear velocity.
Horizon: 300 for Circling Clustering and 100 for Circling and Clustering.
Initial Distribution: We first uniformly sample rough locations for each ball, and then we eliminate overlaps between these positions by executing the physical simulation.
13

---

Under review as a conference paper at ICLR 2023
(a) Six Modes of Clustering
Mode1
Mode2
Mode3
Mode4
Mode5
Mode6
(b) Examples of Circling
(c) Examples of Circling + Clustering
Figure 7: Visualisation of three sub-tasks of Ball Rearrangement. (a): Clustering requires all balls to form into three clusters by colour. (b): Circling requires all balls to form into a circle, neglecting the colour order. (c): Cireling Clustering require all balls form into circle while the balls of the same type are adjacent to each other.
Dynamics: The floor and wall are all absolutely smooth planes. All the balls bounded in 0.3m x 0.3m area, with the radius being 0.025m. We set the friction coefficients of all balls to 100.0 since we observe that setting a amall(e.g.no) not larger than 1.0) friction coefficient does not significantly affect the dynamics. Besides, to increase the complexity of the dynamics. we set the masses and restitution coefficients of all green and blue balls to 0.1 and 0.99, respectively. The masses and restitution coefficients of all red balls are set to 10 and 0.1, respectively. We observe that under these dynamics, the collision may significantly harm the efficiency the rearrangement process. Hence, the agent has to adapt to the dynamics for more efficient object rearrangement.
Target Examples: For Cireling, first random sample a feasible centre the free space. Given centre, we further random sample feasible radius for the target circle. Then we sample circle orientation of the target circle. Given the centre, radius and orientation, we can obtain feasible locations for balls to form circle. Finally, randomly assign these positions to the balls. For Clustering, we directly sample initial ball locations from a Gaussian Mixture Model in Eq. 15. Then step physical simulations remove the collision between balls. The final state serves target example. For Circling+ Clusttrinn it is similar to Circling. We first sample feasible locations for balls form in circle. Then we randomly sample "colourisation starting point', Starting from this point, we colourise the balls in I - G B or R - B G order (50% for each order). For these three tasks, we collect 100,000 target examples. However, demonstrate in Appendix D.2 that the target examples can be reduced to 100 in Clustering.
Support Examples: When sampling support examples, we increase the number of balls from to 3 x 10 so to increase the ball density in the arena. Since our score network is implemented as graph neural network (As mentioned in Appendix C.1), the network can be transferred different dimensions of input. To sample support example, uniformly sample locations for all the balls independently. Then we step physical simulations to remove collisions between balls obtain support example. To collect support examples, repeat the above procedure obtain 100, 000 examples. As shown in Appendix D.2, 50,000 support examples also work well.
A.3 ROOM REARRANGEMENT
Room Rearrangement is built large-scale, synthetic indoor scene dataset 3D-FRONT (Fu et al., 2021). Following (Wu et al., 2022), we use 756 839 rooms for training and 83 for testing. The agent is required to learn scene priors from the target examples (i.e., training set) and rearrange
14

---

likelihood functions are Under review as a conference paper at ICLR 2023
objects into reasonable layouts during the test phase. The state space is the concatenation of obser- vations of each object and the room boundary information. The action space is the concatenation of the two-dimensional linear velocity and -iedimensio angular velocity of each object.
Dataset and Simulator: We clean the 3D-Front dataset Fu et al. (2021) to obtain bedrooms that consist of four walls and three to eight objects. We augment each room by flipping two times and rotating four times to get eight augmentations. We then import these rooms into iGibson Shen et al. (2021) to run the physical simulation. The 756x8 rooms are used for target examples that are used for training the target score network, the classifier- baselines. The other 83x8 rooms are used to initialise the room in the test phase, so the room rearrangement task is performed on the test dataset. The rooms in the training dataset are only used for learning prior knowledge arrange the room.
baselines and the VAE in goal-conditioned
State and Action Spaces: The state consists of aspect ratio R and an object state so R X6 where K denotes the number of objects. The aspect ratio Ta = tanh( (2) where by and by denotes the horizontal and vertical wall bounds. The object state is the concatenation of sub-states of all the objects 5, = s _ ...'', .... s where the sub-state of the i-th object s' E R is consists of 2-D position, 1-D orientation, 2-D bounding bound and 1-D category label. The action is also a concatenation of sub-actions of all the objects a, = [a a² a'''.''.........................
a. For the i-th object, the action a" E R&3 consists of a 2-D linear and a I-D angular velocity. The whole action space is normalised into a 3 x K dimensional unit-box [-1, _]55 by the velocity bounds. Note that the agent-state Sa in this task is the concatenation of agents' positions and orientations while the condition-state Se in this task is the aspect ratio and the concatenation of agents' labels and 2-d bounding boxes.
Horizon: Each training episode contains 100 steps, instead of 250 steps used in Wu et al. (2022).
Initial Distribution: To guarantee the initial state is accessible to the high-density region of the target distribution, we sample an initial state in two stages: First, we sample a room from the 83x8 rooms the test dataset. Then we perturb this room by 1000 Brownian steps.
For more efficient environment reset and physical simulation, we build a 'proxy sim- ulator' based on PyBullet Coumans & Bai (2016-2021) to replace the iGibson simulator. We use iGibson to load and save the metadata of each room. Then we reload these rooms in the proxy simu- lator, where each object is replaced by a box with the same geometry. We set the friction coefficient of all the objects in the room to zero, as the dynamics of the room are complex enough.
Dynamics:
Target Examples: We use the 756x8 rooms from the training set as target examples.
Support Examples: Starting from each state in the target examples, we perturb the state via Brow- nian steps. For every 800 steps, we collect a state and then step simulations to turn into a feasible support example. For each room, we collect 100 times. After cleaning the data, we obtain support set with 448, 754 examples.
B METRIC DETAILS
Here we introduce additional details of PL and CS.
B. LIKELHOLKK (PL)
measures the similarity between given state and target distribution by assigning a pseudo- likelihood function F prozy + R+ At each time step t, the PL-curve reports the averaged PL across all trajectories E prozy (s;)l ). Hence, the PL curve indicates the efficiency and final per- formance of each method. We do not report the PL-curves on Room Rearrangement, since it is hard to describe the human preferences and scene priors by hand-engineering.
The PL curve is only reported in Ball Rearrangement The specific listed follows:
Circling: The - likelihood function is defined as F prozy (s) = exp (oa where og and or denote the standard deviation of the angle between two adjacent balls and the distances from each
15

---

Under review as a conference paper at ICLR 2023
ball to the centre of gravity of all balls, respectively. Intuitively, if set of balls are arranged into circle, then the o, and σg should be close to zero, achieving higher pseudo- likelihood.
Clustering: Different from (Wu et al., 2022), target distribution is six-mode Gaussian instead of a two-mode Gaussian:
Defining the joint centres' positions latent variable C (Cr; Cg. Ct) where Cr, Cg and C denote centres of red, green and blue balls respectively and the above six modes as {cs hsige. the C' obeys a categorical distribution p(C = =G) = f. The pseudo-likelinn likelihood function is a Gaussian Mixture Model:
PGMM)(S) =
Σ
p(C = cx)p(s/C = cc))
p(s|C - ck) = II
N(C, 0.05I)(s*)
II Sigs
N(C 0.05T)(s*)
II 2 SISK
W 0.057)
(15)
Circling+lusserrgg The pseudo-likelhhood function is defined F prozy
exp to,)) exp (ogtoctag)-oc where OR denotes the standard deviation of the angle between two adjacent red balls, and aG and and oc denotes the standard deviation of the positions of red, green and blue centres. Intuitively, the first term exp (apto.) measures the pseudo-likelihood of balls forming a circle and the next term exp (oRtogtag)toc measures the pseudo-likelihodd of balls being clustered into three piles.
B.2 COVERAGE SCORE (CS):
CS measures the diversity and fidelity of the terminal states Sr =. CS reports the Minimal- Matching-Distance
(Achlioptas al., 2018) between ST and fixed set of examples Syt from Ptari Σ min ||Sge - sT|]. This metric can detect mode-collapsing: If an agent only
Egg STEST
moves objects to single mode, the terminal states may be far away from other modes which leads to high value CS (i.e., bad performance).
For Ball Rearrangement: To calculate the coverage score, sample fixed sets examples from the target distribution serving as S.t for Circling, Clustering, and Cireling+ Clustering. respectively. We sample 20 examples for Circling and Circling+ Clusterin and 50 examples for Clastering. Since the balls in the same category can actually be viewed as a two-dimensional point cloud, we measure the distance between two states by summing the CDs between each pile of balls by category.
For Room Rearrangement: The coverage score is calculated by averaging the coverage score in each room condition since the state dimension differs in different rooms. For each room in 83 test rooms, we calculate the coverage score between the eight ground truth states and eight rearrangement results and then the averaged coverage score over the 83 rooms taken the final coverage score for method. We measure the distance between two states by calculating the average 1.2 distance between the positions of the corresponding objects.
C IMPLEMENTATION DETAILS
For all learning- based baselines, we implement all the methods including ours in the same network architecture design and basically the same capacity (e.g., hidden dimensions).
C.I DUALGE
Training Objective: The complete training objective for both target and support networks is the SDE-based score-matching objective proposed by Song et al. (2020):
Bst)) (E (e(())) (p(s()), Va(e) log (og()((((() s()) (16)
where Por (s(() s(0)) .N(s(({; Zlog 20 1)T) and 25 is hyper-parameter.
16

---

Under review as a conference paper at ICLR 2023
Using this objective, obtain the estimated score w.z. different levels of the noise- noise- perturbed target distribution Plato (s(t)) = Poc(s(t) ((0)) (s(0)) (iiuutttnoosuu In this way, we can assign different noise-level t for different tasks conveniently. For all tasks, our method chooses
t t= 0.01.
Network Architecture: In Navigation and Tracking, score networks are simply implemented as fully connected layers. We first encode the agent position s, and noise-level into feature vectors fa and Jr. The state of each obstacle o; is encoded by a shared linear layer to fo.- We then pool the {Jo, J into a single vector. The support score network concatenates the fa. ft and fo and sends them to a linear fusion layer to get the score output. The target score network further encodes goal location into fy and concatenates the fa. ft. fg and fo and sends them to linear fusion layer to get the score output. In Ball Rearrangement and Room Rearrangement, the architecture of the target score network is exactly the same as the score network of Wu et al. (2022) used in Clustering and Cirlcing+ Clluttrrin The support score network exactly the same as the score network of Wu et al. (2022) used in Circling which does not encode the class label into the initial feature.
Reproduction: To entirely reproduce the results and check the details, we recommend running our code released on the Anonymous Github https: /anonyoouuu 4open. sci ence/ // ICLR2 CCLR233Ancnyoouscoceee-e 1426.
C.2 BASELINES
C.3 TARGF
The training objective of TarGF is the same as Eq. 16. And we set / = 0.01 for all experiments for a fair comparison.
For TSrGF(SAC): The implementation is based on the codes released by authors, except that we set = 0.01 for both reward estimation and residual policy for a fair comparison.
For Ta We set r 0.1 and the simulation duration of each timestep At = 0.02. For each agent(object), ORCA only considers the 2-nearest agents as neighbours, since in our experi- ments ORCA often has no solution when the number of neighbours is larger than 2.
C.4 CLASSIFIER BASELINES
These baselines refer the RCE and GAIL in experiments.
RCE and is implemented based on the codes Eysenbach et al. (2021) released by RCE's authors. We only modify 7 0.95, the training steps decrease 0.5 million (i.e., the same number of training steps as other methods) and the model architecture. The architecture of actor and critic networks is implemented the same as ours(ie., the same feature extraction layers and edge-convolutional layers).
GAIL trains two classifiers by discriminating two example sets with agent's rollouts respectively. The discriminator takes only one state instead of two adjacent states as input since the 'ground truth' reward(i.e., likelihood) is defined on the current state. At each training step, we update the discriminator by distinguishing between the agents' and the expert's 's states(for one step) and then update the RL policy under the reward given by the discriminator(for step).
C.5 RL(SAC) NAVIGATION AND TRACKING
This is implemented as ablation of GAIL that replaces the classifiers with ground truth task rewards.
C.6 PRM
We implement PRM from an off-the-shelf open-sourced repository PythonRobotics Sakai et al. (2018) with 16.3k stars. This implementation is also used for navigation tasks. We keep the de- fault configuration and properly rescale our configurations to fit the API.
17

---

Under review as a conference paper at ICLR 2023
C.7 POTENTIAL FIELD
This baseline is also implemented from an off-the-shelf open-sourced repository Python- Robotics Sakai et al. (2018) with 16.3k stars. We set Ç 1.0 for the attractive field and Ç 1.0 for the repulsive field. The distance threshold of the repulsive field is set to be Q = 0.1.
D ADDITIONAL ANALYSIS
D.1 ABLATIONS ON MIXING RATE À AND THRESHOLD d'
We seek to evaluate the sensitivity of our method to the two key hyperparameters, i.e., Ao and d. The experiments are conducted on Clustering for better quantitative evaluation.
We evaluate DualGF with different levels of the initial lambda Åg. As shown in Fig. 8, the PL and CS do not significantly change when the 2 1.0. This indicates our method is not sensitive when it is above a positive threshold.
Lambda(0.5)
Lamidddl
CS1
ACN
Lambdall (5)
0.5
5.0-
Lannbda(.35
25
90
75
100
L
1
Time Step
Figure 8: Ablation study results on the mixing rate de- We fix the threshold d' and evaluate DualGF with different initial lambda.
Further, evaluate DualGF with different levels of the initial threshold d' As shown in Fig. 9, the PI. and CS do not significantly change with all different levels of threshold but the ACN becomes better with the increase of the threshold. This indicates our method not sensitive c'.
Thredhola)
1.0
0,0
Thresholdi)
0.75
CS
0.8
ACN
Thresbola7) Thresboldd)
0.50
0.0
0
25
50
100
Thabalk
Thstakk
Time Step
Figure 9: Ablation study results the threshold c'.
We fix the threshold d' and evaluate
DualGF with different initial lambda.
D.2 ABLATIONS ON THE SIZE OF THE EXAMPLE SETS
This ablation study is conducted Clustering. For simplicity, just fix the X at the initial value.
Target Set: We replace the target score network with ablated versions trained on different scales of the target set. As shown in the figure. 10, the performance of DualGF does not significantly drop with the number of target examples decreasing. Notably, with only 100 target examples, DualGF(Tar le2) achieves comparable performance to the DualGF(Tar Ie5) in PL, CS and ACN.
Support Set: Similarly, we replace the support score network with ablated versions trained on dif- ferent scales of the support set. As shown in the figure. 11, the performance of DualGF significantly
18

---

Under review as a conference paper at ICLR 2023
drops after the number of support examples is smaller than 50,000. This is reasonable since the volume of the free space increases exponentially with the dimension of the state space. Thus, is infeasible to learn good support network with few examples. In cases, using 50,000 to 100,000 support examples works well for 42-dimensional state space (i.e., 21 balls).
109
16.0
Crade
Dual fed)
0.75-
75-
0.59
CS
ACN
5.0
Dua/
DualGF/Tar Tel)
025
TurdF/SCC)
25-
25
100
Time Step
Figure 10: Ablation results on the number of target examples.
Gracle
PL 4.59-
CS
30
ACV 1
DuaKiFiSSp led)
DuakiiPBBp Sod)
Dann Set)
5.0
10-
25
20
75
160
00
0.0
200
Time Step
Figure 11: Ablation results on the number of support examples.
D.3 ABLATIONS ON THE CHOICE OF NOISE LEVEL OF THE SCORE NETWORK
This ablation study is conducted on Clustering. For simplicity, we just fix the X at the initial value.
Target Score Network: We fix the noise scale of the support score network at 0.01 (same as DualGF in the main paper). We then compare the performance of the target score networks conditioned on different levels of noise scale Figure. 12. With the noise level decreasing, performance on PL and ACN slightly drops. This indicates the target useful target gradients under smaller noise levels. However, we observe that the target score network conditioned on smaller noise scale seems be more 'unsafe'. Thus, have specify larger initial value of X to regularise the target gradient. For = 0.005, 0.001, 0.0005, 0.0001, we specify X; 4, 10, 15, 20 respectively.
network can still provide
Oradk
DualCFiTer
10.0
DualGF/7ar 066))
1.56
C31
ACN1
Duaititfer nows)
26
10.
Dual o (69g)
635
16
25c
6/0
86
00
Shacctoe M000
Dwichta AAG
bwatDle Str
Time Step
Figure 12: Ablation results on the noise level of the target score network.
19

---

Under review as a conference paper at ICLR 2023
Support Score Network: Similarly, fix the noise scale of the target score network 0.01 (same as DualGF in the main paper) and A = 3.0 for all experiments. We then compare networks conditioned on different levels of noise scale in Fig-
the performance of the support ure. 13. With the noise level decreasing, the performance on PI. significantly drops while the ACN remains comparable to the original DualGE. In practice, we observe that the target conditioned on a smaller noise scale tends to be more "conservative' i.e., the support gradients are of large magnitude so that they overwhelm the target gradients.
network
Omol
1/66
PaatG///og 0.00
46
10.0
Dust 0.065)
Pl. 0.50
6.75
C31
3.6
ACN1
Diait//(Spp o 061)
6.25
14-
25
50
75
100
naston 1900 Sospooo
Time
Figure 13: Ablation results on the noise level of the support score network.
D.4 IMAGE-BASED DUALGE
The conditional state S. can also be represented as an observation from the environment, e.g., an RGB image. In Figure. 14, we demonstrate the image-based implementation of DualGI. Both the target and support network take an image O₂ 64444443 and agent-state s [ Ya E E as input and output gradient & f O2) R located on the agent-state. The image is simply en- coded by three layers of CNN and then reshaped to a feature vector. This vector then concatenated
with the agent's 's feature vector to be fed into final output linear layer.
tar (sh 0₂)
Sa
(x Ya) E 2
① tar
Network Input
Or
sup (s 0,)
O E 6664-6443
$ sup
Figure 14: The image-based implementation of DualGF on Navigation and Tracking.
We compare this image-based version (i.e., (lalGE((mage)) with learning baselines. No- tably, these baselines still take the low-dimensional state as input. As shown in Figure. 3, Du- alGF(Image) outperforms all the baselines across all the metrics. This indicates our method still effective when the state representation is complicated, such as pixel input.
D.5 SUCCESS RATE BALL REARRANGEMENT AND ROOM REARRANGEMENT
We also report planning success rate (SR) ball and room rearrangement. Since the number of ob- jects in these environments is quite large, so we relax the success threshold to 'less than N collisions in episode" where N denotes the number of objects in each episode. As shown in Figure. 15, Du-
20

---

Under review as a conference paper at ICLR 2023
Table 3: Comparative results Navigation and Tracking. We compare the image-based implemen-
tation of DualGF with state-based baselines.
Navigation (Static)
Navigation (Dynamic) TRS 1
Tracking
TRS f
SR f
SR f
TRS f
SR f
(SAC) GAIL
0.21
0.04
8.3 1
0.4
0.19
0.0
0.21 1 0.02
1.5
1 0.4
0.35
0.01
1.6
1
1.0
0.20 1 0.02
0.1 + 0.1
0.18 1 0.04
RCE
2.2
+ 1.2
0.13
0.01
2.2 +
0.4
0.22 + 0.04
0.1 4: 0.0
0.17 + 0.03
TarGF (SAC)
7.0 0.2
0.19
0.02
4.4
+
0.3
0.19 0.01
0.2 0.1
0.18 + 0.04
DualGF (Image)
16.4 1.0
0.24
0.02
15.7
= 2.7
0.26 - 0.02
0.4 0.0
0.27 = 0.01
aIGF achieves comparable performance with model-based approach TarGF(ORCA) and outperforms all the other learning-base -base baselines except for Room Rearrangement.
1.0
0.5
0.-
Daatr
APS
Daald
Tearsio
Gat
ACE
Daaor
Butl Seg
masione 350
Figure 15: Success rates of all methods. From left to right, we present results on Cireling, Cluster- ing, Circling+ Clustering and Room respectively.
D.6 FAILURES: LOCAL MINIMUM PROBLEM
Similar to Lee Park (2003), the most typical failure of DualGF is the local minimum problem. Figure. 16 demonstrate this problem in Navigation and Room Rearrangement.
the left side, there are some regions in the dual gradient field where the target gradient and the support gradient cancel each other. Besides, the dual gradients around these regions point back to them. Once the agent steps into these regions, it will get stuck and cannot get out.
On the right side, we demonstrate another implicit type of this problem, where the agent is not stuck by the obstacle but by itself. For instance, the target gradient leads a table and shelf to move opposite direction while the support gradient provides 'repulsive force' on them. Thus, both the shelf and the table get stuck in the left corner.
One potential solution is to leverage test-time sampling to increase the optimality of the planning results.
Local Minimum
Dual Gradients
Local Minimum
Local Minimum
Figure 16: The local minimum problem of DualGE, Left: Case study on Ball Rearrungement.
Right: Case study Room Rearrangement.
21

---

V, (s))pss)) is in the same direction with Under review as a conference paper at ICLR 2023
D.7 MITIGATING THE INSTABILITY OF LAGRANGIAN UPDATES
The inherent instability of the Lagrangian updates may lead to bad results as shown in Figure. 17 (a) and (b): When the A increases too fast, the support gradients overwhelm the target gradients so the balls are too "conservative and do not move to the target distribution. When the X drops too fast, the target gradients overwhelm the support gradients that the balls are too "aggressive" and collide with each other.
We can mitigate this problem by setting small learning rate and clipping the gradient. When X is properly updated, we can obtain appealing results as shown in Figure. 17 (c).
(a) At increases too fast.
(b) 4 drops too fast.
(c) A is properly updated.
Figure 17: Success and failure cases of Lagrangian updates.
D.8 ANALYSE THE GRADIENT OF THE PERTURBED DISTRIBUTION
Our score networks trained via denoising score matching are essentially matching the score of the perturbed distribution V. (s)) where
(=(S)= = -f l0 (S(s)p(ss)
(17)
V (S)
V=ac(S(s)p(s)d
When fo is Gaussian kernel, we can derive the closed form of (elor((ss)
le-8ll2
20
aSls C(s, 8) (s 8), C(s,
(18)
V2TO
where C(s, is a scalar function depends on s and s. Thus, Vs ($|s) essentially points the denoising direction, i.e., s. Further, we have:
/ C(s, S((s))
(8)
s
ds
(19)
denoising
divection
averaged
Intuitively, when lies in the zero-density region of (ps)s p($) = O, the gradient the perturbed distribution VaPa (5) still points to the averaged denoising direction to the positive-density region.
To summarise, the score the perturbed distribution (s):
1
V log
VsPa(s)
(20)
Po (s)
where Po (s) obviously. Thus, in the zero-density region of the original distribution, the score networks are estimating the averaged denoising direction to the positive-density region.
D.9 THE LIMITATION OF MANUALLY DESIGNING THE REWARD FOR BALL. REARRANGEMENT
Here we take Circling for example to demonstrate the non-triviality of example- based planning. We leverage the pseudo-likelihood function of Circling manually designed reward function train an RL agent, namely Heuristic(SAC).
As shown in Figure. 18, method achieves significantly higher PL than Heuristic(SAC), which indicates our method better in both sample quality and efficiency. Our method is also better than Heuristic(SAC) in CS, which indicates our method is better in diversity and mode coverage.
22

---

Under review as a conference paper at ICLR 2023
6.75
6.4
ACM 6.9
CS1
0.2
DuaiGF
(S
2.9-
GAIZ
6.0
100
Time Step
Figure 18:
Comparison between DualGF and RL agent trained by manually designed reward. The experiments are conducted on Circling.
D.10 EVALUATION ONLY SUCCESS TRAJECTORIES
In Navigation and Tracking, we report task return (TR) on non-collision trajectories. As shown in Table. 4, our method still outperforms the learning-base baselines across all the tasks and metries.
However, this evaluation will also be problematic since the success trajectories of different methods are different difficulties. For instance, the potential field baseline of only 28% Success Rate yet achieves the highest Task Return. However, we observe that this method only succeeds when the initial state quite near to the goal which is easier for the agent to obtain a higher task return.
Table 4: Comparative results on Navigation and Tracking. We report task return (TR) on non- collision trajectories.
Navigation (Static)
Navigation (Dynamic) SR
Tracking
TR
SR
TR
TR
SR
RL (SAC) GAIL.
1
0,21 0.04
42.4
0.8
0.19
0.01
0.2
0.21
0.02
1.1
0.35 0.01
7.5 + 4.3
0.20
0.02
0.6 +
0.18 0.04
Leaming Based
RCE
15.7 +.9
0.13 0.01
9.8 + 0.1
0.22 + 0.04
0.4 444:00
0.17 : 0.03
TarGF (SAC)
0.19 3.9
0.7 = 0.4
0.09
0.1 = 0.0
0.18 = 0.04
DualGF (wo Sup)
0.010.00
0.07 0.01
13.1 1..19.99
0.09 1 0.03
0.2 0.066.0
0.11 1 0.04
DualGF
81.3+ 0.4
0.37 + 0.02
66.9 + 7,3
0.39 0.05
6.0
0.56 + 0.03
Planning
PRM (Reference)
8.4
0.1 0.44
0.47 = 0.04
0.4
0.16 0.04
Based
Potential Field
77.4
0.00
84.3 0.3
0,28 - 0.07
22.4 1: 2.4
0.09 1 0.02
23


Reviews:
[{"review_id": "in3x7Ewkke", "reviewer": "Reviewer_QU9s", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel path planning approach that learns two gradient fields from two sets of task and support examples, as opposed to learning from whole trajectories or (inverse) reinforcement learning. These gradient fields are mixed at runtime using a heuristic velocity-based controller to generate feasible paths that lead the agent to the goal states in several simulated test scenarios. It is shown that the method outperforms other learning based path planners and is more efficient in finding solutions compared to sampling-based planners.", "summary_of_the_review": "Overall I recommend the acceptance of this paper, it is well written and it introduces a path planning approach that seem to improve over several learning-based alternatives introduced recently in the literature. Note the weaknesses mentioned, although the work is evaluated extensively, we're missing the details of failure cases and why the method works well (across multiple environments).", "strength_and_weaknesses": "* The strength of the paper is in showing how the proposed simple solution works across multiple environments, when evaluated with multiple metrics and compared to multiple alternative approaches. Overall the experiments are extensive and ablation studies are included.\n\n* The paper is well written and the approach is introduced clearly. \n\n* As for weaknesses, I was missing some more details on the tasks and the environment details. Some example successful paths generated by the agent could have been included. Likewise, we would have profited from including some failure example paths, in order to appreciate better what the agent can achieve and when, in what circumstances. \n\n* On a related note, we do not know when the introduced can fail from the paper. When it fails, does it fail slightly or terribly? The ablation studies show the effect of initial lambda as one potential source of issues, from which the lambda update seem to recover. \n\n* We know that Lagrangian relaxation used here to solve the problems is not fool-proof at all. Some details on the potential issues would have been more illuminating.\n\n* I did not read the appendix - if some of these 'improvements' of the main text are mentioned there then I'd suggest putting pointers to the relevant sections in the main text.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OWT-WCT9Hwf", "reviewer": "Reviewer_vd31", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an offline learning approach for path planning. The proposed Dual Gradient Fields (DualGF) model the probability distributions of (1) target states (e.g. goal states) and (2) free space. Then, the problem of finding a path toward a target state while avoiding collision becomes simply following the gradients of the mixture of two probability distributions. The experiments show that the proposed method can find a path without access to online interactions and the ground truth model.\n", "summary_of_the_review": "Overall, the proposed method is novel, simple, and effective. However, the experiments are done in path planning in 2D actions, which does not resolve the concerns about learning reasonable support gradient fields and path planning with non-linear state and action spaces. Adding experiments on a complex, non-linear environment would make this paper strong. But, as of now, I would recommend weak rejection.", "strength_and_weaknesses": "### Strengths\n\n* The proposed method is simple but effective at finding a collision-free path.\n\n* The experiments show better goal-reaching performance as well as fewer collisions in various 2D environments.\n\n* The experiments show that the proposed method scales well with the number of manipulatable objects or agents.\n\n### Weaknesses\n\n* The core of the proposed method is learning a good target gradient field and a support gradient field. However, learning the dual gradient fields purely relies on the target examples and a dataset of randomly sampled states from the free space. The paper claims that the support examples are easy to collect by randomly initializing objects; however, collecting such data is challenging in more complex environments, such as the real world.\n\n* The proposed method simply infers the target gradient field from a set of target states. The experiments are done with linear state spaces (e.g. x, y coordinates), where estimating the target gradient field from a few target examples is straightforward. However, when the state space is complex and nonlinear, such as pixels and non-smooth state changes, the learned target gradient field may not provide meaningful gradients for path planning. \n\n* Moreover, the experiments are done in toyish environments with relatively large datasets for target states (100,000 states). Although the size of the support example data is not mentioned in the paper, to learn a reasonable support gradient field, it must need data that covers most of the states along potential solution paths. This can limit the scalability of the proposed method to general path planning problems.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Mt0ZvRP26O0", "reviewer": "Reviewer_kjqZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors introduced Dual Gradient Field (Dual GF) as an offline learning example-based planning. The two gradients are coming from the target task and the support task. The former encourages the movement towards accomplishing the goal, while the latter enforces safety (e.g. avoiding collisions). Offline, two networks are trained to estimate each gradient using collected samples. During runtime, each gradients are calculated and mixed. The result is passed as an input to a heuristic controller that generates the movement. Authors evaluated their approach across three environments comparing with seven learning, planning, and oracle based techniques. ", "summary_of_the_review": "Overall, I think the paper is going in a great direction, but I believe authors should take a step back for their experimentations in order to turn this into an amazing publication. ", "strength_and_weaknesses": "Strengths \n+ The idea of separating the task from the world limitations is great. The latter can be generalized across various tasks.\n+ The whole idea is simple and easy to implement, although the assumption of having access to the P_tar may not be easy for all cases.\n+ The approach is overall explained well.\n\nWeaknesses\n- Experimentations was the main low light of the paper. Both the methodology and the clarity needs more attention. Regarding the methodology, as part of solution, it is okay to bring the world constraints as soft penalties, yet it is not okay to change the problem based on your solution. If the world does not allow collisions, then merging the quality of colliding and non-colliding trajectories muddies the results, which in turn makes it difficult to judge the effectiveness of your method compared to others. For example, if your path collided in the beginning, it is not meaningful to discuss how close the end of that path was to the target distribution. A better metric to your ACN, is path feasibility: what percentage of generated paths were collision free. Concretely, I recommend focusing on TR (main) and CS (secondary) metrics for collision free paths. If an algorithm fails to find paths it should reflect it in the TR. \n\n- Regarding limitations, while authors discussed the assumption of a simple controller that can move object(s) from s to s', they did not discuss about the situations were the combined gradient may not work well. For example, what if the two gradients cancel each other out and the agent freezes?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8Id7WO9e56", "reviewer": "Reviewer_Msq6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to learn target (goal) and support (obstacle) gradient fields for navigation planning from a set of example configurations. As opposed to many other learning approaches for navigation planning, it does not rely on demonstration trajectories but only on sets of successful target configurations and of freespace configurations. Based on these gradient fields, trained in the form of score networks, a Lagrangian formulation is used to derive a planning algorithm that dynamically mixes the gradient fields to successfully navigate to a target in a new configuration. Studies show that the system can perform well in terms of task success with relatively low computational cost compared to traditional randomized path planners and can address complex re-arranging tasks.\nThe main contribution of the paper lies in presenting an approach that can build a path planner for a task from just a set of example configurations. The studies performed show that this simple approach can be effective at planning for a range of tasks with relatively limited overhead.", "summary_of_the_review": "The paper presents an interesting approach with one novel that attempts to learn a gradient based planning system from state examples without the need for rewards or trajectories. The usability of the approach is demonstrated in a number of examples with tasks of different complexity.\nThe paper, however, lacks discussion of some implicit assumptions and of its relation with traditional mixture potential path planning approaches (this seems important as it seems to be largely an adaptive mixture potential approach with dynamic mixture weight adjustment).\nWhile the experiments cover interesting domains, their detailed description is completely left to the appendix with not even a short description of the obstacle complexity I the paper, making it hard to assess the results. In addition, it seems that a study of the influence of the noise parameter sigma would be of high importance as it seems to drive the propagation of the target gradient field away form the target/goal region.", "strength_and_weaknesses": "The paper presents a new approach to learn path planning from simple examples. A strength here is that as the examples are not trajectories but rather just target and freespace states, the freespace examples often transfer across tasks and can thus be shared for different task objectives in the same types of environments. While this does not apply to the target samples, which need to be specific to the task, they should still be easier to provide to the system than entire trajectories. This reduced effort in providing examples makes the approach interesting for a range of tasks where task objectives can easily be translated into target state examples.\n\nWeaknesses of the paper lie mainly in the lack of discussion of some of the underlying assumptions in the approach and some missing discussions related to relations to other navigation planning work.\nOne implicit assumption that seems to be made (and is not discussed) is that the planning is applied to a holonomic system (i.e. a system with no inherent movement constraints. The reason for this is that in the problem statement section it seems too be assumed that any point in an epsilon ball around the current state is reachable by the system. This assumption makes section 4.3 (the integration with low-level control) problematic as it is not clear that a low-level control for a system with dynamic constraints exists for a step generated by the higher-level planner. It would be important to discuss this.\nA second point that would warrant discussion is the seeming reliance on the right amount of noise in the DSM approach to make the target and support gradient fields work. In the technical description, the target gradient field is described as the gradient of the target state distribution. This, however seems to be very problematic since this gradient would be 0 anywhere outside the target region and thus not produce a gradient towards the target region. The only way to bridge this problem is the gaussian noise which effectively props up the value of the target potential (and thus the target gradient) for non-target points based on there proximity to the target region, basically creating a gradient towards the target region. It would be very useful for the authors to include some discussion regarding the way target gradients in non-target areas are derived (and how the same is achieved for non-freespace regions in the support gradient field). This effect also suggests that the choice of sigma in the DSM algorithm might have significant effect on the performance and a study of the effect of changing sigma would be very useful.\n\nOverall, the approach seems to be similar in character to the mixture of goal and obstacle potential approaches to path planning just that the potentials are learned from samples and that the mixture is dynamically adjusted using the optimization of the Lagrangian formulation. Some discussion of this relation, and in particular a comparison with mixture potential path planning as an additional baseline would be very beneficial. In this context it would also be useful if the authors could compare navigation tasks in more complex environments (multi-room or maze type environments) to study to what degree their methodology to adjust mixture weights (lambda) dynamically overcomes the local minima problem of traditional mixture potential approaches, and to what degree it introduces collision problems due to target gradients overwhelming support gradients near obstacles due to lambda being too small. In this context it might also be beneficial to motivate why only a single step of lambda adjustment is made for each navigation planning step.\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject