Embodied Lifelong Learning for Task and Motion Planning
Authors: Jorge Mendez-Mendez
Leslie Pack Kaelbling
Tomás Lozano-Pérez
Venue: CoRL 2023 Poster (2023)
Keywords: task and motion planning
lifelong learning
generative models

Abstract: A robot deployed in a home over long stretches of time faces a true lifelong learning problem. As it seeks to provide assistance to its users, the robot should leverage any accumulated experience to improve its own knowledge and proficiency. We formalize this setting with a novel formulation of lifelong learning for task and motion planning (TAMP), which endows our learner with the compositionality of TAMP systems. Exploiting the modularity of TAMP, we develop a mixture of generative models that produces candidate continuous parameters for a planner. Whereas most existing lifelong learning approaches determine a priori how data is shared across various models, our approach learns shared and non-shared models and determines which to use online during planning based on auxiliary tasks that serve as a proxy for each model's understanding of a state. Our method exhibits substantial improvements (over time and compared to baselines) in planning success on 2D and BEHAVIOR domains.

Embodied Lifelong Learning for Task and Motion Planning
Jorge Mendez-Mendez MIT CSAIL jmendez8csail .mit.dmit
Leslie Pack Kaelbling MIT CSAIL 1pk@csail.mis mii. edu
Tomás Lozano-Pérez MIT CSAIL. t1p@csail mit mii.edu
Abstract: A robot deployed in a home over long stretches of time faces a true lifelong learning problem. As it seeks provide assistance to its users, the robot should leverage any accumulated experience to improve its own knowledge and proficiency. We formalize this setting with novel formulation of lifelong learning for task and motion planning (TAMP), which endows our learner with the compositionality of TAMP systems. Exploiting the modularity of TAMP, we develop a mixture of generative models that produces candidate continuous parameters for a planner. Whereas most existing lifelong learning approaches determine a priori how data is shared across various models, our approach learns shared and non-shared models and determines which to online during planning based auxiliary tasks that serve as a proxy for each model's understanding of a state. Our method exhibits substantial improvements (over time and compared to baselines) in planning success on 2D and BEHAVIOR domains [1].
Keywords: task and motion planning, lifelong learning, generative models
Introduction
Consider a home assistant robot operating over lifetime in a home. The robot initially comes equipped with number of basic capabilities for planning and control that enable it to execute certain actions, such NAVIGATETO(objeect) and GRASP(object). The robot's user expects it to leverage these abilities to immediately assist with house chores, and to become increasingly capable over time, adapting to the types of problems that arise in its new home environment. This setting evokes a novel lifelong learning formulation for embodied intelligence that necessitates learning in the field, which we formalize and address in this work (Figures and A. I). Notably, we forgo any artificial separation between training and testing: the robot is continually asked solve problems to the best of its abilities, and it free use any data collects improve its knowledge for future use.
One promising tool for tackling this lifelong learning challenge is planning. Planning models comprise relatively independent prediction modules, which permits learning disentangled knowledge and reusing it compositionally. Similar forms of modularity have been shown to provide substantial leverage for training models continually by targeting the learning to individual modules, composing existing modules into new solutions, and adding new modules over time [2].
In this paper, we focus on learning generative models to address the most difficult aspect of task and motion planning (TAMP) [3]: finding continuous parameters (grasps, poses, paths) that guarantee the success of a high-level plan. Our approach learns to generate samples that lead to problem completion, implicitly considering the effect of the current sample on future samples' success. Compared to other applications of generative models, one peculiarity the lifelong sampler learning problem is that it is inherently multitask, since TAMP systems often apply similar basic skills (e.g.. GrASP) to varied object types (e.g., ball, box). At extremes of possible approaches this multitask problem, could leam an independent, specialized sampler for each object type, could learn single, generic sampler across all types. Intuitively, specialized models are likely to yield better
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

k-2 k=3 k=3 o
time
books
leftovers
organize office
close windows
The learning robot will face a sequence of diverse TAMP problems in a true lifelong setting. It will use its current models to solve each problem as efficiently possible, and then use any collected data to improve those models for the future. Images captured from BEHAVIOR [1].
Figure 1:
predictions, but they are trained on a smaller pool of data, making them potentially less robust. We propose hybrid solution that learns both general and specialized samplers, and generates samples by determining online which model is most appropriate given a state. To do so, we pair each sampler with an auxiliary predictor. When drawing samples for solving a problem, the measured error on the auxiliary tasks serves as proxy for the sampler's accuracy at the given state, which we use to construct a mixture distribution that assigns more weight lower-error samplers.
We evaluate the resulting TAMP system for its cumulative performance as it encounters a sequence of diverse problems. This realistic formulation measures how effective the robot is, in aggregate, at solving problems over its entire lifetime. This contrasts with standard (artificial) lifelong learning ovaluations, which instead focus on the final performance over the sequence of problems used for training. Concretely, we evaluate our approach on problems from a 2D domain and from the BEHAVIOR benchmark [1], demonstrating substantial improvements in planning performance.
2 The Lifelong Sampler Learning Problem
Our formulation of lifelong sampler learning emphasizes realistic setting in which robot is deployed in environment and, from that moment on, is continually evaluated for its ability to solve TAMP problems. This section first formalizes the problem in terms of general TAMP systems, and next discusses how it applies specifically to search-then then-samplee (SeSamE) bilevel planning strategies.
2.1 Learning Over a Sequence of TAMP Problems
The lifelong learning robot faces continual stream of planning problems 7(1), ((2) ,....... Each (k) problem -(k] = ( (k))) ~ ((A) is defined by an initial state snnt and goal (*), drawn sequentially from some problem distribution D)&), which is potentially nonstationary (as indicated by the superscript k).. We assume that the robot has access to a sound and probabilisticalyy complete planner; while the planner is guaranteed to solve any problem given sufficient computation time, expect many problems to be infeasible within reasonable time. In consequence, the robot should leverage any available data to focus its planning process on choices that are likely to be successful.
Like the real world, there are no distinct training and evaluation stages. Instead, at each time k. the robot seeks to construct a plan for a new problem (A). We record whether the robot succeeds at the problem within bounded number of samples B, and how many samples attempts. The robot may use any experience collects during the planning attempt as training data to improve its models. We measure performance as the cumulative number of problems solved within given number of attempted samples. A system that learns efficiently from the data available so far will solve new problems more quickly, gaining even more data from successful plans, in a virtuous cycle.
One attribute of this formulation is that the agent is never evaluated on any previous problem- after time k problem ((k) is never exactly encountered again. This departs from existing lifelong learning formulations, which evaluate the agent previous problems ..... -(E-1) (())...,(((_ to measure forgetting.
Yet, avoiding forgetting still contributes to attaining good performance in our setting. Since

---

evaluate the robot continually on new problems, it must generalize from prior experience. If the robot forgets past knowledge that remains relevant and overfits to the latest problems, then will never improve over the distribution of problems that it may face in the future. Even in nonstationary cases, where later problems differ substantially from earlier ones, retaining knowledge of past instances may improve generalization future problems; we demonstrate this empirically in Section 5.3.
In order for the robot to generalize in this fashion, the problem distribution D) must consist of problems that have some common underlying structure, enabling samplers to be shared across problems to serve as the medium through which prior experience informs future planning.
Algorithm 1
2.2 SeSamE Bilevel Planning
skeleton gen wwiiirreeeSrrccSlllaionnaaa
One strategy to solve TAMP problems to separate the search into two disentangled levels to perform bilevel planning [4]. We adopt this strategy and formalize the bilevel planning in a variant of PDDL [5] aug- mented with continuous parameters. The robot will be deployed in a world W (O, R, S, S,O,)). Each object e type 0g E θ is described by feature vector S is characterized by the features describing all objects. The pos- itive predicates r E R produce abstract state s (e.g., IN(bal1, box) A ON(box, floor) An abstract action at = (o,C) is an operator O, with predicate-level preconditions and effects, augmented with a parameterized controller Ca" The parameters of the controller are hybrid: e are discrete typed parameters that specify which objects to apply the controller to, while á are continuous parameters that dictate how the action is applied. For example, NAVIGATETO action applied to table could take continuous parameters & = (Ax, Ay) that specify the target coordinates, relative to the table. An action E A is the execution of a controller with given parameters. We assume that all symbolic elements W are given, and focus on determining the continuous parameters & that result in actions that lead to complete plans. planning problem r (siitt 9) is given by continuous-leve initial state Sinit and predicate-level goal g. satisficing plan æ is sequence of actions a1,-.., which move the robot from Sinit abstret state where s C g.
for j 1,..., N
D. -Loop over skeletons
skel <- saaeleton_gennnext ent -- zeros (enns()))))
D Next skeleton
b> Step sample counter
for 1, 1,... len. (ekeell))
Loop over steps
<|]] - sample(skelli) s(i)); ent [it[i]++
s(i + 1] if not valid 11))
Simulate(s(1) skel o(i))
while ent[i] = M : ent [
Backtrack
:- Set i so latest ent [i]
< M is re-sampled
if i 0: break
Try new skeleton
x. A state
s E r.goal : return skel,
The SeSamE strategy (Algorithm 1) first obtains skeleton plan at the discrete level using standard planning techniques (e.g., A"), and subsequently refines the skeleton into a low-level plan via sam- pling. While generating samples is inexpensive, the simulation step that evaluates each sample can be quite expensive (e.g., due inverse kine- matics, collision checking, path plan- ning). Reducing the number of samples required obtain a satisficing plan, and hence the number of calls to the simu- lator, can substantially improve the effi- ciency the TAMP system, vastly in- creasing the space of problems that can be solved tractable amount of time.
Navigate
Engage
Valid, observed
Success, observed
Valid, learned
Success, learned
Figure 2: Sample distributions optimized for TAMP prob- lem completion. Top: 2D robot tasked with pushing block upward. Bottom: distribution of navigation param- eters that achieve reachability (valid) and that solve the problem (success), generated by an expert (observed) and learned by diffusion model (learned). Samples that con- sider global success yield a distribution over promising (and not merely valid) actions. Diffusion models represent the observed (success or valid) distributions well.
One challenge in this sampling-based for- mulation is specifying distribution that 1) covers the space plausible solutions and 2) generates only promising candi- dates. Learning such a distribution from data, we do in this work, would consti- tute major step toward constructing effective TAMP system.

---

2.3 Samplers as Generative Models of Plausible Candidate Action Parameters
The sampler for each abstract action at should generate action parameters from distribution conditioned on the current state: & ~ p(- s, aT). This distribution should capture the whole space of action parameters that, given the current state, may lead to successful execution of an overall plan. To illustrate this point, consider the block-pusiie -ouchipu domain Figure 2 (see Section 5.1 for details). naïve navigation sampler could bring the robot within reach of the block, but a more intelligent sampler would only consider navigating to the bottom of the block to enable pushing upward.
To learn a generative model that meets these criteria, we require access to a pool of paired states and action parameters (s, Φ) that is representative of the choices that eventually (after successfully sampling all subsequent actions in a skeleton plan) lead to a satisficing plan. Critically, the action parameters that succeed for a given abstract action Pat depend on the samplers for the remaining abstract actions, and as such the data must be collected jointly to ensure that the learned distributions compatible with each other. The lifelong problem formulation of Section 2.1 satisfies this criterion.
3 Nested Models for Sparse Data
In our lifelong learning setting, the robot should use whatever data it has to make the best predictions possible. To handle the sparse-data setting that inevitably occurs early in the robot's experience, we formulate solution that nests predictors of different generality.
In order to train neural generative models as the samplers for our TAMP system, we construct a vector representation of the conditioning variables. The sampling distribution for each abstract action at is given by p(6 s, C. We first represent the state s in terms of the continuous low-level features of the objects involved in the action, Co. following prior work [4]. A trivial second step is to train separate models for each form of controller, PC, since different controllers generally have different no commonalities across their sample distributions. However,
parameterizations, and there each such controller may act on objects of different natures in ways that require substantially different parameters: this distinction is encoded in the discrete variable corresponding to the object types 9,,
Thus, we would like the predictions of our model to be specialized to each individual object type. For this, we can consider the following two strategies (omitting the C subscript for clarity):
1. Learn a single model p( Fan 0.) using, for example, a one-hot encoding of the types. This would enable the learning of a single, shared latent representation of x, the neural net model. However, in cases where the amount of data is small, it is more likely to simply overfit.
2. Learn separate, simpler, model (r( a..) for each discrete value l of 0..
However, early on, when the robot has observed very little data, we may prefer an even more aggressive strategy that pools data over all values of 0,. and learns a single simple model p(ò xo). This model will not be highly specific or accurate, but may learn more quickly to generate reasonable suggestions, due to pooled data. It also important observe that, in the lifelong setting, we may have asymmetric distribution of experience with samplers for different object types, so that some specific models would have substantially more training data available than others, in way that cannot be predicted in advance and that will change over time.
For all these reasons, we propose a nested approach: train both a generic sampler p(o x.)) and a collection of specialized samplers pe(ő εo) for all values l of 6,, Then, we decide online, given an input with 0,, f, based on assessment of prediction reliability, how much to value predictions from the generie versus the specialized samplers, and actually sample from mixture distribution.
To construct this assessment of the reliability of each generative model, we construct an auxiliary training task to predict a variable 2, and augment each training example, so we have (s, Φ, 2). Critically, z must be a value that can be directly measured by the robot, so that the error between its learned predictor f(s, 6) and the actual observed can serve measure of how well trained p(ó s) is in the part of the input space near So, in parallel with training p(ò s), will train fs, d), and in parallel with training each pr( s), will train separate specialized fe(s, 6).

---

At planning time, if the robot must draw sample in state for the action applied to objects with types & = f.r it will use the following mixture distribution:
Pmix s, p(f(s, 2), s)
2)p((
(1)
p(f(s, + ())
where p(fe(s, o), and p(f(s, o), Rt reliability measures of the specialized and generic models at (s, Φ), constructed by comparing their predictions to observed (for example, the inverse of the squared prediction error). Note that, since the auxiliary signals 2 depend on the action parameters d, the sampling process must draw samples from the two mixture components, use them to compute the reliability measures, and then choose between the samples based on the resulting mixture weights.
simple observable geometric properties of the world state as values, restricted to the objects that parameterize the action. Although they would not suffice for selecting good samples 6, since they do not consider the effect of á on future steps nor the interactions with objects outside of e-like a complete simulator would-our ability to predict them accurately is a signifier that the accompanying model has obtained sufficient data in the neighborhood of (s, 3) to consider Φ a good prediction. As an example, consider the NAVIGATETo(bi) from Figure 2. A useful set of auxiliary signals for this case may be: the orientation of the robot facing the block, the distance the center of the block, the distance to the nearest point the block's boundaries, and the relative coordinates of this point and the robot's 's center in the block's coordinate frame. Combined, these signals contain abundant information about the effects of the action, including those relevant for reachability and collision avoidance with the block. A model that learns to map a (s, Φ) pair to accurate predictions of all these signals has likely observed sufficiently similar training pairs to generalize, and therefore likely to generate high-quality samples Φ). See Appendices B.2.2 and B.3.2 for the exact auxiliary signals used in our experiments.
Implementation In our implementation,
action
4 Diffusion Models for Parameter Sampling
We use diffusion models to represent our learned samplers, due to their stability of training and their ability to model complex distributions. A diffusion model transforms Gaussian noise into a distribution over the sample space. To do so, it generates training data by following a forward diffusion process, which progressively adds Gaussian noise to observed training samples, and trains a reverse diffusion process that gradually denoises Gaussian noise to produce sample from the learned distribution
[6]. Concretely, when learning TAMP samplers, the forward diffusion process is given by ((00:7 s) = q( x) II g( @r-1), where each step g( Φt-1) adds Gaussian noise to 6:-1 and q s) denotes the observed distribution of successful action parameters o. The reverse process is the generative model parameterized by иb, and is similarly defined as a Markov chain: Po (Ør:o s) = p(0r) I Pe( Φe, s). where p(ør) is a standard Gaussian prior. Each step pro (b:11 1 di, 8) = N (Ø: - Exp (0t, 8, (), 2 I) estimates the mean of a Gaussian by subtracting predicted noise given a noisy version of the action parameters, the state, and the time step. Once trained, the planner can sample from the model by simulating the reverse process p&p (ØT:0 s).
Separating the learning process into various diffusion models, each in charge of representing specific distribution, as described in Section 3, reduces the difficulty of lifelong training as compared to training a single model to fit all distributions. Even so, each such diffusion model still requires continual training: specialized sampler (e xx) receives additional data as the robot uses it to attempt to solve new problems, and generic sampler p(ò) ax,)) additionally receives data from new object types as the robot encounters them. Given a previous pool of data Zold and newly acquired set of data points Znew, we consider the following training schemes:
Finetuning Starting from the previous model, train on Znew. likely forgetting previous knowledge. Retraining Train new model on Znew and Zold jointly. In practice, this method has been shown to outperform all true continual training approaches, at the cost of high computational expense. Replay Starting from the previous model. train over a balanced sampling of data from Znew and Zola [7]. While more advanced strategies are possible, we found that this simple method matched the retraining performance with only 10% of the training epochs used for retraining.
5

---

Experimental Evaluation
Place A - Placo B
L
Our initial experiments sought to validate that diffusion models can learn useful TAMP samplers and that the mixture distri- bution proposed in Section 3 achieves good performance in an offline setting. The results of these experiments set up our main evaluations over lifelong sequences of 2D and BEHAV- IOR domains. Additional details are provided in Appendix B.
5.1 Visualizing Sampling Distributions
To illustrate the usefulness of our approach to learning sam- plers for TAMP, we created two short-horizon, 2D domains that permit us to visualize the learned distributions. The first domain (Figure 2, top) requires a robot to push block upward. We focus on the navigation action, parameterized by the 2D coordinates, which succeeds if the block becomes reachable. However, the block can only be pushed from the bottom, so most valid actions do not lead to success. The second domain (Figure 3, top) requires placing two blocks in an L-shaped container, where the first block fits in any of the two sections, but the second block only fits in the long section. The blocks placed individually by selecting their 2D pose, but not all valid placements of the first block enable problem success.
Valid, Success, Valid, Success, observed
observed
learned
Isarned
Figure 3: Top: planner must fit two blocks in a container that re- stricts the longer block's placement. Bottom: distribution of parameters for achieving placement of the small block (valid) or solving the problem (success), generated by an expert (observed) and a diffusion model (learned). The distributions match.
We collected two data sets for each domain: one with all valid actions and one with only actions that lead to problem success. We then trained a diffusion based sampler on each data set. Figures 2 and (bottom) show the observed and learned distributions, demonstrating that optimizing for problem success yields distributions that consider the long-term effects of actions, even if future constraints not observable by the sampler. The learned distributions match the observed distributions well.
5.2 Learning Samplers from Fixed Data on 2D Domains
We next studied the impact of using our nested models in solving a variety of TAMP problems in an offline setting. We created five 2D robotics domains, each of which requires navigating, picking, and placing different objects in a target container. The domains vary in the object size ranges, graspable regions, and container size ranges and shapes (see Appendix B.2.1 for additional details). We collected demonstration data for solving K problems from each domain, trained the samplers on the fixed data sets, and evaluated their efficiency solving K' 50 unseen problems, We repeated the evaluation over 10 trials with various random seeds controlling the test problem generation.
Our experiments considered the following (primarily) diffusion-based sampler learning methods:
Specialized:
separate for each typed abstract action
(CD) generic: shared across actions with a common controller within (across) domains
(CD) mixture: our approach mixing specialized and (CD) generic samplers
NSRTs: The sampler learning component of Chitnis al. [4] work mon-diffusion--en based model
Figure 4 shows average results for varying numbers of training problems K (standard errors in Appendix C). As expected, all methods become more efficient at generating good samples as they are trained on increasing amounts data. In particular, the specialized samplers (which observe the least amount of data) are the most inefficient when trained on = 50 problems, but become as efficient as generic samplers upon training on K = 50,000 problems. Our mixtures of nested models are substantially more efficient than alternative methods when trained with little data. Appendix D analyzes various mechanisms for selecting between specialized and generic samplers, demonstrating that using geometric auxiliary signals proxy for sampler generalization is strong choice.

---

Specialized
0 samples
8000
ave 66 solved
Generic
90
4000
Mixture
CD generic
80
mixture
2000
Uniform
NSRT [4]
20
10
10a
10*
102
10a
10
(a) Number of samples per solved problem
(b) Number of problems solved
Figure 4: Results of learning diffusion models as TAMP samplers from offline data. Given sufficient data, all samplers solve the majority of problems efficiently. In the small-data regime (note the log-scale of the x-axis), sharing data across samplers improves sampling efficiency. The mixture distributions learned either individually on each domain (Mixture) across all domains (CD mixture) are best, thanks to their ability to automatically select generic or specialized samplers during planning.
Mixture + Replay
Generic Retrain Finetuee Uniform
Mixture Finetune
Specialized
5.3 Lifelong Evaluation 2D Domains
Specialized + Finetune
cumulative solved
2000
Our lifelong evaluation presented the agent with a sequence of problems from each domain turn (first
domain 1, then domain 2...), imposing a nonstation- ary distribution. The agent attempted to solve each problem given its current sampler, and subsequently used the collected data for additional training. There was no separate test phase: the agent's performance was assessed over its attempts to solve the problems in the sequence (before training on them). We up- dated the models of each method using the retraining. finetuning, and replay strategies of Section 4.
1000
0.0
0.5
1.0
samples 1e7
Figure 5: Lifelong learning results on 2D do- mains (avg. over 10 seeds). The mixture dis- tributions are vastly superior. Finetuning the models directly fails due forgetting, espe- cially with generic and specialized samplers.
Figure 5 shows the cumulative number of problems solved against the number of generated samples. We used retraining for specialized and generic samplers as an approximate upper bound on their performance. Our mixture sampler is substantially more efficient than baselines, and the only approach that outperforms a naïve uniform sampler. In the lifelong setting, agents use their current samplers to collect data, which explains why even the retraining version of the generic baseline fails. The agent uses the model trained previous types to generate samples for novel types; early on, those samplers are overfit to the small set of initial types, causing them to fail solve any problems and preventing the robot from acquiring useful data for subsequent updates. Specialized samplers work reasonably well, but are slowed down by their inability to quickly generate samples when trained over little data. The finetuning approaches, which are known to suffer from forgetting, perform noticeably worse, demonstrating the need to retain knowledge throughout the priblem sequence.
# cumulative solved
2000
5.4 Lifelong Evaluation on BEHAVIOR
1500
1000
We next applied the evaluation protocol of Section 5.3 to 10 families of BEHAVIOR problems (see Appendix B.3 for details). Figure 6 demonstrates that our method en- ables continual learning over this more complex and real- istic domain. Notably, the agent improves even over the starting point, demonstrating the usefulness of our approach even when engineered solutions already in place.
Notume Reo by Spiciliell hatiani Gamett-Rettain
500
0
hand-cratted
500
1000
samples
le3
Figure 6: Lifelong learning results on BEHAVIOR. Our lifelong learner pro- gressively improves over its initial (hand- crafted) samplers, becoming increas- ingly better at solving diverse problems.
hand-crafted samplers that we provided
7

---

Related Work
Lifelong learning Recent literature on lifelong learning has primarily focused on avoiding catas- trophic forgetting [8] in supervised settings [9, 10, 11, 12, 13J. Various techniques achieve this by replaying past data [14, 15, 7], which we adopt. Our focus on enabling robots to become increasingly capable over time via compositionality. Few works have studied compositionality in lifelong supervised [16, 17, 18] and reinforeed [19, 20, 21] domains. Prior lifelong learning work considers training an agent over a sequence of machine learning tasks and subsequently evaluating the system over those same tasks. We propose more natural evaluation setting without train/test splits, where the agent continually seeks to solve TAMP problems and uses the data from those problems to learn. While seemingly this formulation does not require forgetting avoidance (since previous problems are never encountered again), prior work has suggested that knowledge retention useful for generalization to future problems [2]. Our results in Sections 5.3 and 5.4 exemplify this notion.
Learning for TAMP Numerous recent methods seek to broaden the capabilities of TAMP systems beyond engineered solutions via learning. A majority of such methods focus on symbolic aspects of the problem: given a partial symbolic description of a domain, use demonstration data to fill the gaps in symbolie space to enable solving additional problems. This can take the form of learning operators (i.e., action abstractions) [22, 23, 4] or predicates (i.e., state abstractions) [24, 25, 26). Given the difficulty of mapping abstract actions to continuous robot actions, our focus is on learning at the continuous level, specifically in the form of samplers. The most closely related approach, which learns samplers for SeSamE planners, considers a very simple class of regression samplers with a learned rejection classifier [4]. Other methods for learning TAMP samplers use more powerful generative models, but not diffusion models [27, 28, 29, 30, 31J. None of these prior works study the underlying multitask problem, nor do they operate in a lifelong setting as our method does. Other (less) related work uses reinforcement learning to bridge between the discrete and continuous actions [32, 33] or automatically decomposes motion plans into discrete components [34].
7 Conclusion and Limitations
We proposed novel formulation of the embodied lifelong learning problem for TAMP systems, which emphasizes realistic evaluation of the robot it attempts to solve problems over its lifetime. Our solution approach learns a mixture of nested generative models, assigning higher weight to models that attain low error on auxiliary prediction tasks. Our experiments on 2D and BEHAVIOR domains demonstrate the ability of our approach to acquire knowledge over a lifetime of planning.
Limitations of chosen TAMP methods We adopt a SeSamE strategy for planning, which requires faithful environment simulator. In environments that are difficult to simulate (e.g., because they contain many objects), would prefer a different TAMP method; we leave this problem for future work. Moreover, our samplers are conditioned only on the objects involved in the action. Conditioning on all other objects or on the plan skeleton could generate better samples for problem completion.
Limitations of the lifelong setting and approach One direction to improve the learning is to develop better exploration strategies. Our method directly uses the learned models to generate samples; trading off exploration and exploitation could yield additional benefits. Additionall,, our lifelong training replays all past data. In our experiments, this represents only small portion of the computation, but it would become infeasible in longer deployments, so subsampling strategies which our approach can trivially adopt-should be employed. Our lifelong learning formulation builds upon operations that the robot can already execute. would also be valuable to study the autonomous learning of new operations. We encourage future research in this direction.
Limitations of the experimental setting We measure number of samples proxy for planning time. In practice there is trade-off between simulation cost and sampling cost; expensive diffusion models might lead to higher sampling cost than simulation cost in some cases. Relatedly, we have not yet evaluated our approach on physical robots. The facts that our method builds upon a working TAMP system and that is highly data efficient suggests that hardware deployment is probable.
8

---

Acknowledgments
We thank Nishanth Kumar, Willie McClinton, and Kathryn Le for developing and granting us access to the base TAMP system for BEHAVIOR. We also thank Yilun Du for initial discussions and guidance diffusion models. The research of J. Mendez-Mendez is funded by an MIT-IBM Distinguished Postdoctoral Fellowship. We gratefully acknowledge support from NSF grant 2214177; from AFOSR grant FA9550-22- 1-550-22----299 from ONR MURI grant N00014-22-1-2740 from ARO grant W91 1NN-23----002 from the MIT-IBM Watson AI Lab; from the MIT Quest for Intelligence; and from the Boston Dynamics Artificial Intelligence Institute.
References
[1] S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martin, F. Xia, K. E. Vainio, Z. Lian, C. Gokmen, S. Buch, K. Liu, S. Savarese, H. Gweon, J. Wu, and L. Fei-Fei. BEHAVIOR: Benchmark for everyday household activities virtual, interactive, and ecological environments. In Proceedings of the 5th Conference on Robot Learning (CoRL-22), pages 477 377-490, 2022.
[2] J. A. Mendez and E. Eaton. How to reuse and compose knowledge for lifetime of tasks; A survey continual learning and functional composition. Transactions on Machine Learning Research (TMLR), 2023.
[3] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Péres Integrated task and motion planning. Annual Review of Control, Robotics, and Autonomous Systems, 4(1): (11:):2652299) 2021.
[4] R. Chitnis, Silver, B. Tenenbaum, T. Lozano-Pérez, and L. P. Kaelbling. Learning neuro- symbolic relational transition models for bilevel planning. In 2022 JEEE/RSI International Conference on Intelligent Robots and Systems (IROS-22), pages 4166-4173, 2022.
[5] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. Weld, and D. Wilkins. PDDL: The plamning domain definition language. Technical report, Yale Center for Computational Vision and Control., 1998.
[6] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33 (NeurIPS-20), pages 6840-6851, 2020.
[7] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, K. Dokania, P. H. Torr, and M. Ran- zato. On tiny episodic memories in continual learning. arXiv preprint vrXiv:99022 10486, 2019.
[8] M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pages 109-165. Elsevier, 1989.
[9] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Brabska-Barwinskka, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, PNAS, 114(13):551133526 2017.
[10] Zenke, Poole, and S. Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, ICML-17, pages 3987-3995, 2017.
[11] Z. Li and D. Hoiem. Learning without forgetting, IEEE Transactions on Pattern Analysis and Machine Intelligence, TPAMI, 40(12))2935-2999 2947, 2017.
[12] C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner Variational continual learning. In 6th International Conference Learning Representations, ICLR-18, 2018.

---

[13] Serrä, D. Surís, M. Miron, and A. Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. Proceedings of the 35th International Conference on Machine Learning. JCML-18, pages 4548 4557, 2018.
[14] D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems 30, NIPS-17, pages 6467-6476, 2017.
[15] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient lifelong learning with A-GEM. In 7th International Conference on Learning Representations, ICLR-19, 2019.
[16] J. A. Mendez and E. Eaton. Lifelong learning of compositional structures. In 9th International Conference on Learning Representations, ICLR-21, 2021.
[17] T. Veniat, L. Denoyer, and M. Ranzato. Efficient continual learning with modular networks and task-driven priors. In 9th International Conference on Learning Representations, ICLR-21, 2021.
[18] O. Ostapenko, P. Rodriguez, M. Caccia, and L. Charlin. Continual learning via local module composition. In Advances in Neural Information Processing Systems 34, NeurIPS-21, pages 30298-30312, 2021.
[19] E. Brunskill and L. Li. PAC-inspired option discovery in lifelong reinforcement learning. In Proceedings of the 3 International Conference on Machine Learning, ICML-14, pages 316-324, 2014.
[20] C. Tessler, S. Givony, T. Zahavv, D. Mankowitz, and S. Mannor. A deep hierarchical approach to lifelong learning in Minecraft. In Proceedings of the Thirty-First AAAI Conference Artificial Intelligence, AAAI-17, 2017.
[21] J. A. Mendez, van Seijen, and E. Eaton. Modular lifelong reinforcement learning via neural composition. In 10th International Conference Learning Representations, /CLR-22, 2022.
[22] T. Silver, R. Chitnis, J. Tenenbaum, L. P. Kaelbling, and T. Lozano-Pérez. Learning symbolic operators for task and motion planning. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS-21), pages 3182-3189, 2021.
[23] N. Kumar, W. McClinton, R. Chitnis, T. Silver, T. Lozano-Pérezz and L. P. Kaelbling. Over- coming the pitfalls of prediction error in operator learning for bilevel planning. arXiv preprint DXXiv:2088.0737. 2022.
[24] Loula, K. Allen, T. Silver, and J. Tenenbaum. Learning constraint-baser planning models from demonstrations. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS-20), pages 5410-5416, 2020. doi:
[25] A. Curtis, T. Silver, B. Tenenbaum, Lozano-Péred. and L. Kaelbling. Discovering state and action abstractions for generalized task and motion planning. In Proceedings of the the Thirty- Sixth AAAI Conference on Artificial Intelligence (AAAI-22), volume 36, pages 5377-5384, 2022.
[26] T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Pérez, L. P. Kaelbling, and J. Tenen- baum. Predicate invention for bilevel planning. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI-23), 2023.
[27] Z. Wang, C. R. Garrett, L. P. Kaelbling, and T. Lozano-Pérez. Learning compositional models of robot skills for task and motion planning. International Journal of Robotics Research, 40 (6-7):866-894 89886668944111 jun 2021.
[28] Z. Wang, C. R. Garrett, L. P. Kaelbling, and T. Lozano-Pérez. Active model learning and diverse action sampling for task and motion planning. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS-18), pages 4107-4114, 2018.
10

---

[29] J. Striz-Haro, J.-S. Ha, D. Driess, and M. Toussaint. Structured deep generative models for sampling on constraint manifolds in sequential manipulation. In 5th Annual Conference on Robot Learning (ICLR-21), 2021.
[30] B. Kim, L. Kaclbling, and T. Lozano-Pérez. Guiding search in continuous state-action spaces by learning an action sampler from off-target search experience. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, 2018.
[31] B. Kim, L. Shimanuki, L. P. Kaelbling, and T. Lozano-Pérez. Representation, learning, and planning algorithms for geometric task and motion planning. The International Journal of Robotics Research, 41(2):210-23) 2022.
[32] Silver, A. Athalye. B. Tenenbaum, T. Lozano-Pérez, and L. Kaelbling. Learning neuro- symbolic skills for bilevel planning. In 6th Annual Conference Robot Learning (CoRL-22), 2022.
[33] G. Liu, J. Winter, Steckelmacher, R. K. Hota, A. Nowe, and B. Vanderborght. Synergistic task and motion planning with reinforcement learning- based non-prehensile actions. IEEE Robotics and Automation Letters (RA-L), 86)):776-2771, 2023.
[34] J. Johnson, A. H. Qureshi, and M. Yip. Learning sampling dictionaries for efficient and generalizable robot motion planning with transformers, arXiv preprint arXiv:2306. 00851, 2023.

---

Appendices to
"Embodied Lifelong Learning for Task and Motion Planning"
Anonymous Author(s)
Visual Depiction of the Nested Lifelong Sampler Learning Approach
k-1
k-3
k
store
office
windows
init
anCop (borgss, countes] burgert Lndide dorgger
goal
teloged
Buugezz, 02)
TAMP system
(buzqurl, izidee,
2. consinuct plan
avoluede
nou samplers
(a1,
4. train
Aozumulated
samplars
samplers
SA gww.elolooooooooooooooooooooooln [burger) Nevri
Figure A.1: The learning robot will face sequence of diverse TAMP problems in true lifelong setting. It will use its current models to solve each problem as efficiently as possible, and then use any collected data improve those models for the future. Images captured from BEHAVIOR [1].
B Experimental Details
This section provides additional details about the experiments in Section 5 in the main paper.
B.1 Network Architectures and Diffusion Models
All network architectures used throughout work were simple multi-layer perceptrons with two hidden layers of 256 nodes each. For our nested models, the hidden layers were shared between the generative model Pyp and the auxiliary predictor f, and only the output layer was separate-note that there was no sharing of parameters between specialized pt and generic samplers. All training proceeded for 1,000 epochs over the training data in mini-batches of size 512 (when that many samples were available, and a single batch otherwise), using an Adam optimizer with the default hyperparameters of PyTorch, including learning rate of 10-3, The one exception was the replay method for lifelong training, which used 100 epochs of training during model adaptation.
To train the diffusion models, for each point (s, Φ), a time step t E [1,T] was randomly chosen for T = 100, and a sample was drawn from the forward process d; = áva + +cvI- 6 (O, I) is standard Gaussian noise and o₁ is a scaling constant obtained by expanding the expression of q()). The loss function then measured how closely e (ér, x, approximated the true noise C(s, t,) = (ér x, ell. Once trained, the planner sampled from the model by simulating the reverse process Pv (oT:o s).
5i, where
12

---

Books
Cups
Boxes
Sticks
Blocks
Figure B.2: 2D domains used to evaluate our sampler learning approaches. The objects in each domain have properties that ensure that samplers must generate diverse candidate action parameters to solve the problems.
To process the input composed of three vectors (Ør, x. t). the time index t was first processed using sinusoidal positional embeddings [6] of the same dimension æ. Then, the three vectors were concatenated into single input the network. Since the auxiliary predictor f shared the same base layers of the network, we used & 0 as constant input to f.
All inputs and outputs were sealed to [0, 1], except when the range of the variable was less than 1, in which case the given variable was shifted O, but not rescaled.
B.2
2D Domain Experimental Setting
We now provide the details of evaluations the 2D domains.
B.2.1 Domain Descriptions
We created five different 2D domains, specially crafted to require distinct sampling distributions across objects. Figure B.2 depicts the simulated domains. As described in Section 2.2 in the main paper, each problem within a domain is sampled initial state and a goal. In this case, all goals are of the form "place all objects in the container."
Books Rectangular books of sides Wbook [0.5, 1]. bbok [1, 1.5] are scattered in room and must be picked and placed on a rectangular shelf of sides Wsholr E [2, 5), 4shelr E [5, 10). Cups Square cups with sides leup E [0.5, 1] must be picked by the hundle (one specific side) and palced in a cupboard of sides Weupboard E [2, 5], lcupboard E [5, 10). The cupboard is always against a wall.
Boxes Boxes of sides Wbox [0.5, 4bx E [1, 1.5) must be placed on pockets at the extreme ends of a tray of width Whray E [3, and length 4tay E 13].
Sitcks Long sticks of sides Wstick E [0.5, 1], lscck E [5, 6] in a container of sides Weontainer E 3, (3,5), Fcoutaine E [7, 10).
Blocks Small square blocks of sides lblock E [0.25, 0.5) must be place in square bin of sides Ipin E [4, 6). While previous problems contain E [4, 5) objects, these require placing n E [9, 10] blocks.
The robot has three controllers that it can execute: NAvIGATETO(objeect), parameterized by the rela- tive target coordinates normalized by the object's 's size; (objecb) parameterized by the length to extend the robot's gripper and the angle to hold the object at; and PLAcE(object, container). parameterized by the gripper extension.
B.2.2 Auxiliary Geometric Signals
This section describes the auxiliary signals we used train our predictors f, and later describe how those used to construct the mixture distributions for our samplers. We used the following auxiliary signals for each form of controller:
13

---

NAVIGATETO: distance to the nearest point between the robot and the object- in the case of the cup, this signal measured distance to the handle, and in the case of the tray. measured distance to the nearest pocket; coordinates the nearest point on the object's boundaries (in the object's frame); and target coordinates (in the world frame and
object's frame).
PICK: position of the gripper's point (in the world frame and in the object's frame).
PLACE: center of mass of the object (in the world frame and in the container's frame).
Note that all these signals measure the intended effects of an action, but cannot measure the actual attained effect, which would require knowledge of all objects the world. However, by measuring the agent's accuracy in predicting these signals, we can asses how well-trained it the neighboring region of the current state, and use that measure of how well its samples may generalize.
B.2.3 Constructing the Mixture Distribution
In these 2D domains, we created mixture distributions over three mixture components: a generic sampler trained on all object types, specialized sampler for each object type, and a fixed uniform sampler over the parameter space of the controller. We used the inverse of the root mean square error the assessment of reliability, p. For this, first computed (offline) the average prediction error for random guessing via simulation, and assigned this fixed error value to the uniform sampler. Then, we used this value to normalize prediction errors across the various signals.
B.2.4 Lifelong Training Details
In the lifelong setting, upon facing new problem, the agent used its mixture sampler to generate samples for any previously seen object type. For unknown types, the agent used mixture over the generic and the uniform sampler with fixed weights of 0.5. At the very beginning, the samplers were initialized with uniform distribution over the parameter space.
B.2.5 Evaluation Protocols
In the offline setting, generated 50 test problems for each of 10 trials, with varying random seeds controlling the sizes of objects and their placements.
In the lifelong setting, each trial shuffled the order of the domains using the random seed, and presented the agent with sequence 500 problems from each domain. Instead of updating the models after each problem, which would render most updates very minor, we updated the models at intervals of 50 problems, resulting in total of 10 model updates per domain.
In both settings, we used Fast- Downward as the skeleton generator, getting a single skeleton for each problem (i.e., N = 1) and setting the maximum total number of samples to B = 10,000. During search, a maximum M 100 samples were attempted at any given state before backtracking. We did not impose a timeout for these experiments.
B.3 BEHAVIOR Domain Experimental Setting
Next, we describe the precise details of our lifelong learning evaluation on BEHAVIOR problems.
B.3.1 Domain Descriptions
We considered 10 BEHAVIOR problems using the simulated humanoid: boxing books up for storage, collecting aluminum cans, locking every door, locking every window, organizing file cabinet, polishing furmiture, putting leftovers away, re-shelving library books, throwing away leftovers, and unpacking suitcase. Following prior work to adapt BEHAVIOR domains to the TAMP set- ting [23], only actions with PLACEONToP(object,, surface), surffcc. PLACEINSIDE(Object,, surface), target, surface). NAVIGATE-
PLACEUNDER( surface)) PLACENExTTo( To(object), and GRAsP(object, surface) controllers were implemented the continuous level,
14

---

while other actions (e.g., CLEANDUSTY or OPEN) were implemented only at the abstract level and assumed to always succeed if their abstract preconditions held.
B.3.2 Auxiliary Geometric Signals
The auxiliary signals that used to assess each sampler's reliability were:
sine and cosine of the robot's yaw; distance target, nearest point the object's bounding box (in the object's 's frame); distance to the nearest point on the object's bounding box; and robot position (in the object's s frame).
NAvIGATETo:
GRASP: sine and cosine of the Euler angles of the robot's gripper: distance of the gripper to the target and the surface; distance of the gripper the nearest point the target's and surface's bounding boxes; position, and sine and cosine of the Euler angles of the gripper's pose (in the target's and surface's s frames).
PLACCE ... distance from hand and object to surface; nearest points from hand and object surface's bounding box (in the surface's frame); nearest point from hand object's bounding box (in the object's frame); distances these nearest points; positions, and sines and cosines of Euler angles of the gripper's and object's poses (in the surface's frame). For PLACENEXTTO,, we additionally computed the relevant distances, nearest points, and relative coordinates with respect to the target object.
Like in the 2D domains, these signals measure only intended effects, but have no means to effectively measure if those effects are attained (e.g., due to collisions with unforeseen objects).
B.3.3 Constructing the Mixture Distribution
In BEHAVIOR domains, we only considered the trained specialized and generic samplers mixture components, since computing the uniform sampler's s error like in the 2D case would have required precomputing the error of random predictors via simulation, which was prohibitively expensive for BEHAVIOR. In consequence, we used the root mean square error directly (without normalization) to weight the two mixture components.
B.3.4 Lifelong Training Details
In the lifelong setting, we only used the learned samplers for exploration when both generic and specialized samplers had been trained. Whenever a new object type was encountered, hand-crafted samplers were used. At the start of the robot's lifetime, all samplers were initialized to hand-crafter distributions from prior work [23]-nom note that, for BEHAVIOR domains, uniform distribution, like used in the 2D domains, would never complete problems within any reasonable time limit.
B.3.5 Evaluation Protocols
We repeated the BEHAVIOR experiments over four trials with varying random seeds, which controlled both the order of BEHAVIOR problem families and the sampled problems within each family. We trained the agent sequentially on all ten families in each trial. We presented the robot with 96 problems of each family in sequence, and updated models every 48 problems.
We again used Fast-Downwart as the skeleton generator with N = 1. We set the sample bound to B = 1,000, with up to M = 10 samples at each state before backtracking. We did not use a timeout for these experiments.
C Complete Results of Learning from Fixed Data on 2D Domains
Tables C.1 and C.2 show the mean and standard error across trials of the number of samples and number of solved problems in the experiments of Section 5.2 in the main paper.
15

---

Table C.I: Average + standard error of the number of samples needed solve problems in 2D domains after training diffusion models from offline data accompanying table for Figure 4 in the main paper. Variance across trials is very small, demonstrating the statistical significance of our results.
Sampler choice
50 problems
500 problems
5,000 problems
50,000 problems
Specialized
3970.48
2071.54243..0
1541.26224.41
1161.38241.8
Generic
2538.38+644
1830.37 4:48.14
1454. 18
1129.52
Mixture
1904.38 57.88
1469. 15.9..554 622.96
1302.50a:2.as. 50..42
1097. 1197.14::00 144444.555
CD generic
2324.82 45..25
1720.13 132.22
1376. 16:14.6444.
1333.63
CD mixture
1676.47 #36.36
1426. 21 #32.79
1297. 1 88332.8
1134. 7334770447.31
Uniform
0063.76=60.1.
3063. 76
3063.76460.13
1063.762:6013
NSRTs [4]
8472.66=sa.rs
2674.26
2024.20#s6.44
1927. 2277.29
Table C.2: Average = standard error of the number of solved problems in 2D domains after training diffusion models from offline data accoopaynnnnnn table for Figure the main paper. Variance across trials is very small, demonstrating the statistical significance of our results.
Sampler choice
50 problems
500 problems
5,000 problems
50,000 problems
Specialized
76.96+0.08
93.4420.48
95.3220.47
96.4420.36
Generie
89.84
94.322:0.9
95.3610.43
96.28.0032
Mixture
94.28
96.32
95.60
96.4410.39
CD generic
90.6441.33
24.48
95.84
95.60
CD mixture
94.88
95.56=0.33
90.72
96. 06.20=z.41
Uniform
92.76+0.64
92. 20.
92.76
92.76=0.64
NSRTs [5]
23.16+4.89
89.00
93.72#u.40
94.32#u049
D Additional Experiments
In this section, we present ablations and additional experiments to those presented in Section 5 in the main paper.
D.1 Evaluating the Mixture Weight Construction
While the results of our main experiments strongly support the choice of using mixture distributions for generating samples for TAMP, were interested in more clearly understanding the choices of how to construct those mixture distributions. For this purpose, implemented and evaluated five alternative strategies for constructing the mixture distribution from our nested models:
Distance Our first alternative mixture still follows the process training auxiliary models but, unlike our main implementation, uses only a single auxiliary variable: distance to target. This allows to verify whether a collection of auxiliary signals necessary, or a single may suffice.
Reconstruction We similarly create an auxiliary model for directly reconstructing the state features æ. With this, we check the usefulness of including the action parameters á in the auxiliary tasks.
Uniform This strategy simply uses uniform mixture distribution. The purpose of evaluating this technique is to check whether all the gains of mixture distributions come from the mere fact of using a mixture, or the weighting plays an important role.
Proportional This cheating method observes the outcomes of the uniform mixture over all test problems, and computes the mixture weights the proportion of successful samples that were drawn from each mixture component. We include this strategy check whether there may exist some fixed choice of mixture weights that works across all states.
16

---

2000
96
avg # -sampees
avg % solved
Geometry
1750
95
Distance
1500
94
Proportional (X)
Classifier (X)
1250
Uniform
10a
10
104
102
10a
10
# train tasks
# train tasks
(a) Number of samples per solved problem
(b) Number problems solved
Figure D.3: Alternative choices of the mixture strategy to generate samples for TAMP in the 2D domains. Our geometric prediction method is most effective in the low-data setting.
Classifier This additional cheating method also observes the outcomes of the uniform mixture and trains classifier to generate the mixture weights. This enables to study whether it may be possible to train a model to directly choose which sampler to use, as an alternative to using auxiliary tasks as an assessment of reliability.
The results across the five 2D domains are shown in Figure D.3. While all mixture choices perform well, in the low-data regime, which is crucial in lifelong settings, our geometric predictions lead to the highest efficiency across all mixture choices. Notably, the uniform mixture solves the largest fraction of problems. This is expected, given that our mixtures include the uniform sampler over the action space, which is guaranteed to eventually find a successful sample; only the uniform mixture ensures that this sampler is used sufficiently often guarantee solving most problems (albeit less efficiently than other samplers). The reconstruction enror performs worst of all in the low-data setting, but eventually matches the performance of our geometry- prediction implementation; this validates the importance of including the action parameters á as part of the auxiliary signal computation. Neither the strategies that cheat is especially strong, indicating that 1) fixed mixture weights are not sufficiently flcxible and 2) directly predicting which sampler to use, given the state, difficult.
geometry-prediction
D.2 Replay Training Matches Full Retraining in Lifelong Setting on 2D Domains
To validate of balanced replay instead of full retraining, requiring 10x fewer training epochs due to starting from previously trained models, compared the performance of the two methods on the lifelong sequence of 2D domains from Section 5.3 in the main paper. Results in Figure D.4 demonstrate that the two methods perform equivalently.
Rixll
Mixture+Retrail
Mixture+FFintuunn
# cumulative solved
2000
1000
0
2
4
# samples
1e6
Figure D.4: Comparison of retraining vs replay on the lifelong learning evaluation in 2D domains. Replay (which is more efficient) matches the performance of retraining over the sequence of problems.
17


Reviews:
[{"review_id": "x0TmSlwAsb7", "reviewer": "Reviewer_8dBQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a new formulation of Task-and-Motion-Planning — a lifelong learning setting of TAMP. This paper specifically focuses on learning the models for samplers of continuous parameters in TAMP problems.\n", "summary_of_recommendation": "While I like the concept of lifelong learning in TAMP settings, the manuscript lacks the clarity of the problem formulation, and some of the experiments are confusing and unclear. If the authors can significantly improve the paper clarity by addressing my questions, I will definitely increase the score if all the questions are addressed. \n\nAfter rebuttal: I am very borderline on this paper submission. I don't think it's completely convincing to me since they have an important experiment missing that quantifies forgetting explicitly, but other than that, the authors have addressed all my concerns. I am neither up for nor against its acceptance. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Authors defined an action $a$ as the result of applying a controller (Ln98). Is it equivalent to a (sub)goal state? If so, why is it defined as an action instead of a state? \n- Please address the comments on the weakness, mainly about the symbols in formulation, figure 2, limitations, and all the ambiguities in writing.\n- Is the diffusion model a standard model for sampling parameters in TAMP? I believe it definitely needs to be compared against some existing samplers, either a hand-crafted one or an active sampling strategy.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- Propose the formulation of lifelong learning in TAMP setting.\n\nWeakness:\n\nOverview:\nThe major shortcoming of this paper is the definition of “tasks.” I think the definitions are vague and the tasks in each domain have very concrete explanations. Is one task in the 2D domain a sampled initialization + a goal to achieve in the domain? Are the “tasks” equivalent to the “problems” defined in the formulation? Or are “tasks” a subset of “problems”? Without a clear definition, it is very hard to evaluate the applicability of the proposed setting of lifelong learning TAMP, and if it is a reasonable thing to ignore previous tasks. \n\nDetailed points:\n- While it is commonly recognized that in the real world, a person / a robot does not encounter the exactly same situation twice, as for research, not evaluating the previous problem seems strange. I think evaluating prior problems is the best way to quantify how much forgetting an agent experienced and see the correlation between knowledge forgetting and the agents’ performance on new tasks. \n- Ln74-76: I disagree with this statement. Sometimes forgetting past knowledge is good — prior knowledge might be conflicting in performing new tasks. Either the authors formulate problems and define the scope that forgetting knowledge will definitely impact future learning negatively, or this statement is not right to be claimed. \n- The formulation in Section 2.2 is not clear. It is very confusing to see an abstraction action as an element from the space of \\mathcal{O}. At the same time, its notation is not o and its corresponding controller has object-based parameters o. Please make the notations consistent, or at least introduce symbols that would not introduce ambiguities. \n- More importantly, why is “bilevel planning” mentioned here? Is bilevel planning just task and motion planning in this context? If so, can you provide a more detailed explanation before introducing bilevel planning? \n- Figure 2 (Same comments apply to Figure 3): I cannot see the differences in distributions between the “observed” and “learned”. Maybe highlight the differences in Figure 2 so that it’s obvious what the better samples the diffusion model can give.\n- 2 /3 of the points in the limitations have nothing to do with the lifelong learning setting in TAMP. I think those two points apply to other classical TAMP works as well. I think the authors need to mention more of limitations that are more specific to lifelong learning in TAMP.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "iNg-blVD6-", "reviewer": "Reviewer_VjVf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a lifelong learning framework that uses generative models to sample grasps, poses, and paths for a high-level plan. To generate these samples, the authors propose a mixture model that weighs between an object-centric and generalized model for the resulting distribution. Each distribution is modeled using a diffusion model for its ability to capture complex distributions. The authors evaluate their learning strategy in an offline 2D environment and show that a diffusion model that shares data over multiple tasks performs better than models that use specialized datasets. The authors also demonstrate that the proposed framework can learn and improve task performance in a 2D environment and the BEHAVIOR dataset.", "summary_of_recommendation": "The proposed method does reduce the number of samples and improves task performance. But assessing this work's impact is difficult because no experiments or data compare the time required to sample using diffusion models and the time needed to test points and find a valid solution. If the authors can add experiments to clarify this point, it would help better understand the impact of their work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. What do skeleton_gen.next() and 's' mean in Algorithm 1?\n2. What does 'o' represent in line 94? Is it different from the `o' used to represent objects in line 87?\n3. In Section 5.2, when training the model under the mixture dataset, when N=50, how many samples from each task are used?\n4. How many tasks would it solve if the uniform sampler is used in the lifelong environment?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-motivated, and the problem is clearly defined. The following are the main strengths of this paper:\n1. Using a learned sampler, the authors show improvement in reducing the number of samples and improving task performance.\n2. The online training of models efficiently is a challenging problem for the robotics community, and the paper addresses this challenge for diffusion models in this work.\n\nAs the authors mention in the paper (lines 106-109), the simulation step that checks if the sampled point can solve the underlying task is quite expensive. If the authors could provide the timing results for task completion between the learned and uniform sampling methods, it would help validate their claim, especially since it is known that diffusion models fall short in sampling speeds.\n\nHere are a few more suggestions for improving the clarity of the paper:\n1. Although the authors tried explaining their method in Sections 2, 3, and 4, a figure outlining their approach, like in Fig 1 in [1], would make it easier for readers to follow their method.\n3. A video showing the high-level plan for the BEHAVIOR dataset would help understand the complex distributions learned by the diffusion model.\n\n[1] R. Chitnis, T. Silver, J. B. Tenenbaum, T. Lozano-Perez, and L. P. Kaelbling. Learning neuro-symbolic relational transition models for bilevel planning. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS-22), pages 4166–4173.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "hMcoHjZ8T4", "reviewer": "Reviewer_h46E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a method for lifelong learning of continuously new robot task and motion planning (TAMP) problems. This work focuses on learning continuous parameters of a controller (e.g. grasps, poses, paths) that guarantee the success of a (given) discrete high level plan. To achieve this, the proposed method learns a weighted mixture of diffusion models that outputs continuous parameters given the environment state and a parameterized controller. The method is evaluated on a number of 2D domains and 10 tasks from the BEHAVIOR benchmark and the different design choices are assessed experimentally.", "summary_of_recommendation": "An interesting and mostly well written paper which contains a number of original ideas that I think would be interesting to the research community. The proposed method is an original approach to lifelong robot learning, as is the proposed evaluation protocol. I can imagine it inspiring further research. The technical assessment of the method is quite good but there are a few areas where it could be improved. In particular, the lack of a baseline makes it difficult to assess when to use this method compared to already existing approaches. Further the method is only assessed in simulation with no discussion about a path to deployment on hardware.\n\nOn balance this paper currently stands at a weak reject for me but I would be happy to revise my recommendation if the authors address my comments and questions.\n\n=====\n\nFollowing the rebuttal period and discussion I have revised my recommendation to **weak accept**", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "In addition to addressing the points and questions raised in the previous section, please could the authors address the questions below:\n* Line 213: What data does Z_new consist of? Is it just the samples that led to task success?\n* Lines 274-276: Was there a sampling budget or was the sampler permitted to continue sampling until the problem was solved?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n* The paper addresses a relevant and open problem in robot learning — how can robots efficiently and continually learn to solve new tasks? — and the method is well motivated.\n* The proposed evaluation metric — comparing methods based on cumulative performance on a sequence of tasks that never exactly repeat and doing away with train/test splits — is original and interesting. It is also well motivated. I think it is highly relevant to a long term goal of robot learning of deploying robots in homes and hope to see this adopted broadly.\n   * I especially appreciated the discussion in lines 70 - 81 about why avoiding forgetting is still important in the proposed evaluating setting.\n* The proposed method is interesting and to my knowledge original, especially making use of a mixture of learned samplers using diffusion models and learning auxiliary predictors to select the weights between the different models. Both of these choices are practical and interesting and I especially appreciated the implementation discussion in section 3.\n* The set of experiments is mostly well chosen and they shed insight into why the proposed method works.\n   * Sufficient data is provided to support the key method design choices through ablation studies. For example, assessing the different strategies for learning samplers (Fig 5), different approaches for incorporating new data (finetune, replay, retrain (Fig 4, Fig B.3), and for weighting different samplers (Fig A.2)\n   * I enjoyed the discussion in 5.2 and 5.3 - it was insightful.\n   * There are a few additional comparisons that I would like to see (please see below).\n* The paper is mostly clear and well written, and sufficient detail is mostly provided about the method to re-implement it. However there are a few places where the clarity and detail provided could be improved (see below)\n\n**Weaknesses**\n* There are no experiments on a real robot and no discussion about why this is the case or whether the method could be deployed on a real robot in the near future.\n* Whilst the components of the method are well ablated, the proposed method is not compared with any other approaches for solving the same problems. There is also no discussion about why no such comparison is appropriate.\n   * This makes it difficult to assess when to use this method compared to already existing approaches.\n   * [5] seems like a viable candidate. Please could the authors include a [5] as a baseline method or include a discussion as to why this is not appropriate.\n* The method assumes that the robot has access to a sound and probabilistically complete planner.\n   * I am a little concerned about how strong an assumption this is and thus how broadly applicable the proposed method is. It would be helpful to include a discussion of how stringent this assumption is.\n* There are a few areas where the clarity and level of detail provided could be improved\n   * Abstract: “Our method exhibits substantial improvements in planning success” - compared with what? What are the baselines?\n   * Introduction, lines 29-31: Can you provide evidence to support the claim that finding continuous parameters that guarantee the success of a high level plan is the most difficult aspect of TAMP?\n   * Section 2.2: line 93: O does not appear to be defined.\n   * The 1-step and N-step labeling in figures 2 and 3 is a little confusing since it is not referred to in the text. How do they relate to the two data sets collected for each domain (valid action, task success actions)? Additionally, observed and learned look very similar. Presumably this is intended and it would be helpful if the similarities between the two could be discussed.\n   * Fig 4: Is mixture+replay a typo? If not, why is only replay combined with mixture and retrained combined with specialized/generic? \n       * [minor] It would also be nice to see uniform sampling included as a baseline in Fig 4.\n   * Line 253: I appreciate that evaluations were performed over 10 trials however it would be useful if the variance in results could be reported. Also, presumably the reported results in Fig 5 are the average. Please could this be clarified. \n   * No detail is provided on the simulator in which the environments are implemented. To improve the reproducibility of this work, at least some detail about this would be useful.\n   * Related work: “Our focus is on enabling a robot system to become increasingly capable over time via compositionality” This is the first time in the paper that compositionality is mentioned. It is not clear how the proposed method leads to compositional behavior.\n* The evidence provided in section 5.4 does not appear sufficient to establish the benefits of the proposed approach in the BEHAVIOR domain.\n   * No comparison with the uniform baseline or the specialized or generic models is provided in Fig 6 (in contrast it is in Fig 4).\n   * Line 300: Why are the hand-crafted samplers expected to be a strong baseline? Please could the authors provide a brief discussion on this.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "EIaj4fxwhk", "reviewer": "Reviewer_2eBC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper is dealing with the problem of lifelong learning in task and motion planning. The main approach relies on learning samplers for faster planning in subsequent ongoing operations.\nAuthors utilize Diffusion Models to learn samplers. The authors also slightly modify the lifelong learning setting and make it more strict that the agent is never evaluated on any of the previous problems.", "summary_of_recommendation": "I believe the work is interesting, tackling an interesting problem, very useful for the employment of TAMP and uses an appropriate approach. However, I believe it is still not mature enough and relies on the use of Diffusion models without appropriate demonstration and comparison to baselines.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Improving readability would be a minor point as stated in the review.\n\nAlthough it is clear for readers with TAMP background, a more appropriate introduction to samplers would be useful.\n\nMy main concern is in the demonstration domains and learning experiments. It is not fully clear how experiments are done, as there are no training and testing scenarios it is not clear if it is somehow overfitting to the scenario\n. \nBesides that, the contribution of the Diffusion approach is also not visible.\n\nComparison with some other learning-based samplers would be useful. (i.e. LOFT [1])\n\n[1] Silver, Tom, et al. \"Learning symbolic operators for task and motion planning.\" 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written, authors motivate the work and provide relation to some other relevant works in the field. The problem is very useful for the employment of TAMP solutions. However, several open questions and a few stylistic recommendations are stated.\n\nThe presented method is not explained in enough detail. It is basically kept on a high level and an appendix is very much needed to understand the approach.\n\nThe readability could be improved. For example, some of the wording is too strong. e.g., “robot will have to use” → “robot could use”. And some wordings are not directly understandable by non-native English speakers or are not academic language (e.g., conducive, grow stronger, etc.)\n\nDemonstration domains are very simplistic with short-horizon planning and it is not clear what is the distribution of the training examples and how much they differ in the lifelong operation. A more fair baseline for comparison could be provided.\n\nIn the specialized models, one hot encoding is used, which is very limiting for lifelong operation when novel object types can appear. On the other hand, the generic sampler is used on a problem of limited size and therefore the scalability to complex scenarios is not visible.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of lifelong task and motion planning with diffusion models as samplers for grasps, poses and paths as they can capture complex distributions. Experiments demonstrate the benefits of the approach in an offline 2D environment and with the BEHAVIOR dataset. 

Reviewers appreciated the novelty of the work but were initially concerned about the lack of comparisons to other methods and lack of experiments with the real robot. The authors have addressed most of the concerns during the rebuttal phase, particularly with the addition of comparisons in simulation. Authors are encouraged to provide videos demonstrating the sampling strategy and possible demonstrations on the real robot.
Decision: Accept (Poster)