Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph

Authors: utkarsh a mishra
yongxin  chen
danfei  xu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 11:07:13

Abstract: Learning to plan for multi-step, multi-manipulator tasks is notoriously difficult because of the large search space and the complex constraint satisfaction problems. We present Generative Factor Chaining (GFC), a composable generative model for planning. GFC represents a planning problem as a spatial-temporal factor graph, where nodes represent objects and robots in the scene, spatial factors capture the distributions of valid relationships among nodes, and temporal factors represent the distributions of skill transitions. Each factor is implemented as a modular diffusion model, which are composed during inference to generate feasible long-horizon plans through bi-directional message passing. We show that GFC can solve complex bimanual manipulation tasks and exhibits strong generalization to unseen planning tasks with novel combinations of objects and constraints. More details can be found at: https://sites.google.com/view/generative-factor-chaining

Main paper: 
 introduction: 
 Solving real-world sequential manipulation tasks requires reasoning about sequential dependencies among manipulation steps. For example, a robot needs to grip the center or the tail of a hammer, instead of its head, in order to subsequently hammer a nail. The complexity of planning problems increases when multiple manipulators are involved, where spatial coordination constraints among manipulators need to be satisfied. In the example shown in Figure 1 , the robot has to reason about the optimal pose to grasp the hammer with the left arm, such that the right arm can coordinate to re-grasp. Subsequently, the two arms must coordinate to hammer the nail. While classical Task and Motion Planning (TAMP) methods have shown to be effective at solving such problems by hierarchical decomposition [1] , they require accurate system state and kinodynamic model. Further, searching in such a large solution space to satisfy numerous constraints poses a severe scalability challenge. In this work, we aim to develop a learning-based planning framework to tackle complex manipulation tasks with both sequential and spatial coordination constraints.To solve complex sequential manipulation problems, prior learning-to-plan methods have largely adopted the options framework and modeled the preconditions and effect of the options or primitive skills [2, 3, 4, 5, 6, 7] . Key to their successes are skill chaining functions that determine whether executing a skill can satisfy the precondition of the next skill in the plan, and eventually the success condition of the overall task. However, the use of vectorized states and the assumption of a linear chain of sequential dependencies limits the expressiveness of these methods. Consider a task where a robot fetches two items from a box. Intuitively, the skills for fetching one object should not influence the other. However, due to vectorized states and the linear dependency assumption, the skill-chaining methods are forced to model such sequential dependencies. Similarly, a skill intended to satisfy a future skill's condition will be forced to influence the steps in between. Finally, the skill chain representation forbids these methods from effectively modeling multiple-arm manipulation tasks, where concurrent skills must be planned to jointly satisfy a constraint.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. We formulate all factors as distributions of the nodes connected to them. GFC represents spatial factors as classifier-distributions and temporal factors as diffusion model learned distributions. We leverage compositionality of diffusion models to compose spatial-temporal distributions and find the joint distribution of the complete plan directly at inference. (c) Output: Finally, samples drawn from such a joint distribution are symbolically and geometrically feasible solutions of the whole plan.To move beyond the linear chain and model complex coordinated manipulation, we introduce Generative Factor Chaining (GFC), a learning-to-plan framework built on flexible composable generative models. For a given symbolically feasible plan graph, GFC adopts a spatial-temporal factor graph [8] representation, where nodes are objects and robot states, and spatial factors represent the relationship constraints between these nodes. Skills are temporal factors that connect these state-factor graphs via transition distributions. A single skill factor can simultaneously connect to multiple object and robot nodes, allowing for natural representation of complex multi-object interactions and steps that necessitate coordination between multiple manipulators. During inference, this factor graph can be treated as a probabilistic graphical model, where the learned skill factor and spatial constraint factor distributions are composed to form a joint distribution of complete plans. Through 13 longhorizon manipulation tasks in simulation and the real world, we show that GFC can solve complex bimanual manipulation tasks and exhibits strong generalization to unseen planning tasks with novel combinations of objects and constraints. 
related work: 
 Task and Motion Planning (TAMP). TAMP frameworks decompose a complex planning problem into constraint satisfaction problems at task and motion levels [9, 2, 10, 11, 12] . Notably, Garret et al. [1] drew connections between TAMP and factor graphs [8] , representing constraints as factors and objects/robots as nodes. This formalism naturally allows reusing per-constraint solvers across tasks. While classical TAMP relies on accurate perception and system dynamics, limiting scalability, we take a learning approach, though our compositional factor graph representation remains inspired by classical TAMP.Generative models for planning. Modern generative models have been applied to offline imitation [13, 14, 15, 16, 17, 18, 19, 20] and reinforcement learning [21, 22] . In addition to modeling complex state and action distributions, generative models have also been shown to encourage compositional generalization [23, 6, 24] by combining data across tasks [22, 21] . Most relevant to us are Generative Skill Chaining (GSC) [6] and Diffusion-CCSP [25] , both designed to achieve systematic compositional generalization. GSC composes skill chains through a guided diffusion process but fail to solve non-sequential dependencies similar to other skill-chaining methods [4, 5] . Diffusion-CCSP trains diffusion models to generate configurations to satisfy spatial constraints and use exter-nal solvers to plan the manipulation sequence. Our method is a unified framework and generates motion plans to satisfy both spatial and temporal constraints represented in a factor graph.Learning for coordinated manipulation. Coordinating two or more arms for manipulation presents numerous planning challenges [26, 27, 28] , including the combinatorial search space complex constraints for coordinated motion. Recent works have utilized learning-based frameworks [29, 30, 31, 32, 33] in both Reinforcement Learning [29, 31] and offline Imitation Learning [33, 32] . However, most existing works have focused on learning task-specific policies [29, 32] or require multi-arm demonstration data collected through a specialized teleoperation device [33] . In contrast, our factor graph-based representation enables solving multi-arm tasks by composing multiple single-arm skills through inference-time optimization.3 Background Diffusion Models. A core component of our method is based on distributions learned using diffusion models. A diffusion model learns an unknown distribution p(x (0) ) from its samples by approximating the score function ∇ log p. It consists of two processes: a forward diffusion or noising process that progressively injects noise and a reverse diffusion or denoising process that iteratively removes noise to recover clean data. The forward process simply adds Gaussian noise ϵ to clean data as x (t) = x (0) + σ t ϵ for a monotonically increasing σ t . The reverse process relies on the score function ∇ x log p t (x (t) ) where p t is the distribution of noised data x (t) . In practice, the unknown score function is estimated using a neural network ϵ ϕ (x (t) , t) by minimizing the denoising score matching [34] objective E t,ϵ,x (0) [λ(t)∥ϵ -ϵ ϕ (x (t) , t)∥ 2 ]where λ(t) is a time-dependent weight. Several recent works have explored the advantages of diffusion models like scalability [35, 36, 37, 38] and the ability to learn multi-modal distributions [39, 40, 41, 22] . We are particularly interested in the compositional ability [23, 13, 24, 25, 6] of these models for the proposed method.Problem setup. We assume access to a library of parameterized skills [42] π ∼ Π such as primitive actions like Pick and Place. Each skill π requires a pre-condition to be fulfilled and is parameterized by a continuous parameter a ∈ A π governing the desired motion while executing the skill in a state s. For a given symbolically feasible task plan from a starting state s 0 to reach a specified goal condition s goal , generated by a task planner or given by an oracle, the problem is to obtain the sequence of continuous parameters to make the plan geometrically feasible. For example, given a nail at a target location and a hammer on a table, the symbolic plan is to Pick the hammer and Reach the nail. A geometrically-feasible plan requires suitable Pick and Reach parameters such that the hammer's head can strike the nail.Learning for skill chaining. Existing works along this direction model the planning problem as a "chaining" problem: They first model the pre-conditions and effect state distributions for every skill π ∼ Π from the available data and a symbolic plan skeleton Φ K = {π 1 , π 2 , ..., π K } consisting of K-skills is constructed. With this model, they search for the given skill sequence (plan) such that each skill satisfies the pre-conditions of the next skill in the plan. STAP [5] used learned priors to perform data-driven optimization with the cross-entropy maximization method. In GSC [6] , the policy and transition model is formulated as a diffusion model based distribution p π (s, a π , s ′ ) which allows for flexible chaining. While the forward chain ensures dynamics consistency in the plan, backward chain ensures that the goal is reachable from the intermediate states. For a forward rollout trajectory τ = {s 0 , a π1 , s 1 , a π2 , s goal } associated with skeleton Φ 2 = {π 1 , π 2 }, the resulting forward-backward combination based on GSC [6] can be represented asEQUATION 
method: 
 We aim to solve unseen long-horizon planning problems by exploiting the inter-dependencies between the objects relevant for the task at hand. Our method adopts factor graphs to represent states and realize their temporal evolution by the application of skills. While previous works have considered vectorized state representations making it difficult to decouple spatial-independence, we focus on factorized state representations such that the state of the environment is entirely modular, containing information about all the objects in the scenario and the task-specific constraints between them. We use a spatial-temporal factor graph [8] that is transformed into a probabilistic graphical model by representing temporal factors as skill-level transition distributions and spatial factors as constraint-satisfaction distributions. A composition of all the factors jointly represents sequential and coordinated manipulation plans directly at inference and can be solved by sampling optimal node variables using reverse diffusion sampling.States as factor graphs. We define a factor graph {V, F} of a state s consisting of the decision variable V and factor F nodes. Every robot and object is represented as a decision variable node v ∈ V containing their respective state. Factors f ∈ F between nodes in a given state are spatial constraints. For example, a Grasped spatial factor specifies admissible rigid transforms between a gripper and an object. When we construct a probabilistic graphical model from the representation described above, an intuitive way of calculating the distribution of a state, p(s), is the composition of all the factor distributions. Mathematically:EQUATIONwhere p f (S f ) represents the joint factor potential of nodes v ∈ S f ⊆ V, i.e. all nodes involved in a factor 1 and s is the joint distribution of all such nodes. This indicates that the joint distribution of all the nodes must satisfy each of the factors, also explored by Diffusion-CCSP [25] .Skills as temporal factors. To represent transitions between states, we adapt parameterized skills [42] for a factor graph formulation. We define the preconditions of a skill as a set of nodes and factors, thus considering a skill feasible iff the precondition factors are satisfied. For example, for state s 0 illustrated in Figure 1 , the nodes of a factor graph are {L 0 , H 0 , R 0 , B 0 } and the factors existing in this scene are {Grasped(L 0 , H 0 )=True}. Now, since this factor is a precondition of the skill Move(L 0 , H 0 ) that moves the hammer in hand to align with the box, it must be satisfied for the skill to be feasible. The effect of executing a skill creates a new factor graph s ′ by changing the state of the nodes involved and, optionally, adding or removing their factors. This results in a temporal factor between the transitioned nodes of s and s ′ with the continuous action parameter of the skill a π . The skill definitions can be extracted from standard PDDL symbolic skill operator with minor adaptations, following the duality of factor graphs and plan skeletons [1] . Eventually, we solve an optimization problem: satisfying the Aligned, Grasped, and the transition dynamics constraints by finding the correct Move parameters a π1 . Each skill in a plan introduces additional nodes and factors to the factor graph, with added complexity for optimization.Mathematically, we can use the distribution p(s) as established in Equation 2 with all the spatial factors, and represent the temporal skill factor distribution of k th -skill π k as the joint distribution:p π k (s, a, s ′ ) ≡ p π k (S π k , a, S ′ π k ), S π k ⊆ V π k pre which is executable iff the skill's pre-condition s π k pre ≡ {V π k pre , F π kpre } is satisfied by the current state i.e. V π k pre ⊆ V and F π k pre ⊆ F. Once executed, it leads to the transitioned state S ′ π k . Based on the above formulation of a short-horizon transition distribution, we extend to construct a plan-level distribution as already established by GSC [6] and shown in Equation 1. We leverage the modularity of factored states by replacing states s with a set of decision variables S π k in the interest of skill π k . This allows us to chain multiple skills in series and parallel. In such a scenario, the denominator term exists only for certain decision nodes iff they are common in two consecutive skills. We can indeed rewrite Equation 1 as:EQUATIONif we consider that some set of intermediate nodes V i are connected by two sequential skills π i-and π i+ .A key advantage of the factor graph representation is the ability to model multi-arm coordination tasks by connecting the temporal chains of each arm using spatial constraints. Such tasks often require skills to be simultaneously executed on each arm to operate on different or the same objects. We consider two cases for parallel skill execution, where multiple robots are operating on: (1) independent objects and (2) the same object, leading to independent and dependent temporal chains respectively. With our factorized state representation, we can independently control the execution of individual skills correlated with the nodes of interest and calculate the cumulative effect by applying the union of the effects of all the skills to the current factor graph. We consider a scenario shown in Figure 2 (Left). The left and right gripper arm L 0 are holding the pink C 0 and green M 0 cup ({Grasped(L 0 , C 0 )=True} and {Grasped(R 0 , M 0 )=True}) respectively. While both the grippers can independently execute the skill Move to modify separate factors (f π1 1 and f π2 2 ), one can add a constrained relationship factor (µ 1 ) between the two mugs representing a set of transforms that satisfy the precondition of Pour. Such an ability to augment constraints flexibly allows zero-shot coordination planning for unseen tasks at test time even with parallel skill executions on the same object as shown in Figure 2 (Right).Now we have a formulation to construct a symbolic spatial-temporal factor graph plan for a task and chain them using spatial factor and temporal skill factors sequentially or in parallel. To make this plan geometrically feasible, we must find the optimal node variable values. We leverage the expressive generative model to capture the transition dynamics and exploit the compositionality of diffusion models. Given a symbolically feasible factor graph plan, our method, termed Generative Factor Chaining (GFC), can flexibly compose spatial-temporal factor distributions to sample optimal node variable values for the complete plan.Probabilistic model for trajectory plan as spatial-temporal factor graphs. Now, we again consider the spatial graph for representing the state, where the probability of finding a state s is the joint distribution of all the nodes in the factor graph. We will now integrate the spatial factors with the temporal factors considering the compensation term introduced in Equation 2and Equation 3 along with the constraint factors across the chain µ ∈ M as:EQUATIONThis completes the joint distribution of all the nodes in the spatial-temporal factor graph plan considering the temporal factors for all skills with their pre-condition and effect nodes, all spatial factors for all states in the plan, and all intermediate nodes in the temporal chain. It is worth noting that the augmented constraint factors f µ work as a weighing function and can be more precisely represented by f µ (S µ ) ≡ f µ (y = 1|S µ ) for some constraint-satisfaction index y.We align towards diffusion model-based learned distributions to represent the probabilities in the formulated probabilistic graphical model. We transform the probabilities into their respective score functions ϵ(x (t) , t) for a particular reverse diffusion sampling step t and train it using score matching loss. Hence, for sampling a scene-graph for Equation 4, we haveϵ(τ (t) , t) = π k ∈Φ ϵπ k (v (t) k ∈ V π k pre , a (t) k , v (t) k+1 ∈ V π k ef f ect , t) + K k=0 f ∈F k ϵ f (S (t) f , t) - 1 2 v i ∈V i ϵπ i-(v (t) i , t)ϵπ i+ (v (t) i , t) + M ϵ fµ (S (t) µ , t)Such a representation leads to a cumulative score calculation of the joint distribution of all the nodes of interest to the factor using linear addition and subtraction. We can realize from Equation 4.2 that the final score function depends on the composition of all the factors in the spatial-temporal factor graph. While factors f ∈ F are mostly modeled implicitly by the temporal skills, the external factors can be any arbitrary spatial constraints that ensure the satisfaction of the pre-condition of the subsequent skills. Hence, with new additions to the set of external factors µ ′ ∈ M ′ , one can reuse the same temporal skills with added new spatial constraints. The proposed approach is modular as the individual skill factors and constraints can be flexibly connected to form new graphs. We have provided additional details in algorithm 1. 
experiment: 
 In this section, we seek to validate the following hypotheses: (1) GFC relaxes strict temporal dependency to allow spatial-temporal reasoning, performing better or on par with prior works in singlearm long-horizon sequential manipulation tasks, (2) GFC can effectively solve unseen coordination tasks, and (3) GFC is adept in reasoning about long-horizon action dependency while being robust to increasing task horizons. We systematically evaluated our method on 9 long-horizon single-arm manipulation tasks from prior works and 4 complex multi-arm coordination tasks in simulation. We also demonstrate deploying GFC on a bimanual Franka Panda setup in the real world.Relevant baselines and metrics: Our proposed method is based on factorized states and supports long-horizon planning for collaborative tasks directly at inference via probabilistic chaining. In this context, we consider prior methods based on probabilistic chaining with vectorized states (GSC [6] ) and discriminative search-based approaches for solving long-horizon planning by skill chaining: with uniform priors (Random CEM or RCEM) or learned policy priors (STAP [5] ). Since all prior works use sequential planning, we compare the performance of the proposed method on the sequential version of the parallel skeleton. Further information on data collection, training of skill diffusion models and real robot setup is provided in Supp. S4, Supp. S5 and Supp. S6 respectively.GFC relaxes strict linear dependency assumptions. We first evaluate GFC on single-manipulator long-horizon tasks introduced by STAP [5] and also used by GSC [6] . These tasks consider manipulation by reasoning about the usage of a tool (a hook) to manipulate blocks out of or into the robot workspace (sample initial states shown in Figure 3(a-c )). While these tasks are originally designed to highlight linear sequential dependencies, there are steps with indirect dependencies or independence that only GFC can effectively model because of the factorized states. For example, in Rearrangement Push, the picking pose of the cube should not affect the tool use steps. As shown in Table 1 , we observe that the performance of GFC is consistently on-par with the baseline for tasks with strict linear dependencies such as Hook Reach and on-par or better for tasks with more complex Table 1 : We show performance comparison of our method with relevant baselines on 9 single manipulator tasks and 3 two-manipulator tasks based on 100 trials for each of them. The task length shows the relative difficulty of solving them. We also conduct evaluation on 3 extended tasks to show robustness of GFC to task length (|T |) and efficient reasoning about interstep dependencies. dependency structures such as Rearrangement Push. This validates our hypothesis that GFC effectively models sequential dependencies, in addition to independence and skipped-step dependencies in long-horizon tasks. GFC can solve complex coordinated manipulation tasks. Here, we aim to validate that GFC can effectively plan and solve different types coordinated manipulation tasks. We present results on tasks with increased collaboration challenges. First, we consider tasks that require coordination but can be serialized into interleaved skill chains and solved by prior skill-chaining methods. Hammer Place, as shown in Figure S16 , is for one arm to pick a hammer, hand it over to another arm for placemement into a target box. Hammer Nail is an extension where, after hammer handover, first arm picks up a nail and both arms coordinate to move to positions such that the hammer's head is aligned with the nail for the subsequent striking step. The task is illustrated in Figure S16 . As evident from Table 1 , GFC significantly outperforms all baselines in both tasks. The gap is larger in the more challenging Hammer Nail task, which includes additional spatial and temporal constraints as shown in Supp. S7. This demonstrates that GFC can effectively model and resolve both spatial and temporal constraints in complex tasks.GFC can zero-shot generalize to new bimanual tasks by composing single-arm skill chains. The Pour Cup (Figure S11 ) task is to Pick a cup with each arm, Move to position the two cups, and Pour the content of one into the other. GFC can directly reuse Pick and Move skill models and adapt the Strike skill model for the Pour step by adding a new spatial constraint. The constraint that "the cups can only be poured using the open top and not the closed bottom" can be directly added as a factor and optimized globally through guided diffusion process. A quantitative comparison is shown in Table 1 . Finally, we consider the Bimanual Reorientation (Figure S12 ) task where two arms must simultaneously operate on the same object of interest (a pot), lift it up, and rotate it to a target reorientation angle (about z-axis) as illustrated in Figure 4 (Top) for a 45-deg angle. The tasks must be solved via parallel skill chaining with spatial constraints and hence none of the prior baselines can be used. The factor graph ( Figure 2 Right) includes a spatial fixed transform constraint between both the arms and hence the subsequent skills operate while satisfying the constraint. Figure 4 (Bottom) shows a detailed task success rate breakdown given different orientation goals.GFC can handle independence and inconsistent skill chains. Here, we analyze how independent steps in a sequential manipulation chain affects the performance of each method. We consider Hammer Place, where the order of transporting the cube and handing over hammer is interchangeable. As illustrated in Figure 5 , we consider a consistent plan skeleton where sequentially-dependent steps for the two main objectives, i.e., (1) opening lid then transporting cube and (2) picking, handing over, Figure 5 : Linear chaining has limitations. Baseline methods with linear chain assumption suffers from performance drop when given inconsistent skill chains, where steps with sequential dependencies are swapped. GFC retains high success rate using the parallel skeleton. GFC can reason about action dependency while being robust to increasing task horizons. We observe in Figure 6 (left) that while Hammer Place task can be solved by picking or grasping on any end of the hammer handle, Hammer Nail requires more constrained parameter sampling. Further, in addition to the parameter selection along the handle axis, the method also samples suitable orientation (same or flip side) for grasping as shown by two examples in Figure 6 (right). We further give an example of the capability of our method in handling longer horizon inter-step dependencies in Figure S17 and simultaneously being robust with respect to the task length as shown in Table 1 . 
limitations: 
 First, our method does not generate high-level task plans. Solving the full TAMP problem with a unified generative model is an important future direction. Second, our method operates in a lowdimensional state space and hence requires a state estimator. We plan to extend GFC to work with high-dimensional observations. Finally, similar to prior works [4, 5, 6] , our approach operates on parameterized skills. 
conclusion: 
 We presented GFC, a learning-to-plan method for complex coordinated manipulation tasks. GFC can flexibly represent multi-arm manipulation with one or more objects with a spatial-temporal factor graph. During inference, GFC composes factor graphs where each factor is a diffusion model and samples long-horizon plans with reverse denoising. GFC is shown to solve sequential and coordinated tasks directly at inference and reason about long-horizon action dependency across multiple temporal chains. Our framework generalizes well to unseen multiple-manipulator tasks. 
acknowledgement: 
 This project is funded by NSF Award 2409016 and the Georgia Tech AMPF Program. 
 
 Appendix: i.e. a factor f is included iff there is an edge between f and some v ∈ V which also implies v ∈ S f ⊆ V. 


Decision: Strengths:
1. Integrates ideas from composable diffusion-based spatial constraint satisfaction and skill chaining in a unified framework, extending the application to long-horizon, multi-robot coordination.
2. Demonstrates comprehensive experimental evaluation on a variety of tasks, both in simulation and real-world settings, showcasing the method's generalization to unseen tasks.
3. Successfully handles challenging tasks like bi-manual manipulation of shared objects, demonstrating practical utility.
4. The paper is thoroughly organized, with a well-motivated introduction, clear illustrations, and a detailed explanation of the methodology and experiments.
5. Clearly addresses the limitations of the work, including the lack of integration with TAMP and constraints on object states.

Weaknesses:
1. The separate treatment of spatial and temporal factors seems unnecessary and could be simplified, as the current method introduces complexity without clear justification.
2. The framework supports constraints only at discrete timesteps, not throughout skill execution, potentially leading to collisions during motion.
3. Requires a predefined graph structure before applying the diffusion models, which is not sufficiently emphasized or justified.
4. Lack of clarity on whether diffusion models correspond to skill or spatial factor nodes, and how they handle larger graphs.
5. Questions remain about the method's scalability to larger graphs and tighter spatial constraints.

Reviews: review: title: title: Combines diffusion-based inference ideas for enforcing spatial- and temporal- constraints. Paper very well written with extensive experiments.
paper_summary: paper_summary: The paper presents an approach for planning long-horizon trajectories for multi-step, multi-arm manipulation by composing multiple diffusion models in a factor graph framework. The planning problem is represented as factor graph [8]. The nodes of the factor graph are states of robots and objects that are either known or are treated as decision variables be optimized. Factors represent spatial and temporal constraints between nodes (such as pre- and post- conditions of skills, relative pose constraints etc.). Each diffusion model is trained to learn individual factor types. Then, the score function of the joint probability distribution is composed of score functions of individual diffusion factors. This enables modular, composable inference to generate long-horizon plans that show strong generalization to unseen tasks and novel combinations of objects and constraints. An extensive experimental study on simulated and real robots demonstrates that the proposed method solves sequential and coordinated multi-robot tasks such as handover maneuvers, hammering nails etc. with superior performance and generalization compared to existing baselines.
main_review: main_review: ## Strengths

- **Combines ideas from composable diffusion-based spatial constraint satisfaction and skill chaining.** The method combined approaches for composable diffusion based spatial constraint satisfaction from Diffusion-CCSP [25] and diffusion-based temporal skill chaining from GSC [6] in a unified framework. Using composable diffusion models in a factor graph framework is not new [25], however, as far as I’m aware, this is the first work that applies this framework to spatial *and* temporal constraint composition for long-horizon, multi-robot coordination.
- **Experiments are extensive.** The paper performs experimental evaluation on wide variety of tasks — 9 long-horizon single-arm manipulation tasks from prior works and 4 complex multi-arm coordination tasks. Experiments are perform both in simulation and the real world.
- **Experiments demonstrate strong generalization.** The experimental evaluation demonstrates strong generalization to unseen planning tasks with novel combinations of objects and constraints.
- **Effective bimanual manipulation:** Bimanual manipulation of a single, shared object is typically challenging due to the closed kinematic chain. It was satisfying to see in the supplementary video that the proposed method produces bimanual motion plans to manipulate a large cooking pot reasonably well on real robots, by automatically composing skills that were developed for single arms.
- **Paper is well written.** The paper is very well written, thoroughly organized and easy to follow.
    - **Introduction is well motivated.** The introduction motivates the gaps in prior work on skill chaining such as linear dependencies, and the necessity of spatio-temporal factor graphs multi-robot coordinated manipulation.
    - **Experiments section is very well organized.** The section begins with explicitly stating what hypothesis the paper is seeking to validate, and explicitly states the conclusions and key findings from the experiments.
    - **Useful illustrations and running examples.** The running example of the hammering task is very helpful for the user to understand the proposed method. The factor graph illustrations in Fig. 1 and Fig. 2 also help explain the proposed concepts effectively. It was very helpful for Fig. 2 to show examples of factor chaining when (1) robots independently operate separate objects, and (2) when they operate a common object.
    - Appendix Sec. S4 provides a helpful and illustrative explanation of the proposed method.
- **Paper described limitations of the work.** Paper clearly describes the limitations of the work (not integrated with TAMP, uses object state etc.) and the supplementary video shows failure cases of bimanual manipulation of the large pot.

## Weaknesses

- **Factor graph inference algorithm can be simplified by treating spatial and temporal factors similarly.**
    - The proposed method treats factors representing spatial constraints (at the same timestep) *differently* from skill factors representing temporal constraints. The former uses the technique from Diffusion-CCSP where score functions of spatial factors are simply added (based on Eq. 2). The latter uses the forward-backward analysis of GSC [6] which introduces additional terms in the denominator of the probability density (Eq. 3). It seems to me that this separate treatment is awkward and unnecessary. The method can be simplified by just adding the score functions of both spatial and temporal factors following Diffusion-CCSP without additional terms in the denominator.
    - The paper attempts to justify “denominator compensation” in the Appendix Sec. S3 lines 64-75. However the provided justification seems problematic. Specifically, once the expression for $p(\tau | s_0)$ and $p(\tau | s_1)$ are written down (between lines 72 and 73), Eq. (S3) combines them by assuming  $p(\tau | s_0, s_2) \propto \sqrt{p(\tau| s_0) \ p(\tau | s_2)}$. However, it is unclear how this can be theoretically justified. Instead, the expression should be:
        
        $$
         \begin{align*}
          p(\tau | s_0, s_2) &\propto p(s_0, \tau, s_2)
          \propto p(\tau | s_0)\ p(s_2 | \tau)
          \propto \frac{p(\tau | s_0)\ p(\tau | s_2)}{p(\tau)}
          \end{align*}
        $$
        
         My point is that “denominator compensation” seems inaccurate and ad-hoc to begin with, and is probably unnecessary for the method.
        
    - My recommendation is to conduct experiments where denominator compensation is removed, and evaluate whether it is necessary for empirical performance. If not, doing away with denominator compensation will not only simplify the proposed method, but it will also treat spatial and temporal factors in a more consistent and unified way.

- **Framework does not support constraints during skill execution.**
    - The framework supports constraint satisfaction at discrete timesteps, corresponding to the beginning and end of skill executions. This allows applying collision constraints to ensure that the robots are not colliding with other objects only at mode switches (corresponding to states $ \{ s_0, s_1, s_2, \dots \} $ ). But the method cannot guarantee that robots won’t collide *during* the execution of the skills. This is because individual skills are learned independently and modularly (in the absence of other objects and robots) and are executed independently.
    - Similarly, when performing bimanual manipulation of the pot, relative pose constraints between the pot and two end-effectors can only be enforced at the beginning and end of the trajectory. But the closed kinematic chain requires that the relative pose constraints be satisfied *throughout* trajectory execution. How is/can this be achieved in the proposed framework?

## Minor weaknesses

- It would be useful to explicitly cite which prior works the 9 single-arm manipulation tasks (e.g. *Hook Reach*, *Rearrangement Push* etc.) are derived from.
- Typo in Line 90: “multi-model distributions” should be “multi-modal distributions”.
questions: questions: Please discuss and respond to the issues raised in the “weaknesses” section of the review.
limitations: limitations: 3
review_summary: review_summary: Overall, the paper combines approaches for composable diffusion based spatial and temporal constraint satisfaction in a unified framework, and applies it to challenging long-horizon, multi-robot coordination. The paper is very well written. The experimentation is extensive and demonstrates good generalization to novel combinations of objects and constraints. Notwithstanding some concerns about simplifying the inference algorithm, this is a good paper and I recommend a strong accept.
title: title: A factor graph approach for coordinated manipulation
paper_summary: paper_summary: This paper presents a factor graph-based technique for long-horizon task and motion planning. A representation which models states of all agents and objects as nodes and constraints and skills as factors is described. Experiments on simulated and real-world bimanual manipulation tasks demonstrate the success of the method.
main_review: main_review: Overall, the approach is well-motivated and the concept of long-horizon task planning represented as a factor-graph has merit. The bimanual tasks demonstrated on real robots are challenging and relevant. However, the work is lacking sufficient details and clarity to understand the implementation of the factor graph which risks limiting the impact of the work. 

**Strengths:**

* The experiments include challenging bimanual manipulation tasks which the method handles successfully.
* The authors compare against multiple baselines, which strengthens the claims.
* The use of the factor graph for TAMP is well-motivated and grounded in the literature.

**Points of feedback:**

The paper would benefit from a formal statement of the assumptions of the technique and the optimization problem being solved. This would help clarify how this method is more representative than existing methods, e.g. PDDL.  Notably, it is unclear what form the actions and states take (e.g. continuous or discrete), and how the inference problem stated in Eq. (1) is solved. The stated objective is to solve long-horizon planning problems, but the limitations section states that the proposed approach “does not generate high-level task plans.” A clearer formulation would help clarify the output of the method.

Some technical elements could use clarification. Specifically, how the connectivity of the factor graph itself is generated. It seems that the factor graph might encode both discrete (action type, grasped state) and continuous (object poses) variables, which would make this a non-trivial problem to solve (see for e.g. [1, 2]). Furthermore, the paper refers to constraints as probabilistic factors. It is notoriously challenging to formulate planning as inference with constraints, particularly in sampling-based planning (see [3]). How is this achieved in the proposed work?

The results would benefit from further analysis. Beyond success rate, it would be useful to know what causes failures. This is of particular interest for the tasks with low success rate, such as the longer-horizon evaluation tasks (bottom of Table 1), as this is a major claim of the work and the success rates are low, which contradicts hypothesis (3) (line 209). Additionally, the authors may consider including time to completion as a metric to demonstrate the parallel execution benefit.

The claims and methodology would benefit from more clarity in multiple areas. As currently written, it is difficult to follow the details of the implementation. Specifically:
* Multiple terms in the introduction are not defined, including “options” and “vectorized” states. 
* The claim that previous works make a “linear” assumption is a bit unclear. Does this refer to the mathematical formulation of the motion planning problem or a sequential order of events?
* In the problem setup, Section 3, $\pi$, $a$, and $s$ should be related.
* In Section 4, it would be helpful to define vectorized and factorized states explicitly.
* In Table 1, T1, T2, and T3 are undefined and it is unclear how the task length is determined.
* The concepts of “consistent” and “inconsistent” skeletons is unclear. In Fig 5 for example, the inconsistent skeleton appears to be a subset of the consistent skeleton.

*Other points of feedback:*

* The figures of the factor graph are hard to parse due to the heavy use of notation. Figures 1 and 2 are very similar.
* Which skills are available? What spatial constraints exist? What assumptions are there about these skills and constraints? 
* The equations relating to specific examples (Eq. (3) and the unnumbered equation after Eq. (4)) are hard to parse and could be moved to supplementary material.
* References 8, 43, and 46 are duplicates. (*Note: Added after official rebuttal period*)

**References**

[1] Yewon Lee, Philip Huang, Krishna Murthy Jatavallabhula, Andrew Z. Li, Fabian Damken, Eric Heiden, Kevin Smith, Derek Nowrouzezahrai, Fabio Ramos, Florian Shkurti. STAMP: Differentiable Task and Motion Planning via Stein Variational Gradient Descent. Conference on Robot Learning (CoRL), Learning Effective Abstractions for Planning Workshop, 2023.

[2] Alphonsus Adu-Bredu, Nikhil Devraj, Odest Chadwicke Jenkins. Optimal Constrained Task Planning as Mixed Integer Programming. International Conference on Intelligent Robots and Systems (IROS), 2022.

[3] Thomas Power and Dmitry Berenson. Constrained Stein Variational Trajectory Optimization. IEEE Transactions on Robotics, 2024.
questions: questions: The following are suggestions for the revised manuscript:

1. Clarify the problem formulation, assumptions, and methodology (especially how the factor graph is generated and how it is solved)
2. Clarify language and definitions of terms in the manuscript.
3. What accounts for the failures in the results?

See above for details and further feedback.
limitations: limitations: 2
review_summary: review_summary: The paper considers a challenging task and proposes a solution that performs better or on par to proposed baselines. There are multiple areas where the problem formulation, methodology, or results are unclear. Further, the results would benefit from more analysis to explain failures, particularly where they are inconsistent with claims.
title: title: Good work but lack many important details
paper_summary: paper_summary: This paper proposes to use spatial temporal factor graph as a representation of robotics planning problems, called generative factor chaining (GFC). They use factored state representation and states are represented as variable nodes in the graph. Spatial constraint are represented using spatial factor node. GFC uses diffusion models as generative representation for each factor. For a planning problem, assuming the factor graph is given, GFC jointly optimize the diffusion models to obtain the final solution to the whole factor graph. The method is evaluated on both sim and real environments.
main_review: main_review: Pros
- Representing the planning problem as a spatial temporal factor graph is a good idea. The graph representation is more general than linear chains, which is adopted by a few previous approaches. The advantage is shown by the experiments also.
- The authors are clear about what are all the nodes in the graph. The factored state representation makes it easier for the model to learn and do inference.
- The idea of composing diffusion models to solve long-horizon problems is a good one. The current framework have some task level generalization theoretically. 
- I appreciate the real world experiments.

Cons

While the idea is a good one, a few important details are missing out or not emphasized enough. The authors should also make it clearer what is the benefit to prior methods.
- The whole framework still requires a given graph structure, before the diffusion models are used to find the value assignment. This is a very important point but not emphasized enough. I hope the authors can make it clearer and bring it up early in the writing.
- Using factor graph as the representation is not new. It has been explored in the TAMP literature as pointed out by the authors. The major difference to me is differentiating the spatial nodes and the temporal nodes but I don’t see a clear benefit of that. Given a plan skeleton, the planning problem is a CSP and can readily be represented as one factor graph. If the authors want to highlight the representational advantage, it should be made clearer what is the benefit of having spatial and temporal factors.
- Not enough information about the diffusion models. Does one diffusion model correspond to a skill factor node or a spatial factor node, or either one? Also, the input are factored states and it’s assumed there is a state estimator giving all the required information. It should be made clearer too.
- The assumption on different part of the plan being independent is fine if the spatial constraints are not tight. If the task is to place blocks on a narrow shelf, then the placement of the first blocks will affect the placement of the second one, or the collision-free paths the arm can traverse.
questions: questions: - I’d appreciate more details on the diffusion models. e.g. Does one diffusion model correspond to a skill factor node or a spatial factor node, or either one? Are they trained jointly or separately? How well does it handle larger graphs? I imagine when the graph becomes larger, it’ll be harder for the individually trained diffusion models to all find a good sample.
- Generalization to new plan skeletons/graph structures?
limitations: limitations: 3
review_summary: review_summary: I think it's a good work but there are some important points that should be added or emphasized more in the writing. The difference to prior work with factor graph representation should be made clearer also. Update after the rebuttal: I think my concerns are addressed adequately. Recommendation updated.