Spatial reasoning as Object Graph Energy Minimization
Authors: Nikolaos Gkanatsios
Ayush Jain
Zhou Xian
Yunchu Zhang
Katerina Fragkiadaki
Venue: Submitted to ICLR 2023 (2023)
Keywords: Energy Based Model
Robotics
Goal Generation

Abstract: We propose a model that maps spatial rearrangement instructions to goal scene configurations via gradient descent on a set of relational energy functions over object 2D overhead locations, one per spatial predicate in the instruction. Energy based models over object locations are trained from a handful of examples of object arrangements annotated with the corresponding spatial predicates. Predicates can be binary (e.g., left of, right of, etc.) or multi-ary (e.g., circles, lines, etc.). A language parser maps language instructions to the corresponding set of EBMs, and a visual-language model grounds their arguments on relevant objects in the visual scene. Energy minimization on the joint set of energies iteratively updates the object locations till their final configuration. Then, low-level local policies re-locate objects to the inferred goal locations. Our framework shows many forms of strong generalization: (i)joint energy minimization handles zero-shot complex predicate compositions while each EBM is trained only from single predicate instructions, (ii) the model can execute instructions zero-shot, without a need for paired instruction-action training, (iii) instructions can mention novel objects and attributes at test time thanks to the pre-training of the visual language grounding model from large scale passive captioned datasets. We test the model in established instruction-guided manipulation benchmarks, as well as a benchmark of compositional instructions we introduce in this work. We show large improvements over state-of-the-art end-to-end language to action policies and planning in large language models, especially for long instructions and multi-ary spatial concepts.

Under review as a conference paper at ICLR 2023
SPATIAL REASONING AS OBJECT GRAPH ENERGY MINIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a model that maps spatial rearrangement instructions to goal scene configurations via gradient descent on set of relational energy functions over object 2D overhead locations, one per spatial predicate in the instruction. En- ergy based models over object locations of object arrangements annotated with the corresponding spatial predicates. Pred- icates can be binary (e.g.. left, right, etc.) or multi-ary (c.g., circles, lines, etc.). A language parser maps language instructions to the corresponding set of EBMs, and visual-language model grounds its arguments on relevant objects in the vi- sual scene. Energy minimization on the joint set of energies iteratively updates the object locations to generate goal configuration. Then, low-level policies re- locate objects to the inferred goal locations. Our framework shows many forms of strong generalization: (i) joint energy minimization handles zero-shot complex predicate compositions while each EBM is trained only from single predicate in- structions, (ii) the model can execute instructions zero-shot, without a need for paired instruction-action training, (iii) instructions can mention novel objects and attributes at test time thanks to the pre-training of the visual language ground- ing model from large scale passive datasets. We test the model in established instruction eestructionguided guided manipulation benchmarks, as well as benchmark of compo- sitional instructions we introduce in this work. We show large improvements over state-of- oftt--of-hh-assd art end-to-end language to action policies and planning in large lan- guage models, especially for long instructions and multi-ary spatial concepts.
trained from handful of examples
INTRODUCTION
Rearranging objects to semantically meaningful configurations has many practical applications for domestic and warehouse robots (Cakmak & Takayama, 2013). In this work, we focus on the prob- lem of semantic rearrangement depicted in Figure 1. The input is visual scene and a language instruction. The robot tasked with rearranging the objects their instructed configurations.
End-to-end language to action mapping Many works in semantic scene rearrangement have made progress by mapping language instructions directly to actions (Shridhar et al., 2021; Liu et al., 2021c; Janner et al., 2018; Bisk et al., 2017) or object locations (Mees et al., 2020; Gubbi et al., 2020; Stengel-Eskin et al., 2022). Many recent end-to-end language to action or language to ob- ject locations approaches
transformer architectures to fuse visual, language and action streams (Shridhar et al., 2022; Liu et al., 2021c; Pashevich et al., 2021). Despite their generality and their impressive results within the training distribution, these methods typically do show generaliza- tion to different task configurations, for example, longer instructions, new objects present in the scene, novel backgrounds, or combinations of the above (Liu et al., 2021c; Shridhar et al., 2021). Furthermore, these methods cannot easily determine when the task has been completed and they should terminate (Shridhar et al., 2021; Liang et al., 2022) since they do not model explicitly the goal scene configuration to be achieved.
Symbolic planners and learning to plan To handle the challenges of reactive mapping of lan- guage actions, some methods use look-ahead search to infer sequence of actions object arrangements that eventually achieves the goal implied by the instruction (Prabhudesai et al., 2019). Look-ahead search for scene re-arrangement requires dynamics models that can handle complex

---

Under review as a conference paper at ICLR 2023
Langagg Groandnng
SEGGIAL EONOS J0ags
Iul: SOORE
Patworkk
Featss 16 diccle
penting
Lin Scand coM) boiss, pille
Obsect Graph Funry Naxert Mivii
Cangaage CANSINAAAA
Figure I: Scene rearrangement as object graph energy minimization.
multi-object scenes and generalize across varying number of objects and their configurations. which is an open area of research (LeCun, 2022). Latest learning-based planners (Hafner et al., 2019; 2020; Schritwieser al., 2019; Micheli al., 2022) have not shown such generalization abilities (Goyal et al., 2021; Hamrick et al., 2020). As result, planning for scene re-arrangement is dominated by symbolic planners, such as Planning Domain Definition Language (PDDL) planners (Migimatsu & Bohg, 2019; Kaelbling & Lozano-Pérees 2011; Toussaint, 2015; Lyu et al., 2018). Symbolic plan- ners assume that each state of the world, scene final goal states, and intermediate subgoal states can be sufficiently represented in logical form, using language predicates that deseribe object spatial relations. These methods rely on manually specified symbolic transition rules and planning do- mains. They use object detectors for state estimation (Kase et al., 2020) and interface with low-level controllers for object manipulation. Many recent methods use learning to guide the combinatorial symbolic search by predicting directly semantic subgoals, conditioned on the instruction and the visual scene (Xu et al., 2019; Zhu et al., 2020; Drieß et al., 2020). Symbolic state bottlenecks used in these methods usually limits them to modelling pairwise spatial predicates. Spatial configurations that involve multi-ary relations between objects, such as circles, lines, squares, etc., are not easy to achieve because the long sequence of actions needed to put together such a shape configuration is outsourced to the low-level policy since intermediate states cannot be easily described in symbolic form (Figure-1). Under such long action horizon, the low-level policy often fails.
Large language models for spatial planning Recent methods wish to capitalize knowledge of large language models (LLMs) (Brown et al., 2020; Liu et al., 2021b) for spatial planning (Huang al., 2022a) and instruction following (Huang et al., 2022b; Ahn et al., 2022). Works of (Huang et al,, 2022a) showed that LLMs can predict relevant language subgoals directly from the instruction and a symbolic scene description when conditioned on appropriate prompts, without any additional training (Huang et al., 2022b; Ahn et al., 2022; Huang et al., 2022a). The scene description is provided by an oracle (Huang et al., 2022b) or by object detectors (Huang al., 2022b; Chen al., 2022) as list of objects. Predicted language subgoals, e.g., "pick up the can" are directly fed to low- level policies, or engineered controllers (Huang et al., 2022b). Work of (Liang et al., 2022) builds upon LLMs for code generation and predicts actions in the form of policy programs. It capitalizes on knowledge of LLMs for code generation regarding various functions necessary for semantic parsing of diverse language instructions. For the input programmatic scene description it uses lists of objects alongside their spatial box coordinates, missing from previous works. The language model predicts the actions in terms of low level program functions that interfaces with low level manipulation skills and object detection routines. The hope is that with appropriate prompting the LLM will draw from its vast implicit knowledge of code functions associated with natural language, provided in the comments of programs it has seen, to generate programmatic code that solves novel task instances of scene rearrangement. As highlighted in both (Liang al., 2022: Huang al., 2022b), prompting dramatically affects the model's performance and its ability to predict reasonable subgoals, and programmatic code, respectively. this paper, we question whether the language space the most efficient means to reason about objects and their spatial arrangements.
Spatial reasoning as graph energy minimization In this paper, we introduce Spatial Planning as multi Graph Energy Minimization (SPGEM), framework for spatial reasoning for instruction following via compositional energy minimization over sub-grat -rapphs of object entities, one per spatial predicate in the instruction, as shown in Figure We represent each spatial "predicate" as binary or
2

---

Under review as a conference paper at ICLR 2023
multi-ari energy based model (EBM) over graph of object spatial coordinates (LeCun et al. 2006). Predicates and corresponding EBMs can be binary (for concepts such as left, right, front, behind) or multi-ari (for circles, lines etc.). We train EBMs from examples of visual scenes paired with language descriptions of spatial relations, to predict gradients for object locations updates starting from randomly sampled object arrangements through Langevin dynamics minimization (Du al., 2020b). Given a language instruction, e.g., "put all the small objects in a circle inside the plate", neural semantic parser maps the instruction the set of predicates and corresponding EBMs, and grounds the object arguments for each EBM using visual language grounding models (Kamath et al., 2021; Jain al., 2022). Gradient descent on the sum of energies with respect to the object coordinates provides the final object configurations that best satisfy the spatial constraints. Given the predicted locations of objects, we use pick and place visually grounded short term polices (Zeng et al., 2020) to move the objects to their inferred locations.
We show SPGEM generalizes zero shot to complex predicate compositions, e.g., "put all small ob- jects in circle in the plate" when trained from single relation predicate examples, e.g., " AA inside B" or "a circle of A", as shown in Figure I. Contrary to LLMs, SPGEM has been trained with handful of visually grounded examples of spatial predicates. Contrary symbolic planners, engy minimization "searches over" object placements while traversing -on-symboly meaningful scene configurations, and without manual engineering of the transition operators. Contrary end- to-end language to action models, SPGEM infers a spatial abstraction of the goal scene that helps it to better evaluate whether the task has been completed. By representing EBMs over object locations as opposed to object visual pixel appearances (Du et al., 2020a; Liu et al., 2021a) the model can generalize better to zero-shot spatial concepts to novel objects. We build upon recent advances on training energy based models (Mordatch, 2018; Du et al., 2020b). To the best of our knowledge, this the first work that uses EBMs for scene re-arrangement of complex language instructions.
We test SPGEM in scene re- re-arrangement of table top environments in simulation benchmarks of CLiPort (Shridhar et al., 2021) as well as in a new simulation benchmark we contribute with more compositional instructions. We curate multiple train and test splits to test out-of-distribution gener- alization with respect to i) longer instructions with more predicates, ii) novel object colours iii) novel objects iv) novel background colours. We compare method with SOTA language to action mod- els (Shridhar et al., 2021) as well as LLM planners (Huang et al., 2022b). We ablate each component of our model to isolate contributions of perception, language parsing, goal generation and low-level policy modules. Our model outperforms the baselines, especially in complicated instructions.
In summary, contributions are: (i) An energy-based object-centric framework for language- guided goal seene generation. (fi) A modular system for language-guidde guideed scene re-arrangement, that uses semantic parsers, referential language grounding models, EBMs for scene generation and low-level vision-based policies for instruction following. (iii) Contribution of an extensive scene re-arrangement benchmark in simulation with compositional language instructions and shapes. (iv) Comparisons against SOTA end-to-end language to action models and LLMs, and extensive discus- sions of the pros and cons of each. We have included our code in the supplementary file. We will make our code and datasets publicly available upon publication.
2 RELATED WORK
Instructing robots through language Language robot instruction has a long history. Here, we fo- cus our literature review on the task of spatial scene re-arrangement, and we refer the reader the recent reviews of (Tellex et al., 2020; Luketina et al., 2019) for coverage of broader background work. Language is natural means for communicating goals and can easily describe compositions of actions and arrangements seen at training time (Akakzia et al., 2021; Colas et al., 2020; 2022). Language is a much-preferred means of communicating a task than supplying goal images (Nair et al., 2018; Pong et al., 2019; Seita et al., 2021; Wu et al., 2022). We distinguish the following cat- egories in the literature for robot instruction for scene re-arrangement: (i) End-to-end mapping of language to actions language to object locations (Lynch & Sermanet, 2020; Shridhar et al., 2021; Liu et al., 2021c: Stengel-Eske et al., 2022). (ii) Semantic parsers that map a language command and visual scene description to programs over perceptual and control primitive functions (Karam- cheti et al., 2020; Andreas et al., 2015; Walker et al., 2019; Matuszek et al., 2012; Wang et al., 2020; Srinet et al., 2020). These parsers require annotations of instructions with corresponding programs,
3

---

Under review as a conference paper at ICLR 2023
which are typically hard to collect at scale, despite recent progress on this topic through paraphras- ing of instructions (Wang et al., 2015; Berant & Liang, 2014; Wu et al., 2021). Our work proposes semantic parsing framework that instead of predicting action directly predicts EBMs constraints. We show such representation's 's ability to adapt few shots to complex spatial compositions. (iii) Sym- bolic planners that rely on predefined symbolic rules and known dynamic models, and infer discrete task plans given an instruction with logic search (Kaelbling & Lozano-Pérez, 2011; Garrett et al., 2018). These methods can generalize arbitrary goals defined in a problem domain but necessitate the manual design of the planning domain, its predicates, symbolic rules and transition operators. (iii) Large language models (LLMs) that map instructions language subgoals (Zhu et al., 2020; Xu et al., 2019; Huang et al., 2022a;b) or program policies (Liang et al., 2022). Subgoals predicted interface with low level short-term policies or skill controllers. LLMs trained from Internet-scale text have showed impressive zero-shot reasoning capabilities for variety of downstream language tasks (Brown et al., 2020) when prompted appropriately, without any weight finetuning (Wei et al., 2022; Liu et al., 2021b). Researchers in robot learning have repurposed language models for the task of the instruction following (Chen et al., 2022; Lin et al., 2022). Works of (Huang et al., 2022a;b) reasonable sequence of subgoals when prompted with analogous text action plans open loop or closed loop during instruction execution. The scene description usually provided in symbolic form list of objects present and their relations, pre- dicted from open vocabulary detectors (Kamath et al., 2021). Recent works of (Liang et al., 2022; Lin et al., 2022) have also fed as input overhead pixel coordinated of objects to inform the LLM's predictions. The prompts for these methods need to be engineered per family of tasks. It is yet to be shown how the composition of spatial concept functions can emerge in this way.
have shown that LLMs can map instructions
Language Conditioned Scene Generation. A large body of work has explored scene generation conditioned on text descriptions (Johnson et al., 2018; Ramesh et al., 2021; Saharia et al., 2022; Yu et al., 2022; Mansimov et al., 2015). In this work, we seek to make scene generation useful as goal imagination for robot spatial reasoning and instruction following. Instead of generating pixel- accurate images, we generate object configurations by abstracting the appearance of object entities. We show this abstraction suffices for a great number of diverse scene rearrangement tasks.
Energy based models Our work is related to existing work energy-based models (Grathwohl et al., 2019; Mordatch, 2018; Liu et al., 2021a; Du et al., 2020a;b; 2019). Most similar to our work is that of (Mordatch, 2018) which proposes framework for learning to generate and detect spatial concepts with EBMs on images with dots, and (Du et al., 2020a; Liu et al., 202 which demonstrated composability of image-centric EBMs for RGB face and CLEVR image generation. Here we demonstrate zero-shot composability of graph-centric EBMs, their applicability on spatial reasoning, and their utility for vision-based instruction following for scene re-arrangement.
Constraint Guided Layout Optimization: Automatic optimization for object re-arrangement has been studied outside the field of robotics. (Yu et al., 2011) and (Merrell et al., 2011) use few user- annotated examples of scenes to adapt the hyperparameters of task-specific cost functions, which are then minimized using standard optimization algorithms (hill climbing and/or simulated anneal- ing). To learn those hyperparameters from data, these approaches fit statistical models, e.g. Mix- tures of Gaussian, to the given samples. (Fisher et al., 2012) further employ such optimization constraints into an interactive environment, where the user can provide an initial layout and the al- gorithm suggests improvements. All these approaches require expert knowledge to manually design rules and cost function, namely (Yu et al., 2011) identifies seven and (Fisher et al., 2012) eleven expert-suggested criteria for successful re-arrangement. Since they are hand-crafted, these methods do not generalize beyond the domain of furniture arrangement. In contrast, energy optimization is purely data-driven and domain-agnostic a neural network scores layouts, assigning high energy to those that do not satisfy the (implicit) constraints and low energy to those who do, essentially modeling the underlying distribution of valid layouts.
3 METHOD
SPGEM's architecture is shown in Figure 1. We feed an RGB-D image of the scene and a language instruction. A semantic parser maps the instruction to a set of predicate EBMs and their arguments. A visual grounding model grounds the arguments of each EBM to object entities in the scene. Then, the goal configuration is predicted in a symbolic space of bounding boxes by minimizing the to-
4

---

Under review as a conference paper at ICLR 2023
tal energy wrt object locations. Lastly, short-term vision- pick-and- policies move the objects to their goal locations. We describe each component in more detail in the following.
A library of graph energy- models for spatial concepts In our work, a spatial predicate is represented by an EBM that takes as input x the set of objects that participate in the spatial predicate and maps them a scalar energy value (2)). . An EBM defines a distribution over configurations that satisfy its concept through the Boltzmann distribution po (a) e e-Eo(x), Low-energy graph entity configurations imply satisfaction of the language concept and have high probability. An ex- ample of the spatial concept can be generated by optimizing for low-energy configuration through gradient descent on (part of) the input z. We represent each object entity by its 2D overhead centroid coordinates and box size. During gradient descent, we only update the center coordinates and leave box sizes fixed. We consider both binary spatial concepts (in, left of, right of. in front of, behind of) as well as multi-ary spatial concepts (circle, line).
During training, we sample configurations from po, by starting from an initial configuration x0 and refining it using Langevin Dynamics (Welling& & Teh, 2011):
x 2 + c²2
where 2 is random noise, is an update rate hyperparameter and ch is dime-dependent hyperpa- rameter that monotonically decreases as k increases. The role of 2* and decreasing ex is to induce noise in optimization and promote exploration, similar to Simulated Annealing (Kirkpatrick, 1984) After K iterations, we obtain x= .. During training, we learn the parameters 0 of our EBM us- ing a contrastive divergence loss that pushes the energy of sampled x of ground-truth t::
to be higher than the energy
C=Eytapa Fo(x*) - Eg- Eo
where - a sample for the data distribution pD and x a sample drawn from the learned distribution Pa. Compositions of concepts can be created by simply summing energies of constituent concepts, e.g., "a circle of cubes inside the plate", as shown in Figure 1. For detailed implementation and architectural figures of our energy- based models, please see Section-6.1.2.
Semantic parsing of instructions into spatial concepts graphs and their arguments Our parser maps language instructions instantiations of energy-based models and their arguments. Examples of instructions and paired programs are shown in Figure 1. Our parser is a Sequence-to-Tie
model (Dong & Lapata, 2016) with a copying mechanism (Gu al., 2016) which allows it to handle a larger vocabulary than the one seen during training. Specifically, the input utterance is first encoded using a pre-trained RoBERTa encoder (Liu al., 2019), giving sequence of contextualized word embeddings and global representation of the full utterance. Then, a decoder is iteratively employed to either i) decode an operation (EBM or VIMGround)) ii) condition on this operation to decode or copy the arguments for this operation, iñi) add one (or more) child node(s).
Referential visual language grounding We ground noun phrases predicted by our parser with an off-thee thf-the-shelf language vision grounding model (Jain et al., 2022). The input is the noun phrase, e.g., "the blue cube" and the image, and the output is the boxes of all object instances that satisfy the training data.
noun phrase. We finetune the publicly available code of (Jain et al., 2022)
Short-term vision-based manipulation skills Our short-term manipulation skills build upon transporter networks (Zeng al., 2020). Transporter Networks take input an overhead RGB- D image registered at the overhead bird view and predict two robot gripper poses: i) pick pose and ii) a pick-conditioned placement pose. Transporters can model any behaviour that can be effectively represented as two consecutive poses for the robot gripper, such as pushing, sweeping, rearranging ropes, folding, and so on. For more details on transporters please refer to (Zeng et al., 2020). We modify Transporter Networks to be locally conditioned. Specifically, we crop image patch using the predicted pick and place locations from our visual grounding and EBM modules respectively. In this way. the low-level policies know roughly what to pick and where place it, and only lo- cally optimize over the best pick location (within an object of interest) or placement location, at a particular part of the scene, respectively.
Close Loop Feedback: Our model generates its own goal and then goes on to execute it. Once it executes it, we re-detect all relevant objects using our VI.M-gruunder module and check if they
5

---

Under review as a conference paper at ICLR 2023
close to their predicted goal location. Concretely, if the re-detected object's bounding box and predicted goal bounding box has a positive Intersection over Union (loU) we consider the goal to be successfully executed. If we fail to reach the goal, we retry the instruction again starting from the present scene configuration. For more details, please refer to Section-6.2.2.
4 EXPERIMENTS
We test SPGEM in its ability to follow instructions for re-arranging tabletop scenes in simulation and in the real-world. We compare our model against large language model planners (LLM) (Huang et al., 2022b) and end-to-end language to action models (Shridhar et al., 2021).
Our experiments aim to answer the following questions: 1) imization over object configurations compare with LLM planners in predicting object arrangements from spatial instructions? How much this ability scales with the length of the instruction? 2) How does SPGEM compare with state-of-the-an end-to-end instruction to action models across varying instruction lengths? 3) How do different modules of our framework contribute to performance?
How does the proposed set energy min-
Baselines
We compare SPGEM against the following baselines:
(i) CLIPort (Shridhar et al., 2021), a model that takes as input an overhead RGB-D image and an instruction and uses pretrained CLIP encoders featurize the instruction and RGB image, respec- tively, then fuses with depth features to predict pick and place actions using the parametrization of Transporter Networks (Zeng et al., 2020). Its aim is to capitalize on language vision associations already learnt by the CLIP encoders. We use the publicly available code of (Shridhar et al., 2021).
(ii) LLMplanner (Huang al., 2022b) is an instruction following scene rearrangement model that uses Instruct-GPT to predict sequential low-level subgoals in language form, e.g.. "pick the red cube and place it to the right of the blue bowl". These predictions interface with vision and language- conditioned short-term policies, such CLIPort. Scene state description is provided objects in the scene. LLMplanner does not finetune the LLM but instead uses appropriate prompts.
list of
Benchmarks We test our models and baselines for scene re-arrangement in the following bench- marks: (i) The four tasks of put-block-in-boww pack-google objects-seq, pack-google objects- group, and assemble-kits-seg from CLIPort benchmark (Shridhar et al., 2021). (ii) Scene rear- rangement benchmarks that we designed programatically in PyBullet simulator (more details in Section-6.5). These include: a) Spatial Relations benchmark, which contains single pick 'n' place instructions with referential expressions in cluttered scenes with distractors, e.g.. "Put the cyan cube above the red cylinder" b) Compositional-
step scene rearrange benchmark, which con- tains compositional instructions with referential expressions that require object to be re- re-located to a particular placement location, e.g., "put the red bowl to the right of the yellow cube and left of the red cylinder and above blue cylinder". Compositionall group scene rearrange benchmark, which contains compositional instructions with referential expressions that require multiple objects to be re-located, e.g., "put the grey bowl above the brown cylinder and put the yellow cube to the right of the blue ring and put the red bowl above the blue ring and put the blue ring below the grey bowl.". d) Shapes benchmarks, which contain instructions for making multi-entity shapes, such as circles and lines, e.g.. "rearrange all red cubes in a circle"
Evaluation Metrics We use two evaluation metrics: (i) Task Progress was proposed in the Ravens benchmark (Zeng al., 2020) and represents the percentage of the referred objects placed in their goal location, e.g., 4/5 = 0000% for rearranging 4 out objects specified in the instruction. (ii) Task completion rewards the model only if the full rearrangement is complete. For the introduced benchmarks we have oracle reward functions that evaluate whether the task constraints are satisfied.
4.1 SCENE RE-ARRANGEMEN UNDER ORACLE PERCEPTION AND LOW-LEVEL CONTROL
Our goal is to compare spatial reasoning in a language space vs in an abstract visual space for pre- dicting esstucciion-uudde subgoals. We compare model with LLMplanner. In this section, assume oracle perception, that is. oracle referential grounding for both LLMplanner and SPGEM, as well as oracle low-level control policies. We carry out inferred instructions from L.Mpplaner using

---

Under review as a conference paper at ICLR 2023
Step
Step
Step 10
Step 50
"Place the blue box left the green, right the red box above the yellow Put the cyan box inside the blue box. Move the rea box below the magenta box."
Spatial planning via composable concept graph energy minimization. The EBMs handle complex instructions as multiple steps of joint energy minimization. Step 0 indicates the starting point and step 50 the result after 50 gradient steps. Our model "imagines" the goal scene in pixel-abstract but spatially-aware manner. Note that the optimization is not aware of physical constraints: intermediate time steps the objects can overlap, e.g. steps and 10. However, this planning procedure and the intermediate steps, where the energy is high, are not executed.
Figure 2:
oracle controllers that relocate the right objects to locations in the scene that satisfy the predicted instruction while not occupying other objects. Note however though that SPGEM relies on simple pick and place policies that are not language conditioned, while the L.L.planner relies on language- conditioned policies, thus, the oracle control assumption may be more unrealistic the latter case. We forego this difference for the sake of comparison.
We show quantitative results of SPGEM and LLMplanner on compositional-one-stee and group benchmarks in Table 1. Our model outperforms LLMplanner, and the performance gap is larger in more complex instructions. planning, visualize steps of energy minimization for different instructions in Figure 2 and visualize steps of the execution the InstructGPT prompted by us to the best of our capability in Figure 3. As you see, SPGEM trained single predicate scenes shows remarkable compos- ability to much more predicates. Language planning on the other hand suffers from the ambiguity of translating geometric concepts language and vice versa: step-by-step exccution of language subgoals does not suffice for the composition of the two subgoals to emerge. (Figure 3).
elucidate why an abstract visual space may be preferable for spatial
Composition-one-step
Composition-
Method
Task Progress
Task Completion
Task Progress
Task Completion
LLMplanner oracle
82.0
59.0
75.3
29.0
SPGEM oracle
90.8
76.0
88.7
62.0
Table 1: Evaluation SPGEM and LLMplanner with oracle perception and oracle low-level execution policies on compositional spatial arrangement tasks.
4.2 SPATIAL SCENE RE-ARRANGEMENT
Simulation: In this section, compare our model and baselines in the task of instruction-guidee scene re-arrangement. We show quantitative results in Table 2 for the benchmarks of Composition- one-step and Composition- spatial relations and zero-shot evaluated on compositional benchmarks. We further fine-tune the CLIPort on demos from the compositional benchmark. Under all different settings, we vastly out- perform all variants of CLIPort and LLMplanner. We also observe that closed-loop feedback boosts our performance further. Specifically for compositional tasks, our model does not need any re- training because the energy functions are composable, meaning that we can jointly optimize over an arbitrary number of constraints by simply summing the different energy terms, however, we need few synthetic labels from grammar for the language parser. We also show results Spatial Relations and shapes benchmark in Table 7 where we vastly outperform all our baselines. We also show results for cliport benchmark and detailed generalization tests in the Section-6.2.1.
We compare our model with CLIPort model trained on atomic

---

Under review as a conference paper at ICLR 2023
Prompt
Initial
Valid Region
Execution
Exccution
Human: strawberry right
blue cube and to the the apple.
Scone: There an apple, blue cube and a strewberry in the scene.
Put the strewberry to the right of the blue cube
Robot Thought: My goal
l'strawberry right blue cube". "strawberry left apple"]
Robot Action: Put the strawberry to the right of the blue cube
Robot Action: the strawberry to the
the apple
Execulor: Done
Put the strawberry to the apple
Figure 3: Spatial planning via Large Language Models (LLMs). LLM Planners describe their plans in language actions supported by the executor policy. However, this be limiting. Here, the policy supports four atomic relations i.e. left, right, above and below. LLM instructs the executor to put the strawberry the right the cube which the executor can place anywhere in the shaded area. However, in the next instruction of putting the strawberry to the left, the executor violated the first constraint and ended up similar initial configuration. Note that while "middle" a valid English relation that can convey both constraints jointly here, however, a) it is not supported by the executor and b) words do not exist for every configuration especially when the joint constraint increases.
Composition-ne-sttep
Composition-one-step
Composition-group seen-colors
Composition-group
seen-colors
unseen-colors
unseen-colors
Method
10
100
10
100
10
100
10
100
Initial (no movement)
0.0
0.0
0.0
0.11
31.7
31.7
31.8
31.8
CLIPort (zero-shot)
7.0
37.4
37.5
32.6
38.4
C1.
13.0
15.0
14.0
9.01
38.2
38.5
34.7
40.9
L.Mpllnnerr
51.2
53.2
49.4
38.6
39.0
37.1
Ours (zero-shot)
90.0
91.0
92.7
77.2
77.4
77.7
78.4
Ours (scro-sbot + feedback)
91.6
92.0
92.9
91.4
80.8
81.6
81.1
82.4
Table 2: Eyaluation of SPGEM and CLIPort on compositional tasks SPGEM is trained only on atomic relations and tested zero-shot on tasks with compositions of spatial relations. Composition- one-step involves moving only one object to satisfy all constraints specified by the language. Com- positional group task involves moving all objects to satisfy the task constraints. Some language constraints are satisfied already in the initial configuration, and the Initial model captures that.
Real-World: We test our model in the real world using a 7-DoF Franka Emika robot equipped with parallel jaw gripper. We do not do any real world finetuning. Our test set contains 10 language- guided table top manipulation tasks per setting (composition-one-step composition-grat lines). show quantitative results in Table 5, We observe that SPGEM generalizes to the real world thanks to the open vocabulary detector trained in the real world, and the object abstractions in the concept EBMs and low-level policy modules.
circles,
4.3 ABLATIONS
We do ablations and error analysis of our model in Table-4. First, we remove the goal genera- tion from SPGEM by conditioning the place network on the language input instead of the EBM- generated goal image while keeping the pick network and object grounders identical. We observe a drop of 35.1% in accuracy underscoring the importance of goal generation. We then remove our executor policy and instead randomly select pick and place locations inside the bounding box of the relevant object. This results drop of 16% showing the importance robust low-level policies. We do not remove the grounder and parser since they are necessary for goal generation. We then experiment with oracle visual language grounder that perfectly detects the objects mentioned the sentence and we observe gain of 5.1%. We posit that this difference would be higher with real-world scenes. We observe that language parsing in our domain is close to perfect, hence we skip that. We finally evaluate with perfect grounding, language parsing and low-level execution
8

---

Under review as a conference paper at ICLR 2023
left-seen-colors
left -u--ueeenncolors
right-seen-colors
right-unseen-colors
Method
10 demos
100 demos
10 demos
100 demos
10 demos
100 demos
10 demos
100 demos
CLIPort-multi
13.0
44.0
9.0
33.0
29.0
43.0
28.0
44.0
Ours
95.0
95.0
93.0
94.0
89.0
92.0
93.0
96.0
above-seen-colory
above-unseen-colors
below-seen-colors
belov-unseen-colors
Method
10
100
10
100
10
100
10
100
CLiPort-multi
24.0
45.0
22.0
51.0
23.0
55.0
13.0
40.0
Ours
87.0
87.0
89.0
90.0
89.0
90.0
88.0
89.0
circle-seen-coors.
circle-unseen-coors
line-seen-colors
line-unseen-coors
Method
10 demos
100 demos
10 demos
100 demos
10 demos
100 demos
10 demos
100 demos
CLIPort-multi
34.1
61.5
31.2
55.6
48.6
88.2
48.6
88.5
Ours
91.3
91.5
90.2
91.2
98.1
99.0
98.4
99.4
Table 3: Evaluation SPGEM and CLIPort on spatial relations and shapes in simulation.
Model
Accuracy
Task
Performance
SPGEM
77.2
Composition-one-ee
85.6
SPGEM without goal generation
42.1
Composition- -rrupp
75.8
SPGEM w/o learnable policies
Circles
94.0
SPGEM with oracle language visual grounding
82.3
Lines
90.0
SPGEM with everything oracle except goal
88.3
Table 5: Evaluation of SPGEM in the Real World
Ablations of SPGEM the bench- mark compostionssgmup-senncoooos
Table 4:
test the error rate of our goal generator. We obtain 88.3% accuracy thus concluding that our goal generator fails in 11.7% cases. For more detailed error analysis, please refer to Secs. 6.2.3 and 6.2.4 of the Appendix.
4.4 LIMITATIONS
Our model presently has two important limitations. First, it predicts the goal object scene configu- ration but does not have any knowledge regarding temporal ordering constraints implied by physics. For example, our model can predict a stack multiple objects on top of one another but cannot suggest which object needs be moved first. Adding such temporal energy priors is direct avenue for future work. Second, our EBMs are currently parametrized by object locations but many tasks, e.g, manipulation of articulated objects, fluids, deformable objects granular materials, would quire finer grain parametrization in both space and time. Furthermore, even for rigid objects, many tasks would require finer in- in-space parametrization, e-g., would be useful know a set of points in the perimeter of plate as opposed to solely representing its centroid and box for accurately placing things inside it. more details, please refer to Section-6.3.
CONCLUSION
We introduced SPGEM, a modular framework for instruction-guided scene re-arrangement that maps language to object scene configurations via compositional energy minimization over detected object graphs, one per spatial concept. We showed better generalization of our framework to predi- cate concept compositions, novel object combinations, and object scene arrangements never seen at training time. We test our model in diverse table top manipulation tasks in simulation and show im- provements both in and out-of-distribution settings, as well as low training data regimes, against state-of-the-art end-to-end language to action mapping models, and LLM-based instruction follow- ing methods. We contributed new scene rearrangement benchmark for more compositional lan- guage instructions, which we will make publicly available. alongside Our work questions the efficiency of spatial reasoning in language space, and shows a handful of examples suffice to learn EBM concepts, and compose them over novel arrangements.
code, upon publication.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the empirical results, we submit a zip file with an anonymized version of our code as part of the supplementary file. Upon publication, we will make our full code, data and benchmark publicly available on GitHub.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lec, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can, not as i say: Grounding language in robotic affordances, 2022. URL https: / //arxiv.org org/ abs/ / 2204.22222.2 01 01191. 2
Ahmed Akakzia, Cédric Colas, Pierre- Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding Language to Autonomously-Acquinn Skills via Goal Generation. In ICLR 2021 Ninth International Conference on Learning Representation, Vienna / Virtual, Austria, May 2021.
URL https: /haaaaaaa.. inrra. Fr hb/bal-03221111
Jacob Andreas, Mareus Rohrbach, Trevor Darrell, and Dan Klein. Deep compositional question answering with neural module networks. CoRR, abs/1511 11.51111.9,,,,, 2015. URL htl.http///ax:
exg/abs/15t 02799.
Jonathan Berant and Percy Liang. Semantic parsing via paraphrasing. In Proceedings of the 52nd Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1425, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 4
10.3115/11/14-1133. URL https: aalaacaanthaa aclanthology......oo www.nnoooo....g.........oo wwtg
Yonatan Bisk, Kevin Shih, Yejin Choi, and Daniel Marcu. Learning interpretable spatial opera- tions in a rich 3d blocks world. CoRR, abs/1771.003463, 2017. URL
abs/1711 03463.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vors Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: arxiv org/ alss 2005 14165. 2,
Maya Cakmak and Leila Takayama. Towards a comprehensive chore list for domestic robots. In Proceedings of the 8th ACM/EEE International Conference on SumannRRbor Interaction, HRI '13, pp. 93-94, IEEE Press, 2013, ISBN 467145733
Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S. Ryoo, Austin Stone, and Daniel Kappler. Open-vocabulary queryable scene representations for real world planning. 2022. URL http::t//axxi...oootmm
abs 2209.229..20 94.7.4441 2,
Cédric Colas, Tristan Karch, Nicolas Lair, Sean-Michel Dussoux, Clément Moulin- Frier, Peter Ford Dominey, and Fierre-Yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity- driven exploration. CoRR, abs/2002.09253, 2020. URL https: //tp//aavvvv//////a//eee/aaaa org abs//022. 09253. 3
Cédric Colas, Tristan Karch, Clément Moulin-Frice Frier, and Pierre Yves Oudeyer. Vygotskian totelic artificial intelligence: Language and culture internalization for human-like ai, 2022. URL https: //axxiv/ orgg
org/abss abos 2206 6. 01 014
10

---

Under review as a conference paper at ICLR 2023
Li Dong and Mirella Lapata. Language to logical form with neural attention. arXiv preprint arXiv:1601.01280, 2016. 5
Danny Drieß, Jung-Su Ha, and Marc Toussaint. Deep visual reasoning: Learning to predict action
sequences for task and motion planning from an initial scene image. CoRR, abs/2006.05398, 2020. URL https:
mexxivoorg abs 2006. 05398.
Yilun Du, Toru Lin, and Igor Mordatch. Model based planning with energy based models. CoRR, Abss/199.00687, 2019. URL http: / arxiv. org/ abs /109.. 06878. 4
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 6637-6647. Curran Asso- ciates, Inc., 2020a. URL https://prooeeddngg neurips CC httpppppe//////ieeeee. 49856ed47 6ad0 6cd0
pdf. 3,4
Yilun Du, Shuang Li, Joshua B. Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. CoRR, abs/2012.01316, 2020b. URLhttps:/aaxxii
abs/2012. 01316. 3,
Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan. Example- based synthesis of 3d object arrangements. ACM Transactions on Graphics (TOG), (1()):1-11, 2012.
Caelan Reed Garrett, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Stripstream: Integrat- ing symbolic planners and blackbox samplers. CoRR, abs/1802.08705, 2018. URL http: / //arxiv..rgg abs//a//. 08705. 4
Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Mozer, and Yoshua Bengio. Neural production systems. CoRR, abs/2103.01937, 2021. URL https: /ttpaaaavvv////////////../...ava org/ abs 220.. 01937.
Will Grathwohl, Kuan-Chieh Wang, Jörn- Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. CoRR, abs/1912.03263, 2019. URL http: / /aaxivvvv... org/ aabs/// /1912. 03263, 4
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. Incorporating copying mechanism in sequence-to--sequenee learning. arXiv preprint 1rXXiv1603.06399 2016. 5
Sagar Venkatesh Gubbi, Anirban Biswas, Raviteja Upadrashta, Vikram Srinivasan, Partha Talukdar, and Bharadwaj Amrutur. Spatial reasoning from natural language instructions for robot manipu- lation. CoRR, abs/2012.13692. 2020. URL https:
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to con- trol: Learning behaviors by latent imagination. CoRR, abs/1911.011033, 2019. URL http:
/ //arxiv. www.iiiorg/aeo abs///b/aaa/ /1912 001603.....
Danijar Hafner, Timothy P. Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. CoRR, abs/2010.02193, 2020. URL ittps://arxive org/ abs/ 2010. 02193. 2
Jessica B. Hamrick, Abram L. Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims With- erspoon, Thomas Anthony, Lars Buesing, Petar Velickovie, and Théophane Weber. On the role of planning in model-bate based deep reinforcement learning. CoRR, abs/2011.04022 2020. URL https: /ttps///arxiii www.riiwi....ooo.o..ooo.ooooooor alos /201 04021.
Eric Heiden, Miles Macklin, Yashraj Narang, Dieter Fox, Animesh Garg, and Fabio Ramos. Disect: differentiable simulation engine for autonomous robotic cutting. arXiv preprint arXiv:2/05. /2244, 2021. 23
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022a. 2,4, 2, 20
11

---

Under review as a conference paper at ICLR 2023
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models, 2022b. URL htt.tttt///aiiii.orrrr///////// 05608. 2, 3,44
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. arXiv preprint arXiv:2104.33/1, 2021. 23
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In European Confer- ence on Computer Vision, pp. 417-433, Springer, 2022. 3,
Michael Janner, Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguisties, 6:49-61, 2018. doi: 10. 10.11622/111 http://alla_00000000 laraacll_.00000
URL https:////// /aa//clannttt aclanthologyy org/ Q018 1004.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017. 16
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. CoRR, abs/ 0ts//804.01622,, 2018. URL /ttt:///axiiv abs /00/4 622. 4
Leslie Pack Kaelbling and Tomás Lozano-Péree Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, pp. 1470-1477, 2011. doi: 20.1109/CCRA.201.59900
2,
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. MDETR modulated detection for end-to-end multi-modal understanding. CoRR, abs/2104. 12763, 2021. URL https: /aaaiiiivv../ orggrorggoo
wwwp/r/////////o///////////aaaer aba/bb//// 2204.111110 12763.276222 3,4
Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang. Learning adaptive language interfaces through decomposition. CoRR, abs/2010.05190, 2020. URL https://///// /aae:i//axiivaa orgooogg//oo.. a 2010 05190.
Kei Kase, Chris Paxton, Hammad Mazhar, Tetsuya Ogata, and Dieter Fox. Transferable task exe- cution from pixels through deep planning domain learning. CoRR, abs/2003.03726, 2020. URL
https:: //te:////xxiii www.arwwww.w..oreeee..eooeoooere wttp/as.////ooooooooooo../...asr 2003. 03726.
200.//230.......... 2
Scott Kirkpatrick. Optimization by simulated annealing: Quantitative studies. Journal of Statistical Physicx, 34:975-986, 1984. 5
Yann LeCun. A path towards autonomous machine intelligence. preprint posted openreview, 2022. 2, 23
Yann LeCun, Sumit Chopra, Raia Hadsell, Fu Jie Huang, and al. A tutorial energy- based learning. In PREDICTING STRUCTURED DATA. MIT Press, 2006.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control, 2022. URL https: //taxaivvv... org/ alos 2209222222222222222222222 07753. 2,
Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On grounded planning for embodied tasks with language models, 2022. URL https: /ttpps///rxivv //axxiv. org/abs/ 2209. 004 65. 4
Nan Liu, Shuang Li, Yilun Du, Joshua B. Tenenbaum, and Antonio Torralba Learning to compose visual relations. CoRR, 0bx/2111.09299 2021a. URL https: //arri//aiiii/i/ wwt.aawwwwwww.////////.a 092 3,4
12

---

Under review as a conference paper at ICLR 2023
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. CoRR, abs/2107.13586, 2021b. URL htt.t:tttaaatttt.. wttggba/////0o/////o////////g 13586. 2,4
Weiyu Liu, Chris Paxton, Tucker Hermans, and Dieter Fox. Structformer: Learning spatial structure for language-guided semantic rearrangement of novel objects. arXiv preprin! 10.1v:2110.10188, 2021c. 3,3
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19X.v 2019. 5
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N. Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. survey of reinforcement learning in- formed by natural language. CoRR, abs/1906.03926, 2019, URL 1906. 03926. 3
Corey Lynch and Pierre Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020. 3
Daoming Lyu, Fangkai Yang. Bo Liu, and Steven Gustafson. SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. CoRR, abs/181 .00090, 2018. URL http://arxtie wwwg/baaeeee abs / 811 100090.0 2
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention, 2015. URL https: / //t////aaiiii //aaaiivvvrrrgr. org.i.orr//aaaoeoo abts//as////11/ 02793. 4
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.12550 2019. 16
Cynthia Matuszek, Evan V. Herbst, Luke Zettlemoyer, and Dieter Fox. Learning to parse natural language commands to a robot control system. In ISER, 2012.
Oier Mees, Alp Emek, Johan Vertens, and Wolfram Burgard. Learning object placements for re- lational instructions by hallucinating scene representations. CoRR, Abs/2000.08481, 2020. URL
https: /ttp://aaavv//aa/////eeeeaaaavva wwt.:iiiiorg//o/ooeeoeeeeee 220r://bss/00111... 08481.
Paul Merrell, Eric Schkufza, Zeyang Li, Maneesh Agrawala, and Vladlen Koltun. Interactive furni- ture layout using interior design guidelines. ACM transactions graphics (TOG), 30(4):1-10, 2011. 4
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models, 2022. URL htwph:tts///eett /atpsi////aivvea//a org/ abs 22092 00588. 2
Toki Migimatsu and Jeannette Bohg. Object-centric task and motion planning in dynamic envi- ronments. CoRR, abs/1911.04679 2019. URL http::///axxi..r//abbb...../
Igor Mordatch. Concept learning with energy-based models. CoRR, abs/1811.02488 2018. URL
http:/ /ttp/vaxivv/v..////a//eee/aawavl http:///xxvvv.o//aabb/ /8111 02486. 3,4
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. CORR, 0071180770444.... 2018. URL http://
/t/////bbb/100......... mrxiv.org/abs/1e1
Alexander Pashevich, Cordelia Schmid, and Chen Sun.
Episodic transformer for vision-and- language navigation. CoRR, abs/2105.06453, 2021. URL https://arxiv.or abs/
Vitchyr H. Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-super reinforcement learning. CoRR, abs/1903.03698, 2019. URL http:://are worg/abs//ii 903. 03698.
13

---

Under review as a conference paper at ICLR 2023
Mihir Prabhudesai, Hsiao-Yu Yu Fish Tung, Syed Ashar Javed, Maximilian Sieb, Adam W. Harley, and Katerina Fragkiadaki. Embodied language grounding with implicit 3d visual feature repre- sentations. CoRR, abs/1910.01210, 2019. URL
orrg htrg/abb/1990 01210.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Scalable differentiable physics for learning and control. arXiv preprint arXiv:2007.000 022200007.026,,, 2020. 23
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Пya Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102.12092, 2021. URL
https: //tpvvvvvvvv///////////.../.awvv org/ alss//a//// 22t/221222.. 12092. 2.4 4
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image dif- fusion models with deep language understanding, 2022. URL https:/ http///aavvvvvvv////v////.//..aa org/ / 2205. 11487. 4
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan. Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model, 2019. URL
http:// http///rxivvvv rxiv, wrgorrg/oe abs/ / 9991 08265. cite arxiv: 1.08265.
Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Gold- berg, and Andy Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal- conditioned transporter networks. In 2021 IEEE International Conference Robotics and Au- tomation (ICRA), pp. 4568 4575. IEEE, 2021.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021. 1, 3, 6, 19, 25
Mohit Shridhar, Lucas Manuelli, and Dieter Fox, Perceiver-actor: A multi-task transformer for robotic manipulation, 2022. httpps://axxi..orppppp
05451.
Kavya Srinet, Yacine Jernite, Jonathan Gray, and Arthur Szlam. CraftAssist instruction pars- ing: Semantic parsing for voxel-world assistant. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pp. 4693-4714, Online, July 2020. As- URL https:
sociation for Computational Linguisties. doi: 1/235///1220222ll-mm // //acl ant ogy- wwt.//g//2222 acl-ma 3
Elias Stengel-Eskin, Andrew Hundt, Zhuohong He, Aditya Murali, Nakul Gopalan, Matthew Gom- bolay, and Gregory Hager. Guiding multi-step rearrangement tasks with natural language instruc- tions. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Proceedings of the Sth Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 1486-1501. PMLR, 08-11 Nov 2022. URL https::///// /proceeeeeeeeeeeeee ngs. ml r. press/ 64/ sttenel -aw.eeeeeaki2222 html. 1,3
Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3(1):25-55,
2020. doi: 10. 00.1146/amuure--coottu 101119-07111118. URL annurev- seerurev-contrra--o111--
Mare Toussaint.
programming: An optimization- based approach to combined task and motion planning. In Proceedings of the 24th International Conference on Artificial Intelli- gence, I5CAI'55, pp. 1930-1936. AAAI Press, 2015. ISBN 9781577357384. 2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017. 17
Nick Walker, Yu-Tang Peng, and Maya Cakmak. Neural semantic parsing with anonymization for command understanding in general-purpont service robots. CoRR, abs//990.an 007 051 2019. URL http://an rxiv. www./ogg////.oooooo.o..ooo...//e 907. 011.1.11111111 01115.
14

---

Under review as a conference paper at ICLR 2023
Christopher Wang. Candace Ross, Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Learning natural- language to LTL executable semantic parser for grounded robotics. CoRR, abs/2008.03277, 2020.
URL https://aaxii.......
abs/ 2000 03277.
Yushi Wang, Jonathan Berant, and Percy Liang. Building semantic parser overnight. In Pro- ceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), PP. 1332-13422 1342, Beijing, China, July 2015, Association for Computational Linguistics. doi: 10.3115/v1/155-111 URL https:
aat.annoohoooo ogg wwtg
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, 222222201119033
2022. URL https:/ arxiv. org/
11903.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-II), pp. 681-688. Citeseer, 2011. 5
Hongtao Wu, Jikai Ye, Xin Meng, Chris Paxton, and Gregory Chirikjian. Transporters with visual foresight for solving unseen rearrangement tasks, 2022. URL 2202. 10765. 3
orgg /bs/
Shan Wu, Bo Chen, Chunlei Xin, Xianpei Han, Le Sun, Weipeng Zhang, Jiansong Chen, Fan Yang, and Xunliang Cai. From paraphrasing to semantic parsing: Unsupervised semantic parsing via synchronous semantic decoding. CoRR, abs/2106.06221 2021. URL https: /atti//axxxii /aaaaaiivvoo.//.. org/ abs/2106. 06228.
Danfei Xu, Roberto Bartín-Martín De-An Huang. Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Re- gression planning networks. CoRR, abs/1909.13000 13072, 2019. URL http:///axxis www.vvorrraaaa. abs/ 1909.13072 13072. 2,
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation, 2022. URL hitp:://aaxiiii
abs 2206 10789. 4
Lap Fai Yu, Sai Kit Yeung. Chi Keung Tang, Demetri Terzopoulos, Tony F Chan, and Stanley J Osher. Make it home: automatic optimization of furniture arrangement. ACM Transactions on Graphics Proceeinos of ACM SIGGRAPH 2011, v. 30,(4), July 2011, article no. 86, 30 (4), 2011.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rear- ranging the visual world for robotic manipulation. arXiv preprint arXiv:2010.14400 2020. 3, 5, 6, 25
Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, and Yuke Zhu. Hierarchical planning for long- horizon manipulation with geometric and symbolic scene graphs. CoRR, abs/2012.022777, 2020. URL https: /atps://arzzee /arxiv....../////i//aa////avvtvv org/ abs///s/..22/////////////aaaa//a 07277. 2,
15

---

Under review as a conference paper at ICLR 2023
6 APPENDIX
For visualizations and real robot execution videos, please visit our website: https://siter: google
www/ccewcii/////e/..oe......e..e htte//iiew/eeeeee view/spgem/ spgem/
6.1 MODEL DETAILS
Please refer to Algorithm-1 for a simplified pseudo-code for our pipeline. We provide more details in the following sections.
Algorithm Pseudo code for SPGEM's pipeline
Inputa:
img: Input RG3 image, depth: Asscciated depth image, instruction
lang: Input task
Model
VIM-Grounder: A language grounding modules which datects objects mentiened in the sentence
language paraer: Seq-to-Tree model which maps natural language to an executable program
abm abm__1bbas library: library of 3AMs consisting of atomic binary concepts 1ike left, right, above, below and inside and n-ary concepts like circle and lines go taatsrsporttt: low level pick-and-policy which executes the pick and place action conditioned on the goal from EBM
for img, depth, lang dataloader: I load one task inputs
program language parser (lang) the parser spits out program the Grounder and EBKS
intermedlate_outgee for i, prog in erumerate (ormera) $ iterate over program from parser if progl' (moduumm -- 'filter':
- []
object_dess
p' concept') description of object to detect for eg: the left green cube"
object_box = (.MGround(ing, object_desc)
elif prog p ('module') -- 'EBMs'
subjs
subj'' list of lists, rel boxes, e. g. [[1), 10, 911, indexes into Lntermedlate_ourpen
obis - prog[' into intermediata_oute
list of lists, ref baxes, e. g. [12), []), indexes list of str, relation names, e. g, ("right",
rels = prog('rels'] circie"]
picks - pprogl'picke! it 1ist of objects that need to be picked, indexed into intermediate outputs
goal boxes run (on_ebm(subi))
run_ebm (subis, obis, rels, intermed ate_cutpats) # runs the langevin dynamics the required EBKs with appropriate boxes
pick boxes intermediate_outgule boxes]
else:
asser: False
# predict pick and place locations conditioning on bounding boxes of pick and place goals pick_locs, locce_locs - ge_traneporter (img, depth, sick_boxes boxes)
boxes)
# robot consrall actually ly moving the end effector to execute the task low_level_controlleer (pick_locs, place_locs)
6.1.1 TRAINING A SEMANTIC PARSER WITH SYNTHETICALLY GENERATED C'TERANCE-PROGE PAIRS
Domain Specific Language for grounding and Generation We design a Domain Semain-Specific Lan- guage (DSL) which extends the DSL of NS-CL (Mao et al., 2019) (designed for visual question answering in CLEVR (Johnson et al., 2017)) to further predict scene generations, e.g. "put all brown shoes in the green box". Detailed description of our DSL can be found in Table 12.
16

---

Under review as a conference paper at ICLR 2023
O O, ymii ymaa
O Xm m (arymiss Dymaa
fip (anyaax mind
O n entities
x Concat(1X, X, Linear(8, 8, 128)
X fkp rell)
X [= Voenter)
Lincar(6, 128)
LeakyReL.U
LeakyReLLU
Lincar( 128, 128)
Lincar( 128, 128)
LeakyReLU
Transformer Encoder
(4 lavers)
Linear( 128, 128)
Linear( 128, 1)
Lea
Sum over entities
Linear( 128, 1)
E
E
Figure 4: Left: Architecture of the EBM used for binary concepts such as "right of".. The inputs are a "moving" point Om and a "reference point" O, and the output is the energy of their relative placement. Right: Architecture of the EBM used for multi-ary concepts such as "circle". The input is a set of n entities Oz, & I....,7.. n. The output is the energy of this set of entities.
6.1.2 EBMS
Implementation Details: We implement two EBMs, namely a BinaryEBM and a MultiAryEBM, for binary (c.g. "right of"") and multi-ary (e.g. "circle") concepts respectively. The BinaryEBM is used for all tasks that ask for moving objects with respect to some object of reference, e.g. as in the "Put" and "Pack" tasks of the CLIPort benchmark. We train separate EBM for each basic concept, i.c. "left", "right", "inside", "above", "below", etc. A BinaryEBM takes as input two objects and learns move the first object with respect to the second one. A MultiAryEBM receives set of objects and re-arranges them to a shape. Again we build separate EBM for each shape concept, e.g. "circle" and "line".
We implement the BinaryEBM as a relational network that operates on the relative locations between the objects (Figure 4). Then, a multi-layer perceptron (MLP) is used to map the featurized pair to a scalar value, which is the energy.
For the MultiAryEBM, we first featurize each object by applying a linear layer and a LeakyReLU activation on top of its absolute coordinates. Then, we feed the set of object features to a sequence of four Transformer encoder layers (Vaswani et al., 2017) for contextualization. The refined features are averaged into an ID vector which is then mapped to a scalar energy using an MLP. The architecture is depicted in Figure ??.
During training, sample positive and negative examples and optimize for Equation 2. We ad- ditionally use the KL-loss and the L2 regularization proposed in (Du et al., 2020b). We refer the reader that paper for more explanation these loss terms.
Extension to tasks with pose: An important design choice is what parameters of the input we should be able to edit. We inject the prior knowledge that our manipulation domain the objects move without deformations, so we fix their sizes and update only their positions. Our EBMs operate on boxes so that they can abstract relative placement without any need for object class shape
17

---

Under review as a conference paper at ICLR 2023
information. However, EBMs be easily extended to optimize other types of representations, such as pose.
We train an EBM, call it PoscEBM, to optimize the 2D angle of posed object wrt point of reference. This is a common constraint in furniture re-arrangement, c.g. "the armchair should look towards the tv". For this task, parameterize posed object point (its centroid) and an angle (the orientation of its main axis). The PoseEBM takes as input one such object and a reference point. First, it computes the relative location of the posed object wrt to the reference point. Then, the relative location and the angle are concatenated to a vector of three elements. This is fed to an MLP that outputs a scalar energy value. We sample positive examples of scenes where the posed object "looks towards" the point of reference, i.c. its main axis is parallel to the vector connecting its centroid and the reference point.
6.2
EXPERIMENTS
6.2.1
GENERALIZATION EXPERIMENTS
Generalization Benchmark Generalization Tests: We conduct controlled studies of our model's generalization across three axes: a) Novel Colors b) Novel background color of the table c) Novel Objects. In each of these settings, we only change one attribute (i.e. object color, background color or object instance) while keeping everything else constant.
Novel colors: We train the models with ['blue", 'red', "green', "yellow", 'brown', "gray', "cyan'] colors and evaluate them with unseen colors: l'orange', 'purple', 'pink", 'white].. Novel background colors: All models are trained with black colored tables and evaluated with randomly sampled RGB color for each instruction.
Novel objects: We train the models on ["ring", "cube", "cylinder". "bowl"] and evaluate them with ("triangle", "square". "plus", "diamond", "pentagon", "rectangle", "flower", "star", "circle", "hexagon", "heart"].
We evaluate ours and cliport model trained on 10 or 100 demos on spatial reasoning benchmark tasks (involving single left, right, above, below) and composition benchmarks (composition-one-stee step, popposstiiongroupp The results are summarized in the table-6 and in the figure-5.
Percentage Performance drop with novel conditions
CLiPdd
Duns
40
36-8
Accuncty Drop
12:1
Novel Color
Novel Baskground
Novel Dijects
Categary
Figure 5: Generalization comparison between CLIPort and Ours: Average percentage drop by changing color of objects or backgroud color or introducing novel objects (lower is better).
18

---

Under review as a conference paper at ICLR 2023
Attribute
Model
Spatial-seen
Spatial-novel
Composition-senn
Compositon-noeel
10 demos
100 demos
10 demos
100 demos
10 demos
100 demos
10 demos
100 demos
Color
Coot
22.0
47.0
18.0
39.0
25.6
26.8
25.1
24.5
SPGEM (ours)
90.0
91.0
87.0
85.0
83.6
84.2
86.5
84,0
Background
CLiPort
SPGEM (Ours)
90.0
22.0
91.0
47.0
79.0
10.0
68.0
20.0
83.6
25.6
84.2
26,8
77.0
23.7
72.0
23.2
CLi
22.0
47.0
17.0
19.0
25.6
26.8
24.5
24.8
Objects
SPGEM (Ours)
90.0
91.0
86.0
86.0
83.6
84.2
80.9
81.5
Table 6: Generalization experiments of SPGEM and CLIPort in manipulation tasks in simu-
lation.
put-block-in-bowl seen-colors
put-block-in-bowl unseen-colors
packing-guugl--ohjeets seq-scen-objects
pucking-gouugl--obipe
seq-unseen-objes
Method
10 demas
100 demos
10 demos
100 demos
10 demos
100 demos
10 demos
100 demos
CLIPort-mulii
31.0 84.3
82.1
4.8
17.6 95.3
34.8
54.7
27.2
56.4
Ours
93.8
89.0
86.8
94.8
88.0
92.9
working-googlo-objes group-seen-objects
jucking-guogleoojjett group-unseen-ebjeets
ussemblingkitt seq-seen-colors
ussembling-iit kits seq-unseen-colors
Method
10 100
10
100
10
100
100
CLIPort-multi
33.5
61.2
32.2
70.0
38.0
62.6
36.8
$1.0
Ours
86.1
76.8
87.2
79.6
38.4
42.0
40.8
44.0
Table 7: Evaluation of SPGEM and CLIPort on CLIPort benchmark in simulation.
Analysis: We observe that on average on a) Novel Colors our model's performance dropped by 1.% compared to 12.1% % for CLIPort b) on novel backgrounds our drop is 15.1 1% com- pared to 36.8% for CLIPort with most of our failures caused in transporter- based low-level policy c) on novel backgrounds our drop is 4.1% compared to 29.9% for CLIPort.
CLIPort's benchmark: We also show generalization experiments across novel colours and novel objects across the established CLIPort benchmark. Please refer to the original CLIPort paper (Shridhar et al., 2021) for more details of the benchmark. Note that the original CLIPort benchmarks assume access to oracle success/failure information based on which can retry the task for a fixed budget of steps or stop the execution if oracle confirms that the task is completed. We evaluate the CLIPort model without this oracle retry but still with oracle information of how many minimum steps needs to take to complete the task. For example, if the task is "Place all strawberries in the bowl", it has access the oracle information of the number of strawberries, say N, in the scene and it tries to pick and place the strawberries for N steps. Our model, in contrast, fires language grounding model to find the number of task-relevant entities in the scene, without assuming any oracle in- formation. The put-block-in inwwllll and assembling-kitss colours while all remaining tasks test generalization across novel objects. In this bench- mark as well we observe our model generalizes gracefully across novel colours and objects while CLIPort model can suffer significant performance hits. We also outperform CLIPort on all the tasks by a margin except the assembling task when trained on 100 demos. Most of the failure cases there are due to the language grounder getting confused between letters and letter holes.
task tests generalization across
Real World Experiments In our real world experiments, we also use novel objects or novel object descriptions like "bananas", "strawberry", "small objects" which the model hasn't tt seen the simulation training.
Our model's generalization capabilities rely on an open-vocabulary detector and the fact that EBMs and transporter- based low-level execution policy operate on abstracted space in a modular fashion. While CLIPort models can also generalize to novel scenarios, thanks CLIP, however, the action prediction and perception are completely entangled and hence even if CLIP manages to identify the right objects based on the language, it has trouble predicting the correct pick and place locations.
19

---

Under review as a conference paper at ICLR 2023
Benchmark
Precision
Recall
Accuracy
Composition-gron gropp 80.5
82.5
85.0
Composition-onee -ttes-ttion-oe-stee
71.4
19.2
77.0
Table 8: Performance of failure detection using our generated goal
6.2.2 CLOSED LOOP FEEDBACK
We also experiment by adding closed-loop feedback to our model and retry the instruction the feedback mechanism inferred a failure.
Implementation: Our model generates its own goal and then goes on to execute it. Once it executes it, we re-detect all relevant objects using our VLM-ggounder module and check if they are close to their predicted goal location. Concretely, the re-detected object's 's bounding box and predicted goal bounding box has positive Intersection over Union (IoU) consider the goal to be successfully executed. If we fail to reach the goal, we retry the instruction again starting from the present scene configuration (without resetting the environment, since it's big assumption).
Results: As we show in the main paper, adding feedback roughly adds 3.5% performance boost in composition-gun group benchmark and 1% boost in composition-one-te always recover from the failure, however, it would be still important to know if the execution failed so that we can request help from a human. Towards this, we measured the failure detection capa- bilities of our feedback mechanism. Specifically, we measured the precision, recall and accuracy of the failure detection mechanism. The results are Table-8. We observe that all metries are high for composition group benchmark. For composition-ons seeee--oe---eepee ste-step, however, the recall is very low i.c. only 19.2%. This is because in this benchmark, we only need to move one object to complete the task and hence most failures are due to wrong goal generation. Thus, the failure classifier classifies those demos as success, because indeed the model managed to achieve its predicted goal and hence results in low recall. This motivates the use of external failure classifiers in conjunction with internal benchmark is harder because it requires
benchmark. Retrying cannot
goal-checking classifiers. In contrast, composition-grans the model to move many objects and thus results in higher chances of robot failures. These failures can be better detected and fixed by our goal- goal-checker classifier.
Also, note that while previous works like InnerMonologue (Huang et al., 2022a) showed huge im- provements by adding closed-loop feedhack, the improvements for us are smaller. This is hecause, by design, our model is more likely to reach its goal - since EBM generates visual goal and then the low-level policy predicts a pick and place location within the predicted goal, it is very likely to satisfy its predicted goal. Indeed, in the composition-one-stee the model satisfies its goal in 93% cases and in 70% cases in the composition-gre -roupg benchmark. In contrast, InnerMonologue does not have any built-in mechanisms for promoting goal-reaching behaviours and thus the difference in performance with additional goal-satisfying constraints is larger for them.
I Limitations:
There is a lot of potential in designing better goal checkers. As already discussed, we should incorporate external success classifiers like those used by prior literature (Huang et al., 2022a) in conjunction with goal-checking classifiers. Explicit goal generation allows then to check validity of goals directly even before actual robot execution (which makes it safer and less expensive). We leave this for future work. Another promising direction is to add object trackers in the feedback mechanism to keep track of the objects and detect if failure happened. This is important, if for example, we have multiple objects that are visually similar and hence we would need to keep track of which object is supposed to go which goal and retry if it failed to reach its goal.
6.2.3 ERROR ANALYSIS AND ROBUSTNESS TO NOISE
We conduct detailed error analysis of our model on composition-groos benchmark and report results in table-9. We find that robot failures, which include collisions or failure to pick/place objects, result in 6.0% drop in accuracy. Goal generation adds 11.7% to the failure. Visual grounding leads to 5. % errors while we find language parsing to be nearly perfect.
To test robustness of our modules from noisy outputs from parent modules, we add gaussian scaling and translation noise to predictions from grounding and goal generation modules. We add a scaled gaussian noise of varying percentage as fraction of the size of the objects. The results are sum-
20

---

Under review as a conference paper at ICLR 2023
Error Mode
Error Percentage
Robot failure
6.0
Goal Generation failure
11.7
Grounding failure
5.1
Language Parsing failure
0.0
Table 9: Error Analysis of SPGEM in the benchmark
Module
No Noise
Noise=0.01
Noise=0.05
Noise=0.1
Noise=0.5
Noise=0.7
Visual Grounding
77.4
77.1
77.3
77.0
75.8
69.4
Goal Generation
77.4
76.5
76.3
74.6
72.3
71.5
Table 10: SPGEM's robustness to noise in the benchmark compostionssgronp--en-coooossss
marized quantitatively in Table- 10 and qualitatively in Figure-6. We observe that our model is quite robust to noise even as high as 0.5 the size of the object.
Robustness to noise added to various modules
Visual Grounding
Goal Generation
80
60
Accuracy
40
Noise
Noise=0.01
Noise=0.0g
Noise=0.1
Nosie=0.5
Noise=0.7
Noisy scale
Figure 6: Robustness noise added to outputs from visual grounding and goal generation modules
6.2.4 COMPARISON WITH HARDCODED GOALS
We experimented with adding hardcoded goal generation in place of our EBMs for our benchmarks. For binary relations and shapes hardcoded the goal generation. However, for compositions, this non-trivial because we would need to combinatorially search to find a goal configuration which would satisfy all relations. To sidestep this precise issue, while generating training demos in simu- lation, we randomly spawn some objects and subsample a bunch of relations from all possible (*) binary relations form the task description. Hence, for this hardcoded experiment, we directly take the final oracle goal from the generated data as the hardcoded goal. We call this 'Combi- natorial hardcoded goal'. Another reasonable hardcoding approach is to sequentially execute each hardcoded constraint. We call this 'Sequential hardcoded goal'. The quantitative results are in Table- 11. We observe that on spatial relations, circle, lines and combinatorially hardcoded compositions, model is only 3.6% worse average than hardcoded goals. When compared compositions with sequential execution, our model 12.3% better since it generates a goal by considering all constraints jointly without doing any expensive combinatorial search.
21

---

Under review as a conference paper at ICLR 2023
Benchmark
Hardcoded
Ours
Spatial Relations
91.7
90.0
Circles
98.2
91.3
Lines
98.9
98.1
Composition-onn-sttep w/ combinatorial hardcoding
95.1
91.0
Composition-one-ste w/ sequential hardcoding
76.3
91.0
Composition w/ combinatorial hardcoding
76.9
72.4
Composition-grouu sequential hardeoding
62.4
72.4
Table 11: SPGEM's performance against oracle hardcoded goals.
packing-unseen- google objects-seq
packing-unseen google objects-group
put-blocksin--owlls
make-a-cirele
make-a-line
Figure 7: Manipulation tasks in simulation.
6.2.5 EVALUATION METRICS FOR REARRANGEMENT TASKS
For evaluating making-a-cirele task, fit best-fit circle for the final configuration predicted by SPGEM. To do this, we consider the centers the bounding boxes as points. Then, we compute the centroid of those points and the distance of each point from the centroid. This an estimate of "radius". We compute the standard deviation of all this quantity. If this is lower than 0.03, then we assign perfect reward. The reward linearly decreases when the std increases from 0.03 to 0.06. Beyond that, we give zero reward. We tuned these thresholds empirically by generating and distorting circle configuration.
We follow similar evaluation strategy for make-a-line. Here we compute the average slope and fit line to our data. Then we measure the standard deviation the distance of each point from the line. We found that the same thresholds used for circles work well for lines as well.
Step 0
Step
Step 10
Step 50
"Place the blue box right of the red box and below the yellow box. Put the cyan box left of the blue box. Move the red box above the magenta box. Place the green box inside the blue box"
Figure 8: Spatial planning via composable concept graph energy minimization.
22

---

Under review as a conference paper at ICLR 2023
6.3 LIMITATIONS AND PROMISING DIRECTIONS FOR FUTURE WORK
Our model do not take into account physics based priors while predicting goal. Other related methods like CLIPort, Transporter or Language Planners also do not incorporate these priors and we believe incorporating physics into manipulation planning and policy learning remains an open research problem. Incorporating these priors are critical to identify if the generated configura- tion stable or not. It would be even more useful if these priors can be added directly in the goal generation to directly avoid failures during execution. A promising direction is to model physics based constraints as energies. Naturally, one viable solution to achieving that is to leverage simula- tors that provide differentiable simulation, where we can additionally consider the gradients of the scene stability with respect to the spatial configuration of the scene when optimizing our EBMs. Re- cently, number of such differentiable simulators have been proposed (Huang et al., 2021; Heiden et al., 2021; Qiao et al., 2020). Given such a simulator, we can jointly optimize for the physics based constraints and the constraints mentioned in the language jointly to achieve a physically stable goal configuration that our robots can execute.
Physies:
Our goal generation module outputs the final goal configuration for the given instruction, but does not tell the ordering of the subgoals to execute them. For example, when making a tower, the goal generator would place all objects into tower but would not tell which objects to pick up first and thus inadvertently creating occlusions. One solution to this problem is to heuristically pick the order based on objects that are closer to the floor in the predicted scene configuration. However, more general solution would be to add temporal constraints as energy priors which could be an interesting direction for future research.
Temporal Ordering:
Location Parameterization: Different tasks need different abstractions - while many pick and place tasks for rigid objects can be modelled by abstracting them away with bounding boxes, there are tasks that might need finer location parameterization like keypoints or object segmentation masks. The EBMs then sit on top of these finer parameterizations to optimize the energy for the given task. There are multiple ways of getting these parametrizations. The simplest way is to use off-the-shelf object detectors, object segmentors and keypoint detectors to get hicrarchy of param- eterizations and then train simple selection module which conditions on the task to identify the relevant parameterization. Another possible solution would be to design EBMs that sit on a hier- archy of parameterizations similar to the JEPA model proposed in (LeCun, 2022). We leave this exploration for future work.
6.4 EBM ENERGY LANDSCAPE VISUALIZATION
In this section, we visualize the energy landscape of our EBMs. For binary relations, we fix an object's bounding box in the scene and move the bounding box of the other object all over the scene and evaluate the energy of the configuration. We expect the energy to be low in regions which satisfy the relation w.r.t the fixed object and high in other regions. For shapes like circles and lines, we cannot directly plot the energy landscape because we would need to jointly consider all possible combinations of objects in the scene which would be impossible to represent in 2D or 3D landscape. Hence, we fix all but one object in valid circle location and move the free box in all possible regions. We then expect the energy landscape to have low values in regions which complete the shapes and high otherwise. For compositions, objects and score the joint sum of energy for all constraints expecting the energy to be low in regions which satisfy all constraints and high otherwise.
consider single object's 's relation all other
The plots are shown in Figure-9. 9. We observe that the energy landscape is usually smooth and has low energies in valid regions and high energy otherwise.
23

---

Under review as a conference paper at ICLR 2023
Above
Below
Left
Right
Right blue box and left of green box and below red box
Left red box and above blue box and below box
Circle
Line
Right right green and right of blue box
Inside
(Unsatisfiable) Right box and box and above blue box
(Unsatisfiabla) Below
above green box and above blus box
Figure 9: EBM Energy Landscape visualization: In this visualization, the boxes shown here remain fixed, and score the energy of another box relative to the fixed box by moving it all along the workspace. Energy decreases from Red to black in this heatmap.
24

---

Under review as a conference paper at ICLR 2023
Operation
Signature (ObjectSet, ObjectConcept) Ob- jectSet
Semantics
Filter
Filter out set of objects based on some Object Concept like object name (eg. cube) or property (color, material)
BinaryFIBM
(Object A, Object B, Relation) (Pick locations, Place locations)
Executes BinaryEBMs for rearrang- ing Object A and Object B to sat- isty the given binary relation (like left of/right of/inside etc.)
MultiAryEBM
(ObjectSet, Shape Type, Property) (Pick locations, Place locations)
Executes MuliiAryEBMs for the given Shape Type (circle, line, etc.) with specified Properties (like size, position etc.) on of given ob- jects and generates pick and place lo- cations to complete the shape.
Table 12: All operations in the domain-specific language for SPGEM
6.5 BENCHMARK GENERATION
We extend the Ravens (Zeng et al., 2020) and CLIPort benchmark (Shridhar et al., 2021) for spatial reasoning in the PyBullet simulator. For each benchmark, we write template sentence (eg: "Ar- range OBJ1 into a circle") and then randomly select valid objects (and colors) from a pre-defined list. To test generalization, we include novel colors or novel objects in the evaluation set. Once the sentence is generated, we programatically define valid regions which satisfy the relation and then sample empty locations from it to specify object goal locations. We start by placing all objects randomly in the scene and then an oracle, hand-designed, policy picks and places the objects to the desired locations and returns a demo trajectory which involves the raw RGB-D images and pick and place locations. These can be used then to train behaviour cloning policy similar to CLIPort.
6.6 ETIICAL STATEMENT
Mapping language instructions to goals for robot coaching would contribute to the development of intelligent agents that are programmable using natural language, opposed being programmed explicitly or through positive and negative examples. This will facilitate human-robot interaction. Robots will then be able acquire new concepts and commands on-the-fly. adapting to every user's needs and desires, as opposed to maintaining a fixed set of functionalities.
With the potential benefits of easier communication with artificial agents, however, come some drawbacks. An increased ability to program robots would also be followed by more automation and less demand for employees, even for jobs that are now considered "safe". While in the long term, automation would probably be beneficial to society by increasing productivity, in the short term, the process of automation could have a negative impact on many people.
Lastly, while we aim to ease the programming of a robot, we cannot have control over the purposes this technology will be used, e.g. whether robot is tasked for beneficial or catastrophic purposes. Our aim is to enable the technology's 's potential for social good, such the potential of non-expert users, e.g. elderly people, to instruct a robotic assistant.
25


Reviews:
[{"review_id": "w_DH1Qvt1W", "reviewer": "Reviewer_hrSa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors use energy based models (EBMs) and object graph minimization to perform instruction-guided spatial rearrangement. Authors use EBMs to represent each spatial predicate (binary or n-ary). Objects are represented using the 3D or 2D overhead box coordinates. Authors train the EBMs using gradient descent on the sum of energies with respect to the coordinate space, providing the final configurations that best satisfies the instructions. \nUsing this, essentially, the model reason/plan over a pixel-abstract, but bounding-box aware space to interpret the instruction, predicting a goal state. Using a low-level and pick-and-place policy execution, the objects are finally shifted to the goal position. \n\nAuthors make following empirical claims:\n1. The model outperforms the baselines, especially in complicated instructions\n2. Generalization over  1) novel compositions,  2) novel instructions, and 2) novel objects and attributes.", "summary_of_the_review": "The idea of representing spatial predicates as EBMs is novel; alongwith reasoning over the 2D or 3D object coordinate space for instruction-guided spatial rearrangement. I also agree that language space may not be the best option to reason spatially about images. However, some experiments are lacking to support the generalization claims about generalizing to novel objects and attributes, alongwith experiments in real-world setting. Training each EBM to correspond to a 2-ary or n-ary predicate also raises the question of how many spatial relations that this model can support. Also, given the goal configuration is output and needs to be executed by a low-level policy executor, its easy to think of scenarios where the executor may not be able solve this using a greedy policy (stacks, tower of hanoi, inadvertently creating obstacles etc.). So, I am borderline on this. I feel some more experimentation is required, some of which may be non-trivial.", "strength_and_weaknesses": "The idea about moving from spatial reasoning in language space to space of object 2D 3D coordinates and representing predicates though EBMs are quite interesting. I feel there are some challenges to scale the work in real-world settings. Also, a few aspects of the generalization claims seem unsupported through experiments. \n\nA few concerns:\n1. I am wondering is the set of binary and n-ary spatial concepts open? You at least need the knowledge of which is a binary vs n-ary predicate? How do you infer that in care you are not using a list of predicates? I see you are training one EBM per predicate, which means the predicate list should be fixed right?\n\n2. In general, works such as Palm-Saycan [1] have also showed at least one experiment on real environments, such as kitchen environment, moving from table-top ones. As far as I remember, even LLMPlanner [2] uses a real-world table-top environment. I think, showing the work in such setting would help; especially given questions such as the first one.\n\n3. Generalization claim: Also, I do not see experiments performance over novel objects are shown. Among novel attributes, do you only mean shape and color? Is there anything else considered?\n\n4. LLMPlanner such as PaLM-saycan uses success predictors to give some feedback whether goal state has been reached. \nThough such goal state detection through energy minimizer is discussed, I do not see whether such a feedback mechanism exist? Some part has been discussed as physics based issues in Limitations. However, this is important, as the model only outputs final goal configuration; implying many goal configurations may be hard to achieve through pick-and-place policies in reality. What is the effect of this?\n\n[1] Do As I Can, Not As I Say: Grounding Language in Robotic Affordances, Ahn et al. 2022\n\n[2] Inner Monologue: Embodied Reasoning through Planning with Language Models", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gir0OQa3kU", "reviewer": "Reviewer_erLa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an optimization scheme to arrange a set of objects so they conform to a set of spatial constraints, that are either binary directional relationships, or specific spatial arrangements e.g. line or circle. An energy based model (EBM) is learned for each of the constraint type, which maps a set of obvious bounding boxes and locations to score that indicates how well the constraint is satisfied. These models are learned by enforcing that the ground truth layouts scores higher than perturbed layouts. At test time, a gradient-descent scheme is used to optimize the position of the objects, with the guide of the learned EBMs. The paper also incorporates off-the-shelf methods that maps language to constraints, and uses the optimized layout to guide a robotic agent. However, I am not sure if either of these relates to the central contribution of the paper. Evaluation show that the proposed method performs better than general purpose embodied agents on a set of tasks that seem to be tailored towards the proposed method. ", "summary_of_the_review": "I don't know much about robotics, but it appears to me that the core of the method is the learned-prior-guided layout optimization. This works without visual language grounding or the transporter network. I have many concerns with this optimization scheme, especially if it can generalize to more complex scenarios (see weaknesses above). As a result, while I think this method can be potentially useful, I lean towards rejection.", "strength_and_weaknesses": "#Strength\n- Good empirical performance on the set of tasks evaluated. Although these tasks are rather biased towards the proposed method, it is still nice to see.\n- Optimization guided by learnable priors is a good idea and should be fairly generalizable. The ability to learn these priors with a simple MLP over object attributes is also nice to know.\n\n#Weaknesses\n### Novelty\n- Constraint-guided layout optimization is a long studied problem, see earlier works such as \"Make it home: automatic optimization of furniture arrangement\", \"Interactive Furniture Layout Using Interior Design Guidelines\" for optimization, and works such as \"Example-based Synthesis of 3D Object Arrangements\" for learnable spatial priors. In some sense, the types of constraints dealt with in these works are much more complex than those in this work.\n- While learnable spatial priors are nice, the set of relations dealt with these work is rather simple, so it is unclear whether the techniques introduced here can apply to more complex scenarios (many work suggests that neural networks struggle to perform complex spatial reasoning, especially if based on axis-aligned bounding boxes).\n- It is not clear to me what is the significance of the language to constraint module and the robotic planning module. Seems that the work can exist without either of these and neither adds much to the contribution of this work.\n\n### Design Choices\n- I am not sure whether the proposed EBMs can generalize to more complex spatial relations, especially if semantics are involved. Relations such as \"to the left of\" are rathe easy to learn and I am not seeing examples of more complex cases e.g. circle. It seems that the benefit of a learnable prior really starts to emerge for more complex cases, whereas the type of constraints studied in this paper can be easily hardcoded (in a way that allows a gradient-descent based optimization scheme).\n- I am concerned about the optimization landscape formed by the union of all EBMs. It seems that there can easily be scenarios where the optimization get stuck in a local minimum (e.g. object A starts above B, needs to move below B, but additional constraints enforce that A cannot appear to either the left or the right of B). There is a reason why many of these works I mentioned above resorts to methods more complex than simple hill climbing e.g. MCMC with large jumps, simulated annealing, etc.\n- It seems that the method will struggle in cases where not all constraints can be satisfied. Since the learned functions lacks a consistent magnitude, it can be tricky to balance the weights of different constraint and ensure that the most number of constraints can be satisfied. \n\n### Evaluation\n- Would be really helpful to visualize the learned energy function. Since the constraints studied here is rather simple, the models really don't need to learn much to lead to good results. A visualization showing a smooth function that consistently satisfies the constraints can increase my confidence in the proposed method.\n- A baseline with hardcoded rules is needed - the method won't be of much use if it performs much worse than such a baseline.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y86c7q30Gn", "reviewer": "Reviewer_MYGL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper formulates the problem of spatial arrangement of objects as minimizing the energy function over the configurations of the object graph. The graph is constructed by parsing the language instruction and utilizing visual language grounding. The graph energy function is composed of predicates parameterized as neural networks. The model is trained using contrastive divergence with Langevin dynamics. Finally real object arrangement is done by a Transporter Network. Empirical results show that this formulation exhibits zero-shot generalization capability and works better than existing methods on the tasks evaluated.", "summary_of_the_review": "The paper introduces a way to treat of spatial arrangement as object graph energy minimization under the language to action context. The formulation itself isn't new, but the overall solution is interesting and applicable in certain settings. The are important limitations, however.", "strength_and_weaknesses": "Strength\n=======\n- A nice formulation of the spatial arrangement as object graph energy minimization. Although this treatment has been studied for decades, it is still interesting in the context of language instruction for robotics. The setting also requires a minimal amount of training data, as most heavy lifting parts (linguistic, visual and robotic) are handled by existing systems.\n- The empirical results are promising.\n\nWeaknesses\n==========\n- The limitations discussed in the paper (temporal constraints, location parameterization, simple predicates) are important to address, even though energy minimization can be theoretically suitable for solving these problems.\n- There are many moving parts in the overall system, it is important to see how robust the system is if the parts are not too reliable (e.g., semantic parsing, object grounding, robotic failure).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject