Programmatically Grounded, Compositionally Generalizable Robotic Manipulation
Authors: Renhao Wang
Jiayuan Mao
Joy Hsu
Hang Zhao
Jiajun Wu
Yang Gao
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Vision-Language-Action Grounding
Zero-Shot Generalization
Compositional Generalization
Neurosymbolic Learning

Abstract: Robots operating in the real world require both rich manipulation skills as well as the ability to semantically reason about when to apply those skills. Towards this goal, recent works have integrated semantic representations from large-scale pretrained vision-language (VL) models into manipulation models, imparting them with more general reasoning capabilities. However, we show that the conventional {\it pretraining-finetuning} pipeline for integrating such representations entangles the learning of domain-specific action information and domain-general visual information, leading to less data-efficient training and poor generalization to unseen objects and tasks. To this end, we propose \ours, a {\it modular} approach to better leverage pretrained VL models by exploiting the syntactic and semantic structures of language instructions. Our framework uses a semantic parser to recover an executable program, composed of functional modules grounded on vision and action across different modalities. Each functional module is realized as a combination of deterministic computation and learnable neural networks. Program execution produces parameters to general manipulation primitives for a robotic end-effector. The entire modular network can be trained with end-to-end imitation learning objectives. Experiments show that our model successfully disentangles action and perception, translating to improved zero-shot and compositional generalization in a variety of manipulation behaviors. Project webpage at: \url{https://progport.github.io}.

Published as a conference paper at ICLR 2023
PROGRAMMATICALIY GROUNDED, COMPOSITIONALLY GENERALIZABLE ROBOTIC MANIPULATION
Renhao Wang'* Jiayuan M Joy Hsu
Hang Zhao! 1.4.5
Jiajun Wu?
Yang Gao 1445
'Tsinghua University "MIT 3 Stanford University
Shanghai Artificial Intelligence Laboratory
Shanghai Qi Zhi Institute
ABSTRACT
Robots operating in the real world require both rich manipulation skills as well the ability semantically reason about when to apply those skills. Towards this goal, recent works have integrated semantic representations from large-scale pretrained vision-language (VL) models into manipulation models, imparting them with more general reasoning capabilities. However, we show that the conventional cetrraining-fneetunii pipeline for integrating such representations entangles the learning of domain-specific action information and domain -general visual informa- tion, leading less data-efficient training and poor generalization unseen objects and tasks. To this end, propose PROGRAMPORT, modular approach better leverage pretrained VL models by exploiting the syntactic and semantic structures of language instructions. Our framework uses semantic parser recover cutable program, composed of functional modules grounded on vision and action across different modalities. Each functional module is realized as combination of deterministic computation and learnable neural networks. Program execution produces parameters general manipulation primitives for robotic end-effector. The entire modular network can be trained with end-to-end imitation learning objectives. Experiments show that our model successfully disentangles action and perception, translating to improved zero-shot and compositional generalization in a variety of manipulation behaviors. Project webpage at: http::/proooor.gghuubh.
INTRODUCTION
Robotic manipulation models that map directly from raw pixels to actions are capable learning diverse and complex behaviors through imitation. To enable more abstract goal specification, many such models also take as input natural language instructions. However, this vision-language manipu- lation setting introduces perceptual inputs, and correspond this grounded understanding with the desired actions. Moreover, to fully leverage the flexibility of language, the agent must handle novel vocabulary and compositions not explicitly seen during training, but specified at test time (Fig. 1).
problem: the agent must jointly learn to ground language tokens to its
(a) Training Examples
(b) Zero-Shot Gen. Test
(c) Compositional Gen. Test
"Pack the yellow flower into the brown box"
"Push the green blocks into the blue square"
"Pack the ring into the brown box"
"Push yellow flower
into squard"
Figure 1: Zero-Shot and Compositional Generalization: Our framework, PROGRAMPORT, is
capable generalizing to combinations of unseen objects and manipulation behaviors at test time.
cqual contribution

---

Published as a conference paper at ICLR 2023
'put the heart the hole'
'put the heart in the hole'
Program
filter
HOLE
Image Encoder
Text Encoder
filter
HEART
goall
geal
Control
do
PUT
filter
filter
Parameter
Control
Decoder
HEART
HDLE
Parameters
(a) Setraininn--iinetuiinn Paradigm
(b) Program-based Modular Paradigm (ProgramPort)
Figure 2: (a) Existing manipulation models entangle visual grounding from pretrained VL models with task-specific policy learning. (b) Our disentangled approach leverages program-based representation of language semantics to yield faithful VL-grounding and improved genearlization.
To these ends, many recent works have relied on large pretrained vision-language (VL) models such as CLIP (Radford et al., 2021) to tackle both grounding and zero-shot generalization. As shown in Fig. 2a, these works generally treat the pretrained VI. model as semantic prior, for example, by partially initializing the weights of image and text encoders with a pretrained VL model (Ahn et al., 2022; Khandelwal et al., 2022: Shridhar et al., 2022a). These models then updated via imitating expert demonstrations. However, this training scheme entangles the learning of domain- specific control policies and domain domain-independent vision-language grounding. Specifically, that these VL-enabled agents overfit to their task-specific data, leveraging shortcuts during training successfully optimize their imitation learning (IL) objective, without learning a generalizable grounding of language to vision. This phenomenon is particularly apparent when the agent is given language goals with unknown concepts and objects, or novel compositions of known concepts and objects. For example, an agent that overfits to packing shapes into a box can fail to generalize to other manipulation behaviors (e.g. pushing) involving the same shapes.
find
In this work, we introduce PROGRAMPORT, a program- based modular approach that enables more faithful vision-language grounding when incorporating pretrained VL models for robotic manip- ulation. Given the natural language instruction, we first use Combinatory Categorial Grammar (CCG) (Steedman, 1996) to parse the sentence into "manipulation program," based on compact general domain-specific language (DSL). The program consists of functional modules that either visual grounding modules (c.g., locate all objects of a given category) or action policies (c.g., produce control parameter). This enables independent categories or attribute descriptors to their corresponding pixels, and thus disentangles the learning of visual grounding and action policies (Fig. 2b).
directly leverage pretrained VI. model to ground singular,
Our programmatically structured, modular design enables PROGRAMPORT to successfully learn more performant imitation policies with fewer data across 10 diverse tasks in tabletop manipulation environment. We show that after training on subset of objects, visual properties, and actions, our model can zero-shot generalize to completely different subsets and reason over novel compositions of language descriptors, without further finetuning (Fig. l).
BACKGROUND
Problem formulation. We take manipulation primitive-based approach to robot learning. At high level. we assume robot is given set of primitives P (e.g., pick, place, and push). Such primitives are usually defined by interaction modes between the robot and objects, parameterized by continuous parameters, and their composition spans wide range of tasks. Throughout the paper, will be using the simple primitive set P {pick, place) in a tabletop environment as the example, and extend our framework other primitives such pushing in the experiment section.
Therefore, our goal is to learn a policy x mapping an input observation o, at time t to an action a,. Each a, is tuple of two control parameters (Tpick: Tpiace): which, in the pick-and-place setting,
2

---

Published as a conference paper at ICLR 2023
parameterizes the poses for the robot end effector when picking and placing objects, respectively. This formulation also generalizes to other primitives such as pushing, in which case the control parameters are the pre and post-push locations. More specifically, in language-guided tabletop manipulation task, our observation is given by o, (x,,1) 1)). Here, X₁ RR is the top-down orthographic RGB-D projection of the 3D visual scene, while 1, is natural language description of the desired goal. The control parameters Tpicks Tplace both lie in SE(2).
To learn such policy, we assume access to {di, dn each consisting of discrete time-indexed scene-language-acin d; (1,,,,,,,,) a1).
of expert demonstrations D
tuples so that (x, (,))),, The policy π can be trained with an imitation learning objective by randomly sampling a particular tuple at time t from any demonstration.
Vision-language manipulation. Akin Transporter Networks (Zeng al., 2020), our policy is composed of two functions, Qpick and Oplace. outputting dense pixelwise action values over I= and operating in sequence. Qpiic first decides a particular pixel location Tpice in the top-down visual field to pick the target object; Qpicce then conditions the pick location to decide a particular place pose Tplace for the object:
Tpick mrg Qpick ((u,z) x,, Tplace aag max Optace (To) Tpick) <T((T)
(1)
(u.0)eHxW
where poses representing the manipulation end effector primitives are apriori discretized into the set (r), composed of discrete pixel locations and a discrete set of 2D rotation angles. Recently, CLIPort (Shridhar et al., 2022a) proposed to integrate pretrained vision-language (VI.) models into such frameworks. Shown in Fig. 2a, in CLIPort, both Qpick and Quaace are based two-stream fully- connected network (FCN) architectures. The encoders of the FCNs are initialized from pretrained weights of CLIP (Radford et al., 2021), while the control parameter decoders are randomly initialized and optimized. Similar paradigms have also been applied to other manipulation environments such as Khandelwal et al. (2022) and Ahn et al. (2022).
While the pretraining-imettuiing framework can leverage features from the pretrained CLIP encoders inform the model wide array of vision-language semantics, its monolithic structure entangles the learning of aision-language grounding and action policies. This has two drawbacks: poor data efficiency and overfitting. More specifically, since training data will only contain a finite number of compositions of action, category, and property terms, the model usually fails to generalize to manipulation instructions that contain novel compositions of seen terms and unseen terms.
PROGRAMPORT
The key idea of our framework PROGRAMPORT is that reliably utilizing pretrained vision-language (VL) models in manipulation involves leveraging the structure of natural language. Given the visual input and natural language instruction, PROGRAMPORT parses the instruction into program based on a domainn-spcciin of functional modules implemented by neural networks, each fulfilling a specific operation over the vision or action representation. The program executor executes the program upon the visual input and derives the control parameters to action primitives. This compositional design disentangles the learning for visual grounding modules (e.g., locating the objects being referred to) and the action modules (e.g.. predicting the control parameters). Concretely, it enables us to directly leverage pretrained VL models such as CLIP as the visual grounding module without further finetuning, and to only focus on learning action modules, which improves the transparency, data efficiency, and the generalization of the system. A visual overview of our framework is available in Fig. 3.
language (DSL). Each generated program corresponds to hierarchical structure
In the remainder of this section, we first describe the structure of our manipulation programs, and how they are derived from natural language instructions (Section 3)... We then detail the neural modules which realize their execution (Section 3.2). Finally, we specify the training paradigm (Section 3.2).
3.1 PROGRAMMATIC REPRESENTATION OF MANIPULATION INSTRUCTIONS
At the core of the proposed PROGRAMPORT framework is programmatic representation for instruc- tions. Illustrated in Fig. 2b, we use a program to represent the underlying semantics of the natural language instruction. Each program has hierarchical structure composed of functional modules. Each functional module takes the outputs from its predecessors, the visual input, and optionally additional concepts from the language (e.g., red) and computes the outputs (e.g., an attention mask).
3

---

Published as a conference paper at ICLR 2023
Inputs
Grounding Modules
Action Modules
Scene
pick
place
dolpack)
filten/eeaagon
filter
Instruction "Pack the blue hexagon in the orange box"
CCG
Parser
(ilter(bxx)
filten(orangee)
goal(
Figure 3: The execution of manipulation program. A CCG parser first generates a manipulation program. The task-agnostic visual grounding modules then identify visual semantics associated with the desired objects and targets. Finally, the task-specific action modules leverage these grounding results to output the manipulation control parameters.
We present the detailed definition of our domain-specift language (DSL.) in Appendix A.1. All functional modules can be categorized into two groups: visual grounding modules and action modules. Visual grounding modules take as input language descriptors, and return spatial masks over the corresponding locations in the visual scene. In this paper, the only visual grounding module we consider the filter operation, which localizes objects that have certain properties (as filter(blue, ...) will return blue objects in the scene). In contrast, action modules generate control parameters for robot primitives. They take as input the outputs from visual grounding modules. For example, the goal module generates 2D pose that has specified spatial relationship with another object (as in goal(in, ...) will return a location in the stated object). The do(puck, .... ...) operator takes three arguments: the action name pack, the object to be packed, and the goal location, both specified as masks over the input image, and generates the control parameter for the robot's next move.
Semantic parsing with combinatory categorial grammar. To parse language instructions into program structure, we use the Combinatory Categorial Grammar (CCG) formalism (Steedman, 1996). CCG parser is composed of small (less than 10) set of universal categorial grammar rules and domain-specific lexicon. Each lexicon entry maps a natural language word (e.g., red) to its syntactic type (e.g., N/N) and semantic form (e.g., Ax. Ar:ffilter( redd .)). The notation N/N indicates that this word can be combined with a word on its right with type N (noun), and the semantic form indicates that this word corresponds to visual grounding module that selects red objects from the scene.
PROGRAMPORT uses a predefined lexicon, manually specified by the programmers. We choose predefined CCG for three main reasons. First, CCG enables compact definition of domain-specific semantic parser. For each word in the vocabulary, we only need to annotate 1 or 2 entries. Second, the CCG parser is naturally compositional: given only the syntax and semantic forms for individual words, it is able to parse whole sentences with complex linguistic structures such as nested prepositional phrases. Third, and most importantly, the CCG parser can handle unknown words at test time. In particular, following work on syntactic bootstrapping (Abend et al., 2017; Alishahi & Stevenson, 2008), for a novel word (e.g., pack the daxy shape into the box), the parser first "guesses"
syntactic type by ensuring that the entire sentence parses successfully (in this case, the inferred syntactic type is N/N), and predicts its semantic form following prior distribution of p 0(semantics/synuuu under the current lexicon. this case, the predicted semantic form is Az az.filter( filter(daxy,2) daxy, x:).
3.2 PROGRAMPORT ARCHITECTURE
PROGRAMPORT executes the parsed programs by following their hierarchical structures. Overall, both visual grounding and action modules consume outputs from their predecessors and individual concepts parsed by the CCG, instead of the entire language instruction, and produce outputs. By disentangling manipulation in such a structured fashion, our architecture can learn to flexibly leverage vision-semantic priors across myriad of specific manipulation contexts.
4

---

Published as a conference paper at ICLR 2023
Visual grounding modules. The key functionality of visual grounding module is to localize visual regions corresponding to objects with given language-specinn specified property (i.e.. fiteeflleeffoo......... To imbue our manipulation model with semantic understanding of diverse object properties and spatial relationships, including those unseen in the manipulation demonstration data, we borrow from the recent zero-shot semantic segmentation model MASKCIIP (Zhou al., 2022). Given the input image X₁ and singular concept с (...g..lower. we use the image encoder G and text encoder H from a pretrained CLIP model to obtain deep features (x,,) E R "'''""">>> D2 and H(c) RP2,, respectively'. Here, we denote the spatial dimensions of the visual features as H' x W and the feature dimensions of the visual and language streams as D1, D2, respectively.
Recall that vanilla CLIP uses attention pooling to pool the visual features at different spatial locations into a single feature vector, and apply a linear layer to align the vector with the language encoding. We adopt a simple modification from MASKCLIP to achieve a similar semantic alignment of features, while preserving the spatiality of (xx)). In particular, C, and C₁ with weights from the penultimate and last linear layers of the CLIP attention pooling mechanism, respectively. The C, layer is initialized from the value encoder in the QKV-attention layer (Vaswani et al., 2017), while the C layer is initialized from the last linear layer. Applying them sequentially will project the visual feature into the same space as the language feature. Finally, our grounding module F grounding generates a dense mask over image regions of concept e via:
initialize two 1 x 1 convolution layers
grounding Xx, c) = [TILE (H(c)) OC, (C. (g( E R."""",
(2)
where TILE defines a spatial broadcast operator, in this case for the language embedding vector to the spatial dimensions of the visual features, the is the Hadamard product, and the summation is over the feature dimension Dr. This formulation of our semantic module allows us to leverage CLIP's large-scale vision- pretraining generalize semanties open vocabulary categories, including objects and their visual properties. Moreover, since our semantic program has decomposed the input language phrase 1t into individual visual concepts, we can compose multiple filter programs with the intersection operation. Specifically, this is implemented as in NS-CL (Mao et al., 2019) by taking the element-we -ise min of the generated masks, parsing out image regions that do not satisfy the combination of language descriptors.
Action modules. At their core, our action modules extend the architecture from CLIPORT, and finally generate the control parameters (Tpia, Tplace) In the pick-and-plane setting, the visual grounding of the object to be picked (i.e., the mask output by the corresponding filter operations) can be directly interpreted as a distribution for Trick. Therefore, in this section, will focus on how PROGRAMPORT predicts Tolace conditioned on Tpick. the visual grounding of the target location (e.g., the orange box), and the desired relation (e.g., in). The corresponding module is the "goal" operator. There are two design principles. First, besides the image and text, the action modules should also be conditioned on the grounding results from visual grounding modules (i.e., spatial grounding map indicating the position of target objects). Second, we want to prevent overfitting the action module to visual categories presented at training time. Therefore, we extend the architecture for Qplace to take the spatial grounding maps from previous visual modules with the following modifications.
First, we bilinearly upsample or downsample the input masks to the spatial dimensions of different layers within the CLIPort feature backbone, and multiply the interpolated grounding map with these intermediate layers. In contrast to other fusion strategies such as concatenation, this fusion strategy ensures that the grounding map will not simply be ignored by the action modules. Second, we upsample the grounding map to the spatial dimensions of the output action maps, and perform direct element- wise matrix multiplication with the action map. Since the grounding map can be viewed as a mask of the objects being referred to, this multiplication effectively forces the action modules to attend to regions where pick objects and place locations are described in the manipulation instruction. Overall, the end-effector poses in SE(2) can be output by appropriately adapting Eq. (1):
Tplace arg max Up (F grounding) 2 T,e(T)
Optace (TalXe, 1c, Tpick, F gronndnng)
(3)
where Up(-) is an upsampling operator, and @ is the Hadamard product. More details are available in Appendix A.6. In the simple SE(2) pick-and-place setting under the Transporter or CLIPort framework, simply specifying the locations of the object and its target pose is sufficient, and thus do(pack, ...) operates as an identity function.
'In practice we language prompt instead of a single word, following MASKCLIP (Zhou et al., 2022).
5

---

Published as a conference paper at ICLR 2023
Example. Recall that the program for the instruction "pack the blue hexagon in the orange box" is given by do goallfiterrflleereeeeeee
blue), filterfiite(bbx)),, orange). in), pack). As shown in Fig. 3, our first set of filter modules take the imput image and output masks conresponding to "hexagon" shapes and "box" objects. The next stage similarly filters for "blue" and "orange" objects. To compose these adjectives with their corresponding nouns from the previous stage, we take their elementwise min, returning masks corresponding to "blue hexagon" and "orange box", respectively, Next, our goal module takes the masks for the referred objects, which correspond to the picked object and its general place region, and outputs a distribution over SE(2) pose for the object that semantically consistent with the "in" target location. Finally, our do module outputs the final control parameters to the robot end-effector. In this case, it contains Tpic from the execution of (ilee((iler((exaaon) blue), and the Tplace output by the goal module.
Training. Our model can be trained in an end-to-end fashion, given the observation-actio tuples (x,, 1e, a.). via an imitation learning objective. Note that each action a, is itself tuple described by (Tpicks T.lace). Thus, given random observation-actionn tuple (x,, I, ar) sampled from some expert demonstration d; E D at time t, we fit our model by minimizing the cross-entropy loss:
= T [log softmmax (pick((,, u) 4,] ET [log softmax ( (T) |xx, 4,, Tpict)
(4)
4 EXPERIMENTS
In this section, we will show that the modularity of PROGRAMPORT enables better rision-languaee grounding and generalization, which empirically translates to strong zero-shot generalization (i.c., understanding instructions with unseen visual concepts), and compositional generalzation (i.e., understanding novel combination of previously seen visual and action concepts). We also provide validation of our approach on real world experiments in Appendix A.4.
Dataset. Our dataset extends the benchmark proposed by CLIPORT (Shridhar et al., 2022a), in turn based on the Ravens benchmark from Zeng et al. (2020). The CLIPORT benchmark consists of 10 ceguagee-onditionee manipulation tasks in PyBullet simulation environment. Tasks by a Universal Robot UR5e with a suction gripper end-effector. Every episode in a task is constructed by sampling set of objects and properties (such as object type, color, size etc.), and manipulating subset of the objects to some desired conformation, which is described by a language goal.
performed
Model details, training, and evaluation. Our visual grounding modules the weights from the frozen, pretrained ViT-B/16 CLIP encoder. We train for 200k iterations with AdamW and a learning rate of 0.0002. We use either n 100 or 1000 training demonstrations, and evaluate on 100 test demonstrations. Our evaluation metric follows Shridhar et al. (2022a) based on the same CLIP encoders, where a dense score between (failure) and 100 (success) represents the fraction of the episode completed correctly. For baselines, we compare to CLIPORT across two settings: the s single setting, where the model trains and tests on a single task at time, and the mult t setting, where the model trains on all tasks at once, and is independently evaluated on each individual task.
4.1 ZERO-SHOT GENERALIZATION
To evaluate zero-shot generalization, all of our main results in Table use the "unseen" variants of the tasks, as descrihed in Shridhar et al. (2022a). Concretely, the train and test splits of each task involve different objects and object properties, with minimal overlap. For example, out of 11 total possible colors for objects, only 3 are shared for both train and test episodes. Or for tasks involving real-world objects from the Google Scanned Objects dataset (Downs et al., 2022), a total of 56 household items are split into mutually exclusive sets of 37 train items and 19 test items. Crucially, for a fair comparison, these test language descriptors are also entirely missing from the predefined CCG lexicon, and so semantic goal parsing must rely on the probabilistic approach described in Section 3. Our results show that our model effectively grounds object concepts described language with the visual scene, and with the exception of the align-rope task, outperforms our baselines across the benchmark (Table 1). Align-rops is challenging for PROGRAMPORT because it is hard for the CLIP model ground abstract spatial concepts used in this task (e.g. front and back).
Data efficiency. Not only do we outperform the baselines by a large margin, but on many tasks, method does significantly better with 10x fewer demonstrations. For example, on the "packing- shapes" task, our best- rest-performing model trained on 100 demonstrations outperforms its corresponding baseline trained on 1000 demonstrations by 25.0 points.

---

Published as a conference paper at ICLR 2023
packing- box pains
packing-goog
packing-google objects-
separating- small-piles
separating large-piles
objects-seq seq
demonstrations
100
1000
100
1000
100
1000
100
1000
100
1000
CLIPORT (single)
67.1
72.0
64.1
70.8
79.3
82.1
75,9
74.3
62.4
66.9
CLIPORT (multi)
77.1
72.3
66.9
71.6
77.0
81.7
65.9
63.7
54.2
59.1
PROGRAMPORT (single)
72.6
78.0
78.4
79.7
79.0
81.2
76.8
77.9
63.8
69.5
PROGRAMPORT (multi)
82.3
85.7
73.5
84.0
80.7
83,9
77.0
78.8
63.0
68.2
align-rope
packing shapes
assembling kits
put-blocks-in-bowl. in-bolls
towers -anoi
demonstrations
100
1000
100
1000
100
1000
100
1000
100
1000
CLIPORT (single)
84.8
94.1
42.0
38.0
32.2
34.7
36.8
29.9
94.6
99.0
CLIPORT (multi)
81.2
70.3
38.0
32.0
30.2
26.9
55.9
46.7
75.2
69.1
PROGRAMPORT (single)
77.5
86.9
52.0
54,0
40.6
42.4
38.2
38.7
97.6
98.4
PROGRAMPORT (multi)
78.4
89.1
57.0
63.0
39.2
46.8
54.2
57.7
97.0
96.3
Table 1: Zero-Shot Generalization. Training and validation splits for all tasks involve different objects, object properties, or other spatial descriptors.
Input
ProgPort
CLIPort
3d
pili
"Pack
figure
Mlocs
pluce
brown
Figure 4: Qualitative pick and place affordances for PROGRAMPORT and CLIPORT. PROGRAMPORT compositionally reasons over independent visual semantics in zero-shot fashion, correctly picking and placing the target object. CLIPORT identifies both pick object and place location incorrectly.
This data efficiency is derived from our improved grounding properties: by semantically parsing and visually grounding one object concept at time, we more effectively leverage the pretrained VL model. Moreover, note that the pest-performing CLIPORT models for different tasks are mostly the single variants. This suggests the zero-shot performance of CLIPORT relies heavily overfitting to the actions of the specific task, as opposed to leveraging the general visual grounding enabled by CLIP. With PROGR AMPORT, the gap between our ng and mult? model variants is much smaller, and indeed the multi i variants are often the stronger manipulators. PROGRAMPORT thus leverages diverse visual and action information across tasks more effectivelyy, and possesses scalability with respect to both the number of demonstrations as well as the number of tasks.
Improved vision-language grounding. Qualitatively, we show more interpretable and robust affordance maps compared to the baseline (Fig. 4). By grounding single object or property at a time, and then leveraging our DSL operations compose together these masks, we ground the different concepts. For Tpick, the lion figure is extremely small within the visual scene, and positioned in crowded area; our visual grounding modules are still able to compose the concepts of figure and lion together in zero-shot manner, informing accurate pick affordance. Similarly, for T placee the baseline is confused by the visual similarity between the box and the porcelain plate. By contrast, visual grounding modules compose box and brown together focus the left box-like object.
4.2 COMPOSITIONAL GENERALIZATION OF VISUAL GROUNDING AND BEHAVIORS
To evaluate compositional generalization, we introduce 4 new tasks which re-combine basic unary object concepts from the original 10 tasks (Table 2). We do not train directly on these tasks; by evaluating on them only, we assess the ability to reason compositionally about the distinct visual properties, spatial relationships, and manipulation primitives learned at training time. For example, the packing-locatien wwwoaooocooobbe box", where both s shape and loc objects and spatial descriptors seen independently throughout various tasks at training time, but now appear in a test-time context that requires the model to reason semantically over their combination. For full specification of these tasks, please refer Appendix A.2.
task contains instructions of the form "pack the (shape) into the {loc}
7

---

Published as a conference paper at ICLR 2023
packing-color-bor
packing-location
separating-locaaion-iil
pushing-shapes
100
1000
100
1000
100
1000
100
1000
CLIPORT (single)
23.0
24.0
11.0
13.7
48.1
43.8
2.4
3.2
CLIPORT (multi)
19.0
18.2
10.0
8.3
39.9
41.0
2.7
7.6
PROGRAMPORT (single)
64.0
68.0
56.0
62.5
69.2
72.4
32.3
36.9
PROGRAMPORT (multi)
72.0
76.3
62.0
67.1
68.4
70.5
36.6
40.2
Table 2: Compositional Generalization. Models can learn different object shapes, colors, and manipulation properties from different tasks. We show that only PROGRAMPOR hierarchically composes these learned concepts and behaviors together to perform well in more complex tasks.
packing- shapes
packing-google- objects-seq
assembling- kits
separating- small-piles
put-blocks- in-bowls
Semantic Supervision
Method
100
1000
100
1000
100
1000
100
1000
100
1000
MaskCLIP
52.0
54.0
78.4
79.7
40.6
42.4
76.8
77.9
38.2
38.7
PROGRAMPORT (single)
GT
97.0
98.0
95.8
95,9
92.0
93.4
80.6
86.4
97.2
96.3
MaskCLIP
57.0
63.0
73.5
84.0
40.6
42.4
77.0
78.8
54.2
57.7
PROGRAMPORT (multi)
GT
96.0
98.0
96.2
96.4
91.0
94.5
81.2
87.2
99.3
97.7
Table Disentanglement of visual semantics and action. Replacing MASKCLIP grounding maps with ground truth segmentation maps of objects and targets yields near-perfect manipulation.
Length-based generalization to new action settings. Our fourth generalization task (pushing- shapes) is much more difficult, involving language instructions of the form "push the {color) (shape) into the {location] (color} square". Note that this task evaluates both compositional generalization and "length generalization". The latter assesses the ability of the model generalize beyond the assumption of at most one visual descriptor per object in similar training tasks. Furthermore, this task requires the model to zero-shot generalize entirely new actions; only pushing of simple, vogularly-structree blocks is learned at training time, whereas shapes are much larger, irregular, and have different friction properties. PROGRAMPER AMPORT exhibits much better performance here (+32.6 mean reward), compared to the CLIPORT baseline, which fails entirely.
4.3 ABLATION: DISENTANGLEMENT OF ACTION AND PERCEPTION
To illustrate the disentanglement of action and perception in our modular design, we perform an addi- tional ablation study. Specifically, we train our model normally on all 10 main tasks from Section 4.1, and at test time, we choose diverse tasks out the 10. For each test episode, we replace the attention maps generated by the semantic modules with ground truth segmentation maps corresponding to the picked objects and place locations. This effectively poses the question: given a perfect perception of the scene, how would our action modules behave? A properly disentangled model would be able to leverage the noiseless ground truth visual grounding and decide the appropriate actions to perform. As apparent in Table 3, performance across all tasks improves dramatically when provided the ground truth information. This suggests that our action modules do not overfit to mask-specific shortcuts when optimizing the imitation objective, but actually learn to leverage the attention maps provided by the semantic modules in a disentangled manner. Such disentanglement has also been translated to improved compositional generalization for our model.
RELATED WORK
Learning language-guider robot behavior. Many prior works have explored enhancing robotic behavior through annuage-conditione imitation learning (Lynch Sermanet, 2021; Stengel-Eskin et al., 2022; Stepputtis et al., 2020) and reinforcement learning (Andreas et al., 201 8; Luketina et al., 2019; Misra et al., 2017). With the rise of powerful transformer-based architectures, recent works have used large-scale pretrained lasion-language representations to improve performance for tabletop manipulation (Shridhar et al., 2022a), visual navigation (Khandelwal et al., 2022), and real-world skill learning and planning (Ahn et al.. 2022; Huang et al., 2022). For example, CLIPORT shows that by integrating embeddings from the CLIP VL model into the Transporter manipulation framework (Zeng al., 2020), robot can better generalize to manipulating objects in zero-shot manner.
8

---

Published as a conference paper at ICLR 2023
However, in contrast with method PROGRAMPORT, these works do not leverage the semantic structure in natural language to ground individual visual and action concepts. Rather, prior works use the pretrained language model as monolith that consumes the entire language input and outputs latent embedding. This emhedding is not grounded to individual concepts within the language inputs, which leads poor generalization when faced with novel (combinations) object attributes.
Neurosymbolic learning. Prior works have shown neurosymbolic learning, which refers to the integration of deep neural networks for pattern recognition and symbolic structures such as programs for reasoning, as an effective mechanism for enforcing VL grounding (Mao et al., 2019; Wu et al., 2019). Mao et al. (2019) is a representative work that learns to jointly disentangle independent visual concepts and parse natural language sentences, from natural supervision in VQA. The learned concepts can be composed for novel downstream tasks via programmatic (re)composition of neural network modules, enabling zero-shot and compositional generalization. More generally, a structured approach involving neural perception of vision and language, and higher-order symbolic reasoning has appeared previously (Li et al., 2020; Nottingham et al., 2021; Sun & Alexandre, 2013). Beyond improved VL-grounding, this approach has also translated to improved performance and data effi- ciency on many downstream tasks, such as reinforcement learning (Cheng et al., 2019; Verma et al., 2019), scientific discovery (Cranmer et al., 2020; Shah et al., 2020) and robotics (Zellers et al., 2021).
Our work is similarly inspired to combine structured representations with deep learning, in the new setting of robot manipulation. However, instead of learning a small set of visual concepts, we leverage pretrained VL models to cover larger, open-ended set of concepts, and improve our zero-shot capabilities. Moreover, we ground not just visual concepts with their language descriptors, but also abstract actions for robotic primitives, while preserving compositional generalization capability.
Vision-language (VL) grounding. The goal of VL grounding is to identify correspondences between visual input and descriptors in natural language. Common tasks include visual question answering (Antol et al., 2015), VL navigation (Anderson et al., 2018). or VL manipulation (Shridhar et al., 2022a:b). Recent works show that the CLIP (Radford et al., 2021) model trained on 400M image-caption pairs from the web possesses zero-shot VI. grounding capabilities for a wide array of visual concepts (Kim et al., 2021; Shen et al., 2022). In robotics, this has translated to grounding of robotic affordances with natural language in models such as CLIPORT (Shridhar et al., 2022a).
However, we find that the na've approach taken by these works, which leverage only the frozen, pretrained CLIP embeddings, cannot successfully disentangle task-agnostic visual grounding with task-specific action policies. Other works show that this naive integration of pretrained VL models also affects grounding abilities in 3D space (Thomason et al., 2022), or temporal settings such as video understanding (Le & Hoi, 2020). In particular, VI. grounding is often adversely affected by end- to-end training (e.g. via an imitation objective), and the resultant models unable to reason about novel concepts independently, or compose them de novo for more complex tasks (Lin et al., 2022). This shortcoming is addressed by PROGRAMPORT through program-based modular approach.
6 CONCLUSION
In this work, propose PROGRAMPORS a program- based modular framework for robotic manipu- lation. We leverage the semantic structure of natural language to programmatically ground individual language concepts. Our programs are executed by series of neural modules, dedicated to learning either general visual concepts or task-specific manipulation policies. We show that this structured learning approach disentangles general visual grounding from policy learning, leading to improved zero-shot and compositional generalization of learned behaviors.
Future work of PROGRAMPORT includes generalization to more free-form natural language instruc- tions and challenging scenarios where the syntactic context is not sufficient to predict semantics of novel words. Alternate techniques such as extending human-annotated CCG banks (Bisk & Hocken- maier, 2013; Zettlemoyer & Collins, 2012), and leveraging the wealth of pretrained language-to-one co-code models (Chen et al., 2021) can be a concrete path to alleviating this issue. Better representations that can capture interactions between robots and objects (e.g. grasping poses for different objects) is another important extension. Finally, subsequent work may also consider jointly learning and leveraging pretrained representations that more suitable for grounding interaction modes between robots and objects, especially for manipulating difficult articulated and deformable objects in our 3D world (Shridhar et al., 2022b).

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our simulated dataset details are provided in the Appendix A.2. The most salient training details and hyperparameters are presented in Section 4 of the main text. DSL details can be found in Appendix A.1 and architecture details are available in Appendix A.6. Our code will also be publicly available.
ACKNOWLEDGMENTS
This work is in part supported by the Stanford Institute for Human-Centeree AI (HAI), Analog Devices, JPMC, and Salesforce. This work also supported by the Ministry of Science and Technology of the People s Republic of China, the 2030 Innovation Megaprojects "Program New Generation Artificial Intelligence" (Grant No. AA21A50000000,, No. 20222D0161700) as well as grant from the Guoqiang Institute, Tsinghua University.
REFERENCES
Omri Abend, Tom Kwiatkowski, Nathaniel Smith, Sharon Goldwater, and Mark Steedman. Boot- strapping language acquisition. Cognition, 164:1 16-143, 2017. 4
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do As Can, Not As Say: Grounding Language in Robotic Affordances. arXiv:2204.01691, 2022. 8,
Afra Alishahi and Suzanne Stevenson. Computational Model of Early Argument Structure
Acquisition. Cognitive science, 32(5):789-834 2008. 4
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-Languaa Navigation: Interpreting Visually- Navigation Instructions Real Environments. In CVPR, 2018.
Jacob Andreas, Dan Klein, and Sergey Levine. Learning with Latent Language. NAACI-HIT, 2018. 8
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In ICCV, 2015. 9
Yonatan Bisk and Julia Hockenmaier. An hdp model for inducing combinatory categorial grammars. Transactions of the Association for Computational Linguistics, 1:75-88, 2013.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Control Regularization for Reduced Variance Reinforcement Learning. ICML, 2019. 9
Miles Cranmer, Alvaro Sanchez Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering Symbolic Models from Deep Learning with Inductive Biases. In NeurlPS, 2020.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google Scanned Objects: High-Quality Dataset of 3D Scanned Household Items. In ICRA, 2022. 6
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language Models Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In ICML, 2022. 8
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but Effective: CLIP Embeddings for Embodied AI. CVPR, 2022. 2,3,
Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: -ant-annd----uagee Transformer Without
Convolution or Region Supervision. In ICML, 2021. 9
10

---

Published as a conference paper at ICLR 2023
Hung Le and Steven CH Hoi. Video-Grounded Dialogues with Pretrained Generation Language Models. In ACL, 2020.
Qing Li. Siyuan Huang, Yining Hong, Yixin Chen, Ying Nian Wu, and Song-Chun Zhu. Closed Loop Neural-Syy Soural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning. In ICML, 2020. 9
Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On Grounded Planning for Embodied Tasks with Language Models. arXiv:2209.0044 00465, 2022. 9
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. A Survey of Reinforcement Learning Informed by Natural Language. In IJCAI, 2019. 8
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. In RSS, 2021. 8
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro- Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In ICLR, 2019. 5,9
Dipendra Misra, John Langford, and Yoav Artzi. Mapping Instructions and Visual Observations to Actions with Reinforcement Learning. In EMNLP, 2017.
Kolby Nottingham, Litian Liang, Daeyun Shin, Charless C Fowlkes, Roy Fox, and Sameer Singh. Modular Framework for Visuomotor Language Grounding. In Embodied Al Workshop. 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. ICML, 2021. 2,3,9
Ameesh Shah, Eric Zhan, Jennifer Sun, Abhinav Verma, Yisong Yue, and Swarat Chaudhuri. Learning Differentiable Programs with Admissible Neural Heuristics. NeurlPS, 2020.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How Much Can CLIP Benefit Vision-and-Language Tasks? In ICLR, 2022.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort: What and Where Pathways for Robotic Manipulation. In CoRL, 2022a. 3, 6, 8, 13
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Peroeiver-Actor: A Multi-Task Transformer for Robotic Manipulation. In CoRL, 2022b. 9
Mark Steedman. Surface Structure and Interpretation. MIT press, 1996. 2,4
Elias Stengel-Eskin, Andrew Hundt, Zhuohong He, Aditya Murali, Nakul Gopalan, Matthew Gom- bolay, and Gregory Hager. Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions. In CoRL, 2022.
Simon Stepputtis, Joseph Campbell, Mariano Phielipp. Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. In NeurlPS, 2020. 8
Ron Sun and Frederic Alexandre. Connectionist-Symbol Integration: From Unified to Hybrid Approaches. Psychology Press, 2013. 9
Jesse Thomason, Mohit Shridhar, Yonatan Bisk, Chris Paxton, and Luke Zettlemoyer. Language Grounding with 3D Objects. In CoRL, 2022. 9
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention All You Need. In NeurIPS, 2017. 5,
Abhinav Verma, Hoang Le, Yisong Yue, and Swarat Chaudhuri. Imitation- Prooeicte Programmatic Reinforcement Learning. In NewrlPS, 2019.
11

---

Published as a conference paper at ICLR 2023
Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei- Ying Ma. Unified Assual-emantti Embeddings: Bridging Vision and Language with Structured Meaning Representations, In CVPR, 2019.
Rowan Zellers, Ari Holtzman, Matthew Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, and Yejin Choi. PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in 3D World. In ACL, 2021.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. Transporter Networks: Rearranging the Visual World for Robotic Manipulation. In CoRL, 2020. 3, 6, 13
Luke S Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint /2Xiv::/07.1444 2012. 9
Chong Zhou, Chen Change Loy, and Bo Dai. Extract Free Dense Labels from CLIP, In ECCV, 2022.
12

---

Published as a conference paper at ICLR 2023
A
APPENDIX
A.1 DOMAIN-SPECIHIC LANGUAGE
We detail here the operations (Table and types (Table 5) contained within our DSL. The im- plementation of filter. do and goal are given within the main text. The remaining operations are parameter- free. In particular, Scene returns a uniform attention map over the visual imput: ObjUnion and ActionConcat take the elementwise max of their input grounding or action maps, respectively.
Operation
Signature
Semantics
Scene
0 Object
Return all objects in visual scene.
Filter
(Ohject, ObjProp) Object
Return all objects satisfying a vision-language property.
Relate
(Object, Object, ObjRel)
Object
Return all objects satistfying relation between target and reference objects.
Do
((Goal). Action) - Plan
Perform an action according to set of goals.
Goal
(Object, Object, ObjRel) -
Goal
Identify manipulation goal for an object and target relation.
ObjUnion
(Object, Object) Object
Return the union two (sets) of objects.
ActionCencat
(Plan, Plan) Plan
Jointly perform two of actions.
Table 4: All operations in our DSL for vision-language manipulation.
Type
Example
Semantics
Ob ject
Block, Rope, etc.
Manipulation objects.
Ob jectProp
Red, Dark, etc.
Object properties.
ObjRel
In, On, etc.
Object relationships.
Goal
Manipulation target such as distributions over SF(2) poses.
Plan
End-effector control parameters in SE(2).
Action
Pack, Push, etc.
Manipulation primitives.
Table 5: All types in our DSL for vision-language manipulation.
A.2 DATASET DETAILS
Our dataset based off the Ravens benchmark proposed Zeng et al. (2020), and its adaptation in Shridhar et al. (2022a). In particular, for manipulatives, use the same set of 20 synthetic shapes and 56 Google Scanned Objects. For modifying visual appearance, we use the same set of 11 colors Most importantly,
the same split of "seen" and "unseen" objects and colors, training on the "seen" splits and testing on the "unseen" splits. These splits are mutually exclusive for the manipulatives; only 'red', 'green' and 'blue" shared between splits for the colors.
In the remainder of this section, we detail the specific tasks used within our benchmark, including visual scene setup and the success metric used compute the results in Table 1, Table 2 and Table 3.
A.2.1 BASIC TASKS
Packing Box Pairs. Given small boxes of varying colors, and larger, open-topped brown box, the goal is to rightly pack all small boxes of two specified colors into the large brown box. Sizes of all boxes are randomized, and there will be an exact number of the specified small boxes to tightly pack the larger box, in addition to other distractor small boxes. Occasionally, there will be boxes of only
13

---

Published as a conference paper at ICLR 2023
one of the two specified colors; the agent must learn to ignore the missing color and pack the boxes with the one present color.
Success is the fraction of total volume of specified color blocks in the scene that end up packed inside the brown box.
Packing Google Objects Seq. At each timestep, given number of Google scanned objects and an open-topped brown box, the goal is to place a specified object into the box. Each episode is composed of at least and at most timesteps.
Success is determined by the total volume of all specified objects within the brown box up to the given timestep, divided by the volume of all specified objects.
Packing Google Objects Group. Given a number of Google scanned objects from at least two different categories, well open-topped brown box, place all objects of specified category into the brown box.
Success determined by the fraction of volume of specified objects placed within the box,
Separating Small Piles. Given 50 small blocks and two square zones of different colors, positions and poses, push all the blocks into the zone with specified color.
Success determined by the fraction of total blocks within the correct zone.
Separating Large Piles. Given 15 large blocks and two square zones of different colors, positions and poses, push all the blocks into the zone with a specified color.
Beyond size changes, the large blocks also have more mass and lateral friction compared to their small counterparts.
Success determined by the fraction of total blocks within the correct zone.
Align Rope. Given a soft-body rope formed from 20 articulated beads and the outline of 3-sided square, the goal is to connect the two endpoints of the rope two adjacent corners of the square. The language instruction can specify one of four possible configurations:
1. From front left tip to front right tip
2. From front right tip to back right corner
3. From front left tip back left corner
4. From back right corner to back left corner
Success is represented by the degree to which the bead poses after manipulation match the line segment between the two adjacent corners specified in the language goal.
Packing Shapes. Given 4 distractor shapes and specified shape, as well as a open-toppen brown box, the goal is to put the specified shape into the brown box. Shapes are unique.
Success is binary; 1 if the shape is inside the boundaries of the box, and 0 otherwise.
Assembling Kits. Given 5 shapes and a "kit" of similarly-shaper holes of varying poses, the goal is to precisely place each shape into its corresponding hole. Shapes are not unique; in the event that more than one of the same shape exists the field, the language goal will specify a one-to-one mapping between shapes and holes, e.g. "put the orange star into the left star shape hole."
Success is determined by the fraction and degree to which placed shapes match the poses the shape holes.
Put Blocks in Bowls. Given a number of blocks of varying colors, and a (possibly different) number of bowls of (possible different) varying colors, the goal is to place all blocks of specified color into some bowl of a specified color. Note that each bowl will fit at most one block, and is guaranteed that there sufficient number of bowls to achieve the goal.
14

---

Published as a conference paper at ICLR 2023
Success is determined by the fraction of blocks of the specified color being within any bowl of the specified color.
Towers of Hanoi. Ar each timestep, identify and move a ring shape from one of three possible pegs to another specified peg. Each episode contains 7 such timesteps, which is the ideal solution for 3-ring Towers of Hanoi. The sequence moves deterninistically composed apriori using a DFS solver.
Success determined by the fraction of correct ring placements over timesteps.
Packing Prepositions. Given 4 distractor shapes and specified shape, as well two brown boxes, the goal is to place the specified shape into the box with the specified spatial relationship. This spatial relationship given relative to one of the 4 distractor shapes. For example, "pack the flower into the brown box left of the star."
Success binary; if the shape inside the boundaries of the correct box, and 0 otherwise.
A.2.2 COMPOSITIONAL TASKS
After training on basic tasks, we evaluate the baselines and our model on the four compositional tasks described below. In addition the visual scene setup and success metric, also describe their compositional nature and from which basic tasks we expect the model to learn the requisite building block behaviors.
Packing Color Box. Given 4 distractor shapes and I specified shape, as well as two boxes of different colors, the goal to place the specified shape into the box with the specified color. Shapes still unique.
Shapes are seen in the "packing-shaper task and colors are seen in the sracking-box-paa task, as well as the "assembling-kits," "put-blocks-in-bow/i and parpaatinnssmall//ag---ippl tasks.
Note that at training time, the model only ever packs objects into single brown box. Hence, this task assesses the model's s ability to disentangle and separately ground the packing action and the visual properties the box.
Success is binary; 1 if the shape is inside the boundaries of the correct box, and 0 otherwise.
Packing Location Box. Given 4 distractor shapes and specified shape, as well as two brown boxes, the goal to place the specified shape into the box with the specified spatial relationship. For example, "pack the flower into the left brown box." Shapes are still unique.
Shapes are seen the "packing-shaper task and spatial descriptors are seen in the 'assembling task.
Again, the model only ever packs objects into a single brown box training time. This task assesses the model's ability to compose spatial descriptors with the packing action.
Success binary; if the shape inside the boundaries of the correct box, and 0 otherwise.
Separating Location Piles. Given 50 small blocks and two square zones of the color and size, but different positions and poses, push all the blocks into the zone with a specified spatial relationship. For example, "push the pile of purple blocks into the left square."
The small blocks and pushing action are present in the "separating-ssalll-a descriptors are seen in the "assembling-kinn task.
task and spatial
Akin to the "packing- -ocatino-ocaionn--- docation-box" task, this task assesses the model's 's ability to compose spatial descriptors with the pushing action.
Success is determined by the fraction of total blocks within the correct zone
Pushing Shapes. Given 4 distractor shapes and specified shape, and two square zones of the same size, but different colors, positions and poses, the goal is to push the specified shape into the
15

---

Published as a conference paper at ICLR 2023
specified zone. Zones also have spatial descriptor; for example, "push the green ring into the left blue square." Shapes are no longer unique, but rather combinations of color and shape are unique (e.g. there may be two "ring" shapes in the scene, but only one "green ring" shape.)
The pushing action is present in the "separating-smale -aaalsaaag--pilllleee tasks, spatial descriptors are seen in the "assembling-kiss" task and shapes are within the "packing-shape" task.
This task is highly challenging. The shapes pushed at test time are distinct from those seen at training time in saacknngshaapss, and the pushing action is only learned for blocks in the 'separating- small/large-piles tasks. Hence, we test for zero-shot and compositional generalization to novel objects and visual properties, but also to novel actions. Finally, introducing both spatial and color descriptors to the target square zones also adds a length- dimension to the generalization properties.
Packing Nested Prepositions. Given 4 distractor shapes and specified shape, as well as two brown boxes, the goal is to place the specified shape into the box with the specified spatial relationship. This spatial relationship given sequentially, relative to of the distractor shapes. For example, "pack the flower into the brown box left of the star right of the diamond." Shapes are still unique.
Shapes are seen the "packing-shape"" task and relative spatial descriptors are seen in the 'packing- prepositions" task.
Again, the model only ever packs objects into a single brown box training time. This task assesses the model's ability to compose relative spatial descriptors alongwith the packing action.
Success binary; 1 if the shape inside the boundaries of the correct box, and 0 otherwise.
A.3 EXPERIMENTS WITH PREPOSITIONAL PHRASES
Task details are available in Appendix A.2.
packing- peested-prepositoonn
CLIPort ProgramPort 45.0
28.0
17.0
31.0
Table 6: Experiments with relate operator. packing- prepositions evaluates zero-shot generalization whereas packing- prepoittnessssttionn evaluates compositional generalization.
A.4 REAL WORLD EXPERIMENTS
We collect 40 demonstrations of tabletop pick-and-place tasks using a Franka Panda robot with parallel gripper end-effector. Pick objects are common household objects including tape measure, tennis balls. spoons, as well as plastic fruits such as bananas, pears, peaches, etc. These are placed into plates, bowls and blocks of different colors (red, blue. green, purple etc.). For example, language goal may take on the form "pack the banana in the red plate." Success criteria is the same as the "packing-shapes" task in simulation. We train on 30 demonstrations, and hold out the other 10 demonstrations for validation. For test evaluation, we use 10 demonstrations for in mi-iistiibttinn pick objects and place locations, and another 10 with out-of-distribution pick objects and place locations. Test results are shown Table and visualized Fig. 5.
packing-objects
packing-objectt--useen
ProgramPort
70.0
80.0
Table 7: Real world packing objects task. The seen suffix involves new instructions containing the same objects seen in the training set, whereas the unseen suffix evaluates zero-shot generalization to new instructions containing novel objects.
16

---

Published as a conference paper at ICLR 2023
packing-obje packing-objecds-see eedes-eeen
Pack the pear the red plate
Pack the purple block the plate
Pack the mango in the red plate
Pack the baseball in the red plate
Figure 5: Real world packing-shapes task. Pick locations on objects are highlighted in green, and place locations are highlighted in red.
17

---

Published as a conference paper at ICLR 2023
A.5 ADDITIONAL AFFORDANCE VISUALIZATIONS
We present qualitative examples of PROGRAMPORT's and CLIPORT's attention mask across four different tasks in Figures 6, 8, 9. concept, while CLIPORT only outputs masks over the final pick and place locations. For all tasks shown,
outputs masks over every independent visual
able more accurately localize the target locations compared CLIPORT.
Input
ProgPort
CLiPort
brown
Input
ProgPort
CLIPort
Af
Fic-
"Put white blocks orange howl"
Input
ProgPort
CLIPort
"Push the blocks square"
parpe
Input
ProgPort
CLIPort
"Pack the circle
PaU
brss
plns
Figure 9: Qualitative affordance evaluations for different zero-shot and compositional tasks.
18

---

Published as a conference paper at ICLR 2023
A.6 MODEL DETAILS
In this section, we also illustrate and further describe the architectures of our visual grounding and action modules.
Pretrained CLIP
RGB Iupur: IIx W
CLIP RN30
Upsample
(Frezca)
Costr)
Cocro
x 2048
Language Inpux "orange"
CLI RN50
TILE
Chormel Pool
(Frozen)
1024
Previous Grounding Maps W.Wl
7x
Figure 10: Visual grounding module example: filter.
Our visual grounding modules leverage pretrained CL.IP embeddings via a layer- initialization scheme. Formally, CLIP is trained via embedding-level contrastive objective 400M image- caption pairs, which we represent as tuples (x,, l.). CLIP is comprised of an image encoder (ResNet-50) G and text encoder (transformer) H. which featurize their respective components in the input, yielding a visual feature map V and a text embedding u. To generate a visual embedding vector spatial weighted-sie of V is performed via qkv-style attention (Vaswani al., 2017), followed by linear layer F:
== F Σ softmax Emby (AvgPool (V)) Emb; (Viz)²)
Embb(('s))
(5)
where Embg,k.e linear embedding layers and C is some fixed scalar. Image and text embeddings u, v are then orchestrated to the same representation latent space via CLIP's contrastive training objective.
To obtain an attention map over x, representing the language concepts described in I,, we need to maintain the spatiality of the visual feature map V. To this end, remove the summation
pixels i,j. j. and introduce two x 1 convolution layers, C. (initialized with the weights from linear embedding layer Emb,) and C (initialized with the weights from the linear layer F). This is illustrated in Fig. 10.
19

---

Published as a conference paper at ICLR 2023
RGB Input: H x W x
Language Input 'in'
Two Stream Decoder
ARGMAX
Tpick
Pick Map: H
Pick Grounding Map: H 11
(a) Quick submodule
Pick Location Tynck
Grop
Two Stremm Fnender-Deooder
Rotate
RGB Input: x 3
rotated crope
Crpp Featunes: CxCxDxR
Language Iuput 'int"
Decnder
Correlation
ARGMAX
TTheee
Place Features:
Place Mup:
Grounding Map: H.W. 1
(b) Oplner submodule
Figure 11; Action module example: goal.
The two-stream FCN encoder-decoder architecture used in our action modules borrow heavily from CLIPort. However, find that simply treating the pick place grounding maps additional fourth channel to the RGB visual input results in the decoder layers ignoring the visual semantics represented by these grounding maps. To solve this issue, we first bilinearly upsample downsample the grounding map to the spatial resolution of different layers of the decoder. We then directly multiply these grounding maps with the outputs of the decoder layers, explicitly encoding the notion of attention into these latent features.
20


Reviews:
[{"review_id": "h-pPUUze3n", "reviewer": "Reviewer_cgai", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a vision-language-action grounding framework for robot manipulation, with the guidance of a domain specific language Combinatory Categorial Grammar (CCG). CCG first parse the natural language to more structured program, and associate the objects and targets via the grounding module. Then it produces the action parameters via action module. This method has advantages over many tasks than the baseline methods in zero-shot generalization and compositional generalization.", "summary_of_the_review": "It is a good experimental work, with sound techniques and good performance.\nThe paper is well-written and well-motivated.\nI think it is of quality to publish.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well-written and easy to follow.\n\n2. The proposed framework is novel, and has great performance.\n\nWeakness:\n\nThe idea of this paper is somewhat novel, but not entirely refreshing. It seems a combination of different known techniques: CLIP/MaskCLIP, minor modification of CLIPort.\nWith the modularization and structured CCG, zero-shot generalization ability and compositional generalization ability is quite predictable.\nNevertheless, putting all these techniques together and proves the performance gain are also contributions for community.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NmNSfpeR9ss", "reviewer": "Reviewer_LWwk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is an improvement-through-specialization upon prior work in robotic manipulation where Vision-Language (VL) models allow for manipulation of novel objects or objects that are specified using novel characteristics (not used in training). \n\nThe study makes use of a previously described simulation-based robotic manipulation framework (and data?) to train a novel model architecture. Whereas a previous ML modeling study in robotic manipulation (called CLIPort) trained an action module directly from image and text embeddings, this study is able to train for better performance and better generalization by taking advantage of the constrained nature of the available robotic manipulation tasks within a limited framework. This is performed by limiting the space of task descriptions to a Domain-Specific Language (DSL) that can be easily parsed to extract physical entities and their descriptors plus the required task type from a limited (trained-for) set of available manipulation tasks.\n\nIn this manner - the study takes advantage of the stength of VL models to narrow down (mask) the relevant locations for start and end of manipulation tasks (pick A and place in B where A and B can be rich object and location entity descriptions). Specifically, by parsing the commands into task, pick and place entities with a DSL, I believ that they avoid mixing of characteristics between the two entities (which sometimes occures with VLs) and they can use the features that describe objects as filters that progressively limit the possible locations within the image (e.g. \"blue box\" is parsed into a filter for blue and a filter for box whose output can be intersected to produce an accurate location mask). The action module is then trained on the processes masks and is thus disentangled from the task of understanding where the task entities are.\n\nThe results show very significant improvements across several manipulation tasks in terms of performance, generalization to unseen entities and across task types while also exhibiting better data efficiency (within the constrained domain) compared with the previous work.\n", "summary_of_the_review": "The paper presents a novel way of training a robotic manipulation model with help from a Vision Language model for localization of novel objects. It solves a problem encountered in a previous study where it seems that the model learned was unable to efficiently disentangle aspects of scene understanding from aspects of action generation. This was done by limiting the text that describes the task to a domain specific language (as opposed to NLP) and parsing it with a parser that is tailored to the limited task domain.\nThe architecture is able to efficiently learn to manipulate unseen objects or seen objects in manipulation tasks where they were not observed before. An ablation experiment shows that the action module performs nearly perfectly when the VL outputs are replaced with ground-truth segmentation.", "strength_and_weaknesses": "This paper is an exemplary case of harnessing general-purpose modeling of the world (scenes and their descriptions) to solve an engineering problem within a limited domain (a small set of manipulation tasks involving two objects / entities) by combining the strengths of the general-purpose model and programmable human knowledge of the task breakdown so that the the ML modeling that remains to be performed is focused on the aspects of the solution that are still missing.\n\nOne might say that the model architecture in this study benefits from an \"unfair advantage\" over previous work in that the designers used their understanding of manipulation tasks to better constrain the scope of what the VL model contributes to the architecture. I see this as good engineering work for solving robotic manipulation in settings where the set of entities to be manipulated is open ended.\n\nPerhaps a weakness of this line of work, as far as I understand it, is that there exist robotic tasks in which a vision + language model may have more to share with an action module than a set of masks. For example, a situation where the text cound be used to describe a novel task (or any other case where the DSL proposed is too contraining). Such enhancement would require making a new custom parser and extracting new maps / embeddings from the VL model.\n\n- I found equation 2 non-intuitive (due to lack of familiarity with what the Cl and Cv layers signify).\n- In some cases it seems that the pick action is referred to as a 2D (pixel location) vector whereas in others it seems to be a location+rotation.\n- I am not clear on where the observed action is used in eq 4 (is it the (u,v) and tau_i?)\n- Is it correct to say that the only information passed between the VL model and the action model is two heatmaps and a task type (pack or push)? If so, would the authors like to discuss whether the model is missing out on some additional information that the VL model might have?\n\n- An experiment showing transfer of the skills learned in simulation to a real robot would have been super.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C5PXr7Ie-61", "reviewer": "Reviewer_x8wi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of solving visual-language (VL) grounding tasks by leveraging pre-trained VL models. It is motivated by the fact that conventional methods are not data-efficient in the training stage and have poor generalization to unseen objects and tasks. To this end, this paper proposes a framework that first parses an instruction into a program based on a domain-specific language and then composes functional modules, a visual grounding module, and an action module, to execute the program. The experiments show that the proposed framework outperforms a baseline based on a two-stream fully connected architecture in terms of both generalization and data efficiency. The ablation studies suggest that the action module works with ground segmentation maps and does not overfit to individual tasks in a disentangled manner of action and perception. I am leaning toward accepting this paper since it studies a promising research direction and presents a reasonable framework to address the problem with supportive experimental results.", "summary_of_the_review": "I am leaning toward accepting this paper since it studies a promising research direction and presents a reasonable framework to address the problem with supportive experimental results.", "strength_and_weaknesses": "## Paper strengths and contributions\n \n**Motivation and intuition**\n- The motivation for making visual grounding tasks more generalizable by utilizing compositional program-based modules is convincing and intuitive. \n- The limitations of prior works are explicitly mentioned, which strengthens the motivation of this work.\n \n**Technical contribution**\nThe proposed an action module that is conditioned on the grounding map and the extended architecture from CLIPort seem effective for preventing overfitting.\n \n**Clarity**\n- The overall writing is clear. The authors utilize figures well to illustrate the ideas. Figure 1 clearly shows the tasks that this paper deals with and provides task examples to explain the figure.\n- The authors explain their intuitions and reasons well when introducing new ideas.\n \n**Related work**\n- The authors give a clear description of related prior works in learning language-guided robot behaviors, neurosymbolic learning, and vision-language grounding.\n- The authors provide clear comparisons that highlight the difference between their work and related previous work, which makes it easier to follow for those who are not familiar with the background.\n \n**Ablation study** \nAblation studies presented in Table 3 justify the effect of disentangling the visual semantic and the action module, which can be translated into compositional generalization for the model. It is helpful to understand the design choice.\n \n**Experimental results**\nThe experimental results demonstrate the generalization performance of the proposed model.\n \n## Weaknesses and questions\n \n**Clarity**\n- It is unclear to me how we can predict p(semantic/syntax) in the last part of Section 3.1. I wonder what the intuition is to determine in what situation the conditional probability p should be higher given an inferred syntactic type (N/N).\n- It is a little difficult for me to follow the description of the architecture of the visual grounding module in Section 3.2. Providing a diagram explaining the architecture presented in previous works, the modification made in this work, and I/O features between modules and layers, would be helpful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BpXerR2wU8", "reviewer": "Reviewer_dRES", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript considers the task of vision-language manipulation; the proposed approach leverages Combinatory Categorical Grammar (CCG) to parse natural language instructions into a manipulation program (task plan from a domain-specific language), consisting of functional modules.", "summary_of_the_review": "Limited novelty: lacking insights from new methodology or optimization techniques; manuscript predominantly provides an engineering solution\n\nThe problem formulation relies on strong assumptions: regarding the definition of the DSL and regarding the alignment between natural language instructions and the observations/actions; it is not clear that the proposed approach would generalize to more complex instruction-following tasks", "strength_and_weaknesses": "(Strengths)\n\nProvides a framework for combining vision-language models with structured grammar, without further fine-tuning.\n\n(Weaknesses)\n\nSection 1: Many works seek to use pre-trained vision-language models as a frozen semantic prior. It is true that fine-tuning is usually regarded as being too costly, may yield catastrophic forgetting, or simply dismisses the rich experience obtained through the vision-language model's pretext task. However, layer-weight initialization from the pre-trained vision-language model is not without its problems either — chief of which is that the vision-language model is likely not aware of the downstream task. Discussion is missing in the manuscript about why this is not a problem for the proposed approach.\n\nSection 2: This formulation seems to assume that the natural language (sub-)instructions are perfectly aligned in the expert demonstrations. This would be a remarkably strong assumption, compared to other formulations, where the natural language refers to an abstract, high-level goal (with multiple implicit subgoals), and where the actions are at a finer granularity than the high-level goal *and* its implicit subgoals.\n\nSection 3, Conclusion: Missing discussion regarding the limitations of using a predefined lexicon and inability to generalize to arbitrary natural language instructions (as in the popular instruction-following tasks: ALFRED, TEACh, R*R, etc.)\n\nSection 3, Section 4: Missing more complex examples, such as nested prepositional phrases, as alluded to in Section 3\n\nSection 4: The ablation experiment consists of the proposed model under perfect perception conditions (ground-truth visual grounding), compared with an instance of itself without perfect perception, and shows better results under perfect perception. Why is this considered an \"ablation study\", and why is it an informative experiment?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%