Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions
Authors: Elias Stengel-Eskin
Andrew Hundt
Zhuohong He
Aditya Murali
Nakul Gopalan
Matthew Gombolay
Gregory D. Hager
Venue: CoRL2021 Poster (2021)
Keywords: Instruction following
object grasping and manipulation
multimodal fusion
computer vision for robotic applications

Abstract:  Enabling human operators to interact with robotic agents using natural language would allow non-experts to intuitively instruct these agents. Towards this goal, we propose a novel Transformer-based model which enables a user to guide a robot arm through a 3D multi-step manipulation task with natural language commands. Our system maps images and commands to masks over grasp or place locations, grounding the language directly in perceptual space. In a suite of block rearrangement tasks, we show that these masks can be combined with an existing manipulation framework without re-training, greatly improving learning efficiency. Our masking model is several orders of magnitude more sample efficient than typical Transformer models, operating with hundreds, not millions, of examples. Our modular design allows us to leverage supervised and reinforcement learning, providing an easy interface for experimentation with different architectures. Our model completes block manipulation tasks with synthetic commands $530\%$ more often than a UNet-based baseline, and learns to localize actions correctly while creating a mapping of symbols to perceptual input that supports compositional reasoning. We provide a valuable resource for 3D manipulation instruction following research by porting an existing 3D block dataset with crowdsourced language to a simulated environment. Our method's $25.3\%$ absolute improvement in identifying the correct block on the ported dataset demonstrates its ability to handle syntactic and lexical variation. 

Guiding Multi-Step Rearrangement Tasks with Natural Language Instructions
Elias Stengel-Eskin
Andrew H 1*
Zhuohong He
Adit Murali!
Nakul Gopalan² Johns Hopkins University
Matthew Gombolay²
Gregory Hager'
2 GGorgia Institute of Technology
Abstract: Enabling human operators to interact with robotic agents using natural language would allow non-experts to intuitively instruct these agents. Towards this goal, we propose a novel Transformer- based model which enables a user to guide robot arm through a 3D multi-step manipulation task with natural lan- guage commands. Our system maps images and commands to masks over grasp place locations, grounding the language directly in perceptual space. In suite of block rearrangement tasks, we show that these masks can be combined with an existing manipulation framework without re-training, greatly improving learning efficiency. Our masking model is several orders of magnitude more sample effi- cient than typical Transformer models, operating with hundreds, not millions, of examples. Our modular design allows us to leverage supervised and reinforcement learning, providing an easy interface for experimentation with different architec- tures". Our model completes block manipulation tasks with synthetic commands 530% more often than based baseline, and learns localize actions cor- rectly while creating mapping of symbols to perceptual input that supports com- positional reasoning. We provide a valuable resource for 3D manipulation instruc- tion following research by porting existing 3D block dataset with crowdsourced language simulated environment. Our method's 25.3 25.3% absolute improvement in identifying the correct block on the ported dataset demonstrates its ability handle syntactic and lexical variation.
Keywords:
Instruction following, object grasping and manipulation, multimodal fusion, computer vision for robotic applications
Introduction
Reinforcement Learning (RL) is powerful tool for developing perception-driven robotic systems. Most RL algorithms train an agent to perform a single task based on an given reward function. However, what if we would like to adapt extend the learned policy of a robot novel more specialized tasks by providing instructions? Could we exploit previously learned RL policies for a robot to solve novel tasks now specified using natural language? The ability adapt learned policies through natural language to perform novel tasks would open the door to modular integration of perception, language, and action, avoiding the complexities of joint training over all modalities si- multaneously. For example, in our experiments, we constrain general-purper policy using natural language, without needing any language data to train the policy itself.
block manipulation
This kind of modular learning requires injecting guidance from language into the RL-driven action policy while simultaneously addressing the challenges of natural language understanding. Natural language semantics is often underspecified, especially with respect to a physical environment; cor- rectly choosing one out of many potentially valid actions for command such "Move the green block to the left of the right block" involves pragmatic and common sense reasoning. Furthermore, order map from language to action, we need to address the symbol grounding problem [1], i.c.
"Equal contribution; {elias, ahundt] ) jejuueeu "Code and data: https:/
http://stengggguiiiggeggeeeee -wltiiinniiiiiiseeeeeee com/
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Pixel-wise Q values
Input Image
SPOT-Q Network
Masked Q values
"Stack the red block on the blue block"
Patch mask
Pixel-wise mask
Transformer
Figure 1: Schematic of our model. From an input image, pixel-wise Q- D-values are generated using pre-trained network. Simultaneously. Transformer based model produces language language-conditione conditionndd masks, which are heuristically turned into pixel-wise masks over the image. These masks are com- bined with the Q-values to find the highest Q-value that respects the language constraints.
how does one map symbols (e.g. "left", "right", "block", "red") to the physical world? We leverage Transformer encoders learn strategy from data, in place of the manually defined data fusion strategies found in prior work [2, 3, 4, 6, 7, 8].
In work, we factorize language-driven action into learning how following Misra et al. [4]. Learning how to act is accomplished by the SPOT-Q algorithm [9] which teaches agent re-arrange colored blocks into stacks and by grasping and placing them. We use SPOT-Q models pre-trained without any language data for our manipulation experiments, demonstrating that our method lets leverage existing manipulation models without additional training. SPOT-Q identifies regions worth exploring, i.e. containing an object, to encode a set of valid action regions for grasping and placing. We produce segmentation masks that are jointly conditioned on both the language input and an image of the state and apply them with to set of pixel-wise Q-values from the SPOT-Q RL framework. This process ideally provides us with the best action for given state and command, as Figure illustrates.
and learning where act,
Factorizing the learning process this way provides several advantages. Firstly, it offers more efficient path to learning. Separating the language grounding model from the reinforcement learning model leads less complex RL model that receives more direct feedback for tasks. Moreover, the separation of action and language understanding allows us to learn the language component via supervised learning, which is typically more sample efficient. Secondly, the modular nature of the model allows us to swap components easily: we can train masking model on simulated images or real images (cf. Section 5.1), allowing us to use the same action model for both simulated and real tasks without re- re-training. In an end-to-end model, a change to the input space would likely require re-training from scratch. Finally, the factorized model output is more interpretable; when an action is unsuccessful, we can inspect our model to localize the point of failure, be it lack of comprehension mistake while acting. This may not evident from the agent's 's action alone. Note that our implementation of this factorization is not without disadvantages: if the language contains information how act this data could be lost.
In summary, our contributions are:
1. We introduce novel Transformer-based model for mapping language commands to masks over an action space. Our approach's 's modular nature enables us to seamlessly integrate an existing Q-learning framework proposed by Hundt et al. [9] for block manipulation tasks.
2. We show our model succeeds in a variety of simulated and real data settings with orders of magnitude less data than typically available to Transformers, furthering the potential use of Transformers in the low-data regimes commonly found in robotics.
3. We demonstrate this model's 's utility involving a challenging block re-arrangement task with crowdsourced natural language commands containing syntactic diversity well complex natural language phenomena.
existing dataset (introduced by Bisk et al. [10))
4. We release our real and simulated datasets, plus a filtered version of the Bisk et al. [10] dataset containing the subset of examples feasible under realistic model of physics.
2 Related Work
We describe related work the areas of natural language instruction following on robots, Q- Learning for multistep manipulation tasks, and Transformer models for Computer Vision.

---

Robots and Natural Language Commands - We follow the Problem-Solution Sequences (PSS) paradigm [11], where a multi-step manipulation task (e.g. "build a stack") is framed as a sequence of incremental images or state representations. These may be paired with per-step instructions on how to accomplish the intermediate goal (e.g., "stack the red block on the blue block"). Several PSS datasets have been proposed [11, 10] and modeled in 2D [4, 12] and 3D [10]. We translate the 3D dataset proposed by Bisk et al. [10] to a simulated environment in Section 5.2. The sequential nature of PSS relates to the hierarchical RL setting explored by Andreas et al. [13], who learn task- specific sub-policies indexed by linguistic sub-goals. Similar approaches are applied to zero-shot sub-goal sequence generalization 14], we---inaal--odeeree sub-goal execution[ [15], and choosing interpretable linguistic sub-goals [16]. While we also assume access to wise annotated instruc- tions, we focus on guiding low-level robotic manipulation policy with supervised masking model rather than a task-level policy. Gangopadhyay et al. [17] and Sun et al. [18] learn policies guided by symbolic programs, while Perera et al. [19] first parse language into a symbolic representations. Our masking-basee ate symbol ontologies. Language has been used for reward-shaping [20, 21, 22], and reward model learning [23]; we instead use the language-agnostic reward shaping in SPOT-Q. Stepputtis et al. [24] and Lynch and Sermanet [25] use language-conditioned imitation learning for manipulation policies; we instead use Q-learning. TransporterNets [26] learn to perform multi-step tasks from demonstra- tion, but lacks language input. Like our work, Concept2Robot [27] completes language-desctt desccribe tasks by combining learning from demonstration and RL. However, it differs from our system along multiple axes: we examine challenging multi-step tasks with progress reversal[9]: we succeed on much smaller datasets; we consider crowdsourced commands; and we evaluate on real robot data. Nguyen et al. [28] guide object retrieval with language, but do not model placement.
method ties the language directly to the perceptual space, eschewing intermedi-
conditioned navigation [29, 30, 4, 8, 7, 51, frames the problem of choosing an action region as a semantic segmentation task in which each pixel as a separate binary decision: act or do not act. VGP has directed a simulated [5] and real quadcopter [8, 7J. We implement VGP with UNe([31, 4, 7, 8], as a baseline.
Visual Goal Prediction (VGP), introduced by Misra et al. [4] for
Q-learning for Multi-Step Tasks Mohammed et al. [32] reviews RL for robot grasping. Zeng et al. [33] shows Q-Learning outperforms supervised action segmentation for table decluttcring(34, 35, 36), instead, "Good Robot!" [9] segments actions worth exploring to accelerate Q-Learning of single-task, multi-step robot learning, all without language input. "Good Robot!" [9] was also the first to complete multi-step tasks with consideration of progress reversal, a failure case where actions undo previous progress such that (2()) additional actions are required to recover, egg. toppling the tower in partially completed stacking task. We segment actions worth exploring via linguistic input for multi-step, multi-task execution considering progress reversal and low-data 34) conditions.
Transformer Encoders for Vision - Though Transformer encoders, which build representations using self-attention, have become standard in NLP, computer vision tasks are still dominated by con- volutional models, of which UNet [31] is a paradigmatic instance. The use of Transformers in vision tasks has been held back by the quadratic complexity of the Transformer's 's self-attention layer; an image input treated as a sequence of pixels would quickly become computationally intractable [37]. Dosovitskiy et al. [37] introduce minimal modification to perform image recognition using Trans- formers, tiling the image into square patches, which they concatenate into a sequence and augment with learned linear positional embeddings. Though the model presented by Dosovitskiy et al. per- forms image-level recognition well, it cannot perform pixel-level tasks (e.g., semantic segmentation) as-is. Ranftl et al. [38] present an extension this model for pixel-level prediction.
3 Models
We propose ceaasfoomr--baeee model to convert image and natural language inputs into separate masks for grasping and placing, thereby allowing users to guide multi-step manipulation tasks.
We begin by tiling the x n birds-eye image of the scene with C' colored blocks into k x k patches, whose RGB pixel values are concatenated into one vector, resulting in patch vectors I = .1.....i1. The resulting vectors each token in the command string W = Wi.... Wm is embedded into a continuous space resulting in vectors E Cm. and those vectors augmented with positional encodings well.
augmented with fixed sinusoidal positional encoding [39]. Similarly,
The goal is to train model that produces a pair of language-defined spatial masks Mpplece E <R and M/grasp E RR and an optional pair of state reconstructions Serasp Šplace E Cxxxn given by:
3

---

Nr = R ((ooFFFie (Jshared( WT) Wow)) S²= ( Je ( (1:W))os))
(1)
where f. are parameterized by Transformer encoders with output size k. Hm E R H2 E R a is the softmax function, and t E {grasp, place). The reconstruction function R up-samples patches by copying the patch value across each of its constituent pixels, resulting in Note that structurally, the "shared," " "grasp," and "place" Transformers are identical, but while the "shared" Transformer is used both for grasp and place prediction, the others are task-specific.
n image.
We train the masking model separately from the manipulation model. For the latter model, we use pre-trained SPOT-Q model from Hundt et al. [9], which produces pixel-wise Q-values for picking and placing, making it easy to integrate with our approach. Furthermore, pixel-wise Q- method to non-cubie shapes. All masking
values facilitate potential future work on extending models are trained with a multi-task loss, composed of a pixel-wise binary cross-entropy loss for the masks, and an optional auxiliary reconstruction loss:
L.
1 N2
Σ Mi Mog(MM, 1
Σ 1,J
St. log(5) L Ggrasp + Lptace
The reconstruction loss here forces the model to semantically segment the image by identifying pixel with its block ID, or labeling it as background. As in Nguyen and Salazar [40], we use a pre-norm Transformer layer with scaled initialization. Our learning rate schedule is based on Vaswani et al. [39], with a varied number of warmup steps as in Nguyen and Salazar. A comprehensive report of all hyperparameters and a figure of our architecture is given in Appendix A.
Q-Learning- Our model relies on a learned function from states and actions to expected rewards, Q(sr, a). which we learn via the SPOT-Q algorithm [9], which introduces novel reward shaping processes for improving efficiency. More details are given in Appendix A.3. Given pixel-wise n x n state space of an image and two actions, grasp and place, the output of this function can be expressed tensor R "xxxx
4 Datasets and Resources
4.1 Simulation Experiments with Synthetic Commands
We create a dataset of commands and grasp/place masks to train our masking model and demon- strate its performance in simulated environment on a robot arm. Using trajectories from Hundt et al. [9], we annotate successful grasp-and and-place action pairs with synthetic commands generated from a template, and use simulation state information to construct the reference masks M¹ and state infor- mation St. While synthetic language makes simplifying assumptions about the syntactic and lexical diversity of natural language, it provides a controllable test-bed for symbolic reasoning, and is often used to study compositionality [41, 42, 43], as we do in Section 5.1. For row- row-making, the template is "move the color_a block next to the color_b block". and for stacking the template is "stack the color a block on the color block". We train supervised VGP models (cf. Section 3) on these commands, and combine the produced masks with the Q-values from the SPOT-Q framework in simulated environment to execute the actions in order. The visual input to our models is the concate- nation of birds-eye color image and depth image. The training data is augmented by flipping each image across 4 axes, manipulating the command accordingly, as well as by the addition of pixel- wise Gaussian noise drawn from (0, 0.05). This results in train/dev/test splits of 5520 /46 /46 and 8970 /4 /76 for row-making and stacking, respectively, where each sample is an input image and command paired with reference place and grasp mask. Experiments and results on VGP metrics and in the simulation for this data are given in Section 5.1.
4.2 Transfer to Real Images
In order to be useful in real-life settings, our method must also be applicable to real data. However, unlike simulated data, data from real robot is expensive and time-consuming collect. Therefore, it is natural to ask: how much real data is needed to train an adequate VGP model?
The Hundt et al. [9] real robot trial logs include 1496 actions each for row-making and stacking. We split these into an 80/10/10 train/dev/test split and augment the examples by flipping, producing 5980, 747, 748 datapoints. Image collection details are given in Appendix B.2. Given the small size of the real dataset, we pre-train a model on simulated images and fine-tune it with the real images.

---

4.3 Compositionality
In linguistics, compositionality is typically taken to mean that the meaning of a whole utterance is composed of the meanings of its parts; the meaning of "stack the red block on the blue block" is in part made up of the independent meanings of "red" and "blue". Compositionality lets speakers re-combine known symbols in new ways, underpinning human language use. In our task, a compo- sitional reasoner should correctly execute "stack the red block on the blue block" (referred to here as stack (z, b)) without having seen that combination; it should be able to localize red and blue blocks from other color contexts (e.g. stack (I, g), stack (y, b), etc.). However, there is no explicit constraint during training to enforce this; the model could learn to interpret each of the 23 possible color pairs atomically. We test our model's compositional reasoning via its ability to generalize to unseen color combinations. We hold out 6 color pairs during training (26% of the 23 order-invariant pairs) and test exclusively on those pairs. For example, we train a masking model with all color combinations except those made only of red, blue, yellow, and green blocks and then test only on stack ((r, b, y), {r, b, g, y)) examples. We hold-out each subset, training and evaluating model as described above, average the metrics across all subsets, and compare against a random train and test split baseline, where all color pairs are seen at train time. For efficiency, we train on each task separately and do not augment with Gaussian noise.
4.4 Naturalistic Language Commands with Complex Scenes
Language sourced from human speakers is syntactically and lexically diverse, with mistakes, mis- spellings, and other idiosyncrasies that make far more challenging to model than template data. An- alyzing synthetic language showcases our model's ability to integrate discrete symbolic commands with visual perception and action, well as our model's ability to generalize in compositional fashion, but this does not imply that our model generalizes to the complexities of real language.
We examine a 3D block manipulation dataset with human natural language instructions that was introduced by Bisk et al. [10] (cf. Fig. 2). This dataset is set apart from other block-world data by its focus on complex instructions. Rather than using synthetic utterances, instructions were sourced from crowdworkers (9 per example), who were shown a pair of images and asked to provide instruc- tions to robot, with the goal of manipulating the blocks in the first image such that they result in the second image. This collection method result in more syntactically diverse utterances with complex linguistic concepts, such as "mirroring" and "balancing".
The dataset is comprised of a sequence of state rep- resentations paired with natural language instruc- tions on how to transition between states. While the blocks dataset is designed for robot manipulation, several assumptions were made to simplify annota- tion by crowdworkers. Crucially, physics in the en- vironment were turned off while the structures were being constructed and annotated, resulting in some physically infeasible structures. To facilitate research in the blocks domain, and make the dataset more useful for training and evaluating embodied agents, we create a filtered version of the dataset, where physically infeasible examples are removed. Our filtering process removes 49.34%, 43. 43.93% and 38. 23% of train, dev, and test, respectively. Appendix C.1 contains more details on the trans- lation process and data preprocessing. To model the dataset's world state sequences in the VGP framework, we translate the 3D Cartesian coordinates of each block given by the dataset's state rep- resentation into a top-down grid view, where each pixel corresponds to the ID of the block located at that position. We refer to this image as the scene's "state representation."
"Mereedes Benz will move right until it above twitter" Figure 2: Bisk et al. [10] blocks data sample.
While the blocks dataset contains all the state information required to produce image-like state representations, this cannot generally be assumed to be the case. A more likely scenario mimicks Section 4.1, where the model's input is a combined color and depth image. After translating the dataset into the simulated environment used in Section 4.1, we collect color and depth images for the whole blocks dataset. In the original dataset, each block type identified by company logo. Since the logos were downsampled by the simulator, we also assign a color to each type. The demographics and labor practices underlying its original creation are unknown [10, 44], We filtered their dataset without crowdworkers; and it is intended for robotics research, not product deployment. Experiments and results for this dataset are in Sec. 5.2.
5

---

5 Experiments
5.1 Simulation Experiments with Synthetic Commands
An operator guiding a robot through a manipulation task may have a particular outcome in mind: for example, a stack with a particular order of blocks, Language can naturally express such constraints; i.e. "stack the red block on the yellow block." Following the template generation and data annotation in Section 1.1, we train baseline and Transformer VGP models to produce output masks Arrrep and pipluce and combine them with a SPOT-Q model to manipulate blocks in a simulated environment. These masks filtered and thresholded using 0-1 block masks, following Hundt et al. [9]. To evaluate the VGP system in isolation, we report the following metrics:
Block Accuracy is the percentage predictions for which the grasp mask overlaps with the correct block, i.e the model chooses the correct block to pick up. Higher is better.
Idealized Error Score (IES) captures the performance of an idealized robot on the manipulation task, choosing block to move using the grasp mask, and location using the place mask. After the chosen block is teleported to the predicted location, we measure the Euclidean distance between the block's 's new location and its reference location in block lengths. Lower is better.
Our baseline is a UNet-based VGP model; specifically, we extend the LINGUNET model presented by Misra et al. [4] with attentional fusion mechanism. While the original LINGUNET used the final state of an LSTM for the language representation, we use a weighted average of BiLSTM states produced by an attention mechanism. We also ran end-to-end Q-Learning experiment that backpropagates through both the FCN and Transformer, but the model did not converge.
Further details on the masking, as well as the simu- lation environment, can be found in Appendix B.I. The fiterre/thresholled masks Mrsssp and Mplace stacked into single mask M E Raxxx2 A final matrix of Q' of Q values constrained by the language is given by Q' = .M @ Q, where @ is the Hadamard product; Q' is then used to choose an action and a location at which to grasp/place in the simulation.
Table 1: Visual goal prediction (VGP) met- rics for models trained on one or both tasks. Higher accuracy and lower Idealized Error Score (IES, measured blocks), is better. TFMR is Transformer.
Model Train
Test
Block Acc.
IES
UNet
Rows
Rows
57.1%
2.4
TFMR Rows
Rows
84.8%
0.9
UNet Both
Rows
66.5%
1.2
We count stacks and rows as successful only with
TFMR Both
Rows
91.3%
0.6
the specified color order, also tallying the number of actions this required and the percentage of times the gripper picked up the correct block. We contrast our model with the SPOT-Q framework alone, which has no access to the order constraints, and with our UNet baseline. Each model was evaluated with the same SPOT-Q model and tested on 100 trials. Hundt et al. [9] set a 30 action limit for failed trials; we added several evaluation heuristics to terminate failed trials when an irrecoverable state was reached, measuring color ordering errors. Stacking tasks use 8 colors and low-making tasks use 4.
UNet Stacks
Stacks
27.9%
1.0
TFMR Stacks Stacks
81.5%
0.4
UNet
Both
Stacks
75.2%
0.6
TFMR
Both
Stacks
90.4%
0.4
Table 2: Simulation Metrics. Higher task and color grasping success percentages are better. A lower average number of actions per trial is better.
Table shows the visual goal prediction met- rics for all model variants trained on one or both tasks. We see that the Transformer-based mod- els (TFMR) uniformly out-perform the UNet baseline across all metrics, often by large mar- gins. For both UNet and Transformer archi- tectures. training on two tasks simultaneously covers a broader range of semantically differ- ent examples with a corresponding increase in performance on all metrics. Notably, these ex- amples differ by task: firstly, row- www-making uses a subset of the colors available in stacking. Secondly, while the grasp data across both tasks is roughly identical, the data for place masks is quite different. For example, both "stack the blue block on the red block" and "move the blue block next to the red block" result in grasp masks over the blue block, but whereas the place mask for stacking picks out the block on which to stack (the red block), the row mask specifies a location near the target block. Thus, it is noteworthy that combining tasks benefits not only the block accuracy, but also the IES.
Task
Model
Task
Color %
Actions
Rows
RL only
10%
31.2%
15.3
Rows
UNet
10%
32.5%
19.2
Rows
TFMR
53%
76.7% 7
12.3
Stacks RL only
0%
13.0%
NaN
Stacks UNet 0%
15.2%
NaN
Stacks TFMR 29%
78.6%
11.6

---

Table 2 shows the task completion percentage for the best UNet and Transformer models (trained on both tasks) in the simulated environment, as well as the percentage of successful grasps that picked up the correct color, and the average number of actions taken to complete a successful trial. In the aggregate, we see that high performance on VGP metries does not translate to equívalent performance in the simulation environment. For example, both for stacking and row-making, the percentage of correct color grasps is substantially lower that the VGP equivalent metric, block ac- curacy. For the Transformer model, this may be due to the imprecision of the patch-wise tiling of the image: when heuristically combined with a common-sens, mask, a patch may cover multiple adjacent blocks. This is an inherent limitation of our approach. When compared to the RL-only and UNet baselines, our Transformer- -baseed model succeeds in significantly more trials. Additionally, the mean number of actions across successful trials is lower, indicating that the Transformer -bssed masks result in fewer actions per successful trial. Note that for stacking, both baselines succeeded on 0 out of the 100 trials; due toppling and occlusion, stacking poses significantly harder task than row-making, and it is harder to recover from a misplaced block, even if the trial is not ended. We see that even a fairly high block accuracy and low IES are insufficient for high task performance. This highlights the one of the main challenges of multistep tasks: small error rates compound across multiple actions, yielding low task performance. A full error analysis is given in Appendix B.3 Data Ablation
Transfer to Real Images- IES of row- row-making and stacking models when trained on increasing subsets of real data. We see that for both rows and stacks, performance improves slightly as we in- crease the amount of training data, with diminishing re- turns after 30% of the available data (1794 examples) are used. Given that Transformers typically require several orders of magnitude more data to reach good performance [45, 40, 46), the fact that our architecture can be trained with mere thousands of simulated training examples, and successfully finetuned with even fewer real examples, extracted from only a few hundred real im- ages, represents a significant achievement. For a proof-of-coneee demonstration of our model run- ning on a real robot, see Appendix B.4 and the supplementary materials.
In Figure 3, plot the
1.0
rows
stacks
0.5
0.0
598
1196
1794
2302
2000
3588
4186
4784
5382
5950
Real images
Figure 3: IES as function of the % of real fine-tuning data used. IES is error measured in blocks, lower is better.
By generating held-out data splits as described in Section 4.3, can test the ability of our model to generalize compositionally. i.e. to recombine known symbols in new contexts. A model with compositional color representations will be able to combine colors across contexts; for example, if it has seen red blocks in the context of green blocks, it should generalize to blue block context. We would therefore expect the model perform roughly on par with a model trained on a i.i.d. data split, where all contexts are seen at train time. Conversely, if the model has learned to interpret each utterance atomically, we should expect much lower held-out performance.
Compositionality
Our results are reported in Table 3. Both rows and stacks see slight drop in block accuracy from the baseline. Nevertheless, the lack of a large gap between the base- line and the held-out conditions suggests that the model has largely learned to generalize compositionally to un- seen block combinations. This suggests that, although the model certainly has the capacity to memorize 23 combi- nations, and although it is not trained to perform compo- sitional reasoning, it learns to bind symbols to percepts in a way that generalizes across contexts (i.e. "red block" refers to one particular block, independent of its context). These results are promising but limited to simulated language; accordingly, in Section 5.2 we evaluate our model with real language data.
Table 3: VGP metries on a held-out sub- set of unseen color pairs. Baseline mod- els trained on a randomly-shuffled split. Task Data Split Block Acc. IES
Stacks Random 85.7%
0.4
Rows Random 89.1% %
1.7
Stacks Unseen 84.8 %
0.6
Rows Unseen 84.3%
1.3
5.2 Naturalistic Language Commands with Complex Scenes
In this experiment we model the naturalistic data described in Section 4.4, where state representa- tions are coupled instead with diverse crowdsourced commands, in contrast to the template gener- ated commands in Section 5. We train models on state representations (cf. Section 4.4) as well as birds-eye color and depth images. This highlights the flexibility of our model: we are able to train from images and from state representations with no modifications to the architecture.
Large pre-trained encoders have been shown to encode useful linguistic features at a number of levels of analysis [47, 48, 49]; accordingly, we make use of bert -bate-aae--- base--ased [50] as an input feature, contrasting it with GloVe [51] word embeddings. Tuning and metrics are as in Sec. 5.1.
7

---

Table 4: VGP metries show that best Transformer mod- els out perfoorr UNet. both on state and image inputs.
Table 4 shows the VGP metrics for models trained from image-like representations of the state (described in Section 4.4) as well as from images. For brevity, only the best model for each embedding type is re- ported, with full results in Appendix C.5. On state representations, the Transformer- based models show a clear advantage over the baseline, with IES being substantially lower for both Transformer models. When using GloVe embeddings with state representation input, 2 x 2 patches resulted in the best performance; patch size of 4 yielded the best results for BERT emheddings. Overall, while the BERT--based model has slightly higher block accuracy, the IES are equal between the two models when trained on state representations. While these metries measure per-step performance, these results suggest that our masking model can be applied to real as well as simulated language.
Model
Input
k Block Acc.
IES
UNet GloVe
State
56.5%
3.2
UNet GloVe
Image
63.4%
3.1
TFMR + GloVe
State
2
88.4%
2.1
TFMR + GloVe
Image 2
88.7%
2.8
TFMR- + BERT
State
4
90.5%
2.1
TFMR + BERT
Image
83.6
3.4
In Table 4, we also see that the performance shifts substantially when changing the input from state to images. While the baseline improves slightly in IES, the Transformers' performance decreases dramatically. These results stand in contrast to those in Section 5.1, where a Transformer trained on images unequivocally outperformed the baseline. Nevertheless, the best Transformer model still outperforms the baseline. We speculate that the baseline models may perform better here because the architecture developed for image segmentation problems. Interestingly, while BERT-based Transformer performed well when trained from state, the same models' performance here is greatly diminished. Improving the Transformer's real-world performance is direction for future work.
Data Translation and Filtering In Section 5.1, we observed that high performance in a static eval- uation against a dataset did not entail equally high performance when using a realistic robot for multi-step tasks in simulated environment. In its current form, the blocks dataset can only be eval- uated using static metrics. Furthermore, the images provided in the dataset are taken from a fixed 45 degree projection, and our method assumes access birds-eye images. To obtain birds-eye images of the environment and to facilitate further research into combining rich, naturalistic language with action in manipulation tasks, we translate the dataset into the same environment used in Section 5.1, and filter the dataset to only physically feasible examples; this process is described in Section 4.4.
Table 5 reports the VGP metrics on the the physically feasible subset of the data for the Transformer models. Firstly, we see that performance decreases across most met- rics. However, for UNet, IES improves on the physically feasible subset. In contrast,
Table 5: VGP metrics for models trained on images of the physically feasible subset of the blocks dataset. indicates significant improvement over UNet as per the Wilcoxon ranked-sign test.
Model
Input k
Block Acc.
IES
UNet + GloVe
Image
56.4
3.0
the Transformer models decrease in perfor-
TFMR+ + GloVe
Image 2
84.4"
3.0
mance across both metrics. Nevertheless. the Transformer model still performs on par with the baseline on IES, and substantially better on block accuracy. These results suggest that the physically feasible subset is more challenging to model, even when evaluating the models statically. In other words, even when the evaluation and models are physically unconstrained, modeling only physically feasible scenes may be harder than modeling all scenes. This result highlights the diffi- culty of developing embodied agents and the importance of testing them in realistic environments.
TFMR + BERT
Image 2
78.9*
3.6
6 Conclusion
We have presented a general approach to multi-step rearrangement and manipulation tasks that fac- torizes each step into learning how to act and learning where to act. To decide where to act, we have presented a Transformer-based model for converting state representations and natural language instructions to masks over an action space that learns with remarkably few examples. We have shown the ability of our model to act in a realistic simulated environment for two multi-step block rearrangement tasks, and naturally learn a compositional mapping from text to actions in a percep- tual space. We have demonstrated that our model performs well under both these realistic physical assumptions and with real and complex natural language via dataset of crowdsourced instructions.
8

---

7 Acknowledgements
We thank the anonymous reviewers and the area chair for their input and engagement. We would also like to thank Benjamin Van Durme, Tae Soo Kim, and Jonathan Jones for their helpful feedback. This material is based upon work supported by NSF Award #1763705 and by the Office of Naval Research under grant N00014-19-1-2000 Elias Stengel-Eskin is supported by an NSF GRFP.
References
[1] S. Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3), 1990.
[2] C. Paxton, A. Hundt, F. Jonathan, K. Guerin, and G. Hager. CoSTAR: Instructing collab- orative robots with behavior trees and vision. ICRA, 2017. URL btttts://arxiv..reee abs/ /111 06145.
[3] A. Hundt, Jain, C.-H. Lin, C. Paxton, and D. Hager. The costar block stacking dataset: Learning with workspace constraints. IROS, 2019, 2019.
[4] D. Misra, J. Langford, and Y. Artzi. Mapping instructions and visual observations to actions with reinforcement learning. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages
2017.
[5] V. Blukis, N. Brukhim, A. Bennett, R. A. Knepper, and Y. Artzi. Following high-level navigation instructions on a simulated quadcopter with imitation learning. arXiv preprint arXiv: 1806: 1806.00047 2018.
[6] V. Blukis, Y. Terme, E. Niklasson, R. A. Knepper, and Y. Artzi Learning to map natural language instructions to physical quadcopter control using simulated flight. In Conference on Robot Learning (CoRI.,, 2019.
[7] V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding and mapping for natural language robot instruction following. arXiv preprint arXiv:2011.077 2020.
[8] V. Blukis, Y. Terme, E. Niklasson, A. Knepper, and Y. Artzi Learning map natural language instructions to physical quadcopter control using simulated flight. In CoRL, 2020.
[9] A. Hundt, B. Killeen, N. Greene, H. Wu, H. Kwon, C. Paxton, and G. D. Hager. "Good Robot!": Efficient reinforcement learning for multi-step visual tasks with sim to real transfer. IEEE RA-L, 5(4), 2020.
[10] Y. Bisk, K. J. Shih, Y. Choi, and D. Marcu. Learning interpretable spatial operations in a rich
3d blocks world. In AAAI, 2018.
[11] Y. Bisk, D. Yuret, and D. Marcu. Natural language communication with robots. In NAACL HLT 2016, 2016.
[12] S. Gubbi Venkatesh, A. Biswas, R. Upadrashta, V. Srinivasan, Talukdar, and B. Am- rutur. Spatial reasoning from natural language instructions for robot manipulation. arXiv:2012. 13693, 2020.
[13] J. Andreas, D. Klein, and Levine. Modular multitask reinforcement learning with policy sketches. In JCML'17 17 Proceedings of the 34th International Conference Machine Learning Volume 70, pages 166-175, 2017.
[14] Oh, S. Singh, H. Lee, and Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In ICML'17 Proceedings of the 34th International Conference Machine Learning Volume 70, pages 2661-2670, 2017.
[15] E. A. Brooks, Rajendran, R. L. Lewis, and S. Singh. Reinforcement learning of implicit and explicit control flow in instructions. arXiv preprint arXiv:2102.13199 2021.
[16] Y. Jiang, S. Gu, K. P. Murphy, and C. Finn. Language an abstraction for hierarchical deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 32, pages 9414-9426. 2019.

---

[17] B. Gangopadhyay, H. Soora, and P. Dasgupta. Hierarchical program -riigeere reinforcement learning agents for automated driving. arXiv preprint arXiv:2103.13861, 2021.
[18] S.-H. Sun, T.-L. Wu, and J. J. Lim. Program guided agent. In International Conference on Learning Representations, 2019.
[19] Perera, J. Allen, C. M. Teng, and L. Galescu. Building and learning structures in a situated blocks world through deep language understanding. In SPLU 2018, 2018.
[20] P. Goyal, S. Nickum, and R. Mooney. Using natural language for reward shaping in reinforce- ment learning. In IJCA/ 2019, 2019.
[21] P. Goyal, S. Nickum, and R. J. Mooney. PixI2r: Guiding reinforcement learning using natural language by mapping pixels rewards. arXiv preprint arXiv:2007.15543, 2020.
[22] T.JJ. Fu and W. Y. Wang. Semi-supervised policy initialization for playing games with lan- guage hints. In NAACL 2021. ACL, June 2021.
[23] D. Bahdanau, F. Hill, J. Leike, E. Hughes, A. Hosseini, Kohli, and E. Grefenstette. Learn- ing to understand goal specifications by modelling reward. In International Conference on Learning Representations, 2018.
[24] S. Stepputtis, Campbell, M. Phielipp. Lee, C. Baral, and H. B. Amor. Language- conditioned imitation learning for robot manipulation tasks. Neurips 2020, 2020.
[25] C. Lynch and Sermanet. Language conditioned imitation learning over unstructured data. In RSS 2021: Robotics: Science and Systems, 2021.
[26] A. Zeng, P. Florence. J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. CoRL, 2020.
[27] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipula- tion concepts from instructions and human demonstrations. In Proceedings of RSS, 2020.
[28] T. Nguyen, N. Gopalan, R. Patel, M. Corsaro, E. Pavlick, and S. Tellex. Robot object retrieval with contextual natural language queries. arXiv preprint arXiv:2006. 13253, 2020.
[29] N. Gopalan, E. Rosen, G. Konidaris, and S. Tellex. Simultaneously learning transferable sym- bols and language groundings from perceptual data for instruction following. Robotics: Sci- ence and Systems XVI, 2020.
[30] J. Roh, C. Paxton, A. Pronobis, A. Farhadi, and D. Fox. Conditional driving from natural language instructions. Conference on Robot Learning, pages 540-551, 2019.
[31] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI. Springer, 2015.
[32] M. Q. Mohammed, K. L. Chung, and C. S. Chyi. Review of deep reinforcement learning-based object grasping: Techniques, open challenges, and recommendations. IEEE Access, 8, 2020.
[33] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In IROS. IEEE, 2018.
[34] D. Kalashnikov, A. Irpan, Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. CoRL, 2018.
[35] I. Akinola, J. Varley, and D. Kalashnikov. Learning precise 3d manipulation from multiple uncalibrated cameras. In ICRA, 2020.
[36] S. Song, Zeng. J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. In IEEE RA-L, volume 5, 2020.
10

---

[37] A. Dosovitskiy, L. Beyer. A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, Heigold. S. Gelly, Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[38] R. Ranftl, A. Bochkovskiy, and V. Koltun. Vision transformers for dense prediction. arXiv preprint arXiv:2103.13413, 2021.
[39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polo- sukhin. Attention is all you need. In Neurips, 2017.
[40] Q. Nguyen and J. Salazar. Transformers without tears: Improving the normalization of self-attention. arXiv preprint
2019.
[41] B. Lake and M. Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequeen recurrent networks. In ICMI.. PMLR, 2018.
[42] B. M. Lake, Linzen, and M. Baroni. Human few-shot learning of compositional instructions. arXiv preprint 191.::9911.0587, 2019.
[43] L. Ruis, J. Andreas, M. Baroni, D. Bouchacourt, and B. M. Lake. A benchmark for systematic generalization in grounded language understanding. arXiv preprint arXiv:2003.5166, 2020.
[44] M. L. Gray and S. Suri. Ghost work: How to stop Silicon Valley from building a new global
underclass. Eamon Dolan Books, 2019.
[45] M. Popel and O. Bojar. Training tips for the transformer model. arXiv: 00247, 2018.
arXiv preprint
[46] A. Araabi and C. Monz. Optimizing transformer for low-resource neural machine translation. arXiv preprint urXiv:2011.02266 2020.
[47] J. Hewitt and P. Liang. Designing and interpreting probes with control tasks. In EMNLP- IJCNLP, 2019.
[48] I. Tenney, D. Das, and E. Pavlick. Bert rediscovers the classical nlp pipeline. ACL, 2019.
[49] G. Jawahar, B. Sagot, and D. Seddah. What does bert learn about the structure of language? In ACL, 2019.
[50] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL HLT. ACL, 2019.
[51] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation.
In EMNLP. 2014.
[52] E. Rohmer, S. P. N. Singh, and M. Freese. Coppeliasim (formerly v-rep): versatile and scalable robot simulation framework. In IROS 2013, 2013, www.coppeliaroooties.oo com.
[53] P. Khullar, K. Majmundar, and M. Shrivastava. NoEl: An annotated corpus for noun ellipsis in English. LREC 2020, 2020.

---

A Model
A.1 Transformer
Figure 4 shows a schematic of our Transformer-based VGP model, run on real robot data. The inputs are bird's eye color image (and depth image) and language command. The image is tiled into an array of non-overlapping square patches, which are flattened into a one-dimensional list by concatenating each row of tiles. The tokens of the language command are then concatenated to the end of that list. This concatenated input is passed through a shared set of Transformer encoder layers, which encodes the dependencies between words and the image. The output of the shared Transformer is then passed to two identical Transformer encoders with separate parameters, one for predicting grasp mask, and one for predicting place mask. This bifurcation is based the observation that there are commonalities between producing the two types of masks (e.g. localizing colors, delineating blocks) but that the final tasks of producing grasp and place masks distinct.
A.2 Hyperparameters
The number shared and combined layers is treated as a hyperparameter n E [0, 2, 4, 6]. as well as the number of warmup steps E [100, 400, 1000, 4000). the dropout probability E [0.25, 0.33, 0.40), and the loss weight ratio 7 E [0.01, 0.1, 0.2]. Each model has a hidden dimension of 256 and ferd-forward dimension of 512. Due to the larger number of hyperparameters, sample 70 random parameter configurations.
...SEP] stack the blue block the red block
Shared Transformer
Grasp Transformer
Place Transformer
stack the blue block on the redi
block...
Figure 4: Our Transformer- baasedd based Visual Goal Prediction model, as run on real robot data. It takes an image and an instruction as input, then produces separate masks for grasping and placing.
A.3 Q-Learning
Our model relies on a learned function from states and actions to expected rewards, Q(5t, a). This function is learned in the context of Markov Decision Process (S, A, P. R, composed of of states S, a set of actions A, a transition function P : S x S x A - IR, a reward function R: A + R, and discount factor 7, where 7 < At time chooses action a, = (8t). where æ is a policy. The function Q. approximating the expected reward for each state-action pair, can be used to extract a deterministic policy that maximizes the expected reward: T5t) = argmax (st,, a), Va E A. We learn Q to maximize the reward R over time by minimizing (2 a.) - yel. where Yt R(st+1, a,) + 7 ((++1))). definition of R has major impact on the learning efficiency of the agent. Specifically, relying single reward the end of task yields very sparse reward signal and results inefficient learning- In the multi-step tasks considered here (stacking and row-making) there are intermediate subtasks that naturally lend themselves to reward shaping, i.e. providing smaller intermediate rewards to the agent when the chosen action reflects an incremental progression towards the goal state. The SPOT-Q algorithm, introduced by Hundt al. [9], uses intermediate rewards to shape the learning of Q(s,, a) for row-making and stacking. We use SPOT-Q to Iearn the function Q a). Given
agent observes state S, and
multi-step task, the
12

---

pixel-wise n x n state space of an image and two actions (grasp and place), this function can be expressed as a tensor E R=xxx
B Experiment
B.1 Combination Heuristics
In the case of stacking, following Hundt et al. [9], we intersect each mask with common-sena mask that assigns value of 0 where there are no blocks (i.e. an empty space cannot be the location of grasp place). For row-making, we allow placing in empty locations, but intersect the grasp mask with a common-sens, mask. For place and grasp actions in stacking, and grasp actions in row-making, we then find the maximum location s" in mask M: s* argmax M. Because the reconstruction function copies a patch value across all of its pixels, for a patch size of p there will be p² pixels with the value of s*. We intersect this patch with the common-sense mask to find the single block with the highest value in M. For place actions in row- mow-making, we simply threshold M to obtain set of valid patches to intersect with the Q values. Both processes produce binary masks.
B.2 Simulation
We use the CopelliaSim simulator [52] for experiments involving a simulated robot.
Images All real images were collected via a UR- 5 robot, Robotiq 2f-85 2-finger gripper, and Prime- sense Carmine RGB-D camera. All of these con- ditions are different from the simulator, except the URS. The images were collected under varied lighting conditions. The data is saved in the same file format as the experiments in Tables I and 2. with birds-eye view, 224 x 224 RBG and depth- map images, etc.
EDU
Early Termination
The step-by-step nature of
the instructions means that the agent is sometimes unable reverse course: for example, if the first instruction were "stack the green block on the blue block" but the agent mistakenly grasped the red block and placed it on the green block, the green block would become unreachable. Since neither the Q-value model nor the language understand- ing model were trained on unstacking tasks, there would be way to reverse course here without adding an external observer. Thus, in these cases, we terminate the trial. Similarly, if the stack is toppled, we terminate the trial, as without an external observer cannot determine which step of the process we have regressed to, as the partial toppling of stacks not unusual. Because blocks are typically not occluded in row-making, we do not need these heuristics, and the only way for trial fail is by timing out after 30 actions.
Figure 5: Simulation Environment
Simulator
For experiments involving simulated robot, the CopelliaSim simulator [52] used. The simulated agent collects observations via fixed RGB-D camera, whose images are project birds-eye view. The agent operates over discretized spatial and angular action space, and movement to a particular location and angle is performed by an inverse kinematics solver built into the simulator.
B.3 Error analysis
As mentioned in $5.1, stacking tasks may terminate early for a number of reasons, including inre- versible actions that make successful stack impossible. We conduct an error analysis determine what percentage of the failures are due to errors in the perceptioon/eeasoning component (i.e. the masking) and what percentage are due to physical failures of the grasping arm. We find that 61. .97% of the failures could be attributed to timing out after 30 actions. Qualitatively, we found that this
13

---

often happens when block tumbles outside of the work area, rendering it impossible to grasp. An- other 7. 7.04% can be attributed toppled stacks, where a stack height of 2 or 3 was reached before place action destroyed the whole stack, ending the trial. Finally, 30.99% were due to incorrect block orderings which occluded crucial blocks; this due to the 21 .1.4 of grasp actions that picked up the incorrect block.
B.4 Proof-of-Concept
Due to restrictions induced by the ongoing COVID-19 crisis, were unable to sufficient tri- als to include quantitative real results. Nevertheless, we include a proof-of-concee demonstration video in which
the Transformer-based masking model with real robot arm successfully complete a real stack in the bottom to top order green, yellow, blue, red.
C Experiment
C.1 Data Translation and Preprocessing
Note that in the original dataset, 4.82% of the examples have multiple blocks moving in a single frame. As this breaks the assumptions made in $ 1, we ignore these examples both while training and evaluating.
For training, we convert the 64 x 64 state image grids into binary masks, where all elements are zero except the pixels corresponding to the block which is moved.
C.2 Data Filtering
We filter out physically infeasible examples, which are typically due to unstable structures toppling or blocks being placed in overlapping areas. Because the images captured from the simulated envi- ronment are a lower resolution than the originals, many of these logos are difficult to read in the simulated images; to aid with block discrimination, we assign a color to each block in addition to its logo. Table reports the same statistics given in Bisk et al. [10] on the filtered subset of the data.
Dataset
Configs
Types
Tokens
Utterances
Mean Length
Bisk et al. [10]
100
1,820
233,544
12,975
18.0
Physically Feasible
100
1,434
133,083
7,398
18.0
Table 6: Dataset Statistics for feasible subset of the Bisk et al. [10] data
C.3 Dataset Ambiguity
A remaining limitation of the dataset is that some descriptions are ambiguous and not reliably ac- tionable, since annotators did not attempt to execute the described actions. Some of the natural language descriptions are ambiguous, as depicted in Fig. 7.
C.4 Qualitative Error Analysis
Based our results in Tables 4, we qualitatively examine some of the errors made by the Transformer-basse based model, where we observe several patterns. Looking at the validation examples with the highest IES, we observe that they often correspond to instances where the source block was mis-identified, leading the wrong block to be moved, and yielding a large IES. More interestingly, in the cases where the correct block was moved, the instruction often contains higher-order concepts as well as linguistic complexities such as ellipsis. For example, in Figure there a reference to a "disconnected square", which is rare, higher-order geometric concept. In addition, the subsequent clause lacks an explicit reference back to the square; the annotator chose to leave this reference salient position by the previous clause. This type of
as implicit, given that the square is raised noun- noun-phrass phrase ellipsis is common in natural language [53], and reflects the type of advanced pragmatic reasoning required handle natural language.
Other examples of a higher-order concept instruction that the model performs poorly on are: "Take the Mercedes Benz block on the Burger King block without hiding the right top edge" (hiding),
14

---

"There is a disconnected "square" at the bottom. Place Shell on top of the lower left corner."
Figure 6: A particularly challenging example for the model. Not only does this involve higher- order concept (disconnected square) but it also incorporates noun- noun-phrase ellipsis.
"Add Shell as the second block in the four-block row."
Figure 7: Ambiguous command makes reference future steps: row of 4 is later constructed, but at the time of construction, the agent has no access to this information.
"targt [sic] goes 1/2 under Adidas with the right side hanging off" (hanging), and "take the Stella Artois block and place it on top of the Nvidia block, lined up perfectly" (perfectly). The model also struggles with long coreference chains, even when the anaphora that is to the right of the Starbucks block and make it the highest block on the board by placing on the Mercedes block. It should be in line with the bottom block."
explicit, e.g. "Place the block
15

---

C.5
Full results for Experiment 2
Table 7: Full VGP results for the blocks dataset
Model
Embedding
Recon. Loss
Input
k
Block Acc.
IES
UNet
GloVe
Yes
State
55.3
3.3
UNet
GloVe
No
State
56.5
3.2
UNet
GloVe
Yes
Image
53.6
3.1
UNet
GloVe
No
Image
63.4
3.1
Transformer
GloVe
Yes
State
4
90.7
2.2
Transformer
GloVe
No
State
4
92.8
2.3
Transformer
GloVe
Yes
State
2
90.8
2.3
Transformer
GloVe
No
State
2
88.4
2.1
Transformer
BERT
Yes
State
4
88.9
2.6
Transformer
BERT
No
State
4
90.5
2.1
Transformer
BERT
Yes
State
2
89.0
2.4
Transformer
BERT
No
State
2
78.7
2.8
Transformer
GloVe
Yes
Image
4
88.9
3.4
Transformer
GloVe
No
Image
4
85.5
3.6
Transformer
GloVe
Yes
Image
2
79.3
3.2
Transformer
GloVe
No
Image
2
88.7
2.8
Transformer
BERT
Yes
Image
4
89.5
3.5
Transformer
BERT
Image
4
72.1
3.8
Transformer
BERT
Yes
Image 2
90.1
3,3
Transformer
BERT
No
Image 2
83.7
3.1
Table 8: Full VGP results for the blocks dataset on the physically feasible subset.
Model
Embedding
Recon. Loss
Input
k
Block Acc.
IES
UNet
GloVe
Yes
State
47.8
3.1
UNet
GloVe
No
State
49.1
3.1
UNet
GloVe
Yes
Image
45.5
3.0
UNet
GloVe
No
Image
56.4
3.0
Transformer
GloVe
Yes
State
4
88.2
2.4
Transformer
GloVe
No
State
4
90.3
2.4
Transformer
GloVe
Yes
State
2
89.1
2.6
Transformer
Glo Ve
No
State
2
85.1
2.4
Transformer
BERT
Yes
State
4
86.8
2.7
Transformer
BERT
No
State
4
88.3
2.2
Transformer
BERT
Yes
State
2
88.4
2.6
Transformer
BERT
No
State
2
72.0
3.0
Transformer
GloVe
Yes
Image
4
86.3
3.4
Transformer
GloVe
No
Image
4
83.0
3.7
Transformer
GloVe
Yes
Image
2
75.7
3.2
Transformer
GloVe
No
Image
2
84.4
3.0
Transformer
BERT
Yes
Image
4
88.3
3.7
Transformer
BERT
No
Image
4
66.2
3.7
Transformer
BERT
Yes
Image
2
88.5
3,4
Transformer
BERT
No
Image
2
86.3
3.5
16


Reviews:
[{"review_id": "tHlq7yYI57D", "reviewer": "Reviewer_h8PP", "summary": "This work leverages recent vision models to perform natural language guided block stacking.  They use synthetic language and a real crowdsourced resource (filtered for physical viability).  Limited experiments (due to covid) are performed on a physical manipulator.  The task requires understanding spatial language expressions for both identifying the referent and target location.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Novel model architecture for the task with good comparison baselines\n2. Analysis of simple stacking and rows in both synthetic and real images.\n3. Curation of the Bisk et al. dataset for robotic tasks and evaluation on this real language setting\n4. Manipulation performed with simulated UR-5\n5. Basic manipulation demonstration of simplest setting on physical manipulator\n\n**Weaknesses**\n1. Insufficient analysis of the difficulties of real language (e.g. behavior in abstract language settings)\n2. Limited physical manipulator experiments which also means limited analysis of the causes for the model's transfer errors to the real setting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors present and demonstrate the utility of novel models for language guided control/actions based on recent advances.  They do basic comparisons against language and task complexity and include experiments both on simulated and real control platforms. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "The primary limitation of the current work is limited error analysis which makes it difficult to diagnose what the next steps are for the research community and ideally more physical experiments (though this is not a limitation that can be expected to be addressed in the current climate.)"}}, {"review_id": "l5s80ryn6o", "reviewer": "Reviewer_1smN", "summary": "The main contributions of this paper could be concluded as following:\n(1) Proposed a transformer-based model to build mapping from natural language instructions to real scenes with pixel-wise Q values and masks which are compatible with both supervised learning and reinforcement learning methods.\n(2) Apply the proposed transformer-based model on multi-step rearrangement tasks and compare it with a baseline model:Unet-based VGP model in metrics of block acc and IES\n(3) Give out a dataset based on Bisk dataset mentioned in paper with both real and simulated samples to for training model for multi-step rearrangement task in more realistic scenes\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tInnovatively proposed the method to mapping input image with natural language instructions with pixel-wise Q-value and masks which is feasible in practice and useful\n2.\tThe proposed model was validated that it can converge fast with much fewer samples than traditional transformer model in training phase.\n3.\tDesign a complete experiment setting to compare propose model with baseline in aspects of training efficiency, genralization toward different taks types et al.\nWeakness:\n1.\tThe mapping lack explainability, a theoritical explaination or citiation should be presented to clear the reasonability of propose mapping method\n2.\tData ablation experiments should be more comprehensive. Fisrt, generalizability should be taken into consideration, a different dataset could be used to validate the propose model can still improve fast or not. Second, robustness should be validated. For example, add some noisy into train set.\n3.\tThe released dataset need to validated in several aspects such as data type balance,reasonablity of insturctions and size\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend this paper for its contributions on proposing  a transformer-based model to solve multi-step rearrangement task in realisitc simulated environment as it provides an innoative and feasible method to map natrual language instructions to input scene images and the experiments shows that it could be more effective and efficient in training than most transformer-based models.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "1.\tExplainability of propose pixel-wise mapping method\n2.\tA more comprehensive validation of effiency of propose transformed method\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "P4_J5DGYLBp", "reviewer": "Reviewer_gQ8o", "summary": "This paper proposes using natural language instructions for multi-step rearrangement tasks, by mapping images and instructions to masks over pick or place locations. The model is factorized into 2 components — a pretrained SPOT-Q model independent of language, and a language-conditioned model proposed by the authors. The predictions from these two components are combined to obtain the final action.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Experiments are conducted both with synthetic and real images, as well as templated and crowd-sourced language. The results show that the proposed method outperforms previous methods on both visual goal prediction and on task completion metrics.\n- The factorized model is interesting, and allows reusing pretrained models, thereby reducing data requirements.\n\nWeaknesses:\n- The authors discuss the advantages of using a factorized model, but do not discuss the limitations. For example, language might have more information than \"where to act\", like \"put the red block *gently* on the blue block\".\n- Some analysis of how the performance changes as the no. of steps in the task is increased could be informative.\n- It seems like the natural language experiments (based on the dataset from Bisk et al) involve moving only a single block. If this is true, it doesn't represent the \"multi-step\" rearrangement setting being addressed in the paper, and needs to be made explicit.\n- The experiments on compositionality aren't particularly informative — it seems pretty intuitive that the model will learn concepts like \"red block\" and \"blue block\" independently. A more interesting question is — can the model learn to compose different object attributes? E.g. if it is only shown \"small red blocks\" and \"large blue blocks\" during training, can it generalize to \"large red blocks\"?\n- The experiments with the physically feasible subset are not convincing — (1) are the differences statistically significant?, (2) could the difference be because the physically feasible dataset is smaller in size?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed approach is sound, and the experimental results show that it improves over existing methods, but some additional experiments could make the paper better.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "- More convincing analysis of physically feasible subset.\n- Clarify whether the natural language setting is single-step or multi-step.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "OHgAZJ9B0Vy", "reviewer": "Reviewer_kWqo", "summary": "\nThis paper addresses the problem of learning to perform block rearrangement tasks described by natural language instructions. To this end, the paper proposes a framework that decouples the problem into two sub-problems: where to act and how to act. Specifically, it proposes a Transformer-based model to map images and language instructions to masks over grasp and or place locations. Then, a pre-trained policy performs actions according to its Q function that provides pixel-wise Q values and predicted masks. The experiments show the proposed framework can reliably solve the task. Ablation studies verify design choices such as language models (GloVe and BERT), the encoder architecture (UNet and Transformer), etc. I believe this work studies a promising research direction (i.e. language instruction guided agents) and proposes a reasonable framework to address the problem. Yet, I am concerned with the narrow scope of the problem and the limited novelty and technical contributions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n## Paper strengths and contributions\n\n**Motivation**\n- Leveraging natural language descriptions to instruct robots is easy and natural especially for non-expert users and flexible.\n- Employing natural language instructions for learning a multi-task policy allows leveraging semantic similarity across tasks, which potentially allows generalizing to unseen tasks.\n- Decoupling learning where to act and how to act could potentially be more sample efficient compared to learning an RL policy from scratch purely from reward signals. Also, such as framework is more interpretable since we can tell if it fails to predict correct locations to apply actions or it just cannot precisely perform actions.\n\n**Technical contributions**\n- Predicting segmentation maps indicating where to grasp or place seems effective\n- Employing a Transformer to process natural language instruction yields better performance compared to UNet.\n\n**Ablation study**\nAblation studies justify some of the design choices:\n- UNet vs. Transformer: the results show that employing a Transformer model yields better performance compared to an UNet baseline.\n- GloVe vs. BERT: the BERT-based Transformer has slightly higher block accuracy and performs well when trained from states.\n\n**Experimental results**\nThe experimental results show that: \n- The proposed Transformer-based visual goal prediction (VGP) model can predict segmentation maps for grasping and placing reasonably well in terms of block accuracy and idealized error score (IES).\n- The proposed framework achieves a higher task completion percentage with a lower average number of actions per trial.\n- The proposed framework can learn from thousands of simulated training images and successfully be fine-tuned with around 2k real images to achieve reasonable performance (with an IES around 0.7).\n- The proposed framework can deal with unseen color pairs.\n\n## Paper weaknesses and questions\n\n**Novelty**\nThe novelty, from both the views of theoretical analysis aspect and empirical contribution perspective, is not clear or not explicitly mentioned.  Therefore, I do not find enough novelty from any aspects while the efforts put in for setting up and evaluating the proposed system are highly appreciated. To me, this seems more like a system paper than a learning paper.\n\n**Two-stage pipeline**\nThis paper proposes to decouple the block rearrangement problem into where to act and how to act. While I do recognize the advantage of this decoupling scheme, there are still some issues:\n- Error propagation: errors produced in each stage can accumulate and lead to undesired behaviors.\n- The intermediate representations: intermediate representations (such as segmentation maps in this case) need to be chosen beforehand and the chosen one may not be the best for the entire system. The justification suggesting that a segmentation map is a good representation for this task is missing from this paper. Some other choices, such as object detection boxes, could have been evaluated.\n\n**Limited problem scope**\nWhile the proposed framework and ideas are applicable to many problem families, the only task considered in this paper is block rearrangement. I encourage the authors to apply the proposed framework to other domains to increase the significance of this work.\n\n**Related work**\nIt would make the related work section more comprehensive by including some prior works that explore using formal languages (i.e. programs) as a task representation, including\n- Modular multitask reinforcement learning with policy sketches\n- Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning\n- Programmable agents\n- Program Synthesis Guided Reinforcement Learning\n- Program Guided Agent \n- Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving\n- Reinforcement Learning of Implicit and Explicit Control Flow in Instructions\n\n**RL only baseline**\nIt is not fair to compare the proposed framework against the RL-only baseline this way since the RL-only baseline does not leverage the segmentation map supervision and does not learn from other tasks like \"SPOT-Q\". Some fairer comparisons could be: (1) learning the RL policy with an auxiliary task of predicting segmentation maps, (2) learning a multi-task RL policy to solve the block rearrangement tasks with multiple configurations.\n\n**Unseen color pairs**\nWhile the ability to generalize to unseen color pairs (e.g. stack the red block on the blue block) is evaluated in this paper, the evaluation setup is very limited. It limits the number of unseen color pairs to only 1 but all other 55 combinations are seen. I would say holding out at least 10-20% makes more sense.\n\n**Grid downscaleing factor**\nInstead of predicting segmentation maps with the same size as the input images, this work proposes to downscale it. However, the downscaling factor would depend on the size of the object of interest and therefore it is not flexible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "\nI believe this work studies a promising research direction (i.e. language instruction guided agents) and proposes a reasonable framework to address the problem. Yet, I am concerned with the narrow scope of the problem and the limited novelty and technical contributions.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "\nDescribed in the strengths and weaknesses section.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the author response and the discussion phase, the reviewers agree that there is room for improvement, but appreciate both the key idea of the paper of using pre-trained models to efficiently learn language-conditioned tasks and the effort of demonstrating results going from real language all the way to low-level robot control. The results on a real robot are also appreciated and make it particularly well-suited for CoRL. I recommend accept.

--------------------------------------

This paper seems to be somewhat borderline. It presents an interesting method to a relevant problem, includes some ablations to understand the importance of different design choices, and includes experiments with natural language along with some experiments on a real manipulator.

The paper also has several weaknesses, including the following:
* The experiments are limited to block re-arrangement tasks, rather than a more general class of rearrangement tasks. It does not seem like the method is limited to such simple block-based problems, but the experiments do not validate its generality.
* There are a number of experiments that would strengthen the empirical analysis, including a stronger RL baseline, more analysis on the failure cases with natural language, and more experiments on a real robot. (see comments from reviewers kWqo and h8PP). Including this more detailed analysis and comparisons would make it possible for the reader to identify/understand important future steps in this research area.
* The related work section does not discuss the relation of this work to a number of relevant works. Beyond the works mentioned by reviewer kWqo, there are additional prior works that study rearrangement & manipulation tasks from language instructions [1,2,3].

The final result of the paper will depend on the author response and reviewer discussion. If some of the weaknesses can be addressed, then it will most likely be above the bar.

* [1] Learning to Understand Goal Specifications by Modeling Reward. ICLR 2019.
* [2] Language as an Abstraction for Hierarchical Deep Reinforcement Learning. NeurIPS 2019.
* [3] Language Conditioned Imitation Learning over Unstructured Data. arXiv 2020 / RSS 2021
Decision: Accept (Poster)