CLIPort: What and Where Pathways for Robotic Manipulation
Authors: Mohit Shridhar
Lucas Manuelli
Dieter Fox
Venue: CoRL2021 Poster (2021)
Keywords: Manipulation
Pre-trained Models
Vision Language Grounding
CLIP

Abstract: How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.

CLIPORT: What and Where Pathways for Robotic Manipulation
Mohit Shridhar 1.f Lucas Manuelli 2 Dieter Fox 1.2
'University of Washington 2NVDDI mahrêca. washington. 1mamuell@@nvidia.co foxêcs washaaswwwinngttnnc...
iioorr.github... io
Abstract: How can we imbue robots with the ability to manipulate objects pre- cisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize new goals quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vi- sion and language by training large-scale internet data, however these repre- sentations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based ma- nipulation. Specifically, we present CLIPORT, a language-conditionel imitation- learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specifee tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments simulated and real-world settings show that our approach data efficient in few-shot settings and generalizes effectively to seen and unseen se- mantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better comparable to single-task policies.
Introduction
Ask a person to "get a scoop of coffee beans" or "fold the cloth in half" and they can naturally take concepts like scoop fold and ground them in concrete physical actions within accuracy of few centimeters. We humans do this intuitively, without explicit geometric or kinematic models of coffee beans or cloths. Moreover, we can generalize minimal set of examples on what needs to be achieved. How can we imbue robots with this ability efficiently ground abstract semantic concepts in precise spatial reasoning?
broad range of tasks and concepts from
Recently, a number end-to-end frameworks have been proposed for vision-based manipulation 3,, 5J.. While these methods do not any explicit representations of object poses, instance seg- mentations, or symbolic states, they can only replicate demonstrations with a narrow range of vari- ability and have notion of the semantics underlying the tasks. Switching from packing red pens to blue pens involves collecting a new training set [2], or if using goal-conditioned policies, involves the user providing goal-image from the scene [5, 6]. In realistic human-robot interaction settings, collecting additional demonstrations or providing goal-images often infeasible and unscalable. A natural solution to both these problems is to condition policies with natural language. Language pro- vides an intuitive interface for specifying goals and also for implicitly transferring concepts across tasks. While language-grounding for manipulation has been explored in the past [7, 9, 10], these pipelines are limited by object-centric cettrricceppeeee representations that cannot handle granular or deformable ob- jects and often do not reason about perception and action in an integrated manner. parallel, there has been great progress in learning models for visual representations [11, 12] and aligning represen- tations of vision and language [13. 14, 15] by training on large-scale internet data. However, these models lack fine-grained understanding on how to manipulate objects, i.e. physical affordances.
i Work done partly while the author was a part-time intern at NVIDIA.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

[A]
"wign right corter"
the heragon
gray latter shape
bluee blocks green
and blocks
30
[8]
01
"pack the yoshi youhi figure in the brown box"
"pack all the blus and black sneaker objects in the brown box"
"put the brown block on the blue and red blocky"
"push the pile of purple blocks into the green square"
"move the ring the middle of the stand"
"pick all the cherrie and put them in the box"
pack the scissors in the brown box"
'move the rook one block forward"
"fold the cloth half
"sweop the beans blue zone"
Figure J. Langunge-
Language-Conditione Conditioned Manipulation Tasks: CLIPORT is broad framework applicable to a wide range langange conditioned 10 simulated tasks with 1000s of unique
manipulation tasks in tabletop settings. We conduct large-scale experiments Ravens instunces per tusk. See Appendix A for challenges pertaining to each task. CLIPORT can even learn one multi task model for all 10 tasks that achieves
compurable performance single-tate one multi-task model for 9 real-world tasks (k-o; only 5 shown) trained with just 179 imago-action pairs.
Similarly. demonstrate our approach
Panda manipulator
To this end, we propose the first framework that combines best both worlds: end-to-end learning for fine-grained manipulation with the multi-goal and multi-task generalization capabilities of lision-language grounding systems. We introduce a two-stream architecture for manipulation with semantic and spatial pathways broadly inspired by (or vaguely analogous to) the two-stream hypothesis cognitive psychology [16, 17, 18J. Specifically, we present CLIPORT, a language- conditioned imitation-learning agent that integrates the semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Transporter has been applied to wide range of rearragement tasks from industrial packing [2] manipulating deformable objects [6]. The key insight of the approach formulating tabletop manipulation as series of pick-and-phe
affordance predictions, where the objective is to detect actions rather than detect objects and then learn policy. This action-centric approach to perception [19] is data efficient and effective at circumventing the need for explicit "objectness" in learnt representations. However, Transporter is tabula rasa system that learns all visual representations from scratch and so every new goal or task requires collecting new set of demonstrations. To address this problem, we bake in strong semantic prior while learning policies. We condition our semantic stream with visual and language-goad features from a pre-trained CLIP model [1]. Since CLIP is pre-trained to align image and language features from millions of cange-caption pairs from the internet, it provides a powerful prior for grounding semantic concepts that are common across tasks like categories, parts, shapes, colors, texts, and other visual attributes, all without top-down pipeline that requires bounding boxes or instance segmentations [13, 14, 15, 20J. This allows us to formulate tabletop rearrangement as series of Iinguage-conditions affordance predictions, predominantly vision-based inference problem, and thus benefit from the strengths of data-driven paradigms like scale and generalization.
To study these benefits, we conduct large-scale experiments in the Ravens [2] framework with simulated suction-gripper robot. We propose 10 language-condittoned tasks with 1000s of unique instances per task that require both semantic and spatial reasoning (see Figure a-j). CLIPORT is only effective solving these tasks, but surprisingly, can even learn a multi-task model for all 10 tasks that achieves better or comparable performance to single-task models. Further, our evaluations indicate that our multi-task model can effectively transfer attributes like "pink block" across tasks, having never seen pink blocks the word 'pink" in the context of the evaluation task. We also demonstrate our approach on a Franka Panda manipulator with one multi-task model for real-world tasks trained with just 179 image-action pairs (see Figure I k-o).

---

In summary, our contributions are as follows:
An extended benchmark of language-grounding tasks for manipulation in Ravens [2].
Two-stream architecture for using internet pre-trained vision-language models for condi- tioning precise manipulation policies with language goals.
Empirical results on a broad range of manipulation tasks, including multi-task models, validated with real-robot experiments.
The benchmark, code, and pre-trained models are available at: diiport.gtthui ii.
2 Related Work
Vision-based Manipulation. Traditionally, perception for manipulation has centered around object detectors, segmentors, and pose estimators [21, 22, 23, 24, 25, 26]. These methods cannot handle deformable objects, granular media, or generalize unseen objects without object-specific training data. Alternatively, dense descriptors [27, 28, 29] and keypoint representations [30, 31, 32] forgo segmentation and pose representations, but do not reason about sequential actions and struggle to represent scenes with variable numbers of objects. On the other hand, end-to-end perception-to- action models can learn precise sequential policies [2, 4, 6, 33, 34, 35], but these methods have lim- ited understanding of semantic concepts and rely on goal-images to condition policies. In contrast, Yen-Chen al [36] showed that pre-training on semantic tasks like classification and segmentation helps in improving efficiency and generalization of grasping predictions.
Semantic Models. With the advent of large-scale models [37, 38, 39J. number of methods for learning joint vision and language representations have been proposed [13, 14, 15, 20, 40]. How- ever, these methods are restricted to bounding boxes instance segmentations, which make them inapplicable for detecting things like piles of coffee beans or squares on chessboard. Alternatively, works in contrastive learning forgo tow-down object-deeection and learn continuous representations by pre-training on unlabeled data [1 12]. Recently, CLIP [1] applied a similar approach to align vision and language representations by training on millions of image-caption pairs from the internet.
Language Grounding for Robotics. Several works have proposed systems for instructing robots with natural language [7, 8, 9, 10, 41, 42, 43, 44, 45, 46, 47]. However, these methods use dis- entangled pipelines for perception and action with the language primarily being used to guide the perception. As such, these pipelines lack the spatial precision necessary for tasks like folding cloths. Recently, Lynch et. al [48] proposed an end-to-end system for grounding language in continuous control, but it requires several hours of human telcoperation data for single simulated desk setting.
eetion-recognitiin networks [49, 50, 51] and audio- recognition systems [52, 53]. In robotics, Zeng et. al [54] and Jang et. al [55] have proposed two- stream pipelines for affordance predictions of novel objects. The former requires goal-images and the latter is restricted to one-step grasps with single-category goals. In contrast, our framework provides a rich and intuitive interface with composable language commands for sequential tasks.
Two-Stream Architectures are prevalent
3 CLIPORT
CLIPORT an imitation-learning agent based on four key principles: (1) Manipulation through two-step primitive where each action involves a start and final end-effector pose. (2) Visual rep-
resentations of actions that are equivariant to translations and rotations [56. 57]. (3) Two separate pathways for semantic and spatial information. (4) Language-conditioned policies for specifying goals and also transferring concepts across tasks. Combining (1) and (2) from Transporter with (3) and (4) allows us to achieve generalizable policies that go beyond just imitating demonstrations.
Section 3.1 describes the problem formulation, gives an overview of Transporter [2], and presents our language-conditioned model. Section 3.2 provides details on the training approach.
3.1 Landuage-Condittoned Manipulation
We consider the problem of learning a goal-conditioned policy ग that outputs actions a, given input 7t (o., It) consisting of visual observation o, and an English language instruction l;:
π (()) T 1:) a (Tpick, Tplace) E A
(I)
3

---

Downomping
Muttipyy
Semantic Ventral
RGB HxWX3
Sentence Encoder (Frozen)
Affordance HxWx1
"pack all the blue and yellow boxes in the brown box"
1024
Spatial Dosal
RGB-D HxWx4
Transporter ter ResNet (Untrained)
Dense Features HiWWed
2. CLIPORT Two-Strream Architecture. An overview of semantic and spatial
semantic
CLIP ResNet50 [1] to encode RGB input, and its decoder layers are conditioned with tiled languuge features from the CLIP sentence encoder. spatial stream
decoder layers are laterally of dense pixelwise Reatures that is used for pick place affordance predictions. This same two-stream architecture is used in all Fully- Convolutioual-eeee fna. Quury. and Phay with faca is used to predict pick actions, and Quuy and dayy are used to prodict place actions. See Appendix for
RGB-D inpat,
semantic stramm
outpur map
architecture.
The actions = (Tpick Tplace) specify the end-effector pose for picking and placing, respectively.
We consider tabletop tasks where Tpick, Tplace E SE(2). The visual observation o, is a top-down orthographic RGB-D reconstruction of the scene where each pixel corresponds to a point in 3D space. The language instruction I either specifies step-by-step instructions e.g. "pack the scissors" + "pack the purple tape" etc., or single goal description for the whole task e.g "pack all the blue and yellow boxes in the brown box". . See Figure 4 for specific examples.
We assume discrete-time input-action pairs Çi = (, (02, 12, a2),...) where a, (Tpick, Tplace) corre- sponds expert pick-and-place coordinates timestep t. These expert demonstrations used to supervise the policy π.
to dataset D (6i, -2, Sn of expert demonstrations with associated
Transporter for Pick-and-Plan manipulation. The model first (i) attends to a local region to decide where to pick, then (ii) computes placement location by finding the best match through
The policy is trained with Transporter [2] to perform spatial
correlation of deep visual features.
composed of two action-valus modules (Q-functions): The pick module Qpick decides where to pick, and conditioned on this pick action the place module Oplace decides where to place. These modules are implemented as mally-Convolutionl--Neee (FCNs) that are translationally equivariant by design. As we will describe in more detail below, we extend these networks to two-stream architectures that can handle language input. The pick FCN fpick takes input 7: (or, lt) and outputs dense pixelwise prediction Qpick H of action-values, where are used predict the pick action Tpick:
Following Transporter [2, 6]. the policy
Tpick argmax Qpick ((u, v)))))
(u,v)
Since o, is an orthographic heightmap, each pixel location (u, v) can be mapped to a 3D picking location using the known camera calibration. fpick is trained in a supervised manner predict the pick action Tpick that imitates the expert demonstration with the specified language instruction at timestep f.
The second FCN & Qqerry takes in 7, [Tick]], which is c X crop of o, centered at Tpick along with the language instruction It, and outputs a query feature embedding of shape Rexexd The third FCN
&key consumes the full input 7/ and outputs a key feature embedding of shape Hx The place action-valuee Qplace are then computed by cross-correlating the query and key features:
Splace A Ar|ies Tpick) (4query (7r [pick]) 4 (xr) [A]]
(3)

---

where Ar E (2( represents potential placement pose. Since is an orthographic heightmap, rotations in the placement pose ÂT can be captured by stacking k discrete angle rotations of the crop before passing it through the query network & Queery Then pluce = argmax Ar Qpaace (Ar/xx Tpics) placements in the expert demonstrations, For all models, we use c = 64, k 36 and d = 3. As in Transporter [2, 6]. our framework can be extended to handle any motion primitive like pushing, sliding, etc. that can be parameterized by two end-effector poses at each timestep. For more details, we refer the reader to the original paper [2].
where the place module is trained to imitate
Two-Stream Architecture. In CLIPORT, we extend the network architecture of all three FCNs fpick, & querr and Payy from Transporter [2] to allow for language imput and reasoning about high-level semantic concepts. We extend the FCNs to two-pathways semantic (ventral) and spatial (dorsal). The semantic stream is conditioned with language features at the bottleneck and fused with inter- mediate features from the spatial stream, See Figure 2 for an overview of the architecture.
The spatial stream is identical to the ResNet architecture in Transporter- - a tabula rasa network that takes in RGB-D input o, and outputs dense features through an hourglass encoder-decoder model. The semantic stream uses a frozen pre-trained CLIP ResNet50 [I] to encode the RGB input o, up until the penultimate layer o, Ve : R 7 and then introduces decoding layers that upsample the feature tensors to mimic the spatial stream v ((-1) + v. (") RxxwxC at each layer I.
The language instruction 1, is encoded with CLIP's Transformer- based sentence encoder to produce a goal encoding I, - g: IR 1024 This goal encoding g: is downsampled with fully-connected layers to match the channel dimension C and tiled to match the spatial dimensions of the decoder features such that g: + g RaxwxC_ The decoder features are then conditioned with the tiled goal features through an element- product v! ① g!" (Hadamard product). Since CLIP was trained with contrastive loss on the dot-product alignment between pooled image features and language encodings, the element-wise product allows us use this alignment while the tiling preserves the spatial dimensions of the visual features. This language conditioning is repeated for three subsequent layers after the bottleneck inspired by Uin [58]. We also add skip connections to these layers from the CLIP ResNet50 encoder to utilize different levels of semantic information from shapes to parts to object-level concepts [59]. Finally, following existing two-stream architectures in video- action recognition [51]. we add lateral connections from the spatial stream to the semantic stream. These connections involve concatenating two feature tensors and applying
conv to reduce the channel dimension [v.") ④ Si .d TBh - IRx where vū and d, are the semantic and spatial tensors at layer respectively. For the final fusion of dense features, addition for fpick and 1 x 1 conv fusion for & Qquery and &key worked the best empirically. See Appendix C for details on the exact architecture.
(1)
3.2 Implementation Details
Training from demonstrations. Similar to Transporter [2] we train CLIPORT through imitation learning from set of expert demonstrations D {G,, S2, Sn consisting of discrete-time input- action pairs S = {(01, 11, a1). (O2, 12, az),... .). During training, we randomly sample an input- action pair from the dataset and supervise the model end-to-end with one-hot pixel encodings of demonstration actions Ypick "x x and Ypiace R xW xk with & discrete rotations. In simu- lated experiments with the suction-gripper, we use k 1 for pick actions and k: = 36 for place actions. The model is trained with cross-entropy loss: L. = - -Ey,a |log Vpick) - Ey [logVpaace] where Ypick softmax Qpick (u, v) (y))) and Vpluuee softmax Optace (11,, v,, w 17,,, Tpick)). Com- pared to the original Transporter models that were trained for 40K iterations, we train our models for 200K iterations (with data augmentation; see Appendix E) account for additional semantic vari- ation in tasks- randomized colors, shapes, objects. All models are trained on single commodity GPU for 2 days with a batch size of
(Ex.allog
Training multi-task models. Multi-task training is nearly identical to single-task training except for the sampling of training data. First, we randomly sample task, and then select random input- action pair from that task in the dataset. Using this strategy, all tasks are equally likely to be sampled but longer horizon tasks are less likely to reach full coverage of input-action pairs available in the dataset. To compensate for this, we train all multi-task models 3x longer for 600K iterations or 6 GPU days.
We cannot use depth information with CLIP since it was trained with RGB RGB-only image-caption pairs from the internet.
5

---

4 Results
We perform experiments both in simulation and hardware aimed at answering the following ques- tions: 1) How effective is the language-conditineed two-stream architecture for fine-grained manip- ulation compared to one-stream alternatives and other simpler baselines? 2) Is it possible to train a multi-task model for all tasks, and how well does it perform and generalize? 3) How well do these models generalize to seen and unseen semantic attributes like colors, shapes, and object categories?
4.1 Simulation Setup
Environment. All simulated experiments are based on Universal Robot URSe with suction gripper. The setup provides a systematic and reproducible environment for evaluation, especially for benchmarking the ability to ground semantic concepts like colors and object categories. The input observation top-down RGB-D reconstruction from cameras positioned around a rectangular table: one in the front, one on the left shoulder, and one on the right shoulder, all pointing towards the center. Each camera has a resolution of 640 x 480 and noiseless.
Manipulation Tasks. We extend the Ravens benchmark [2] set in Py- Bullet [60] with 10 ingguaee-condtiione manipulation tasks. See Figure for examples and Ta- ble 3 for challenges associated with each task. Each task instance is constructed by sampling a set of objects and attributes: poses, colors, sizes, and object categories. 8 of the 10 tasks have two variants, denoted by seen and unseen, depending on whether the task has unseen attributes (e.g. color) at test time. For colors: Teeen colurs = (yellow, brown, gray, cyan) ) and T Tunseen colors = forange, purple, pink, white) with 3 overlapping colors Tan = {red, green, blue} used in both the seen and unseen spilts. For packing objects, we use 56 tabletop objects from the Google Scanned Objects dataset [61] and split them into 37 seen and 19 unseen objects. The lan- guage instructions are constructed from templates for simulated experiments, and human-annotater for real- world experiments. For more details about individual tasks, Appendix A.
Lunguage-Conditionn
Evaluation Metric. We adopt the (fail) to 100 (success) scores proposed in the Ravens bench- mark [2]. The score assigns partial credit based on the task, c.g. 3/5 + 60.0 for packing 3 out of objects specified in the instructions, or 30/56 = 53.6 for pushing 30 out of 56 particles into the correct zone. See Appendix A for the specific evaluation metric used in each task. During an evalua- tion episode, an agent keeps interacting with the scene until an oracle indicates task- cosk-coppeeiinn We report scores on 100 evaluation runs for agents trained with 1, 10, 100, 1000 demonstrations.
seen
unseen
4.2 Simulation Results
200
100
Intppomenttle
Table presents results from our large-scale experiments in Ravens [2] and Figure 3 sum- marizes these results with aver- age scores across seen and unseen
80
80
ASSOACAl
Calon inuil
60
40-
20
splits.
10
100
1000
10
100
1000
4 of Train Demos
of Train Demas
Baseline Methods. To study
Figure 3. Average scores across seen and unseen splits for all lasks in Table 1.
the effectiveness of our two-stream architecture, Transporter-only and CLIP-only. Transporter-only is the original Transporter [2], or equivalently, the spatial stream of CLIPORT with RGB-D input. Although Transporter-only does not receive any language goals, it shows what can be achieved through chance by exploiting the most likely actions seen during training. On the other hand, CLIP-only is just the semantic stream of CLIPORT with RGB and language input. CLIP-only shows what can be achieved by fine-tuning a pre-trained se- mantic model for manipulation without spatial information, particularly depth.
broadly compare against two baselines:
Two-Stream Performance. Figure 3 (seen) captures the essence of our main claims. The perfor- mance of Transporter-only saturates at 50% since it doesn't use the language instruction to ground the desired goal. CLIP-only does have a goal, but lacks the spatial precision to go the last mile and thus saturates at 76%. Only CLIPORT (single) achieves more than 90%, which indicates that both the semantic and spatial streams are crucial for fine -reiiead manipulation. Further, CLIPORT (sin- gle) achieves 86% on most tasks with just 100 demonstrations, showcasing its efficiency.
In addition to these baselines, we present various ablations and alternative one-stream and two- stream models in Appendix F. To briefly summarize these results, CLIP is essential for few-shot

---

packing-hone hox- hoxp-sirss scon-colors
packing box pairs unseen-colors
packing seen- goonle objects-seq
packing-unseen-
packing unseen- gnseen-gnoee objects-soq
packing seen google objects-group
packing Beaenng-useen--ee google objects-group
Method
100
1000
10
100
1000
100
100
1000
10
1000
100
1000
Transporser-only 12)
44.2
55.2
54.2
52.4 34,6
48.7
47.2
54.1
26.2
39.7
45.4
46.3
19.9
29.8
28.7
37.3
60.0
54.3
61.5
59.9
46.2
54.7
49.8
52.0
CLIP-only
38.6
69.7
88.5
87.1 33.0
65,5
68.8
61.2
29.1
67.
89,3
95.8
37.1
49.4
60.4
57.8
52.5
62.0
89.6
92.7
43.4
65.9
73.1
70.0
RN50 BERT
36.2
64.0
94.7
90.3
31.
52.7
65.6
72.1
32,9
48.4
87.9
94.0
29.3
48.5
48.3
56.1
46.4
52.9
76.5
86.4
43.2
52.0
66.3
73.7
CLIPORT (single)
51.6
82.9
92.7
98.2
65.3
68.6
71.5
14.8
59.5
86.8
96.2
27.2
65.5
71.
52.7
67.0
94,0
61.5
66.2
78.4
CLIPORT (multi)
66.8
88.6
94.1
96.6
59.0
69.7
76.2
71.4
41.6
78.4
85.0
84.4
40.7
51.1
65.8
70.3
71.3
84.6
89.6
88.3
68.4
69.6
78,4
80.3
CLIPORT (mulliiattr)
46.2
72.0
86.2
80.3
35.4
45.1
78.9
87.4
48.6
69.3
84.8
stack block pyramid leq-seen-colee
stack black pyramid saq-unseen-colors
separating piles seen-colors
separating piles
lowers of hamoi
lowers: of hunai
unseen-coloes
seq son-soon-colors
seq-unseen- colors
100
1000
10
1000
100
1000
1
100
1000
10
1000
100
1000
Transporter-only [2]
4.5
2,3
5.2
4.5
3.0
4.0
23
5.8
42.7
52.3
42.0
48.4
41,2
49.2
44.7
52.3
25,4
67.9
98.0
99.9
24.3
44.6
71.7
80.7
CLIP-only
6.3
28.7
55.7
54.8
2.0
12.2
18.3
19.5
43.5
55.0
84.9
90.2
59.9
49.6
73.0
710
9.4
52.6
88.6
45.3
24.7
47.0
67.0
58.0
RN50 BERT
5.3
35.0
89.0
97.5
6.2
12.2
21.5
30.7
31.8
47.8
46.5
46.5
33.4
44.4
41.3
44.9
28.0
66.1
91.3
92.1
17.4
75.1
85.3
89.3
CLIPORT (single)
28.3
64.7
91.1
98.8
13.7
24.3
31.2
41.3
54.5
59.5
93.1
98.0
47.2
51.0
76.6
75.2
59.4
92.9
97.4
100
56.1
89.7
95.9
99.4
CLIPORT (multi)
33.5
75.3
96.8
96.5
23.3
26,8
31.7
22.2
48.9
72.4
90.3
89.0
56.6
62.6
64.9
62.8
61.6
96.3
98.7
98.1
60.1
65.6
76.7
68.7
CLIPORT (multi attr)
51.5
59.3
79.8
49.9
48.2
59.8
56.7
allign rope
packing unseen shapes
assembling Kiis-seq colors
assemblinngg kits-sey unseen colors
put- blocks-inn bo-bowls seen colors
put- blobblock-inn bowls
coloes
10
100
1000
10
100
1000
10
100
1000
1
100
1000
10
1000
100
1000
Transporte-oolly [2]
6.9
30.6
33.1
51.5
20.0
22.0
22.0
5.8
28.6
7.8
17.6
25.6
28.4
16.8
31.3
62.7
64.7
11.7
17.2
14.8
18.7
CLIP-only
13.4
48.7
70.4
70.7
13.0
28.0
44.0
50.0
0.8
9.2
19.8
23.0
2.0
4.6
10.8
19.8
23.5
60.2
93.5
97.7
11.2
34,2
33.2
44.5
RNSO-BERT'
3.1
25.0
63.8
57.1
19.0
25.0
32.0
44.0
22
5.6
11.6
21.8
1.6
6.4
10.4
18.4
13.8
44.5
81.2
91.8
16.2
23.0
30.3
23.8
CLIPORT (single)
20.1
77.4
85.6
95.4
21.0
26.0
40.0
37.0
12.2
17.8
47.0
66.6
16.2
18.0
35.4
34.8
23,5
68.3
92.5
100
18.0
35.3
37.3
25.0
CLIPORT (mult))
19.6
49.3
82.4
74.9
25.0
35.0
37.0
31.0
114
348
46.2
52.
7.8
21.6
29.0
25.4
54.0
90.2
99.3
100
32.0
48.8
55.1
45.8
CLIPORT (multi-attr)
7.6
10.4
43.8
34.6
23.0
41.8
66.5
75.7
training demonstrations (I, 10, 100, or 1000). The challenges pertuining to each task are desorihed in Appendix A. CLIPPORT (single) models models are trained on seen splits, and evaluated 1000T demonstrations where T = 10. CLIPORT (multi-attr) indicate CLIPORT (mult)) models trained on seen-and-umseen splits from all lasks except for that one particular heldout lusk, which it is trained only the seen split. See Figure for an overview with average scures.
Table 1. Language- Contuitioon Test Resalts. Task scores (meun 9) from 100 evaluation instances
both seen and unseen
CLIPORT (multi) models trained on splits of tasks with IT, 10T, 100T, and
learning (i.e. n 2 10) in lieu of semantic stream alternatives like ImageNet-trained ResNet50 [62] with BERT [38]. Image-goal models outperform CLIPORT (single) in packing Google objects, but this is only because they do not have solve the language-groundigg problem.
Multi- Performance. In realistic scenarios, we want the robot to be capable of any task, not just one task. We investigate this through CLIPORT (multi) in Table I with one multi-task model trained on all 10 tasks. CLIPORT (multi) models are trained only on scen-splits of tasks, so an unseen attribute like 'pink' is consistent throughout single and multi-task settings. Surprisingly, CLIPORT (multi) outperforms single-task CLIPORT (single) models in 41/72 = 57% of the eval- uations in Table 1. This trend is also evident in Figure 3 (seen), especially in instances with 100 demonstrations or less. Although CLIPORT (multi) is trained on more diverse data from other tasks, both CLIPORT (multi) and CLiPORT (single) have access to the same amount of data per task. This supports our premise that language is a strong conditioning mechanism for reusing concepts from other tasks without learning them from scratch. It also validates a trait of data-driven approaches where training on lots of diverse data leads to more robust and generalizable representations [1, 63]. However, CLIPORT (multi) performs worse on aager-horizon tasks like align-rope. We hypothesize that this is because longer-horize tasks get less coverage of input-action pairs in the dataset. Future works could use better sampling methods that balance tasks according to their average time horizon.
Generalizing to Unseen Attributes. Tasks that require generalizing to novel colors, shapes, and objects are more difficult and all our agents achieve relatively lower performance on these tasks, as shown in Figure 3 (unseen). However, CLIPORT (single) models do substantially better than chance, i.e., Transporter-only. The lower performances are due to the difficulty of grounding unseen attributes such as 'pink' and 'orange' in the language instruction "put the pink block on the orange bowl", when the agent has never encountered words 'orange'. 'pink' or their corresponding visual characteristics in the context of the physical environment. Although pre-trained CLIP has been exposed the attribute 'pink', it could correspond different concepts in the physical setting depending on factors like lighting condition, and thus requires at least few examples to condition the trainable semantic decoder layers. Additionally, we notice that CLIPORT (single) is also less As evidenced in
prone to overfitting compared to Transporter-onl task in Table 1, Transporter-only suffers from a performance drop because of rings with unseen colors despite the fact that Tower of Hanoi can be solved without attending to the colors and simply focusing on the ring size. We hypothesize that since CLIP was trained on diverse internet data, enables our agent to focus task-relevann concepts while ignoring irrelevant aspects of the task.
7

---

"pack yellow and blocks into the brown box"
"put green letter shape the right shape hole"
paak white tape the brown box"
"unfold
"sweep into the yellow zone
t+2
t=1
t=1
t=1
Input
Pick
Place
Figure 4, Affordance predictions from CLIPORT (multi) models in sim (left two) and real settings (right throc). More examples Appendix H.
Transferring Attributes across Tasks. One solution for dealing with unseen attributes is to explic- itly learn these attributes from other tasks. We study this with CLIPORT (multi-attr) in Table 1 and Figure (unseen). For these models, CLIPORT (multi) is trained on both seen-and-unsedd splits from all tasks except for the task being evaluated on, for which was only trained the seen split. As such, this evaluation measures whether having seen pink blocks in put-blocks-in salocks-in-ooe bow--uuwllennn colors helps solve "pack all the pink and cyan boxes" that such explicit transfers result in significant improvements. For instance, on the put-blocks-in- bowls-unseen-colon task for 1000, CLIPORT (multi)'s performance increases from 45.8 to 75.7.
Results indicate
4.3 Real-Robot Experiments
(Samples)
Test
We validated our results in hardware with Franka Panda manipulator. See Appendix D for setup de- tails. Table reports success rates for a multi-task model trained and evaluated 9 real-world tasks. Due to COVID restrictions, we could not conduct large- scale user-studics,
Stack Blocks
5 (13)
10
70.0
Pui Blocks in Bowl
5 (10)
10
65.0
Pack Objects
13
10
60.0
Move Rook
4 (29)
10
70.0
Fold Cloth
9 9
10
57.0
Read
(26)
report small train (5- 10 de- mos) and test sets (5-10 runs) per task. Overall, CLI- PORT (multi) is effective at few-shot learning with just 179 samples, and the performances roughly correspond to those in simulated experiments, with simple block ma- nipulation tasks achieving ~ 70%. We estimate that for more robust real-world performance at least 50 to 100 training demonstrations that the model sometimes exploits biases in the training data instead of learning to ground instruc- tions. For instance, in Put Blocks in Bowl, the training consisted of only datapoint on "yellow blocks" being placed inside "blue bowl". This made it difficult to condition the model to place "yellow blocks" in non-blue bowls. But instances with just one or two examples where a colored block went to different colored bowls was sufficient to make the model pay attention to the language. In summary, unbiased datasets containing both good coverage expected skills and invariances, and decent number of training demonstrations, are crucial for good real-world performance.
Loop Rope
4 412)
10
60.0
Sweep Beans
5 (23)
5
60.6
Pick Cherries
(26))
5
75.0
Table 2. Success rates (5b of multi-takk model trained an evalunted real-world indicate total image-action pairs, in Figure
Figure 1). Samples
necessary, evident in Figure 3. Interestingly. we observed
Conclusion
We introduced CLIPORT, an end-to-end framework for language-conditione fine-grained manip- ulation. Our experiments, specifically with multi-task models, indicate that data-driven approaches to generalization have yet to be fully-exploited in robotics. Coupled with the right action abstrac- tion and spatio-semantic priors, end-to-end methods can quickly learn new skills without requiring top-down pipelines that need task-specific engineering.
While CLIPORT can solve range of tabletop tasks, extending it to dexterous 6-DOF manipula- tion that goes beyond the two-step primitive remains a challenge. As such, cannot handle com- plex partially-observable scenes, output continuous control for multi-fingered hands, or predict lask-completion (see Appendix I for an extended discussion). But overall, we are excited by the confluence of data and structural priors for building scalable and generalizable robotic systems.

---

Acknowledgments
All simulated experiments were facilitated through the Hyak computing cluster funded by the STF at the University of Washington. We thank Mohak Bhardwaj for help with the Franka setup at UW. We are also grateful to our colleagues Chris Xie, Jesse Thomason, and Valts Blukis for providing feedback on the initial draft. This work was funded part by ONR under award #1 140209-405780.
References
[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I, Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2 [cs], Feb. 2021.
[2] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong. V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference Robot Learning (CoRL), 2020.
[3] Akkaya, M. Andrychowiez, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.
[4] D. Kalashnikov, A. Irpan. P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. Conference on Robot Learning (CoRI.,, 2018
[5] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski. C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous mullii-ask robotic reinforcement learning at scale. arXiv preprint arXiv:2104. 08212, 2021.
[6] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. IEEE International Conference on Robotics and Automation (ICRA), 2021.
[7] M. Shridhar and D. Hsu. Interactive visual grounding of referring expressions for human- interaction. Proceedings of Robotics: Science and Systems (RSS), 2018.
[8] C. Matuszek, L. Bo, L. Zettlemoyer, and D. Fox. Learning from unscripted deictic gesture and language for human-row robot interactions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014.
[9] M. Bollini, S. Tellex, Thompson, N. Roy, and D. Rus. Interpreting and executing recipes with cooking robot. In Experimental Robotics, pages 481-495. Springer, 2013.
[10] D. K. Misra, J. Sung, K. Lee, and Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research 2016.
(IJRR), 35(1-3):281
[11] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 15597-6007. PMLR, 2020.
[12] K. He, H. Fan, Y. Wu. S. Xie, and R. Girshick. Momentum contrast for unsupervised vi- sual representation learning. In The IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-9738, 2020.
[13] J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic rep- resentations for vision-and aedioon-nd-lamnn lan-uane tasks. In Advances in Neural Information Processing Systems (NeuRIPS), 2019.
[14] Y. C. Chen, L. Li, L. Yu, A. EI Kholy, F. Ahmed, Z. Gan Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. European Conference on Computer Vision, pages 104-120. Springer, 2020.

---

[15] H. Tan and M. Bansal. Lxmert: Learning modaltyy encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing (EMNL.P), 2019.
[16] D. H. Hubel and T. N. Wiesel. Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. Journal of neurophysiology. 28(2):229-22) 289, 1965.
[17] M. Livingstone and D. Hubel. Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science, 24044853),740-749, 1988.
[18] A. Derrington and P. Lennie. Spatial and temporal contrast sensitivities of neurones in lateral geniculate nucleus of macaque. The Journal of physiology, 357(1):219-244 1984.
[19] J. J. Gibson. The ecological approach visual perception: classic edition. Psychology Press, 2014.
[20] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. arXiv preprint MxXiv:214.11766. 2021.
[21] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In The IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[22] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. In Proceedings of Robotics: Science and Systems (RSS), 2018.
[23] M. Zhu, K. G. Derpanis, Y. Yang. S. Brahmbhatt, M. Zhang, C. Phillips, M. Lecce, and K. Danilidis. Single image 3d object detection and pose estimation for grasping. In 20/4 IEEE International Conference on Robotics and Automation (ICRA), pages 3936-3943. IEEE, 2014.
[24] A. Zeng, K.-T. Yu, S. Song. D. Suo, E. Walker, A. Rodriguez, and J. Xiao. Multi-view self- supervised deep learning for 6d pose estimation in the amazon picking challenge. In 2017 IEEE international conference on robotics and automation (ICRA), pages 1386-1383. IEEE, 2017.
[25] X. Deng, Y. Xiang, A. Mousavian, C. Eppner. T. Bretl, and D. Fox. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3665-3671. IEEE, 2020.
[26] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. In Conference on Robot Learning (CoRL), pages 1369-1378. PMLR, 2020.
[27] R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference on Robot Learning (CoRL), 2018.
[28] P. Florence, L. Manuelli, and R. Tedrake. learning. IEEE Robotics and Automation Letters, 5(2):492 2019.
in visuomotor policy
[29] P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gon- zalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference Robotics and Au- tomation (ICRA), pages 9411 1-9418. IEFF, 2020.
[30] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. In International Symposium Robotics Research (ISRR), 2019.
[31] D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems (NeuRIPS), 32: 10744-1034, 2019.
[32] X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige. Keypose: Multi-view 3d labeling and keypoint estimation for transparent objects. The IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11602-11610, 2020.
10

---

[33] K. Zakka, A. Zeng. J. Lee, and S. Song. Form2fit: Learning shape priors for generalizable assembly from disassembly. In 2020 IEEE International Conference on Robotics and Automa- lion (ICRA), pages 9404-9410. IEFE, 2020.
[34] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed- loop grasping from low-cost demonstrations. IEEE Robotics and Aulomation Letters, 5(3): 4978 4985, 2020.
[35] Y. Wu, W. Yan, Kurutach, L. Pinto, and Abbeel. Learning to Manipulate Deformable Objects without Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2020.
[36] L. Yen-Chen, A. Zeng. S. Song, P. Isola, and T.-Y. Lin. Learning to see before learning to act: Visual pre-training for manipulation. In IEEE International Conference on Roboties and Automation (ICRA), 2020.
[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeuRIPS), 2017.
[38] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North Amer- ican Chapter of the Association for Computational Linguistics (NAACL), 2018.
[39] A. Dosovitskiy. L. Beyer. A. Kolesnikov, D. Weissenborn X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale. In International Conference on Learning Representations, 2020.
[40] Yu, J. Tang, W. Yin, Y. Sun, H. Tian, H. Wu, and H. Wang. Ernie-vl: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006. 16934, 2020.
[41] Y. Bisk, D. Yuret, and D. Marcu. Natural language communication with robots. Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 751-761, 2016.
[42] J. Thomason, S. Zhang, R. J. Mooney, and P. Stone. Learning to interpret natural language commands through human-robot dialog. Twenty-Fourth Fourth International Joint Conference on Artificial Intelligence (IJCAI). 2015.
[43] J. Hatori, Y. Kikuchi, S. Kobayashi, K. Takahashi, Y. Tsuboi, Y. Unno, W. Ko, and J. Tan. Interactively picking real-world objects with unconstrained spoken language instructions. In Proceedings of International Conference Robotics and Automation (ICRA), 2018
[44] Y. Chen, R. Xu, Y. Lin, and A. Vela. Joint Network for Grasp Detection Conditioned on Natural Language Commands. arXiv:2104.066 00490099000000 [cs], Apr. 2021.
[45] V. Blukis, R. A. Knepper, and Y. Artzi Few-shot object grounding for mapping natural lan- guage instructions to robot control. In Conference on Robot Learning (CoRL), 2020.
[46] C. Paxton, Y. Bisk, J. Thomason, A. Byravan, and D. Fox. Prospection: Interpretable plans from language by predicting the future. In International Conference on Robotics and Automa- tion (ICRA), pages 6942-6948, IEEE, 2019.
[47] S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAA/ Conference on Artificial Intelligence (AAAI), 2011.
[48] C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07643 2020.
[49] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. arXiv preprint arXiv:1406.2199 2014.

---

[50] C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1933-1941, 2016.
[51] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 2202-6211, 11, 2019.
[52] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen, Slow-fast auditory streams for audio recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 855-859. IEEE, 2021.
[53] F. Xiao, Y. Lee, K. Grauman, J. Malik, and C. Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.081 08740, 2020.
[54] A. Zeng. S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, al. Robotic pick-and-place of novel objects in clutter with aulti-affordaace grasp- ing and cross-domain image matching. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3750-3757. IEEE, 2018.
[55] E. Jang, S. Vijaanaraaiimhan P. Pastor, J. Ibarz, and S. Levine. End-to-end learning of se- mantic grasping. In Conference on Robot Learning (CoRL), Proceedings of Machine Learning Research. PMLR, 2017.
[56] R. Kondor and S. Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning (ICML), 2018.
[57] T. Cohen and M. Welling. Group equivariant convolutional networks. In International confer- ence on machine learning (ICML), 2016.
[58] D. Misra, A. Bennett, V. Blukis, E. Niklasson, M. Shatkhin, and Y. Artzi Mapping instruc- tions to actions in 3d environments with visual goal prediction. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
[59] G. Goh, N. C. t. C. V. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah. Multimodal neurons in artificial neural networks. Distill, 2021. wttp::illllll///2//ml//oom
neurons..
[60] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
[61] Google scanned objects dataset, 2020. URL http:://app..tppttoooooooootooot htt.r//arrrreeeeeeooeeeeeeoooooo
org/
[62] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In The IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[63] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lec. 12-in-1: Multi-task vision and lan- guage representation learning. In The IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[64] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, al. Pytorch: An imperative style, high perfformaee deep learning library. arXiv preprint arXiv:
2019.
[65] V. Sanh, I.. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[66] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierar- chical image database. In 2009 IEFE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
12

---

[67] J. Mao. C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv: 1904. /2584, 2019.
[68] D. Ding, F. Hill, A. Santoro, and M. Botvinick. Object-based attention for spatio-temporal rea- soning: Outperforming symbolic models with flexible distributed architectures. arXiv preprint 2012:02012,08508, 2020.
[69] E. M. Bender, T. Gebru, A. McMillan-Majon and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623, 2021.
13


Reviews:
[{"review_id": "Q9ykp0uZTrA", "reviewer": "Reviewer_HUjp", "summary": "The paper proposes a language-conditioned modification of Transporter (Zeng’20). Pretrained CLIP (Radford’21) modules are used for the visual and the language stream with fine-tuning on language-annotated demonstrations. The proposed model succeeds on a range of tabletop spatial manipulation tasks in simulation and real world. The model outperforms chance performance (i.e. no language input) on most tasks. Using depth as an additional input somewhat improves performance. Multi-task training improves data efficiency, especially when trained on larger datasets. The model somewhat outperforms standard pretraining of the vision and language models separately (e.g. ImageNet and BERT models), because it leverages joint vision-language pretraining (see App. Tab 4).\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n- The paper presents a promising system that can perform spatial manipulation tasks from language and visual input.\n- Several interesting experiments and ablations on this system are presented.\n- The proposed method leverages large-scale internet datasets via using the pretrained CLIP model, which is a promising direction for robotic control.\n\nWeaknesses:\n- As the paper notices on l319, the base Transporter architecture is quite limited in terms of tasks it can perform and it’s unclear whether the finding of this paper will generalize to more general architectures.\n- The paper claims that “CLIP is essential for few-shot learning in lieu of alternatives like ImageNet-trained ResNet50 [56] with BERT [35].” That does not follow from the comparison in the App. Tab 4. In fact, ImageNet ResNet + BERT outperforms CLIP in the few-shot regime on two of the 4 tasks. \n- In general, the experimental evaluation is highly confusing and lacks many baselines that one would want to see. For instance, the no pretraining baseline or the ImageNet ResNet + BERT baseline are the natural choice, but they are omitted in favor of chance performance, a minor architecture ablation, and an ablation that uses more data. More informative baselines are provided in the appendix. Perhaps just switching the baselines in the paper for the baselines in the appendix would solve this problem? \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "\nThe paper attacks an interesting problem and presents a system that has many of the right parts to attack that problem. I am leaning towards accepting the paper, but the presentation of the baselines needs to be significantly improved for that.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "\nSee weaknesses", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "FLd2UnhAjde", "reviewer": "Reviewer_WZS2", "summary": "The authors combine a large pre-trained CLIP vision-and-language model with a Transporter network in order to facilitate training a multi-task policy conditioned on natural language inputs with limited data. They compare against CLIP and Transporter baselines on both trained language-conditioned tasks and transferred languaged-conditioned tasks. They find that this method is able to generalize effectively to novel combinations of attributes (such as seperating the notions of 'blue' and 'cube', so that they can solve tasks involving blue cubes after having seen tasks with blue non-cubes and non-blue cubes). In tasks involving this kind of conceptual decomposition, they dramatically outperform other baselines. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nI find this particular kind of of generalization across attributes extremely interesting, especially considering that they achieve it solely through end-to-end training. Additionally, the demonstration that large pre-trained networks for non-robotics language and vision tasks can be used to enable more efficient training training for robotics seems significant, as data and pre-trained models for these domains are much more accessible. I was also pleasantly suprised at the amount of training time used to achieve this level of result -- 6 GPU days is a decent amount but feasible for most labs. \n\nWeaknesses: \nIt's not entirely clear how the authors get from the affordance prediction to the actual actions on the robot. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This result is both immediately useful and offers up new and promising opportunities for future research. The use of natural language is both incredibly convenient for use and adding new tasks, and suprisingly data-efficient. The work additionally offers interesting new avenues for how large pre-trained networks from other domains could be used to enable more efficient robot learning. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "It's not entirely clear how the authors are moving from the affordance prediction to actions on the actual robot. Since the labelled data all seems to be labelled images rather than action sequences, my best guess would be that the authors are using the affordances to select pre-programmed manipulations to take, and what parameters to call those manipulations with. I'd like to understand this better though, so a little more clarification would be nice. "}}, {"review_id": "CXu-6RFR1p3", "reviewer": "Reviewer_Hq8X", "summary": "The paper presents a framework for learning language-conditioned, vision-based manipulation tasks. The high-level idea is to separate the spatial aspect of manipulation (e.g., where to pick and place) from the language conditions that specify the manipulation target. The goal is to generalize to new language commands by leveraging a pretrained vision-language model. The visual manipulation framework builds on the Transporter network (Zeng et al., 2020), which models multistep manipulation as a dense image prediction problem. The language condition is supplied through a pretrained CLIP vision-language embedding model (Radford et al., 2021). By conditioning the Transporter prediction on the CLIP embedding and training on a fixed set of demonstrations, the framework is shown to zero-shot generalize to follow new language commands (e.g., picking blocks of unseen colors). The method is evaluated on a set of simulated and real-world tabletop manipulation tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like the overall premise of the paper: large-scale vision-language models should be able to help visual-manipulation models to generalize better. I also like the overall execution of the paper: the proposed method is a simple extension of the Transporter model to incorporate language conditions and is evaluated extensively on both simulated and real-world manipulation tasks. However, a few doubts remain on the main thesis: aligning vision-language representation with visual-manipulation representations brings “the best of both worlds”. \n\nFirst of all, to what extent does the “spatial / geometric” aspect of manipulation benefit from the “semantic” aspect of manipulation? Intuitively, CLIP is an excellent medium for generalizing to unseen visual concepts and combination of visual concepts in *visual recognition tasks*. However, I struggle to understand what kind of synergies between vision-language embedding and manipulation can be exploited beyond, e.g., correctly recognizing a new object that’s not shown in the demonstration because the pretrained CLIP model has that object before. \n\nFor example, if the Transporter model is never trained with pick-and-place an awkwardly-shaped or a slippery mug, correctly recognizing that it is a mug shouldn’t help with its manipulation. In other words, if pick-place tasks are mostly about understanding the geometric/dynamic aspect of the objects, why would a vision-language model help with its generalization? We may further extrapolate this argument: if we have a detection model that can correctly localize novel objects (e.g., 2D position & orientations) given language commands, to what extent do the language-conditioned Transporter network benefit from the CLIP embedding? \n\nSecond, language commands contain information beyond object categories and attributes. For example, in the example given by the paper “pack all the blue and yellow boxes in the brown box”, to what extent does the relationship “in the” constitute a reusable linguistic concept beyond identifying a task? For example, would the model be able to generalize to “place a dice in the mug'' or “grasp the mug in the cabinet”? I understand that the paper does not claim generalization to novel relationships or non-object-related concepts. But it is important to discuss such limitations or even present negative results.\n\nMinor:\n- \nLine 165: what does it mean to “downsample” the goal embedding? Are there downsampling operations like strided 1-D convolution involved or is it just a linear projection?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am overall neutral about the paper. I'm recommending weak reject right now but it's more like a \"borderline\". I will likely change my rating after seeing additional discussion / caveats about the main thesis of the paper and further experiments regarding the limitations.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See \"weaknesses\"", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: CLIPort combines CLIP (a joint language and image embedding) with TransporterNets (a method for performing data efficient BC on sparse action manipulation problems) to enable multi-task NLP conditioned policy learning. Multiple reviewers praised the techniques ability to generalize via CLIP pretraining and it’s leveraging of large-scale internet datasets, as well as the perceived data efficiency of the approach. The authors addressed the bulk of reviewer feedback, adding clarifications for limitations as well as additional real-world experiments and baselines. The paper achieves compelling real-world results and should be of interest to the CoRL community.
Decision: Accept (Poster)