Visually-Grounded Library of Behaviors for Manipulating Diverse Objects across Diverse Configurations and Views
Authors: Jingyun Yang
Hsiao-Yu Tung
Yunchu Zhang
Gaurav Pathak
Ashwini Pokle
Christopher G Atkeson
Katerina Fragkiadaki
Venue: CoRL2021 Poster (2021)
Keywords: robot learning
visual representation
interactive perception

Abstract: We propose a visually-grounded library of behaviors approach for learning to manipulate diverse objects across varying initial and goal configurations and camera placements. Our key innovation is to disentangle the standard image-to-action mapping into two separate modules that use different types of perceptual input: (1) a behavior selector which conditions on intrinsic and semantically-rich object appearance features to select the behaviors that can successfully perform the desired tasks on the object in hand, and (2) a library of behaviors each of which conditions on extrinsic and abstract object properties, such as object location and pose, to predict actions to execute over time. The selector uses a semantically-rich 3D object feature representation extracted from images in a differential end-to-end manner. This representation is trained to be view-invariant and affordance-aware using self-supervision, by predicting varying views and successful object manipulations. We test our framework on pushing and grasping diverse objects in simulation as well as transporting rigid, granular, and liquid food ingredients in a real robot setup. Our model outperforms image-to-action mappings that do not factorize static and dynamic object properties. We further ablate the contribution of the selector's input and show the benefits of the proposed view-predictive, affordance-aware 3D visual object representations.

Visually-Grounded Library of Behaviors for Manipulating Diverse Objects across Diverse Configurations and Views
Jingyun Yang"
Hsiao- Yu Tung"
Yunchu Zhang"
Gaurav Pathak
Ashwini Pokle
Christopher G Atkeson
Katerina Fragkiadaki
Carnegie Mellon University
(j Lingyuny, htung,
apokle, ega, kfragki2) (araaree @andrew. cmu, edu
Abstract: We propose a visually-grounded library of behaviors approach for learning to manipulate diverse objects across varying initial and goal configu- rations and camera placements. Our key innovation to disentangle the stan- dard image-to-action mapping into two separate modules that use different types of perceptual input: (1) a behavior selector which conditions on intrinsic and semantically-rich object appearance features to select the behaviors that can suc- cessfully perform the desired tasks on the object in hand, and (2) library of behaviors each of which conditions on extrinsic and abstract object properties. such as object location and pose, to predict actions to execute over time. The selector uses semantically-rice 3D object feature representation extracted from images in differential end-to-end manner. This representation is trained to be view-invariant and affordance-awart using self-supervision, by predicting vary- ing views and successful object manipulations. We test our framework on pushing and grasping diverse objects in simulation as well as transporting rigid, granular, and liquid food ingredients in real robot setup. Our model outperforms image- to-action mappings that do not factorize static and dynamic object properties. We further ablate the contribution of the selector's input and show the benefits of the proposed view new-prreicttiee affordance-and aware 3D visual object representations.
Keywords: Robot Learning, Visual Representation, Interactive Perception
Introduction
Object manipulation in unstructured environments is challenging since methods to manipulate ob- jects largely depend on the object's visual appearance. One approach to capture the dependence between actions and visual features is to learn a direct mapping from image to actions with deep neural networks [1, 2, 3]. Despite their flexibility, such end-to-end image-to-action mappings have been shown be data aggressive [4], and cannot easily generalize across objects, camera view- points, or scene configurations [5].
Approaches that abstract away object details and encode only a subset of their properties, e.g., their 3D locations and velocities [6] or 3D keypoints [7, 8] make the state-to-action mapping easier to learn with less data. However, this abstraction may substantially limit the range of objects that a policy can handle, since useful information (object shapes, softness, weight, and material, for example) for the downstream task may be ignored. The challenging question is: how a framework for object manipulation that uses abstract representations for sample efficient behavior learning, but at the same time is capable of utilizing semantically-rice representations for handling diverse objects and views.
design
We propose goually-grounded library of BEhaviors (V-BEs), a hierarchical framework for vision- based object manipulation. Our main contribution is that the two levels of our policy hierarchy use different visual representations. At the lower level of the hierarchy, a behavior library contains set of distinct behaviors each of which operates on an abstract object state representation that captures
Equal contribution.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Generalize across different objacts
Generalize across different camera viewpoints
Generalize across different initial configurations
We propose novel policy representation that generalizes to unseen objects, object place- ments and camera views.
Figure I:
dynamic properties of objects in the environment, such as object and gripper positions over time. Behaviors can be redundant, use different state abstractions, be closed loop open loop, and be learned or manually engineered with different algorithms [9, 10, 11]. At the higher level of the hierarchy, selector is trained using environment interactions to associate objects to behaviors that can successfully manipulate them. This selector takes as input RGB-D images of the scene, maps them to semantically-rich and view-invariant& 3D object feature representations [12), and outputs the behavior that best suits the presented object and task. The decomposition of visual information into a perceptually rich selector network and behaviors with state abstraction makes the behaviors easier to learn with reinforcement learning or be designed by an engineer, and the selector casier to learn by trial-and-error learning as does not bear the burden of producing low-level actions. At test time, we use 3D object detectors to localize the object, and present its location and size to the low-level controllers when necessary.
Our second contribution is proposing a visual representation for our behavior selector that is robust changing camera viewpoint (with respect to non-changing robot torso), as well object place- ments. We use the differentiable 2D-to-3D neural networks of Tung et al. [12] to map an input RGB-D image alongside camera intrinsics and extrinsics transform 3D visual scene feature maps from the current camera viewpoint to the world frame, ensuring that visual features are bound to a static world frame-that coincides with the frame of the agent's body-and do not change dramat- ically while the camera moves. The 3D visual feature representations learn based on environment interactions to embed objects close the behaviors that can manipulate them. They are further trained from the auxiliary task of view prediction using multi-view camera setup, which encour- ages the model complete missing information from the current view so the 3D visual feature maps are consistent across views and through occlusions.
We test the proposed model in both simulated and real robot setups. In simulation, we consider two manipulation tasks: pushing and grasping. We show that our model can manipulate diverse objects and generalize the learned skills to unseen objects with varying object starting positions, initial poses, goal locations, and camera viewpoints. We show that our model outperforms existing end- to-end image-to-action mappings [4] and state abstracted object-to-action mappings that use only 3D object locations [6]. We ablate the contribution of our higher-level 3D feature representation used in our selector and then compare against 2D pretrained baselines. Lastly, we test our model in a real robot setup where the robot transports rigid, granular, and liquid objects onto plate. We provide supplementary materials including additional qualitative results attps://yjy00s github. wttprooects//--beeeeee
2 Related Work
Libraries of behaviors have been considered in many previous works, and library elements have been called action, motion, behavioral, or motor primitives, parameterized policies, options, chunks, macros, subroutines, behavioral units, and skills, as well as many other terms. However, most previous work either assumes that the environment state is known and does not use perception, or has different approach to perception than ours. Work of Neumann et al. on modular policies [13] defines motor primitive as a mapping from robot's joint angles and velocities and not the locations of objects. Rather, object locations are used as context for selecting a motor primitive. This means i) motor primitives are "blind" the object 3D location throughout the episode, and ii) object appearance is discarded which means the library of primitives cannot handle object variability. Work of de Silva [14] on parameterized policies similarly considers as the contextual vector the desired goal location of a dart, and learns a mapping from this location to policy parameters, where each policy operates only over the robots' joints and represented as a Dynamic Motion Primitive (DMP) [15, 16, 17J. To extend DMPs to tasks involving physical
Libraries of Behaviors:

---

PO OR/OUNT +.0.00
Retrieval Keys
Behavior Library
x
Policy
a
Action
Query (Visual Features)
IT
Behavior 1
3D centroids and 3D poses to aucioonmapinng
IT
Behaviar 2
a
Action
X
Policy
Action
iTy
Behavior
Image to action mepping
V-BE (Ours): visually-groudded library of behaviors
Figure 2: Comparison of our model with previous approaches. In contrast to state-to-action or image-to-action mapping, the proposed framework decomposes policy into a behavior selection module and a library of behaviors to select from. The decomposition enables these modules to work on different representation: the selection module operates a semantically-rion visual feature space, while the behaviors operates in an abstract object state space that facilitate efficient policy learning.
interactions, Kroemer et al. [18] parameterize DMPs based on the distance of an object part from the gripper, but only consider known object 3D shapes. Strudel et al. [19] use a depth frame as input to a meta-policy that selects skills to execute. However, the use of modularity over skills is for temporal sequencing: there is no mixture of behaviors per manipulation task, rather an image to action mapping is learned with behavior cloning from demonstrations.
Deep Reinforcement and Imitation Learning for Object Manipulation: Current work in (deep) reinforcement learning typically considers only one monolithic policy learned via reinforcement learning [9, 20, 21], imitation learning
[22, 23, 1], a combination of the two [24, 25]. Most approaches take as input 3D object and gripper centroids, orientations, and velocities, and ignore contextual visual information [26]. Existing methods are commonly trained in fixed environment and evaluated in terms of their ability to discover a policy rather than their ability to generalize to previously unseen circumstances, e.g. novel objects and views, which the goal of this work.
Methods that do aim to generalize across objects learn a mapping from images-to io-actions [4], depth- [28, 29]. They have been successful in various tasks includ- ing object grasping [27, 30] and object pushing from a fixed camera view [31, 4]. Seminal works of [4, 32] train image-to- to-ge-t--cctionn deep neural network by imitating trajectories obtained from planners that operate in a low-dimensional state space. Nevertheless, such monolithic mage-to-action deep mappings don't generalize well across different camera viewpoints [5, 33, 34] and objects. Works that attempt to transfer visuomotor policies learned in simulation to the real-world often require identical placement of the camera in the real world [35, 6]. Florence et al. [36] train manipulation policies parameterized by the 2D or 3D locations of designated set of visual descriptors obtained from RGB-D images, and show their policies generalize across objects of the same object category, e.g., shoes of different shapes. The proposed work further allows generalization across object cate- gories, and no explicit descriptor selection or optimization is necessary. Furthermore, our framework permits each behavior to use different state representation.
to-actions [27] nointcloud-to-action
Hierarchical Reinforcement Learning (HRL): Most existing HRL methods [37, 38] focus on discovering behaviors and temporally sequencing them for long-horizon tasks. This paper instead uses hierarchical vision-based policy architecture to improve model generalization across objects, configurations, and viewpoints.
Please check supplementary materials for additional discussion of related work.
3 Methods
Problem Setup We consider the problem of manipulating (c.g. grasping, pushing, etc.) diverse sets of objects from various initial configurations and camera viewpoints. At training time, the agent has access to training objects O RGB-D images from J viewpoints Ui...., UJ E V, and it can interact with the training object from randomly selected viewpoints. At test time, the agent has to perform the same manipulation task as at training time on test objects Ows from selected viewpoints. Performance is measured by the success rate of manipulating all test objects with random initial configurations and camera viewpoints.
Our framework is comprised of two major components - (1) a library of behaviors IT = (m: 1,2,..., K) and (2) a selector function G. Given an RGB-D image of the scene captured from camera view denoted by J, (T, and an object 3D bounding box o, the selector G obtains the
3

---

Seterrer Selector
Retrieval
Crap Object
GRNNS
Input Image J
M
Retiooal Kay ng
Bohavior Ri
Execute Selected Behavior
GRNN
Behavfors
The robot interacts with traninn
3D-w--30 Liting
objects using behaviors the llbrary collect binary success failure interaction
collected Interaction labels are then used to train the behavior selector.
M (Cameru Frans)
Input Image
(lumm)
Figure 3: Overview of the proposed framework.
probability of successfully manipulating the object when applying behavior n, the object by com- puting a query object feature representation F(J,, o) and compares it with learned key embeddings K; x(Ti), i = I,.,, K associated with each behavior mi
G(T,, O, TH &. x) = ((F o; o), ks)) E [0, 1),
where Φ is the learnable neural network parameters for function F, (: is the inner product opera- tion and a is the sigmoid function. We will detail the feature dimension of F(l,, o; o) and in the next section. At test time, we select the behavior स, that has the highest probability of leading to successful manipulation.
Each behavior πT, uses its abstract state representation for the objects to manipulate, for exam- ple, 3D object locations and 3D poses, and does not necessarily take into account other aspects of the input image. Each behavior is designed orientations. To handle a diverse set of objects and orientations, our selector learns to pick different behaviors for different objects or orientations. One advantage of the modularity of our policy archi- tecture is that each behavior can use its own state abstraction, such as object 6D-poses, 3D bounding boxes, part-based 3D boxes, or 2D or 3D object keypoint locations [7]. Our framework supports integrating a wide range of existing models, representations, and controllers as selectable behaviors.
learns to handle only a subset of object shapes and
In the rest of this section, detail the architecture and training details of the behavior selector in Section 3.1, and describe how we acquire library of behaviors for grasping and pushing -the two robot manipulation tasks we evaluate our framework on-inn Section 3.2.
3.1 Visually-Grounde Behavior Selector (G)
The behavior selector a classifier that learns object-centric visual feature representation F(.,, Φ) for the object box in image view I,, and behavioral key embeddings x² (i [ύ=18 I,..., K) for the behaviors in the library to retrieve behaviors compatible with a particular object.
Training the selector with interaction labels. We learn the object feature representation and be- havioral keys through trial-and-erres In each trial, our agent applies a randomly sampled behavior πi on an object in the workspace which results in binary success or failure outcome / {0, 1}. which we call interaction labels. Agent interactions are organized as tuples of the form (J,, o, n; E). With the interaction experience as training data T, we train the feature encoder f and behavioral keys H; n(T;)) (i =1, 1,..., K) with the loss:
Caffora (K, ②)
Σ (Ju,o,
BCE(C, G(L..
Σ (L.,.
where BCE(y, = y - log(p) - -(1 (1- (11-y)log
y) log(1 - pp) is the binary cross entropy loss.
3D object feature representation. Our object feature representation F(I.. o; Φ) is computed using Geaaetry-aware Recurrent Neural Networks (GRNNs) of Tung et al. [12], which are end-to-end differentiable architectures that map single RGB-D image or a set of multi-view images to 3D feature representation of the scene the image(s) depict. Given posed image I., GRNNs obtain scene-centric 3D feature representation M GRNN (T.) E R H DxC through differentiable

---

2D-to-3D operations, 3D convolution-bass refinement, and 3D rotation operations that align the 3D feature representations with the robot's coordinate frame (as opposed camera frame), with W. H, D denoting the width, height and depth of the scene map and C' ddenotes the feature dimension of the 3D scene feature map.
From the scene map M. we obtain object-centric feature representation F(J,, o) crop o) e 164x64x64x32 by cropping the scene map using fixed-size axis- axis-aligned box o, centered around the object we wish to manipulate. The feature cropping operation is similar to the one used in Mask- RCNN [39]. The retrieval policies keys are also learned in the same representation space. Thus, both F Fl Ø) and K4, (i = 1, K) have length 64 x 64 x 64 x 32 in the experiments.
3D object detector. We learn a detector with 3D Mask-RCNN built on top of the GRNNs feature encoder [12]. We use groundtruth 3D object boxes at training time, and predicted 3D object boxes test time, where we train our representation detect objects in 3D.
View prediction and occupancy prediction as an auxiliary task. We use view prediction and occupancy prediction an auxiliary task to help image encoder generalize better in its ability to select behaviors. These two self-supervised prediction tasks have been shown to provide a useful pretraining or co-training objective for 3D object detection in [40]. Given an input posed image I and a query view q", the overall sulf--uper prediction loss reads:
Csett-pred (Ø, (0, 0, =
Σ |Po(GRNN& (1" IM + |Occ,, (CRNN, ( occ"!l1,
(3)
view prediction
occupancy prediction loss
where Po (M. q) is a projection function that projects a 3D feature map M from the query viewpoint q to a 2D feature map and decodes it to target image I using an image decoder with neural network weights 0, Occ,, (M) E 564x64x64 is voxel occupancy prediction function that predicts a 3D occupancy map from an input 3D feature map M using a single 3D convolution layer with weights 17, and occ" is the estimated occupancy map computed from all available input views in the nhth data point by voxelizing the unprojected point clouds from all available depth images. We train the model from unlabelled multi-view images captured around the table by simply moving the cameras, capturing the images, and recording the corresponding camera locations.
The final objective for training the affordance-based visual features is
minimize [(s, Φ, 0, n1 = Leell prea (Ø, 0, 1) + Aa Latlond Φ),
(4)
k.
where A is a hyperparameter for balancing the two losses.
3.2 Building a Library of State Abstracted Behaviors
Any existing behaviors, whether engincered learned using reinforcement or imitation learning, can be included in our library. This flexibility is a contribution of our modular architecture. this paper, we consider three common manipulation tasks: pushing, grasping, and transporting. We build appropriate behavior libraries for each.
In pushing, the behaviors are deterministic goal conditioned policies a; = (s((,))) g) that map a state of the environment and the robot s, = [s s7] and a goal state g to an action as time step t. The environment state of is the 3D object centroid and the robot state of is the gripper 3D location, pose, and whether it is opened or closed. Actions include 3D translation, opening (position control), and closing (force control) of the gripper. A goal state g is target centroid location for the object. We use a total of 25 goal conditioned policies one is trained from the whole set of objects, while the others are trained on disjoint subsets of object configurations organized based on object category and initial poses. We train all policies using deterministic policy gradients (DDPG) [25] with goal relabelling (HER) [9] while randomizing initial and goal object 3D locations.
In grasping, we design controllers π ( prrssp pprasp.ggraapy which given a 3D grasping point perasp R3 relative to the center of the object and a grasping 3D angle garasp E IR,, move the gripper (open loop) to the grasping 3D point location, close it, and move it to the desired goal location. The grasping angle yerwey consists of two numbers describing the yaw of the gripper and the elevation angles between the gripper and the table surface. When the elevation angle smaller than 90 degrees (not top-down grasps), we constrain the gripper to point toward the center of the object on
5

---

the x-y plane. We manually select 30 different controllers including top-down grasps with different yaw orientations (top- grasps) and grasps from the side with different elevation angles of the gripper (side-grasps). We empirically found that these parameterized controllers are quite stable and can be shared across multiple objects. More details are provided in the supplementary materials.
4
Experiments
Our experiments aim to answer the following questions: (1) Does the proposed serrary-bssed ap- proach outperform existing methods that use single combined perception and policy module, ei- ther using 2D images, 3D object locations, or 3D scene feature maps as input? (2) Is the proposed view-invariant and affordance-aie
3D feature representation a necessary choice for the selector? (3) Does the method work on a real robot? We test our model on grasping and pushing wide variety of objects in the MuJoCo simulator [41] and further test a transporting task on real-world Franka Panda robot arm.
4.1 Simulation Experiments
Our simulated environment consists of Fetch Robot equipped with a parallel-jaw gripper. The robot is positioned in front of table of height 0.4m. To obtain the visual observations, on each episode we choose 3 random cameras from cameras placed at 30 nominal different views including 10 different azimuths ranging from 0 to 360 combined with 3 different elevation angles from 20", 40, 60% All cameras are looking at the center of the table top, and are 0.5 meter away from that point. All images have size 128 x 128.
In the grasping task, the agent has to grasp an object and move it specified target location above the table. We use 274 distinct object meshes from 6 categories in ShapeNet [42] including toy buses, toy cars, cans, bowls, plates, and bottles. The materials and densities of all objects are identical. We randomly split the dataset into 207 training objects, and 67 testing objects. After augmenting the meshes with random scaling from 0.8 1.5 and random rotations around the vertical z-axis, we get a total of 800 distinct object configurations (object instance and pose), 600 for training and 200 for testing. At the start of each episode, an object placed in an area of 30cm x 16cm around the center of the table, and goal is sampled uniformly 10 ~ 30cm away from the gripper's initial position. An episode successful if the object centroid is within 5em of the target at the final timestep.
Task Deseriptions:
In the pushing task, the agent has to push an object placed on the table specified target location. We use 100 objects from 12 categories in ShapeNet [42]: baskets, bowls, bottles, toy buses, cameras, cans, caps, toy cars, earphones, keyboards, knives, and mugs. After augmentation and splitting to train and test sets, we obtain 615 training object configurations and 200 for testing. The initial and the goal position of the object are both uniformly sampled to be within 15cm of the center of the table along both x-axis and y-axis, although we resample if that location is already in the goal area. An episode is successful if the object centroid within 5em of the goal within 50 timesteps.
Baselines: We compare our method with various learning and non-learning based methods for object manipulation:
(a) Single Behavior w/ Abstract 3D State (Abstract 3D) 19, 25J: policy takes as input ground truth 3D bounding box of the object and gripper and outputs actions.
(b) Single Behavior w/ Abstract 3D State and 2D Images (Abstract 3D + Image): a policy takes both RGB-D images and the ground truth 3D bounding box as inputs and outputs actions. Our architecture resembles that of [43], but we further include ground truth object position as extra inputs to the model. For fair comparisons to other methods, the model only takes as input the current state as opposed to the states in 5 past steps, as in [43].
(c) Single Behavior w/ 3D Feature Tensor (Contextual 3D): a policy takes as input RGB-D images and the ground truth 3D bounding box and outputs actions. Different from (b), the model first transforms the image into a view-invariant 3D feature tensor using GRNNs [12], then converts the 3D feature tensor into a feature vector though three 3D-convolutional layers and fully connected layer, and concatenates it with the rest of the inputs to predict actions.
(d) Ours, Library of Behaviors w/ Visual Selector (V-BEs): Our model takes the same input as (b) and (c). The 3D bounding boxes are used as input to all the behaviors. The RGB-D images are transformed into 3D affordance-awant visual features and treated as input to the selector.

---

Single Behavior
Labrary of
Ablation study on the selector's
Behaviors
visual feature representation
Abstract 3D
Abstract 3D
Contextual
V-BEs
V-BEs w/
V-BEs wo tune-tunigg
19. 25]
Image
3D
(Ours)
2D features on Interaction Labels [45, 12]
grasping
0.30
0.35
0.20
0.78
0.46
0,31
pashing 0.83
0.70
0.10
0.88
0.81
0.46
Table 1: Success rates on grasping and pushing unseen objects. We also ablate the proposed method
with selectors operating on varying representations.
We train the baselines with different learning methods including hehavior cloning [4], DDPG-HER [11, 9] and DAGGER [44]. We report the best performance we got by training the model with these different methods. We also attempt to make all the models have similar number of parameters so the comparison is fair. However, larger networks are empirically harder to train and do not converge well, so we instead increase the number of parameters in smaller networks until their performance saturates. For pushing, we found that using DDPG-HER is enough to lean a good Abstract 3D policy from scratch. For abstract 3D Image, we found it is critical to use behavior cloning from expert demonstrations to obtain good policies. The expert demonstrations are obtained from trained expert policies on single objects. For Contextual 3D, we include DAGGER to enforce behavior cloning during execution. To train the grasping policies, we further include human demonstrations in the replay buffer when training it with DDPG-HER. Both abstract 3D + Image and Contextual 3D are trained with DAGGER since offline behavior cloning is insufficient.
4.2 Single Behavior versus Library of Behaviors
We compare the proposed model with models that do not use a library-based approach, i.e., single behavior approaches. As shown in Table I, our method outperforms all the single behavior base- lines. Abstract 3D performs well, but since it does not use any visual information, its performance saturates at around 0.8 for pushing and 0.3 for grasping. Abstract 3D performs poorly for grasping. The learned behaviors do not transfer well to new objects. Adding a 2D image helps, but not dramat- ically (see Abstract 31) + Image in Table 1). Although 3D feature maps obtained from GRNNs are semantically rich and can handle varying viewpoints, the mapping to actions is harder learn due to the higher dimensionality of the 3D scene map, resulting in under-fitting models. Our model takes advantage of both abstract and semantically rich representation and thus can handle better object variability and transferability. The combinatorial nature of the proposed method allows the model to capture the multi-modality in trajectory generation.
4.3 The Necessity of Building the Selector with the Proposed 3D Representations
Next, show the importance of using view-invariant 3D visual feature representations and fine- tuning the selector with interaction labels. We compare our method with two baselines: (a) a model with selector that learns the visual affordance features using 2D visual features extracted from 2D) CNNs, and (a) a model with a selector that operates over 3D visual feature representation learned only with the view and occupancy prediction loss, as suggested in [45, 12), without fine tuning with interaction labels. See Table 1 for the results. Our method significantly outperforms these two baselines, which shows the importance of both proposed components. To fully test the power of existing 2D CNNs, we also tested 2D feature selector with existing VGG network [46] pretrained on ImageNet and fine-tuned on our interaction labels. However, the performance ( a success rate of 0.78 on pushing) does not differ too much with shallower 2D CNNs trained from scratch.
4.4 Transporting Task on a Real Robot
We test our model on 7-DOF Franka robot arm equipped with a parallel-jaw gripper (using [47]'s software stack) for a transporting task, where the robot needs to transport various rigid, granular, or liquid food ingredients from random initial positions and poses onto plate (see Figure 4). We set up Intel RealSense RGB-D cameras that have full view of the workspace around the the center of the table. In each trial, an object is placed in 50cm x 30cm region on the table. and the goal is transport all objects to a plate 25cm to the left of the starting region. Granular objects and liquids are placed inside containers in the beginning of the trial. An episode is considered successful if the object is successfully transported into the plate. For granular objects and liquid, an episode considered successful if at least half of the total quantity ends up in the plate.
7

---

Cormin
Trainin Objects
Testing Dojects
Workepace Setup
Figure 4: Real robot experimental setup. We set 4 cameras around the table that cover different viewpoints of the workspace. 3 out of 4 cameras are shown in the right most image. An extra topdown camera used, but clipped from the image.
Figure 5: Sample real robot executions. Each row shows an execution trial during test time. Our robot can successfully transport rigid, granular, and liquid food ingredients to target plate.
We construct our library of behaviors with 26 controllers, in which 13 of them are pick-and-place controllers from various grasping angles and the other 13 are pick-and-pour controllers using the same grasping angles as the pick-and-place ones. At evaluation, we randomly select 3 views from possible camera views to obtain RGB-D - images as inputs to the learned selector. We use 20 rigid objects, 20 granular objects, and 12 bottles of liquids for our experiment, and we split them into 38 training object and 14 testing objects. To train our model, we collect total of 3510 interaction labels by running the 26 behaviors the training objects with random initial poses and using the labels to finetune the visual selector with the objective specified Equation (4).
Our model achieves success rate of 88.6% % on the test set. We compare our model with two base- lines: 1) an image-to-action model trained with behavior cloning (Image), and 2) a hierarchical model that uses library of behaviors and a selector with 2D representations (V-BEs with 2D fea- tures). We use the data collected during the interaction label collecting process as the data used to train both baselines. For the Image baseline, make it work in simulation where there is more data. It may need more data to learn a general and robust policy in the real world [48]. For the V-BEs with 2D features baseline, 38.0% which is worse than the proposed model. This again shows the importance of operating the selector in the proposed new-invariiant and object-centric 3D feature space. Sample executions and transporting objects in clutter are visualized in Figure 5 and included in our supplementary video.
not able to get it to work at all, while did
get success rate of
Conclusion
We have presented V-BEs, a hierarchical policy architecture where 3D object visual feature repre- sentations are used to select from a library of behaviors. The proposed modular architecture supports both low-level behaviors and the selector to be learnt data efficient manner. We have shown re- sults on pushing and grasping diverse objects in simulation and in the real world, across diverse viewpoints. Our method outperforms image-to-acti well as policies that operate on 3D locations and velocities alone. Since our framework is sample- efficient and simple to run, we can easily deploy complex transporting skills on a real robot arm.
monolithic policies of previous works, as
8

---

Acknowledgments
This work supported by Sony AI, NSF award No 1849287. DARPA Machine Common Sense, an Amazon faculty award, and an NSF CAREER award.
References
[1] T. Zhang, Z. McCarthy, Jow, D. Lee, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. ICRA, pages 1-8, 2018.
[2] D. Gandhi, I.. Pinto, and A. Gupta. Learning to fly by crashing. In 2017 JEEE/RSJ Interna- IEEE, 2017. [3] D. Pathak, P. Mahmoudich, G. Luo, P. Agrawal, Chen, Y. Shentu, E. Shelhamer, J. Malik,
tional Conference Intelligent Robots and Systems (IROS), pages
A. A. Efros, and T. Darrell. Zero-shot visual imitation. In ICLR, 2018.
[4] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1333 2016.
[5] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and
C. Finn. Robonet: Large-scale multi-robot learning, 2019.
[6] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki,
A. Petron, M. Plappert, G. Powell, A. Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[7] Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and S. Savarese. Keto: Learning keypoint representations for tool manipulation. arXiv preprint arXiv:1910.11977 2019.
[8] L. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. 2020.
[9] M. Andrychowicz, Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. arXiv preprint arXiv: 1rXiv:1707.01495 2017.
[10] S. Feng, X. Xinjilefu, C. G. Atkeson, and J. Kim. Optimization based controller design and implementation for the atlas robot in the darpa robotics challenge finals. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages (028-1035. 2015.
[11] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. M. O. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wier- stra. Continuous control with deep reinforcement learning. CoRR, 0bbs/1509.025 2016.
[12] H.-Y. F. Tung, R. Cheng, and K. Fragkiadaki. Learning spatial common sense with geometry- aware recurrent networks. Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[13] G. Neumann, C. Daniel, A. Paraschos, A. Kupesik, and J. Peters. Learning modular policies for robotics. Frontiers in computational neuroscience, 8:62, 2014.
[14] B. D. Silva, G. Konidaris, and A. Barto. Learning parameterized skills, 2012.
[15] E. Rückert and A. d' Avella. Learned parametrized dynamic movement primitives with shared synergies for controlling robotic and musculoskeletal systems. Frontiers in computational neuroscience, 2013.
[16] S. Schaal. Dynamic movement primitives-a framework for motor control in humans and hu- manoid robotics. In Adaptive motion of animals and machines, pages 261-280. Springer, 2006.
[17] Stulp, E. Theodorou, and S. Schaal. Reinforcement learning with sequences of motion primitives for robust manipulation. Robotics, IEEE Transactions on, 1oi:10.1109/TRC 20t.//RR...01..21299..
12 2012.
[18] O. Kroemer and G. S. Sukhatme. Learning relevant features for fo skills using meta- level priors. CORR,
2016. URL http://aaww....rr/a///6666.. 04439.
[19] R. Strudel, A. Pashevich, I. Kalevatykh, Laptev, J. Sivie, and Schmid. Learning to combine primitive skills: A step towards versatile robotic manipulation, 2019.

---

[20] O. Kiline, Y. Hu, and G. Montana. Reinforcement learning for robotic manipulation using simulated locomotion demonstrations, 2019.
[21] N. Wahlström, T. B. Schön, and M. P. Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXivv/552.0222: 2015.
[22] N. Ratliff, J. A. Bagnell, and S. S. Srinivasa. Imitation learning for locomotion and manipula- tion. In 2007 7th IEEE-RAS International Conference Humanoid Robots, pages 392-397, 2007.
[23] M. Sieb, X. Zhou, A. Huang, O. Kroemer, and K. Fragkiadaki. Graph-structured visual imita- tion. In Conference on Robot Learning (CoRL), 2019.
[24] L. Fan, Y. Zhu, J. Zhu, Z. Liu, O. Zeng, A. Gupta, J. Surreal: Open-source reinforcement learning framework and robot manipulation benchmark. In Conference on Robot Learning, pages 767-782. PMLR, 2018.
S. Savarese, and L. Fei-Fei.
[25] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint arXiv: 10087, 2017.
[26] OpenAI, I. Akkaya, M. Andrychowicz. M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's s cube with robot hand, 2019.
[27] Mahler, J. Liang. S. Niyaz, M. Laskey, R. Doan, X. Liu, J. Aparicio, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metries. 07 2017. doi: 100:10.11 15677/RSS.SSS.. 2017.XIII.055
[28] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. CoRR, abs/ (bbs/1905.1050. 2019, URL http://arxiv..ortt orgg
10520.
[29] Y. Qin, R. Chen, H. Zhu, M. Song, J. Xu, and H. Su. S4g: Amodal single- view single-shot se(3) grasp detection in cluttered scenes, 2019.
[30] L. Pinto and Gupta. Supersizing supervsip: Learning to grasp from 50k tries and 700 robot hours. CoRR,
2015. URL http://arxivy org/ abbs/1599. 06825.
[31] P. Agrawal, A. Nair, P. Abbeel, J. Malik, and S. Levine. Learning poke by poking: Experi- ential learning of intuitive physics. arXiv preprint
07911. 2016.
[32] T. Tosun, E. Mitchell, B. Eisner, J. Huh, B. Lee, D. Lee, V. Isler, H. S. Seung, and D. Lee. Pixels to plans: Learning non-prehensile manipulation by imitating planner. arXiv preprint WrXiv:194.00300 2019.
[33] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Wiele, V. Mnih, N. Heess, and J. T. Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference on Machine Learning, pages 4344-4353. PMLR, 2018.
[34] L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542. 2017.
[35] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-oo to-canonical adaptation networks. In CVPR, pages 12627-12637, 2019.
[36] P. Florence, L. Manuelli, and R. Tedrake. sulf-supe correspondence in visuomotor policy learning, 2019.
[37] T. Kulkarni, K. R. Narasimhan, A. Saeedi, and J. B. Tenenbaum. Hierarchical deep rein- forcement learning: Integrating temporal abstraction and intrinsic motivation. arXiv preprint arXiv:1604.0 06057, 2016.
[38] J. T. Springenberg, K. Hausman, M. Riedmiller, N. Heess, and Z. Wang. Learning an embed- ding space for transferable robot skills. 2018
[39] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick. Mask R-CNN. CoRR, abs/1 703.06870, 2017. URL http:://arxivv org/ aa/aabbs//00 06870.
10

---

[40] A. W. Harley, S. K. Lakshmikanth, Li, X. Zhou, H.-Y. F. Tung, and K. Fragkiadaki. Learning from unlabelled videos using contrastive predictive neural 3D mapping. ICLR, 2020.
[41] E. Todorov, T. Frez, and Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012.
[42] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1r
2015.
[43] Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, pages 1-8. IEEE, 2018.
[44] S. Ross, G. J. Gordon, and J. A. Bagnell. No-regret reductions for imitation learning and structured prediction. In In AISTATS. Citeseer, 2011.
[45] A. W. Harley, Li, K. Lakshmikanth, X. Zhou, H.-Y. F. Tung, and Fragkiadaki. Embod- ied view-contrastive 3d feature learning. arXiv preprint arXiv:/906.000 2019.
[46] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recog- nition. arXiv preprini arXiv: 1409. 1556, 2014.
[47] K. Zhang, M. Sharma, J. Liang, and O. Kroemer. A modular robotic arm control stack for research: Franka-interface and frankapy. arXiv preprint urXiv:201 2020.
[48] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. CoRR, abs/1603.02199, 2016. URL http:/ //////axxii /arxivavvv.../aa/ orggabs//603.. 0b99//00.160011999.


Reviews:
[{"review_id": "g-t6aQxUjNV", "reviewer": "Reviewer_7MZX", "summary": "The paper proposes a two stage hierarchical approach for learning to manipulate objects in a way that generalizes across objects, configurations, and views. The lower level consists of a library of learned or engineered behaviors that can handle a subset of these variations. These behaviors then become discrete actions of a higher level policy, which operates on an estimated 3D scene representation, transformed into the robot frame. The scene representation is then cropped and mapped onto a latent vector representation that is jointly learned with a vector representation for each behavior. At inference time, the behavior is chosen based on the similarity of these vectors. The method is evaluated in simulated and on a real robot in grasping, pushing, and transport tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method combines a variety of deep learning machinery and models in a structured way. The idea of having a two-stage approach to generating behavior is not new but the way how this is done here is very original and there are many ideas in there that make a lot of sense, e.g. jointly training the input and behavior representations, making generalization over viewpoints explicit via self-supervised objectives and via estimating the scene representation in 3D and transforming it into the robot frame.\n\nThis already good paper could be further improved by extending the experimental section with qualitative results that visualize the components of the system. How good are the learned 3D representations in sim and in real? What does the embedding space of the behaviors look like? What are the typical failure cases of the proposed method?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a nice paper that includes a lot of relevant ideas that should be interesting to the field. The paper appears correct and significant to me. It includes experiments in simulation and on a real robot. The experimental evaluation could be improved with additional analysis (see above).\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "If possible, additional evaluations as discussed above would strengthen the paper.\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "_P41A6VhHs", "reviewer": "Reviewer_XqcZ", "summary": "\nWithin manipulation tasks (this work focuses on grasping and adjacent skills), different behaviors are needed to generalize to different kinds of objects and their various orientations - for instance, pinching a container by its lip is specific to a certain family of objects but not others. While in principle it should be possible to train a single pixel-to-actions, goal-conditioned policy to subsume all these kinds of perceptual and control generalization, the authors opt to push the challenge of generalization to the perception stack only. Given some perceived object representation, and a fixed set of open-loop grasping “scripts” with different angles and grasp elevation angles, select the script most likely to succeed at the task. Concretely, RGB-D image goes into a GRNN, which creates a view-invariant 3D scene representation. The object-centric features are “cropped” from the 3D scene representation, which are the same feature space as the retrieval policies (64x64x64x32). The alignment between the object centric 3D representation and policy-specific key results in a fixed open loop pattern being played out. \n\nAn HRL interpretation of this work is that it’s an options policy for a 1-step MDP that learns which of N pre-defined goal-conditioned behaviors to select. The authors show that decoupling perception and planning in this manner leads to better performance compared to an end-to-end perception and policy, and also that their choice of object +behavior representation is superior to 2D images, 3D object locations, and 3D scene feature maps. Generalization to novel objects, novel viewpoints, and object poses is demonstrated on a real robot. \n\nThis is essentially an extension of Redmon and Angelova 2014 to 3D space and multiple viewpoints. \n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths\nPolicy is able to generalize to new objects and poses, from different viewpoints, on tasks that would be quite challenging for an end-to-end system to learn\nAuthors demonstrate that alternate object representations do not generalize as well to varying viewpoints and object configurations.\n\nWeaknesses\nAlthough the authors demonstrate that their choice of feature representation generalizes better than a naive 2D image representation, the downstream behaviors are very simple, scripted behaviors that are hand-tuned such that succeeding at the task means classifying one of K classes correctly. \nThe “library” is not really a library of different skills, but rather one or two behaviors with a discretized set of yaw and grasp elevation parameters, which drastically limits the effective DoF of behaviors to small, discrete action spaces. As mentioned in the summary, I believe the closest relevant work to this is not multi-task or options or even skills learning, but just visual grasp point detection generalized to 3D, as the learned “classifier” is just discriminating between candidate grasp poses.   \nState-of-the-art closed-loop grasping behavior from 2D images is already able to generalize to multi-view (Sadeghi et al 2017), novel objects and orientations (Kalashnikov '18). Given that the proposed work is essentially an open-loop grasping system (much, much easier and less general in regards to the skills that can be learned), I would have liked to see stronger generalization capabilities in the library.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The overall framework is quite interesting - pushing the problem of generalization as much as possible into perception + a standard classification problem (embedding alignment), and one that can be learned via hindsight data collection with a library of skills. I would have voted to accept this if this framework were applied to a more challenging library of skills, but instead as applied, there is *no* library skill learning on the real robot - it is merely just applying a classifier to a variety of scripted motions (with specific grasp angles and heights such that the classifier merely needs to pick the right grasp pose). The choice of the term \"selector\" sort of obfuscates this, but at the end of the day, it's really just a classifier over a set of memorized trajectories. I think the \"selector\" terminology would be more appropriate if the system were able to generalize in that \"key\" space, i.e. select new skills from embeddings that were not used at training time.\n\nThere is some novelty in the investigation of which 3D object representations computed from RGB-D are most ideal for predicting the \"options policy\", ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "I'd like the authors to comment on how this proposed framework is any different from simply parameterizing a K-way classifier for memorized trajectories, where the classifier is parameterized using a dot product on a learned object representation layer (but otherwise equivalent to a classifier).\n\n"}}, {"review_id": "KRWCQG9wuB", "reviewer": "Reviewer_GybB", "summary": "The main contribution of this paper is a novel pipeline for learning grasping and pushing actions that integrates both raw sensory data as well as more abstract object features. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe main strength of the paper in my opinion are the results with the real robot (personally, I would focused mostly on these as opposed to the results in simulation)\n\nThe main weakness of the paper is the comparison against a weak baseline, almost designed to fail. There clearly are end-to-end systems designed for grasping and pushing that in theory should be able to achieve better results then the baselines (though I understand that training such systems is an art in itself and how hypermarkets are decided can have big impact on the results). The other main weakness of the paper is the lack of many details without which it is a bit difficult to judge the level of contribution (see issues later in the review). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper brings in some new ideas and the evaluation with a real robot is great. There are still many issue to be addressed and if there was a \"borderline\" option between weak accept and weak reject, I would have picked that one. \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "\nIn section 3, the 3D bounding box would work for relatively simple objects in which the bounding box would have little else but the object inside of it; what about more complicated objects? Also, is the bounding box aligned with the principal axis of the object if there is one?\n\nIn section 3.1 when describing the training, is the randomly chosen behavior still condition on the goal for the particular trial? Also, at this point in the paper it is not clear how do you deal with behavior parametrizations, e.g., push can be parametrized with force, direction, etc?\n\nWith regards to the 3D object detector, is this learned for each individual object or do you learn a detector that works on all objects, even ones not seen at training? Also, were the test objects in the experiments excluded from the training data for the object detector?\n\nSection 3.2, line 193, why not also change in orientation of the end effector when it comes to the action space? Currently it is stated that it can only change its position (translation) but not orientation. Ideally, the dimensionality of the action space should be stated explicitly. \n\nSection 3.2, line 195, needs more detail here -- what is the difference between the 25? just the goal? also, how are the objects split into train and test and why is one behavior trained on all objects?\n\nSimilarly, for grasping, it not at all clear where these 30 grasping controllers come from and also it is not clear how the grasping point relative to the center of the object is actually selected (or who selects it)\n\nIn section 4.1, there needs to be more detail on the simulation environment, in particular, how does it model things like texture, friction, etc.? Are all the objects assumed to have the same material and density and only the shape is different?\n\nWith regards to object augmentation in line 228, does this mean that on average, from each original training object, you get 3 augmented objects and configurations?\n\nWith regards to the sentence, \"An episode is successful if the object centroid is within 5cm of the target at the final timestep.\", how is it decided when an episode ends, is there a fixed limit?\n\nWith regards to how objects are split into train and test, do the categories play a role e.g., do you ensure that the test set only has objects from novel categories or do you ensure that for each category, some objects are in train and some in test?\n\nFor the grasping pick and place task, it would be nice to have a comparison of a traditional approach, like Agile grasp or some other classifier used for point selection and then planning to move it to a target location -- at least, what are your thoughts on how your system would compare against a more traditional approach? The current baseline comparisons are OK but in some ways, it feels like there were selected because they were doomed to fail (there is also the whole issue of how hyperparamters are selected for your approach vs baselines and without any principled methodology for doing so, I can only assume that the graduate students spent any time tuning any parameters mostly for the proposed approach)\n\n\nsome minor issues:\n\nfigure 3 could use a longer caption\n\nline 63 of the intro and at other places talk about the object being \"diverse\" but it is not explained, diverse with respect to what? all the objects seem mostly rigid, simple shapes so what are they diverse in terms of?\n\nline 7 in the abstract, the phrase \"in hand,\" is a bit confusing as the object is not always in he robot's hand\n\nThe paper often puts \"object location and pose\" next to each other, but pose already includes location information so it's a bit redundant. \n"}}, {"review_id": "4nLRiO7Kous", "reviewer": "Reviewer_UjV5", "summary": "This works re-factors the problem of manipulation of diverse objects to building a diverse set of policies and building a data-driven policy selector. They show significantly better results on unsee objects than baselines without such a re-factor. Ablations show the importance of the view-predictive and affordance-aware representations", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Approach seems very data efficient and seems to generalize to new objects much better than the baselines.\n* This approach may include a variety of policy types: learned or rule based.\n\nWeakness:\n* There is no baseline number reported for the real robot result, it seems like a challenging set of tasks and some baseline here would be more compelling.\n* This work advocates for building a library of behaviors including new controllers, it would been nice to see some discussion around the trade-off between building these libraries versus just collecting more data for the single-policy approach.\n* The interaction label process seems tedious, are there any heuristics used? For example, don't use pick and place if the task requires a pour. Are the baselines also retrained with these interaction label data in some way.\n* It is surprising that the baselines perform so poorly in Table1. What are the success rates on the train objects?\n* Were there any cases where this approach can tell you if you are missing a behavior to tackle the task?\n* How were the real robot controllers designed? Did you have knowledge of the test objects during that design?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In some limited settings, this approach seems very efficient for manipulation tasks, however, it is not clear how generalizable the result is. Stronger baselines on real robot results would have made this work more compelling. Additionally, some discussion of the trade-offs of having to develop and maintain libraries of policies and the additional cost of the interaction labeling versus single-policy approach with more data. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* Include a baseline for the real robot result.\n* Some discussion/analysis of why the sim baselines in Table 1 are so poor\n* Discussion about when no appropriate behavior is in the library, does the selector give you a signal about this?"}}]

Author rebuttal:

Meta-review: Strengths:

- The view invariance of the proposed method (a result of the use of GRNNs) is a key strength and seems to generate over views better than the baselines.

- The high level structure of the approach where the perceptual module and the behavior keys are trained jointly was viewed as a strength. The fact that the low level policies could be learned or rule-based was also viewed as a strength.

Weaknesses:

- Two reviewers asked for stronger baselines. This is in reference to Table 1 where the proposed method is evaluated on a class of grasping tasks and a class of pushing tasks. The baseline methods perform terribly on these tasks and on grasping in particular. However, there are lots of grasp control/detection methods available now that can do very well with grasping, especially in the single-object scenario that we consider here. There are also a variety of push control methods that can similarly do well. 

- There was a concern that the fixed set of behaviors made the approach overly simplistic. Essentially, in the real robot exps at least, the agent looks at the scene and then selects one of a fixed set of motion scripts. Joint learning of behavior policies could help here. Another approach might be to interpolate amongst learned skills at test time.

- Multiple reviewers asked for more comprehensive real robot exps.

Post Rebuttal:

The authors added a significant amount of content and comparisons into the appendix (supplemental), including grasp comparisons which we appreciate, as well as the t-sne plot which was cool. However, if this work is viewed through the lens of "k-way" classifier over pre-scripted behaviors, then (as the authors point out) the remaining contribution is the space in which the scene is represented. Possibly this is really prior work rather than a new contribution here. Nevertheless, the application to manipulation goes beyond prior work and was viewed positively.

Decision: Accept (Poster)