VIOLA: Object-Centric Imitation Learning for Vision-Based Robot Manipulation
Authors: Yifeng Zhu
Abhishek Joshi
Peter Stone
Yuke Zhu
Venue: CoRL 2022 Poster (2022)
Keywords: Imitation Learning
Robot Manipulation
Object-Centric Representation

Abstract: We introduce VIOLA, an object-centric imitation learning approach to learning closed-loop visuomotor policies for robot manipulation. Our approach constructs object-centric representations based on general object proposals from a pre-trained vision model. VIOLA uses a transformer-based policy to reason over these representations and attend to the task-relevant visual factors for action prediction. Such object-based structural priors improve deep imitation learning algorithm's robustness against object variations and environmental perturbations. We quantitatively evaluate VIOLA in simulation and on real robots. VIOLA outperforms the state-of-the-art imitation learning methods by $45.8%$ in success rate. It has also been deployed successfully on a physical robot to solve challenging long-horizon tasks, such as dining table arrangement and coffee making. More videos and model details can be found in supplementary material and the project website:  https://ut-austin-rpl.github.io/VIOLA/.

VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors
Yifeng Zhu', Abhishek Joshi', Peter Stone Yuke Zhu!
1 The University of Texas at Austin Sony AI
Abstract: We introduce VIOLA, an object-centric imitation learning approach to learning closed-loop visuomotor policies for robot manipulation. Our approach constructs object-centric representations based on general object proposals from a pre-trained vision model. VIOLAA uses a transformer-based policy to reason over these representations and attend to the task-relevant visual factors for action prediction. Such object-based structural priors improve deep imitation learning algorithm's 's robustness against object variations and environmental perturbations. We quantitatively evaluate VIOLA in simulation and on real robots. VIOLA out- performs the sate-of-the-art imitation learning methods by 45.8% in success rate. It has also been deployed successfully on a physical robot to solve challenging long mong-horizon tasks, such dining table arrangement and coffee making. More videos and model details can be found in supplementary material and the project website: https://ut-austin-ra wwt.usiiipp..giuuu.oo////....... github.io/VIOLA.
Keywords: Imitation Learning, Manipulation, Object-Centric Representations
Introduction
Vision-basse manipulation is a critical ability for autonomous robots to interact with everyday en- vironments. requires the robots to understand the unstructured world through visual perception to determine intelligent behaviors. In recent years, deep imitation learning (IL) [1-4] has emerged as a powerful approach to training visuomotor policies on diverse offline data, particularly human demonstrations. Its success stems from the effectiveness of training over- parameterized neural net- works end-to-end with supervised learning objectives. These models excel at mapping raw visual observations to motor actions without manual engineering. While deep IL methods often distin- guish themselves from reinforcement learning counterparts in their scalability to long-horizon tasks, burgeoning body of recent work pointed out that II. methods lack robustness to covariate shifts and environmental perturbations [5-11]. End-to-end visuomotor policies are likely to falsely associate actions with task -ireeeeeevant visual factors, leading poor generalization in new situations.
In this work, we endow imitation learning algorithms with awareness about objects and their inter- actions to improve their efficacy and robustness in vision-based manipulation tasks. As cognitive science studies suggest, explaining a visual scene as objects and their interactions facilitates humans learn fast and make accurate predictions [12-14]. Inspired by these findings, we hypothesize that decomposing visual scene into factorized representations of objects in the scenes would enable robots reason about the manipulation workspace in a modular fashion and improve their general- ization ability. To this end, we develop an object-centric imitation learning approach, which infuses structural object- abject-based based priors into the model architecture of visuomotor policies. Training policies with these priors would make it easier for the model to focus on task-relevant visual cues while discarding spurious dependencies.
The first and foremost challenge of such an object-centric approach is to determine what consti- tutes an object and how objects are represented. The definitions of objects are often fluid and desendepnndent for manipulation tasks. This work studies the notions of objects operationally and considers them as disentangled visual concepts that inform the robot's s doecision-making works have explored learning visuomotor policies with awareness of objects, but they are limited to simple control domain [6], single object manipulation [15], or require costly annotations for object detection [16]. We are motivated by the recent advances in visual recognition, in particular, image models for generating object proposals [17, 18), iee., localized bounding boxes on 2D images. These object proposals capture general priors of "objectness"" across appearance variations and object cate-
Previous
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Rebot
Commande
sisueil observation
Objeci-cantic rep ressenntation
Mult--weed acll-attentinn
Action generation
object proposels
Task-relevant
Task-inalevart
Figure I: VIOLA first obtains a set of general object proposals from raw visual observations. It extracts object features from the proposals to build the object-centric representation. The transformer- based policy uses multi- head self- to reason the representation and identify task-relevant regions for action generation.
gories. They have served as intermediate representations for downstream vision tasks, such as object detection and instance segmentation [19-21]. In this work, we investigate using object proposals from a pre-trate te-irrined vision models as object-centric priors for visuomotor policy in manipulation and use the object proposals as starting point to build our object-centric representations.
We introduce VIOLA (Visuomotor Imitation via Object-centric LeArinng,, an object-centric imita- tion learning model to train closed-loop visuomotor policies for robot manipulation. The high-level overview of the method is illustrated in Figure 1. VIOLA first uses pre-trained Region Proposal Network (RPN) [18] to get a set of general object proposals from raw visual observations. We ex- tract features from each proposal region to build the factorized object-centric representations of the visual scene. These object-centric representations are converted into a set of discrete tokens and processed by Transformer encoder [22]. The transformer encoder learns to focus task-relevant regions while ignoring the irrelevant visual factors for decision-making through the multi-head self- attention mechanism when trained on supervised imitation learning objectives.
We compare VIOLA against state-of- state-of-the-are fff-the-rt art deep imitation learning methods for vision-based ma- nipulation in simulation and on real robot. We use simulation systematically evaluate the poli- cies' performances and generalization abilities in the canonical setting (i.e., testing in the training distribution) and under three challenging variations, including initial object placements, presence of distracting objects, and camera pose perturbations. Our quantitative evaluations show that VIOLA outperforms the most competitive baseline by 45.8% in success rate. When visual variations such as jittered camera views introduced, VIOLA maintains its robust behaviors of precise grasping and manipulation, while end-to-end learning methods would fail to reach the target objects. VIOLA also produces visuomotor policies to solve three challenging real-world tasks with a small set of 50 demonstrations, including a multi-stage coffee-making task where VIOLA achieves 60% success rate while baseline methods fail entirely.
Our contributions with VIOLA are three-fold: 1) We learn object-centric representations based on general object proposals and design transformer-based policy that determines task-relevant proposals to generate the robot's actions; 2) We show that VIOLA outperforms state-of-the-art baselines in simulation and validate the effectiveness of our model designs through ablative studies; and 3) We show that VIOLA learns policies on real robot to complete challenging tasks.
2 Related Work
Imitation Learning (IL.) for Manipulation. IL has been an established paradigm for acquiring manipulation policies for decades. It can be roughly categorized into non-parametric and parametric approaches. Non-parametric approaches, such as DMP and PrMP, can effectively acquire manipula- tion behaviors through a small number of demonstrations [23-26). However, they typically focus on open-loop trajectory generation and fall short in handling high-dimensional observations. Paramet- ric approaches, especially neural networks, have shown promise in vision-based manipulation. Nev- ertheless, these approaches are susceptible distributional shifts and observation noises [1-4, 27]. Object-centric priors have been previously explored in imitation learning policies to overcome the issues above [15, 16]. However, these previous works either focus on the manipulation of single object instances, or requires costly annotations for pre-training object detections. Based on the same conceptual idea as previous object-centric imitation learning, VIOLA uses pre-trained RPN to introduce object proposals as object-centric priors into the end-to-end IL policies, thus improving their robustness towards visual variations and solving tasks that involve complicated interaction with multiple objects.

---

Visual Representations for Visuomotor Policies. Various types of intermediate visual rep- resentations have been explored for visuomotor policy learning. Object bounding boxes have been commonly used as intermediate visual representations [28, 16, 29]. However, they require fine-tuned or category-specífic detectors and cannot easily generalize to tasks with previously unknown objects. Recently, deep learning methods have enabled II. to train policies end-to-end on raw observations [2, 3]. These methods are prone to covariate shift and causal confusion [6], resulting in poor generalization performances. Similar to our work, a large body of literature has looked into incorporating additional inductive biases into end-to-end policies. Notable ones include spatial attention [30-32] and affordances [33-38]. However, these representations are purposefully designed for specific motion primitives, such as -nd--lace,, limiting their abilities to generate diverse manipulation behaviors.
Object-Centric Representation. Object-centric representations have been widely used in visual understanding and robotics tasks, where researchers seek to reason about visual scenes in a modular
way based on the objects presented. In robotics, poses [39-4]] or bounding boxes [28, 16, 29] are commonly used as object-level abstractions. These representations often require prior knowledge about object instance/category and do not capture fine-grained details, falling short in applying to new tasks without previous unknown objects. Unsupervised object discovery methods [42, 43] learn object representation without manual supervision. However, they fall short in handling complex scenes [42, 43], hindering the applicability to realistic manipulation tasks. Recent work from the vision community has made significant progress in generating object proposals for various down- stream tasks, such object detection [19-21] and visual-language reasoning [44, 45]. Motivated by the effectiveness of region proposal networks (RPNs) on out-of- distiitiibttion images [21], we use object proposals scaffold our object-centric representations for robot manipulation tasks.
3 Approach
We introduce VIOLA, an object-centric imitation learning approach vision-based manipulation. The core idea is to decompose raw visual observations into object-centric representations, on top of which the policy generates closed-loop control actions. Figure 2 illustrates the pipeline. We first formulate the problem of visuomotor policy learning and describe two key components of VIOLA: 1) how we build the object-centric representations based on general object proposals, and 2) how we use transformer-basee architecture to learn policy over the object-centric representations.
3.1 Problem Formulation
We formulate a robot manipulation task as a discrete-time Markov Decision Process, which is de- fined as a 5-tuple M (8, A, P. R. 7). where S is the state space, A is the action space, P(-|s, a) is the stochastic transition probability, Ris, a, 5') is the reward function, and E [0, 1) is the dis- count factor. In our context, S is the space of the robot's raw sensory data, including RGB images and proprioception, AA is the space of the robot's 's motor commands, and π S - A is closed- loop sensorimotor policy that we deploy on the robot to perform a task. The goal of learning a visuomotor policy for robot manipulation is to learn policy π that maximizes the expected return BIE:=0 R(s,, de; st+1)]-
In our work, we use behavioral cloning our imitation learning algorithm. We assume access to set of N demonstrations D = { where each trajectory т, is demonstrated through telcopera- tion. The goal of our behavioral cloning approach is to learn policy that clones the actions from demonstrations D.
We aim to design an object-centric representation that factorizes the visual observations of un- structured scene into features of individual entities. For vision-based manipulation, we assume no access to the ground-truth states of objects. Instead, we use the top К object proposals from a pre-trained Region Proposal Network (RPN) [18] to represent the set of object-related regions of an image. These proposals are grounded on image regions and optimized for covering the bounding boxes of potential objects. We treat these K proposals as the K (approximate) objects and extract visual and positional features to build region features from each proposal. For manipulation tasks, reasoning about object interactions is also essential to decide actions, To provide contextual informa- tion for such relational reasoning, we design three context feutures: global context feature from the workspace image to encode the current stage of the task; an eye-in-hand visual feature from the eye- in-hand camera to mitigate occlusion and partial observability of objects; and a proprioceptive fea- ture from the robot's is states. We call the set of region features and context features at time step t as the
3

---

Objoc-Cantr Representatin
VIOLA
Feature Encoder
lhe H
Sinusoidal Encoding
Context fealures
Adcition
Ragion features
Concateniatin
GMM
Temporal Postion
Transformer
Feature Encoder
h
Oe
St
Task Space Command
H+1 Steps
Observations
Action Token
Feature Encoder
Bounding Box Positore
Region Features
Top K P Pooossss
Feature Paoling
RPN
Spatial Feature Map
Spatial Schmax
Workspaon Image
Eye hrnd Imagee
ReaNett-10
Propriccsption
linser
Projection
Features
Figure 2: Model architecture of VIOLA. At time f, computes the per-step features h, using the top K object proposals. Then it constructs the object-centric representation by composing per-step features from the last H++ time steps with their temporal positional encodings. Transformer-hased policy reasons over output latent vector the action token, â. which is passed through multi- layee- perceptron (MLP) generate actions.
per-step feature h4 Finally, our object-centric representation 2 is a composition of per-step features from the last H 1 time steps with their temporal positional encoding, describe in Sec. 3.2.
3.2 Object-Centric Representation
This section describes how to build the object-centrie representation 2, VIOLA first obtains general object proposals using pre-trained RPN. VIOLA computes region features from proposals and ob- tains a per-step feature h, using the region features and the three context features which we describe next. VIOLA builds 2, through the temporal composition he from a history of observations.
General Object Proposals. At each time step I, we generate object proposals using pre-trained RPN on the workspace image. We select the top K proposals with the highest confidence scores, which indicate regions that contain objects with the highest likelihood. The intuition of using pre- trained RPN is that it captures good priors of "objectness" in RGB images through the supervision of natural image datasets. Recent work shows that an RPN trained on an 80-class dataset can generalize 2000-class dataset [21]. Our preliminary studies suggested that such a generalization ability also holds for localizing objects on raw images in our simulation and real-world tasks regardless of the domain gaps. We use the pre-trained RPN from Zhou al. [21] localize regions with objects.
Region Features. For our policy to reason over objects and their spatial relations, we need to iden- tify what objects each region contains and where these regions are from the top K proposals. To encode this information, we design a visual feature and a positional feature for each region. To extract the visual feature from region, we learn spatial feature map by encoding the workspace image with the ResNet-18 18 module [46] and extract the visual feature using ROI Align [17]. We use learned spatial feature map for visual features rather than from a pre-trained feature pyramid network in detection models because we share the same objective of localizing objects as object detectors but different goals for the downstream tasks - pre-trained feature pyramid networks are optimized for visual recognition tasks, but we need actionable visual features that are informative for continuous control. Such design choice supported by our ablation experiment in Appendix B.2 For positional features, encode the coordinates of bounding box corners images using sinu-

---

soidal positional encoding [22] (more details in Appendix A). We flatten each visual feature and add it to the positional feature of the same region to obtain a region feature.
Context Features. We extract the region features for the policies to reason over individual objects.
However, they are insufficient for decision-making in vision-based manipulation tasks, so we intro- duce three context features to assist decision-making. As regions only encode local information, we add global context feature to encode the current task stage from observation, which is computed from the spatial feature map of the workspace image using Spatial Softmax. During manipulation, the robot's gripper often occludes objects in the workspace view, so we add an eye-in-hand feature from eye-in-hand images to represent the occluded objects. We also encode the robot's 's measure- ments of its joints and the gripper into proprioceptive features for the policy to generate precise actions based on the robot's states. We aggregate the context and region features at each time step t into a set, which we refer to as per-step feature h4-
Temporal Composition. We build object-centric representation 2t through the temporal compo- sition of he that captures temporal dependencies and dynamic changes of object states. Building policies over a sequence of past observations rather than the most recent observation has been shown effective by prior work [1. 27]. In our work, the temporal composition also increases the recall rate of object proposals, making the policy more robust in the case of detection failures, More specifi- cally, 2, built from a set of per-step features { from the last H + 1 time steps. To encode add sinusoidal position encoding of the temporal positions {PE.) ==0 to the per-step features, resulting in our object-centric representation 2; { @ PE Our ablative studies (see Sec. 4) indicate the importance of temporal positional encoding that retains the temporal ordering of features, especially when using our transformer-bis based policy.
their temporal ordering,
3.3 Transformer-based Policy
We desire a policy that focuses on task-relevant region features in Z₁ generate actions. Regions that associate with task-relevant objects facilitate the accurate prediction of actions, while regions with task-irrelevant objects are likely to confound the policies. We seek to use transformer [22] as the policy backbone, which allows policies to reason over objects and their relations with its self- attention mechanism. The core of a transformer an encoder layer which consists of a multi-head self-attention block (MHSA), a normaliaoali [47] function, and a cully-connected neural net- work (FFN). transformer encoder layer takes as input sequence of n latent vectors (zi...., 2x) (also called tokens) and outputs sequence of n latent vectors (y1..... Bn)) A MHSA block con- sists of multiple self-attention blocks in parallel, which computes attention over all the tokens and computes weighted sum of token values. We refer readers to Vaswani et al. for more details [22]. In short, each transformer encoder outputs = FFF([aayrNNrmmMMS) (MSHA( (X)), where each row of Y is output latent vectors y that corresponds to z,. Our transformer-baat based policy is stack of multiple transformer encoder layers, which allows for a higher degree of expressiveness over input tokens compared to single layer.
For our policy, we tokenize our object-centric representation 2t, treating each region and context feature vector as an input token. To make action generation attend more to task-relevant region features than task-irrelevant ones, we append a Iearnable token, action token, to the input sequence of a transformer. The action token design resembles the specific class tokens in natural language understanding tasks [48] or visual recognition tasks [49], which are used for outputting latent vectors for downstream tasks. Similarly, we can get the output latent vector å, from the action token, which learns to attend to task-relevant regions through action supervision. In the end, we pass à through a two-layered fully fully-connected network, followed by a GMM (Gaussian Mixture Model) output head, which has been shown effective to capture the diverse multimodal behaviors in demonstration data [50, 27].
Implementation Details. Here we describe several key implementation details. Full implementa- tion details in Appendix A. We set K 20 in simulation, which offers a nice trade-off between object recall rates and computation efficiency (see Appendix B).. In the real world, we set K = 15 to attain high recall rate on real images. For temporal composition, we use H = 9 be consistent with the setup in BC-RNN [27]. The action token is a learnable parameter randomly initialized from the normal distribution.
We use color augmentation and pixel shifting [4, 27, 49] during training to encourage the gen- eralization ability of policies. Additionally, we adopt random erasing [51] to prevent policies from overfitting to specific region features. Concretely, we randomly apply random crasing during
5

---

Initial State
Goal State
Initia State
Goal State
Intiaa State
Goal State
(a) Serting
(h) Stacking
(c) BUDS-Kitchen
Goel
Initial State
Goal
State
(d) Mhing-PPateeoor"
(e) Blning-BBwl
(f) Nake-
Figure 3: Visualization of initial and goal configurations for the simulation and real-worke tasks.
Variunts
[54]
[6]
BC-RNN [27]
VIOLA-Patee
VIOLA
Sorting
Canonical
62.8
1.0
87.6 1.1
Placement
1.9 0.7
8.3
11.7 + 1.0
48.5 + 2.2
68.3 + 1.5
Distractor
14,5 + 1.9
26.0# 11.4
46.7 4.5
58.6 4.2
74.4 5.7
Camera-Jitter
Stacking
Canonical
14.9 +
1.
0.8
71.2 1.0
71.3 1.0
Placement
1.6 0.3
0.34 0.5
0.0 0.0
28.4 1.9
46.7 + 0.2
Distractor
5.6 1.4
5.6
144
41.4:51
38.6
Canera-Jitter
0.3
0.9
1.0
17.6
29.3 1.7
BUDS-Kitchen
Canonical
0.0
0.0
0.0
0.6
84.2
Placement
0.0
0.0
0.0
3.1
S8.4 +
Distractor
0.0
0.0
0.0
10.7
73.2 6.2
Camera-Jitter
0.0
0.0
0.0
2.
41.2 1.7
Table 1: Success rates (%) in simulation setups over 100 initializations with repeated runs 3 random seeds.
training with a probability of 0.5. Random erasing fills in Gaussian noise into a randomly selected region whose size is 2% 5% of the image with aspect ratio ranging from 0.5 to 1.5.
4 Experiments
We design our experiments to answer the following questions: 1) How well does VIOLA perform against state-of-the-ar end-to-end imitation learning algorithms? 2) How does it take advantage of object-centric representations? 3) What design choices are essential for good performance? and 4) Is VIOLA practical for real-world deployment?
4.1 Experimental Setup
We conduct quantitative evaluations in simulation and real-world tasks validate our approach. The simulation tasks are designed with the robosuite [52] framework and used for quantitative compar- isons between VIOLA and baselines. We also validate our design choices through ablative studies. We design three simulation tasks, Sorting, Stacking, BUDS-Ki Kitsen, and three real-world tasks, Dining-PlatoForke Dining-Bowl, Make-Coffer that cover a rich spectrum of manipulation be- haviors that combine prehensile and non-noon-eeeeeee prehenenee motions. We visualize their initial and goal con- figurations in Figure We design these tasks understand if the use of object priors benefits policy learning along two axes of task characteristics: large placement variations of objects and multi-stage long-horizon execution. For the first axis, we design Sorting and Stacking to have large initial ranges of object placements. For the second axis, we use BUDS-Kitchi Kitchen, multi- long- ong-horizon manipulation task from prior work [4]. The real-world tasks are designed to resemble real-world ev- and Dining-Bory Bowl for dining table arrangement, Make-Coffee
eryday tasks: Dining-PlateFon for espresso coffee making. Full details tasks and data collection are included in Sec. C.
Evaluation Setup. To systematically evaluate the efficacy and robustness of the learned policies, design the following testing setups in simulation: 1) Canoni all the objects and sensor configurations follow the same distribution as seen in demonstrations; and 2) Three testing variants, namely Placement, Distractor, and Samera-Jitter. The design of these variants is inspired by the MAGICAL benchmark [53] for evaluating systematic generalization of imitation learning policies. The design principle testing variants is to keep task semantics canonical setup. At the same time, we identify the three challenging axes of variations for learning- based manipulation, namely initial object placements (Placement), distractor objects presented in the scene (Distractor), and camera pose jitters (Camera-Jitter),
same as in

---

BC RMN
VOLL
Models
Canonical
Placesent
Distractor
Conera-Jitter
75
80.0
Base
0.0 0.0
0.0 00000.0
0.0 0.0
0.0 00:=00 0.0
4 Temporal Observation
69.5 + 2.5
1 +12
1.3.7 + 3,4
19,99 + 1.5
20.0
03
Temporal Positional Encoding
1
50.9 1.5
+ Region Visual Features
72.8 2: 0.9
1 + 1.2
1 54.6 4.6
49.2 4: 1.6
Region Positional Features
-00.2 + 2.9
0.36
122.0 5.7
46.5 + 1.9
Dinino -PI FateFork
Dining -Bome
Make-Coffee
Random Erasing (=VIOLA)
+87.6
18.3
74.4255.7
50.7:0066
Table 2:
The effect of VIOLA model designs on success rates (%) in Sorting task. Changes larger than 2% are annotated with f 4.
Figure Success rates (%) in real robot tasks.
4.2 Experiment Results.
We answer question (I) by quantitatively comparing VIOLA with the following baselines: BC: behavioral cloning that conditions on current observations; OREO [6]: behavioral cloning method that learns object-aware discrete codes using VQ-VAE [55] and learns policies thereon; BC- RNN [27, 56, 15, 57]: a state-of-the-art method that uses a temporal sequence of past observations with recurrent neural networks. VIOLA-Patch: variant of VIOLA, where we use regular grid of image patches as inputs to the policy rather than proposals, similar to ViT [58]. To make a fair evaluation, we randomly generate 100 initial configurations and repeat runs with three different random seeds. We evaluate all policies on the same set of pre-generated initial contigurations.
Table shows that VIOLA outperforms the most competitive baseline BC-RNN by 50.8% success rate in Canonical and 44.1% in three testing variants. VIOLA's strong performance suggests the advantage of object-centric representations for visuomotor imitation. OREO's results suggest that learning object-aware discrete codes via unsupervised learning does not consistently improve performance over simple BC for all tasks. The comparisons between BC, OREO, and the other methods using temporal windows show the importance of temporal modeling for these algorithms to achieve high performances and robustness in complex vision-based manipulation domains.
Table 1 shows that VIOLA-Patch has comparable performance VIOLA in some evaluation tups. It shows that the transformer backbone can attend to patch regions that cover task-relevant visual cues, even though the patch division is agnostic to objects. Nonetheless, VIOLA still per- forms better than VIOLA-Patch, especially in the long-horizon task BUDS-Kitches suggesting regions with complete coverage of objects is critical for the success of VIOLA, highlighting the importance of the object priors.
Ablative Studies. To answer questions (2) and (3), we use ablative studies to validate our model's design and show how takes advantage of the object-centric priors. Table 2 quantifies the effects of the transformer backbone, the use of temporal windows, object proposal regions, and our data augmentation technique. We start with our base model, a transformer model that only takes the current observations input. Second, we add the temporal window of observations similar to BC-RNN. Results show that the use of temporal observation is key to unleashing the power of the transformer architecture, making the performance of this ablated version comparable to BC-RNN. Nonetheless, this model does not encode temporal ordering as the transformer model is invariant to input permutation. In the next row, we add temporal positional encoding [22] the input sequence, which produces a method that outperforms the top baselines.
In the next two rows, we procedurally add visual and positional features of regions to prove that our model does exploit the object priors. Results show that visual features alone only improve our model's 's robustness to Distractor. When using both visual and positional features, the model performs better in Canonical and Distractor, but worse in Placement and Camera-Jitter. We hypothesize that this ablated version overfits to locations of proposal boxes demonstrations; therefore, it generalizes worse in Placement and Camera-
where the distribution of object locations on 2D images shifts from demonstrations. To mitigate such overfitting, we use Random Erasing [51] achieve the highest success rates in all evaluation setups.
Real Robot Evaluation. We compare VIOLA against the SOTA baseline, BC-RNN, on all three real-world tasks. We evaluate in 10 different initial configurations and repeat times in the same configuration for each policy evaluation. To mitigate potential human bias introduced by setting up objects before each trial, we employ the A/B testing paradigm for evaluation. That is, we reset the environment while being agnostic to the next policy to execute. The entire evaluation process for one initialization is repeated until all evaluation trials are completed.
7

---

Ctjjett Propocats
Hegione top Weights
Top-1
Top 2
Temporal
Woeght
Top 2
Top-3
Figure 5: Visualization of transformer attention over regions. When the robot is to grasp the k-cup, VIOLA'S 's top attention is over the k-cup across observations while it takes the robot fingers and coffee machine into account to help spatial reasoning. When the robot is to close the coffee machine, VIOLA's top-1 attention all over k-cups, the robot gripper, and the coffee machine, and it reasons over these regions to generate actions.
The quantitative evaluation in Figure 4 shows that VIOLA outperforms BC-RNN by 46.7% success rate on average. Qualitatively, we observe that VIOLA can robustly grasp k-cups or open the coffee machine in the Make-Coffes task, while BC-RNN tends to reach to wrong positions where grasps are missed or to release the gripper mistakenly. In the supplementary videos, we include the evaluation rollouts of VIOLA for all the tasks. We also qualitatively visualize the attention of VIOLA in Figure 5 by showing the top 3 regions with the highest attention weights at each timestep of temporal observation. The figure shows that when the robot is to grasp the k-cup, VIOLA's top attention is over the k-cup while it also takes the robot fingers and coffee machine into account to close the coffee machine, VIOLA's 's top-1 attention
facilitate spatial reasoning. When the robot spreads over the k-cup, robot gripper, and the coffee machine, and the policy generates actions for closing through spatial reasoning over these regions.
5 Conclusion
We present VIOLA, an object-centric imitation learning approach to learning closed-loop visuo- motor policies for robot manipulation. Our approach uses general object proposals to build the object-centric representation. This representation encodes the visual and positional features from the proposal regions and context features of global scene information and robot states. We design transformer-bast based policy to identify task-specific relevant regions for action generation. The results show the superior performances of VIOLA compared to state-of-the-art baselines in simulation and the real world. We also validate our model designs through ablative studies, showing how each model component impacts policy performance.
Limitations. While VIOLA has achieved great success in learning robust visuomotor policies, it suffers from common limitations of learning from offline demonstration datasets. One future direction is to use this object-centric representation in an online manner so that it can improve over roll-out experience over time. Also, we use a pre- rre-trained RPN without adaptation, which may fail to generalize to manipulation scenes with aggressive distributional shifts from natural images on which the RPN is trained. A possible remedy is to fine-tune the RPN while training the policy. Another limitation of our approach is that we only consider monocular RGB images in the input data, which lacks the 3D information of objects. This formulation prevents VIOLA from excluding the background visual elements in the input, which would limit the generalization ability of VIOLA in variants of large visual changes, such as table texture change. For future work, we would like to extend VIOLA to using depth images so that distracting background elements such as tables can be disentangled from the object representations.
Acknowledgments
The authors would like to thank Yue Zhao for the insightful discussion on the project and the manuscript. This work has taken place in the Robot Perception and Learning Group (RPL) and Learning Agents Research Group (LARG) at UT Austin. RPL research has been partially supported by the National Science Foundation 1ONS-1955523, FRR-2145283), the Office of Naval Research (N00014-22- --2044,, and the Amazon Research Awards. LARG research is supported in part by NSF
11S-172,,57, FAIN-2019844). ONR 2-0333), DARPA, GM, Bosch, and UT Austin's Good Systems grand challenge. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.
ARO (W91INF-19-
8

---

References
[1] A. Mandlekar, F. Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420. IEEE, 2020.
[2] A. Mandlekar, D. Xu, R. Martin-Marttn, S. Savarese, and L. Fei-Fei. Learning to general- ize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06085, 2020.
[3] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imita- tion learning for complex manipulation tasks from virtual reality teleoperation. In 2018 JEEE International Conference on Robotics and Automation (ICRA), pages 5628 5638. IEEE, 2018.
[4] Y. Zhu, P. Stone, and Y. Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters, 2022.
[5] Ross, G. Gordon, and Bagnell. reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[6] Park, Y. Seo, C. Liu, L. Zhao, T. Qin, Shin, and T.-Y. Liu. Object-aware regularization for addressing causal confusion in imitation learning. Advances in Neural Information Processing Systems, 34, 2021.
[7] P. De Haan, D. Jayaraman, and S. Levine. Causal confusion in imitation learning. Advances in Neural Information Processing Systems, 32, 2019.
[8] C. Wen, J. Lin, T. Darrell, D. Jayaraman, and Y. Gao. Fighting copycat agents in behavioral cloning from observation histories. Advances in Neural Information Processing Systems, 33: 2564 2574,2575,22 2020.
[9] Codevilla, E. Santana, A. M. López, and A. Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9329-9338. 2019.
[10] J. Zhang and K. Cho. Query-efficient imitation learning for end-to-end autonomous driving. arXiv preprini arXiv: 1605 06450, 2016.
[11] I. Kostrikov, O. Nachum, and J. Tompson. Imitation learning via off-policy distribution match- ing. arXiv preprint 1912.00112 2019.
[12] K. Greff, Van Steenkiste, and Schmidhuber. On the binding problem artificial neural networks. arXiv preprint arXiv:2012.05208 2020.
[13] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[14] E. S. Spelke and K. D. Kinzler, Core knowledge. Developmental science, 10(1):89-96, 2007.
[15] Florence, L.. Manuelli, and R. Tedrake.
in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492-499, 2019.
[16] M. Sieb, 7. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki. Graph-structuree visual imita- tion. In Conference on Robot Learning, pages 979-9899 PMLR, 2020.
[17] K. Hc, G. Gkioxari, Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[18] S. Ren, K. He, R. Girshick, and Sun. Faster r-enn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.

---

[19] Z. Cai and N. Vasconcelos. Cascade r-cmn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154- 6162, 2018.
[20] X. Zhou, D. Wang, and P. Krähenbühl. Objects as points. arXiv preprini arXiv: 2019.
[21] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty- thous-t classes using image-level supervision. arXiv preprint arXiv:2201.02605, 2022.
[22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polo- sukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[23] S. Schaal. Dynamic movement primitives-a framework for motor control in humans and hu- manoid robotics. In Adaptive motion of animals and machines, pages 261 261-280. Springer, 2006.
[24] J. Kober and J. Peters. Learning motor primitives for roboties. In 2009 IEEE International Conference on Robotics and Automation, pages 2112-2118. IEEE, 2009.
[25] A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann. Probabilistic movement primitives. Advances in neural information processing systems, 26, 2013.
[26] A. Paraschos, C. Daniel, J. Peters, and G. Neumann. Using probabilistic movement primitives in robotics. Autonomous Robots, 42(3):529-555 2018.
[27] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108, 03298, 2021.
[28] D. Wang. C. Devin, Q.-z. Cai, F. Yu, and T. Darrell. Deep object-centric policies for tonomous driving. In 2019 International Conference on Robotics and Automation (ICRA), pages 8853 88599. IEEE, 2019.
[29] C. Devin, P. Abbeel, T. Darrell, and S. Levine. Deep object-centric representations for gener- alizable robot learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7111-7118. IEEE, 2018
[30] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010. 14406, 2020.
[31] D. Seita, P. Florence, J. Tompson, F. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng- Learning to rearrange deformable cables. fabrics, and bags with goal-conditioned transporter networks. In 202/ IEEE International Conference Robotics and Automation (ICRA), pages 4568-4575. IEEE, 2021.
[32] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[33] M. Breyer, J. J. Chung, L. Ott, R. Siegwart, and J. Nieto. Volumetric grasping network: Real- time 6 dof grasp detection in clutter. arXiv preprint arXiv:2101.0f 132, 2021.
[34] Z. Jiang, Y. Zhu, M. Svetlik, K. Fang, and Y. Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. arXiv preprint arXiv:2104.01542 2021.
[35] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. In The International Symposium of Robotics Research, pages 132- 157. Springer, 2019.
[36] D. Kulkarni, A. Gupta, C. Jonescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning object keypoints for perception and control. Advances neural information processing systems, 32, 2019.
10

---

[37] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[38] R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint
2018.
[39] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, Fox, and S. Birchfield. Deep object pose es- timation for semantic robotic grasping of household objects. arXiv preprint arXiv: /809, 10790, 2018.
[40] S. Tyree, J. Tremblay, T. To, J. Cheng, T. Mosier, J. Smith, and S. Birchfield. 6-dof pose esti- mation of household objects for robotic manipulation: An accessible dataset and benchmark. arXiv preprint arXiv:2203.05001, 2022.
[41] T. Migimatsu and J. Bohg. Object-centric task and motion planning in dynamic environments. IEEE Robotics and Automation Letters, 5(2):844 851, 2020.
[42] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Ler- chner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901. 1 1390, 2019.
[43] Locatello, D. Weissenborn, Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. Advances in Neu- ral Information Processing Systems, 13:11525-11555 2020.
[44] W. Su, X. Zhu, Cao, B. Li, L.. Lu, F. Wei, and Dai. VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: 0908.99988300 2019.
[45] Y.-C. Chen, L. Li, L. Yu, A. EI Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. European conference on computer vision, pages 104-120. Springer, 2020.
[46] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[47] J. I.. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 1607.06455 2016.
[48] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert:
Pre-training of deep bidirectional
transformers for language understanding. arXiv preprint arXiv: 180.0:88100 2018.
[49] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 110044113499 13649, 2020.
[50] Z. Wang, A. Novikov, K. Zolna, J. S. Merel, J. Springenberg, E. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[51] Z. Zhong, L. Zheng, G. Kang, S. Li, and Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
[52] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martín. robosuite: modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. /2293, 2020.
[53] S. Toyer, R. Shah, A. Critch, and S. Russell. The magical benchmark for robust imitation. Advances in Neural Information Processing Systems, 33:18284-18295, 2020.
[54] C. Finn, X. Y. Tan, Duan, T. Darrell, S. Levine, and Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 512-519, IEEE, 2016.
[55] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.

---

[56] R. Dinyari, P. Sermanet, and C. Lynch. Learning to play by imitating humans. arXiv preprint arXiv:2006.06877 2020.
[57] R. Rahmatizadeh, Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA), pages 7758-3766 IEEE, 2018.
[58] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- ers for image recognition at scale. arXiv preprint 2rX1v:2010.11929, 2020.
[59] K. Hsu, M. J. Kim, R. Rafailov, J. Wu, and C. Finn. Vision-based manipulators need to also from their hands. arXiv preprint arXiv:2203. 126777777777... 2022.
[60] B. Burgess-Limerice C. Lehnert, J. Leitner, and Corke. Eyes on the prize: Improved per- ception for robust dynamic grasping. arXiv preprint arXiv:2204.1.N5 13879, 2022.
[61] O. Khatib. A unified approach for motion and force control of robot manipulators: The opera- tional space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, 1987.
[62] T. He, Z. Zhang, H. Zhang, Z. Zhang. J. Xie, and M. Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 558-567, 2019.
[63] R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang. Look closer: Bridging egocentric and third peridopeeoonnoo views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 2022.
12


Reviews:
[{"review_id": "uRhuyQ6UOXy", "reviewer": "Reviewer_9cZF", "summary": "The paper presents VIOLA, an object-centric imitation learning method for manipulation. The idea is that object- and task-specific knowledge should be attended to for a manipulation policy. To achieve this, VIOLA uses a pre-trained object detector to find regions of interest, which are fed into a transformer-based policy. The paper introduces three simulated tasks (sorting, stacking, kitchen) and three real-world tasks (plate+fork setting, bowl retrieval+setting, and making coffee). VIOLA is evaluated on these tasks along with 3 baseline imitation learning methods (BC, OREO, BC-RNN).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Object-centric information seems useful and a transformer gives a path for how to use such information.\n* The evaluation strategy is thorough. I appreciate the out-of-distribution tests (placement, distractor, camera jitter) and the A/B testing paradigm to mitigate human bias (L283).\n* The make coffee task seems challenging and linked to a realistic application.\n* VIOLA-Patch (a variant of VIOLA without object-specific patches) is a great baseline to differentiate performance due to transformers.\n\nWeaknesses\n* The method is somewhat complex, which makes it challenging to determine the key aspect of VIOLA's success. Data augmentations are not all ablated or used consistently across baselines (unclear). I'm glad the random erasing was included in the ablation, but I'm not sure how much the other augmentations (color, pixel shifting, etc.) contribute to VIOLA's success. It would be nice if all baselines used the same augmentations, or if not, include motivation for why the augmentations are specific to the method. The eye-in-hand image should also be used consistently.\n* Does K (the number of object proposals) need tuning for each environment? It would be more compelling to use the same K in simulation and the real world (just stick with either 15 or 20). L205-206 says \"we set K=15 which are found to attain a high recall rate on real images\" - but I'm not sure how this was selected. Can K be easily inferred from the demonstrations or was any online tuning done here?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The motivation for this work is strong, and I like the experimental setup. I have some concerns regarding the complexity of the method (see weaknesses above), but if these are addressed then I believe the work can be a strong contribution to CoRL.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I'm surprised at how small the object proposal patches are (visualized in figure 5). For example, I would consider the coffee maker as a whole to be a single object, but one patch cannot capture this. Would the method struggle to be scale-invariant (e.g. if the size of objects or distractors becomes much larger or smaller in the image)?\n\nIt appears as though Dining-PlateFork and Dining-Bowl were initially going to be a single sequential task. Was this the case? Was there a failure here due to its long-horizon nature?\n\nL117-118: \"Our goal is to learn the policy... that maximizes the expected return.\" This phrasing makes it sound like reinforcement learning methods will be evaluated, but to my knowledge, the reward is not being maximized anywhere (and there is no reward function defined for any task). Furthermore, I can't find a description of how successes are determined (in computing the success rate).\n\nSome of the descriptions could be clearer (no need to respond to these):\n* I don't understand what the global context feature is doing (L161).\n* Figure 2 is overly complex, perhaps partially due to the complexity of the method and partially due to too many details being included in the figure.\n* Is \"sinusoidal position encoding of the temporal positions\" (L174) common? I don't entirely follow the purpose of this here. Do we expect a sinusoidal nature in these manipulation tasks?"}}, {"review_id": "gOl6KgfKY-i", "reviewer": "Reviewer_Cdbg", "summary": "This paper proposed a model architecture for behavior cloning. The key element of the architecture is to feed region proposals from an RPN into a transformer. Results in simulation and real suggest this model works well.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- Cool real-world videos.\n- Nice looking architecture figure.\n- Results in sim and real suggest the model works pretty well.\n- Figure 5 attention rankings are interesting.\n\n# Weaknesses\n\n- An inexperienced researcher would read this introduction and think that this paper has come up with the idea of using object centric representations in visuomotor policy learning. But of course this is not the case. See for example Transporter Networks, CoRL 2020, which isn’t an object centric method but reviewed different prior works in this area. \n- This also goes for the title, “VIOLA: Object-Centric Imitation Learning for Vision-Based Robot Manipulation” which seems to suggest the authors have invented this idea. A more appropriate title might be “Region Proposal Transformers for Behavior Cloning” or something like that.\n- The paper refers to its method as an algorithm but I think it should probably be referred to as a model architecture. The algorithm it’s evaluated with is behavior cloning.\n- Why does section 3.1 talk about MDPs? There is no concept of rewards, or transition functions, or discount factors that is used in this paper. This paper just studies behavior cloning. The MDP paragraph copy paste is irrelevant to this paper. “Our goal is to maximize the expected return…” is not true, instead the method’s goal is to fit human demos.\n- The details of the architecture are probably not sufficiently described in order to implement it from reading the paper.\n- Why is [22] cited for doing behavior cloning with an RNN? This has been done well before [22], for example:\n  - R. Rahmatizadeh, P. Abolghasemi, L. Bo ̈lo ̈ni, and S. Levine, “Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration,” ICRA 2018\n  - P. Florence, L. Manuelli, R. Tedrake, “Self-supervised Correspondence in Visuomotor Policy Learning,” RA-L 2019\n- IL methods suffering from covariate shift has been studied at least since DAgger, Ross et al, 2011. Not a thing only from 2019 onwards, as it is cited here.\n- Even specifically RNN for BC with a GMM (also called MDN) was done in Rahmatizadeh et al noted above.\n- Overall I believe the academic question of the text should be more specifically “do region proposals help do more effective behavior cloning?” With this more precise framing of the text, I’m not sure the experimentation is very thorough for this question. The only comparison that gives a hint here is the Patch rather than RPN question. \n- Although the real world demos are in some ways impressive looking, they don’t seem to require much closed loop feedback.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am conflicted on this paper. On the one hand the robot demos are pretty cool, and it seems to work pretty well. Meanwhile, I listed several key weaknesses with the presentation and novelty of the paper in the Weaknesses section. I’m pretty borderline on this paper, I would currently lean reject but could consider moving up to accept if the authors address the weaknesses I listed.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See Weaknesses discussion.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "MSnX4PtyjHU", "reviewer": "Reviewer_77ch", "summary": "The paper presents an object-centric imitation learning approach to learning closed-loop visuomotor policies. The policy relies on object region proposal and features and a transformer based policy, as network structure, to improve generalization across view points, initializations and distractors. The policy outperform naive baselines and several ablated methods in both simulation and the real world. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\nThe paper introduces an interesting object-centric approach for closed-loop visuomotor policy learning, and demonstrates robustness across viewpoints, initializations, and distractors under systematic simulation and real world evaluations. \n\nCompared to the naive end-to-end approaches, VIOLA is more structured and makes use of transformers to attend to task relevant parts in the spatial-temporal domains.\n\nThe implementation details for making the transformer policy to work with temporal-spatial data, if released, can benefit the community in general.\n\nWeakness:\n\nOne of the main contributions seem to be using a pretrained FPN for proposing object bounding boxes, which is claimed to be more generalizable. Are there comparisons of this intermediate representation to other vision pipelines such as mask predictions or object detections? I can imagine the recall rates for these methods for finding interesting objects would be lower.\n\nThe applicability of pretrained object detectors can be limited in some cases such as deformable or granular object manipulations. The region proposals seem to make the network focus less on the irrelevant parts, but the generality and robustness of the approach still need more experiments to verify. A depth camera might also be able to help filter out backgrounds in some cases. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a well-thought and clearly written approach to imitation learning through object-centric representation and transformer backbones. The method outperforms baseline methods in simulation and in the real world. There are still several weaknesses regarding the general applicability and performance of the methods. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The architecture design seems to contribute a lot to the success rates in the network ablation study. I wonder if the performance gap to BC-RNN is still big if we vary the number of demonstrations or increase the BC network size. \n\nSimilarly, I wonder if the context information from spatial feature map, eye-in-hand image, and proprioception also provided to the simple BC baselines. I.e. how much the region proposals help with the transformer policy, v.s. How much the transformer policy outperforms vanilla networks. \n\nThe fact that random erasing helps a lot in table 2 is unclear to me. Is it because that the policy is overfitting to the K patches and some random masking would help with generalization? \n"}}, {"review_id": "Ckvx0me1IQE", "reviewer": "Reviewer_n5Tr", "summary": "The paper presents a transformer-based object-centric Visual Imitation Learning method, topping relevant state-of-the-art methods. The method is shown in several simulated tasks, introduced in the paper, including a multi-stage environment, and in the real world. Ablations show that the method is able to effectively deal with perturbations, jitter and object placement variations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This work is very good and shows a significant improvement over previous works. \n- The design of the method is simple and elegant, with ablations justifying most design choices\n- Clear and well presented\n\nWeaknesses:\n- The works lacks experimentation with more practical concerns such as generalization across similar objects, severe occlusion, and even more significant distractors than the ones shown.\n- A concern is that the distractors shown in the supplementary files are minimal.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I have recommended strong accept, because I am willing to argue for this paper’s acceptance, and believe it is a positive contribution dealing with an important problem. \n\n---\n\nUpdate: Most of the concerns raised were answered by the authors. My evaluation remains unchanged as a Strong Acceptance.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Technical issues and questions:\n- Hand-eye camera images are used as input to deal with local occlusion, but this choice is not discussed in detail. Would it be self-sufficient to have only the eye-in-hand camera? How much does the agent benefit from it being there? \n- Random erasing, which seems to be key to using regional features, could use more explanation within the background/method of the paper. \n- As per L151-L154, the model would be relying on a trainable RPN instead of a frozen one. Is the ResNet also not pre-trained? Wouldn’t this reduce the ability of the agent to generalize across similar objects (e.g., trained on one screwdriver, and then manipulating another). Moreover, it would be interesting to compare the frozen networks versus the trainable one used in the paper’s ablations (rather than in supplementary materials).\n- Choice of ResNet18 over other backbones itself is unclear – modern transformer models might provide better features, it would have been interesting to see. \n\nMinor issues:\n- L20: Typo: “its success of it”\n- Related Works, the cited Sieb et al. [24] has relevance in more than just the bounding box representation it uses since its very closely related to the proposed method, in that it is also vision-based object-centric imitation learning and the given summary of it doesn’t do it justice. \n- L187: MHSA is redefined\n"}}]

Author rebuttal:

Decision: Accept (poster)