Know Thyself: Transferable Visual Control Policies Through Robot-Awareness
Authors: Edward S. Hu
Kun Huang
Oleh Rybkin
Dinesh Jayaraman
Venue: ICLR 2022 Poster (2022)
Keywords: visual foresight
dynamics models
visuomotor control
video prediction
planning
transfer

Abstract: Training visual control policies from scratch on a new robot typically requires generating large amounts of robot-specific data. How might we leverage data previously collected on another robot to reduce or even completely remove this need for robot-specific data? We propose a "robot-aware control" paradigm that achieves this by exploiting readily available knowledge about the robot. We then instantiate this in a robot-aware model-based RL policy by training modular dynamics models that couple a transferable, robot-aware world dynamics module with a robot-specific, potentially analytical, robot dynamics module. This also enables us to set up visual planning costs that separately consider the robot agent and the world. Our experiments on tabletop manipulation tasks with simulated and real robots demonstrate that these plug-in improvements dramatically boost the transferability of visual model-based RL policies, even permitting zero-shot transfer of visual manipulation skills onto new robots. Project website: https://www.seas.upenn.edu/~hued/rac

Published as a conference paper at ICLR 2022
KNOW THYSELF: TRANSFERABLE VISUAL CONTROL POLICIES THROUGH ROBOT- AWARENESS
Edward S. Hu Kun Huang
Oleh Rybkin
Dinesh Jayaraman
GRASP Lab, University of Pennsylvania
(hued, huangkun, oleh, dinesh)) @seas www.n.
ABSTRACT
Training visual control policies from scratch on new robot typically requires generating large amounts of robot-specific data. How might we leverage data previously collected on another robot reduce or even completely remove this need for robot-specific data? We propose "robot-aware control" paradigm that achieves this by exploiting readily available knowledge about the robot. We then instantiate this in a robot-aware model-based RL policy by training modular dynamics models that couple a transferable, robot-aware world dynamics module with robot-specific, potentially analytical, robot dynamics module. This also enables us to set up visual planning costs that separately consider the robot agent and the world. Our experiments tabletop manipulation tasks with simulated and real robots demonstrate that these plug-in improvements dramatically boost the transferability of visual model-based RL policies, even permitting zero-shot transfer of visual manipulation skills onto new robots. Project website: https 7/www. seas upenn.ne........m wdw.-eued//dehaeee
INTRODUCTION
Raw visual observations provide versatile, high-bandwidth, and low-cost information stream for robot control policies. However, despite the huge strides in machine learning for computer vision tasks in the last decade, extracting actionable information from images remains challenging. As result, even simple robotic tasks such as vision- based planar object pushing commonly require data collected over many hours of robot interaction to learn effective policies. This data collection cost would be amortized if the learned policies could transfer reliably and easily to new target robots. For example, a hospital adding new robot to its robot fleet could simply plug their existing policies and start using it immediately. Going further, other hospitals looking to automate the same tasks could purchase a robot of their choice and download the same policy models.
However, such transferable policies are difficult to achieve in practice. Even when the task setting, such as the hospital, remains unchanged, the changed visual appearance of the robot itself leads to out-of of-disttibbttion inputs for visual policies pre- trained other robots. This issue particularly affects manipulation tasks: manipulation involves operating in intimate proximity with the environment, and any cameras set up to observe the environment cannot avoid also observing the robot.
There is a way out of this bind: most robots are capable of highly precise proprioception and kinesthesis to sense body poses and movements through internal sensors. We propose to develop "robot-aware" policies that can benefit from distinguishing pixels corresponding to the robot agent from those corresponding the rest of the "world" in image observations.
Our robot-aware policies treat the robot and the world differently, to their advantage. While this general principle applies broadly all visual controllers, we demonstrate the advantages of robot awareness in model-based reinforcement learning (MBRL) policies. MBRL policies work by planning through visual dynamics models that are trained to predict the consequences of agent actions. First, we inject robot awareness into the dynamics model, by factorizing it into robot-specific "robot" dynamics and robot-aware "world" dynamics. Figure shows a schematic. Our experiments show that composing these two modules permits reliably transferring visual dynamics models even across robots that look and move very differently. Next, we design a robot-aware planning cost over the separated robot and world pixels. We show that this not only allows visual task reward specifications to transfer from a source to a target robot, it even leads to gains on the source robot itself by allowing the policy to reason separately about the robot and its environment.

---

Published as a conference paper at ICLR 2022
Visual Foresight
Gial Emage
Controller
Execution
Robot Aware Cosl
Inage
Anslytical
Robot Model
Pw
P
Leaned
World Medal
P P(o') Pai (r') Par (0m.r'
P(o'l o.a) = PRz(r'l () Par (0% a'.r.
Step I:
Learn robot-aware world
Step 2: Zero-shot prediction / /ontrol with unseen robot R2
model from videos of robot
Figure 1: (Left) As part of our method, robot-aware control (RAC), we propose factorize visual dynamics into a rohot and world model. (Right) The world model is then readily reused with a new robot, permitting plug-and-play transfer.
We evaluate our method, robot-aware control (RAC), on simulated and real-world tabletop object pushing and pick-and-place tasks, demonstrating the importance of each robot-aware component. We show that RAC policy trained on a single, low-cost 5-DoF WidowX arm can transfer entirely "zero-shot" to very different 7-DoF Franka Panda arm in the real world. To our knowledge, this is the first demonstration zero-shot visual skill transfer between real-world robots, and also the first instance of transfer after training on only one source robot.
BECOMING AWARE OF THE ROBOT AGENT IN THE SCENE
We set up our robot-aware control approach by starting with the question: what is required to be "aware of the robot" in a vision-based manipulation setting?
First, a robot-aware agent must locate the robot in the visual observation from the camera. Consider, for example, manipulation task involving a standard robot arm, as Fig T At each discrete timestep t, the input to the agent's policy is an RGB image observation o(t) from a static camera observing the robot workspace. Since the robot must operate in close proximity with objects in the workspace, these images contain parts of the robot's 's body as well as the rest of the workspace. We define a visual projection function T(-) that maps from true robot state r and "world" (i.e., the rest of the workspace) state w to the image observation o(t) 1 (r((t,, w(t)). Robot states r could be joint angles, and world states w could be object poses in the workspace.
Note that the robot's state r(t) is readily available in high precision through internal joint encoders. Further, a full geometric specification of the robot shape and joint degrees freedom may be assumed available (usually from the manufacturer). Finally, techniques for camera-te to-boot calibration increasingly robust and efficient, to the point of acquiring calibration from even a single image when the robot shape is known above (Labbé al.) 2021 Lee al.] 2020). Combining these, may simply project the articulated robot shape through the camera parameters to obtain the visual projection the robot in the scene, see Supp. A.II for details. When parts of this projection may be occluded, we may further use RGB-D observations to easily identify and handle them. We represent this as an image segmentation mask M, (t) of the same height and width as image observations ott): M, is 1 on robot pixels and 0 elsewhere. We show examples of such masks in Fig This simple process effectively spatially disentangles robot pixels from world pixels in image observations o()). This spatial disentanglement is key building block for our robot-aware visual policies, since it permits treating robot and world pixels differently!
Second, with common high-level robot action spaces used in reinforcement learning (such as end- effector displacement actions), may even be possible to analytically prediet future robot states r' given the current state and the action (r, robot-aware control, and we may instead simply learn the robot-specific dynamics model P, from a small amount of experience. However, as we will show in our experiments, analytical P, models work well for our manipulation tasks, and further, they permit fully zero-shot transfer to a new robot.
r'. This is striet requirement for
AAtemmttive techniques permit automatically "self relffreeconnzzing" the robot in scene Michell al. 2004 Na
tale al. 2007 Edsinger & Kemp 2006 Yang al. 2020) even without prior knowledge of robot embodiment.
2

---

Published as a conference paper at ICLR 2022
3 TRANSFERABLE ROBOT- MODEL-BASED RL POLICIES
Above, we have described how policy might be "robot-aware" by spatially disentangling the robot its observations, and utilizing knowledge of its dynamics. We now show how such robot- awareness enables effective visual control policies that can be readily transferred between robots. In particular, we instantiate a robot-aware model-based reinforcement learning agent. Continuing in the manipulation setting from Sec 2 assume that tasks are specified through goal image that exhibits the target configuration of the workspace. Given the goal image 0g = T(rg. Wg), the agent must execute a sequence of actions a = (a(0), a(t + 1), .T)) to reach that goal, where T is a time limit. Success is typically measured by how close the final world state is the goal image. For example, the goal image might show a specific goal orientation of objects on a table, in which case success might be measured in terms of how close the objects are their goals.
To solve such tasks, we inject robot-awareness into a widely used model-based reinforcement learning paradigm for robots: visual foresight (VF) (Finn& & Levinee 2017 Ebert al. 2018 Standard VF involves two key steps:
Visual dynamics modeling: The first step is to perform exploratory data collection on the robot to generate dataset D of transitions (o(t), a(t), oft + 1)). Dropping time indices, sometimes use the shorthand (o, a, o') to avoid clutter. Then, a visual dynamics model P is trained on this dataset 'D to predict of given and
inputs, i.e. o' x P(o,a), a).. When the robot state is available, it is sometimes included as a third input to to assist in dynamics modeling.
Visual MPC: Given the trained dynamics model P. VF approaches search over action se- quences ag to find the sequence whose outcome will be closest to the goal specification Og. as predicted by P. For outcome prediction, they apply P recursively, as o(o(0), a) = P(P a())), a(I))....
Then, they pick the action sequence a a'(o(0), o,)) arg min, (()(o(0), a), og), where the cost function C' is commonly the mean pixel-wise &g error between the predicted image and the goal image. Sometimes, rather than measure the error of only the final image, the cost function may sum the errors of all intermediate predictions. For closed-loop control, only the first action a" (0) is executed; then, a new image o(1) is observed, and a new optimal action sequence of (o(1), o,) is computed, and the process repeats. For action sequence optimization, we found the cross-entropy method (CEM) (De Boer et al. 2005) to be sufficient, although more sophisticated optimization methods (Zhang et al. 201 9 Rybkin et al. 1021) could also work well.
To train reliable dynamics models P. visual foresight approaches commonly require many hours of exploratory robot interaction {Finn & Levine, 2017 Ebert et al. 2018) even for simple tasks. Data requirements may be reduced little by interleaving model training with data collection: in this case, data collected by selecting goal images and then running visual MPC using the most recently trained dynamics model to reach those goals. However, such goal selection must be designed to match the target tas,(s), which may trade off task generality for sample complexity.
3.1 IMPROVING VISUAL FORESIGHT THROUGH ROBOT- AWARENESS
Having trained one data-hungry visual dynamics model P on one robot, do we still need to repeat this process from scratch
new robot aiming perform the same tasks? In standard VE, the answer is unfortunately yes, for two main reasons: (1) With the new robot, all observations o(l) - [r (l), w()) are out-of-domain for P. Further, the visual dynamics of the new robot may be very different, as between green 3-DOF robot arm, and red 5-DOF robot arm. (2) As described above, standard VF approaches operate by aiming match the goal image. However, since the observations contain the robot, they commonly require the task specification image og [(Tg. g wg) itself to contain the robot plausible goal-reaching pose. This makes task specification more difficult, and also robot-specific: it is impossible to plan to reach 0g using different robot. We show how VF policies may overcome these obstacles through robot-awareness.
3.1.1 ROBOT-AWARE MODULAR VISUAL DYNAMICS
How can aabot-swareness be useful in visual dynamics modelling? Recall from Sec 2 that the robot
is both spatially disentangled and its dynamies are known to function (r a) r'. where ' denotes the next timestep. Here, we propose to exploit this by effectively factorizing the visual dynamics into world and robot dynamics terms.
3

---

Published as a conference paper at ICLR 2022
World state
Prediction
Next world
Lw
Ow
Ow
ow
Action (dx, dz)
Curr. robot state
Future robot state
Encoder
Learned
Decoder
Analytical
P
L.TTM
C
Objective
r'
The visual dynamics architecture is composed of an analytical robot model Pr, and the learned Pa. world model. During robot transfer, the target robot's analytical model used.
Figure 2:
Given the robot state r and the image observation o, we first compute the projected robot mask M, as above, then mask out the robot in the image to obtain product. See Fig 2 for examples of such masked images. Now, we train world-only dynamics model Pw by minimizing the following error, summed over all transitions in the training dataset D:
(1 M2), where o is the pixel-wise
L = (O,,
dull:
where the robot state computed using the known robot dynamics, r' (r, a). Treating all dynamics models as probabilistic, this is equivalent to the following decomposition of full visual dynamics into robot dynamics module P. and world dynamics module Pa:
P(r d'w |r, Ow = (r (r,))r Pa (or(o low, r,,', (r,",,))
What advantage does this modularity offer? First, since P., largely captures the physics of objects in the workspace, we hypothesize that can be shared across very different robots. Second, P. is commonly available for every robot "out-of-the" the-box" described in Sec 2 requiring data collection or training. Together, this paves the way for zero-shot transfer of visual dynamics.
Specifically, suppose that a robot-aware world-only dynamics module Pwr has been trained robot arm R2 with robot dynamics PRI for various manipulation tasks as seen in Figure The full dynamics model, used during visual MPC, would be PRI P.. in equation 2 Then, given new robot arm R2 with dynamics PR2, its full dynamics model P2 = PR2 Pw. is available without any new data collection at all. We validate this in our experiments, demonstrating few-shot and zero-shot transfer of P.. between very different robots.
3.1.2 ROBOT-AWARE PLANNING COSTS
Recall that the visual MPC stage in VF relies the planning cost function C'ô 0g), which measures the distance between predicted future observation ô and the goal Og. is clear that for any task, the ideal planning cost is best specified some function of the robot configurations and the world configurations w. However, since only image observations are available, it is common in VF to aim minimize pixel-wise error planning cost (Tian et al.) 2019 Nair & Finn 2020 Jaaaraman et al. 2019). such as C'ô 0g) = ||ō o, 11.. Instead, our spatially disentangled observations o - (r, owl make it possible produce the decomposed robot-aware cost:
C(f, dw rg. Ow.g) = A| -ryll + 11ó - ou.gll
(3)
The first term in this expression the robot cost scaled by À and the second term is the world cost. We compute robot cost that is usable by all robots such end effeetor distance, and compute the world cost by measuring pixel-wise distance over the world region of the predicted and goal image.
Why should this decomposed cost help? This robot-aware cost makes it possible to separately modulate the extent to which robot and world configurations affect the planning cost. To motivate this, observe that the basic pixel-wise cost suffers from key problem: it is affected inordinately by the spatial extents objects the scene, so that large objects get weighted more than small objects. Indeed, Ebertt al. (018) report that the robot arm itself frequently dominates the pixel cost in
4

---

Published as a conference paper at ICLR 2022
T-0
T-1
T-2
T-3
Goal
Real
Cost
Pixel
Heatmap
Coof
0.8
RA
Pooe Coui R.A.
Heatmap
Timesiop
Figure 3: The behavior of the pixel and RA cost between the current images of a feasible trajectory and the goal image. (Left) The first row shows the trajectory. The next two rows show the heatmaps (pixel-wise norm) between the image and goal for each cost. The heatmaps show high cost values in the robot region, while the RA cost heatmaps correctly have zero cost in the robot region. (Right) The relative pixel cost fails to decrease as the trajectory progresses, while the RA cost correctly decreases.
manipulation. This means that the planner often selects actions that match the robot position in goal image, ignoring the target objects. See Figure for visual example of the pixel cost behavior and its failure compute meaningful costs for planning, As result, even the task involves displacing an object, the goal image is usually required to contain the robot in a plausible pose while completing the task (Nair & Finn 2020). For our goal of robot transfer, this is an important obstacle, since the task specification is itself robot-specific. Even ignoring transfer, robot-dominant planning costs hurt performance, and gathering goal images with the robot is cumbersome. Our approach does not require robots to be in plausible task completion positions in the goal images. In fact, they may even be completely absent from the scene. our experiments, we use goal images without robots, and sometimes with humans in place of robots. To handle them, could set À to 0 in the above equation to instantiate a cost function that only focuses on the world region.
Some VF-based methods do handle goal images without robots; however, they typically require all input images for dynamics and planning to also omit the robot, so that image costs are exclusively influenced by the world configurations w (Wang et al., 2019 Pathak al. 2018 Agrawal al.) 2016). For closed-loop controllers, this often means extremely slow execution times, because, at every timestep, the robot must enter the scene, execute an action, and then move out of the camera view. This also eliminates tasks requiring dynamic motions.
Finally, there are efforts to learn more sophisticated cost functions over input images Co og) (Nair et al.; (Sermanet et al.) 2018 Srinivas et al., 2018 Yu et al., 2019 Tian et al.) 2021 For example, Nair et al. train a latent representation to focus on portions of the image that are different between the goal and the current image, and show that costs computed over these latents permit better control on one robot. These approaches all learn the cost contributions of different objects or regions, from data. Instead, we directly segment the robot using readily available information. While we restrict our evaluation to basic pixel-basee costs, we expect that these other costs will also benefit from spatially disentangled inputs, i.c., Ce ow, Fgg. ow.g).
3.2 IMPLEMENTATION DETAILS
We summarize some key implementation details about the robot-aware model here, and refer the rest to Supp- A.I For implementing the learned world dynamics model Pw. we extended the authors' implementation of the SVG architecture (Denton & Fergus 2018) to be conditioned robot actions and states. SVG consists of convolutional encoder, frame predictor LSTM, and decoder alongside learned prior and posterior network. The encoder network takes in the RGB image, current mask. and future mask as a 5-channel 64x48 image. It outputs a spatial latent of dimension (256, 8, 6). Actions and end effector poses then tiled onto this spatial latent before being fed into the convolutional LSTM. The convolutional LSTM output is then fed into the decoder which outputs the predicted image. Refer algorithm and for world model training and testing pseudocode.
The world model P., is a CNN that must take the world pixels o as input and predict future world pixels oy.. Since it cannot selectively process partial images Own must instead feed in
5

---

Published as a conference paper at ICLR 2022
full rectangular image. However, if trained Pa on full images of the training robots without modification, then test images with new, different-looking robots would lie outside the training distribution of Pw. To induce some invariance to the robot appearance in Pws the robot is masked (producing the appearance of a black robot) at training as well as at test time. Note that the world model is still trained based on Eq. T which only penalizes errors in world pixels. We use Movelt, PyRobot, and MuJoCo to implement the analytical robot dynamics for the various robots.
4 EXPERIMENTS
Our experiments focus zero and sew-shot transfer across robots for pushing and pick-and-plan (Finn & Levine 2017; Ebert al., 2018 Dasari et al., 2019): robot arm must move objects to target configurations, specified by goal image, tabletop. We aim to answer: (1) How does the robot-aware controller compare against standard controllers and a domain adaptation bascline when transferred to new robot? (2) Which robot-aware components in the visual foresight pipeline are most important for transfer?
Baselines. For evaluating robot-aware dynamics models, we compare against prior visual foresight methods (Ebert et al. 2018, Finn & Levine (017)) which use action-con conditiond models that solely rely on image input (VF), as well as models that rely on robot state and images State) For our full robot-aware pipeline, we experiment with combinations of predictive models and cost functions to probe their transferability new robots. Hereafter, we will refer all controllers by the names of the dynamics model and the cost, e.g.. RA/RA for the full robot-aware controller, and VFt+State/Pixell for the baseline VF controller. See Table for all controllers. We also compare against an unsupervised domain translation approach (CycleGAN+ VF VF+-Stae/Pixeel) based on the human-to-robe imitation policy of Smith et al. (2020). that learns a mapping between target and source robot images for transfer. Note that this policy is few-shot since it is trained test robot images. See Supp. for additional details about baselines, calibration, data, training, and evaluation.
Transfer settings. We evaluate the transferability of robot-aware control in various configurations:
b Simulated zero-shot transfer from one robot. Simulation permits a "straight swap" of the robot with perfectly controlled viewpoints and environments. We conduct pushing and pick-and-place experiments by training policies on a 5-DoF WidowX200 arm and evaluating these models zero-shot on an unseen 7-DoF Fetch arm. For the prediction experiment, we train on 10k trajectories of length 30 of the WidowX200 arm performing random actions on the tabletop with several objects, and evaluate on 1000 trajectories of the Fetch robot. See Supp. A.9 A.. for more prediction experiments. For the control experiments, we transfer the VidowX200 trained policies to the Fetch robot and perform pushing and aed--and-place tasks. The CycleGAN (Zhu et al. 2017) (authors' implementation) was trained using 1 videos (12k frames) each of the training and test time robots. See Supp. A.3 for more details on control experiments.
Real zero-shot transfer from one / multiple robot(s), We evaluate zero-shot prediction and control of an unseen Franka robot (Fig. 4 from models trained on 1=.8k videos single WidowX200 robot, well as models trained on multi-robot dataset (Fig. [ that randomly perturbs the objects. The multi-robot models are pretrained on 83k videos including 82k RoboNet (Dasari et al.) (2019) videos of Sawyer, Baxter, and WidowX robots, and 1k videos from the above Widow X200 dataset. We used a subset of RoboNet for which we were able to manually annotate camera calibration matrices and robot CAD models (see S.II.A.II). After training, we deploy the models to the unseen Franka and perform pushing tasks.
Next, we evaluate zero-shot predic- tion of an unseen Modified Wid- owX200 (Fig. 4) from models trained on the multi-robot dataset. To mod- ify the appearance and dynamics of the WidowX200 we swapped out the
14 000
Figure 4: WidowX200, Mod. WidowX200, Franka black 1 14cm forearm link with a silver 20cm forearm link, and added a foam bumper and sticker. We collected 25 videos of the modified robot for evaluation.
p Few-shot transfer. Finally, we follow the few-shot transfer setup of Robonet (Dasari et al., 2019 by first pretraining the models on RoboNet data, and then finetuning on <400 videos of the Widowx200. We then evaluate the prediction and control performance the WidowX200 on pushing tasks.
Note that the Widow X and W200 are different robots.

---

Published as a conference paper at ICLR 2022
Table 1: World prediction evaluations, reported average world region PSNR and SSIM over timesteps. We compare against conventional models that use image and/or state (VF and VF-tet
Few WidowX2000
Model
Fetch (Train
WidowX200 (Train robots)
Fraika (Train robols)
Franka
WidooX2000
(Train on robots)
Widox
PSNKT
SSIM 1
PSNK
SSIM
PSNK
SSIM
PSNK
SSIM 1
PSNK
SSIM 1
VP
39,9
0,985
32.86
0.939
90.59
3,18
0,913 + 0.04
28.31 333
0.901 + 1.04
28.42 3.43
0.905 + 0.04
VF+-State
33:05
0.941
31.03 + 3.17
0,915 4 0.04
29.19 3.02
0,901 1: 0.04
28.31 4 3,18
0.894 1: 0.05
(Ours)
0.944
0.03
0.04
0.03
Ground Truch
Ground Truth T-5
RA Pred. T-5 5
Baseline Prod.
T T-0 0
T-5
Zero-shot Fetch (Sim)
Zero-shot Franka
Model outputs of the zero-shot experiments. First row: the RA model is able to predict contact with the green box, while the baseline (VF++State) predicts none. Next row: the RA-model models the downward motion of the octopus more accurately. Refer to the website for videos.
Figure 5:
Results: Transferring visual dynamics models across robots. the Table shows quantitative prediction results for average PSNR and SSIM per timestep over a 5-step horizon, computed over the world region. Peak signal to noise ratio (PSNR) and structural similarity index metric (SSIM) widely used metrics used for evaluating video prediction approaches. PSNR scales the inverse MSE error logarithmically, and SSIM measures similarity over local patches instead of per- ver-p.xel. The robot-aware world dynamics module P. outperforms the baselines across all zero-shot and few-shot transfer settings, simulated and real, on both metrics. To supplement the quantitative metrics, Fig 5 shows illustrative examples of 5-step prediction, where recursively run visual dynamics models to predict images 5 steps out from an input.
We show more examples of prediction results in the Supp. and website. Some key observations: first, the Vt+-State baseline overlays the original robot over the target robot rather than moving the target robot (see Figure 5 bottom right). Next, RA produces relatively sharper object images and more accurate object motions, as with the pink octopus toy in Figure Finally, while both VF+SState and RA predict motions better for large objects, VF+ State more frequently predicts too small, or even zero object motions (see Figure top right). Our RA approach can use known robot dynamics for accurate robot state predictions, which in turn reduces uncertainty in predicting object interaction. As we will show next, these performance differences in object dynamics prediction are crucial for successfully executing manipulation tasks.
Results: Transferring visual policies across robots. We now evaluate our full robot-aware MBRL pipeline on simulated and real pushing tasks, as well as a simulated pick-and-place task. This jointly evaluates the RA dynamics models and the RA planning cost Pushing tasks are specified by single goal image, while pick-and-place tasks are specified by three goals for picking, lifting, and placing. After each evaluation episode, we measure the distance between the moved object's 's current and goal position. We define a successful episode as a distance under 5cm for the pushing task, and under 2.5cm for the pick-and-place task. For each task, we use an object-only goal image where the
We evaluate the RA cost in isolation by planning with perfect dynamics models in Supp. evaluate the RA cost with human goal images.
A.6 In Supp.
A.7
7

---

Published as a conference paper at ICLR 2022
RAC
Baseline End
Start
Gool
End
Push Melon
10 Down
Pick Block Place Right
Figure 6: Example results from the control experiments (full videos on website). We show the start, goal, and end states of RA/RA and baseline (taee/itt/)ixel),, The goal image overlaid on top of the end state for visual reference.
Table 2:
Zero-shot control results, All models are trained on a single robot Widdox200) unless denoted otherwise. Note the CycleGAN baseline is trained 12k images of the test-time robot.
Dynamics
Fetch Push (Sim.)
Fetch Pick-and-place (Sim.)
Franka Push
Cost
Model
(Real)
CycleGAN+VF... Staate
Pixel
12/20 (60%)
0/20 (0%)
VF+tState
Pixel
0/20 (0%)
0/20 (0%)
0/30 (0%)
RA
Pixel
0/20 (0%)
0/20 (0%
0/30 (0%)
VF-+State Statet
12/20 (60%)
4/20 (20%)
6/30 (20%)
RA
RA
18/20 (90%)
8/20 (40%)
22/30 (71%)
VtatStatc (Multi-robut)
Pixel
11/30 (36%)
RA (Multi-robot)
27/30 (90%)
object is in the correct configuration, but the robot is not in the image. For pushing, vary the goal position and object per trial and for pick-and-place, we vary the block initialization and goal position. See Supp. A.3
and A.5 for more details.
Table 2 and shows that RA/RA achieves the highest success rate in all tasks and transfer settings. Note that method's success rate (40%) on pick-and-place success rate of 55% on the source robot (see Supp. 10 all source robot results), which shows our method is robust to robot transfer. We observed two primary failure modes of the VF baselines in zero-shot scenarios. First, both Statettt/Pixell and RA/Pixel, which optimized the pixel cost, tend to retract the arm back into its base, because the cost penalizes the arm for contrasting against the background of the goal image seen in Figure 6 Next, consistent with the prediction results, Vt+State/Pixel and VF-+State/RA suffer from predicting blurry images of the training robot and inaccurate object dynamics, which negatively impacts planning.
much worse than method's
The CycleGAN baseline performs moderately well in pushing (60% success) but completely fails in pick-and-place (0% success). Qualitatively, the CycleGAN baseline correctly moves to pick the block, but nearly always fails by selecting an unstable grasp, where the block slips through the gripper. Note that the CycleGAN itself is trained correctly: it successfully produces visually high quality domain- translated images of Fetch-to- WidowX (see website). However, transfering dynamics models for grasping requires very precise domain translation, which CycleGAN is unable achieve even when trained with 12k images of the target robot.
The failure of the baselines transfer zero-shot is expected, since generalizing to robot single robot dataset is daunting task. One natural improvement is to train on
from training multiple robots before transfer facilitate generalization. We evaluate RA/RA and Statextp/ in this multi-robot pretraining setting. where they are trained on RoboNet videos in addition to our WidowX200 dataset. As Table ("Muuliirobott rows) shows, the State////ix improves with training on additional robots, but still falls far short of RA/RA trained on even single robot. RARRA improves still further with multi-robot pretraining.
8

---

Published as a conference paper at ICLR 2022
> Few-shot transfer. In this setting, dynamics models are pretrained on RoboNet and finetuned on ~400 videos of the WidowX200. As Table 3 shows. RA/RA significantly outper- forms the baseline controller. Due to the baseline's 's inability to model object dynamics aside from the largest object, it suc- ceeds only on pushes with the largest object. The RA model adequately models object dynamics for all object types, and is
Table 3: Few-shot transfer to WidowX200 control results.
Dynamics Model
Cost
Success
VF RA
Pixel
4/50 (8%)
RA
40/50 (80%)
able to succeed at least once for each object (see Supp. A.5 for object details).
OTHER RELEVANT PRIOR WORK
Several works have focused on transferring controllers between tasks (Duan et al., 2017 Finn et al., 2017b), environments (Finn et al. 2017a). and simulation-to-real (Tobin et al. 2017). but relatively little attention has been paid to few / zero-shot transfer across distinct robots. Aside from RoboNet (Dasari et al. 2019), discussed above, few prior works have studied providing the robot morphology as input to model-free RL policies, enabling transfer to new robots (Chen et al. 2018
Devin et al. 2017
Wang al. 2018).
Devin et al. 2017 train modular policies containing a robot module and a task module, both learned from data, but requires data gathered on the new robot for training the robot-specific module before transfer is possible. [Chnn et al. (2018) propose to train a "universal" policy conditioned on the robot hardware specification, but must train on many robots to permit transfer, and operates on low-dimensional states. In contrast, RAC transfers visual policies after training even on a single robot. For simulated snake and centipede robots with repeated chained segments, NerveNet (Wang et al.) 2018) demonstrates transferable state-based locomotion skills by running a graph neural network policy over the known chain structure. This permits transfer, for example, from a centipede with chained segments to one with 10 identical chained segments. RAC targets more challenging settings: visual object interaction and manipulation on real robots with varied morphologies and appearances. RAC is also technically distinct: it trains model- goal-reaching controller, while these prior methods (Devin et al.
2017 Chen et al., 2018 Wang et al.) 2018) train model-free task-specific policies. [Ricarrd et al. (2021) concurrently propose decomposed world and robot state dynamics model to enable zero-shot transfer across robots in state-based navigation with sim-to-real training, whereas we evaluate our method on transfer of vision-based manipulation skills and train on real data.
Other works train visual representations that generalize to different embodiments, such as humans or other robots, and use them infer rewards for RL (Zakka al. 202 Zhou al. 2021 Smith iet al. 2020 Sermanet et al. 2018). However, these methods require access to images or significant experience on the test-time robot. For example. Zakka al. (202T) learn task-specific rewards (e.g. block pushing) and require ~ 100k steps on the test robot to learn. In contrast, RAC's ' world dynamics model can be reused across tasks, and can achieve multiple tasks through goal image specification.
Finally, our work has interesting connections to "contingency awareness" in cogmitive science ( (Watson 1966), that has also been studied for RI. (Bellemare et al., [20T Choi et al., 2018): in simple 2D Atari games, they show that learning to localize the agents in image observations can improve model-free -fee visual reinforcement learning and exploration. We operate in more complex real-world robotic settings, and focus on transferring model-based policies across agents.
6 CONCLUSION
We have studied the challenging task of transferring learned visual control policies for object manipulation across robot arms that might be very different in their appearance and capabilities. Our arobot-aware control" (RAC) paradigm, when used in widely used model-based RL algorithms, convincingly transfers zero-shot to unseen real and simulated robots for object manipulation for the first time, and also yields large gains for few-shot transfer. RAC benefits from a world model that aims capture the physics of the environment independent of the robot as be fully transferable. Implemented with pixel models and costs, RAC's world model is still subject to limitations: it is tied to the andeeffeccor action space and even to the shape of the source robots, which may inhibit transfer in more extreme settings. Further, the operating environment and its physics might differ from one robot to the next. We will aim to address these limitations and challenges in future work.

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGEMENTS AND DISCLOSURE OF FUNDING
This work was partly supported by an Amazon Research Award to DJ. The authors would like to thank Karl Schmeckpeper and Leon Kim for technical guidance, the anonymous reviewers for their constructive feedback, and the Perception, Action, and Learning Group (PAL) for general support.
8 REPRODUCIBILITY STATEMENT
To ensure reproducibility, we will release the codebase that contains our video prediction and control algorithms, well as weights for our trained models. The supplementary contains details about hyperparameters (Supp. A.2) and architecture (Supp. A.T). A. For datasets, we will release our subset of RoboNet labeled with robotic masks, and our WidowX video dataset. We provide details of our mask labeling process in Supp. A.II. and the labeling code will be in the codebase. See the website for the code https: :tws:://www.. seas upenn. aedu/-hued/raa rac
REFERENCES
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. arXiv preprint arXiv:1606.07419, 2016.
Mare Bellemare, Joel Veness, and Michael Bowling. Investigating contingency awareness using atari 2600 games. Proceedings of the AAAI Conference on Artificial Intelligence, (6(1):864-88) 871, Sep. 2021.
Tao Chen, Adithyavairavan Murali, and Abhinav Gupta. Hardware conditioned policies for multi- robot transfer learning. In Advances in Neural Information Processing Systems, 2018.
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi. and Honglak Lee. Contingency-awar exploration in reinforcement learning. arXiv preprint 1 2018.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning, 2019.
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International Conference Machine Learning, 2018.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In IEEE International Conference on Robotics and Automation, 2017.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, llya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. In Advances in Neural Information Processing Systems, pp. 1087-1098, 2017.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.0083 2018.
Aaron Edsinger and Charles C Kemp. What can i control? framework for robot self-discovery. In 6th International Conference on Epigenetic Robotics, 2006.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In IEEE International Conference on Robotics and Automation, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1166-1135, 2017a.
10

---

Published as a conference paper at ICLR 2022
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference on Robot Learning, pp. 357-368, 2017b.
Dinesh Jayaraman, Frederik Ebert, Alexei A Efros, and Sergey Levine Time-agnostic prediction: Predicting predictable video frames. In International Conference Learning Representations, 2019.
Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic. Single-view robot pose and joint angle estimation via render & compare. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1654-1663, 2021.
Timothy E Lee, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Oliver Kroemer, Dieter Fox. and Stan Birchfield. Camera-to-robov pose estimation from single image. In IEEE International Conference on Robotics and Automation, pp- 9426-9432, 2020.
Philipp Michel, Kevin Gold, and Brian Scassellati. Motion-based robotic self-recognition. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2004.
Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. In International Conference on Learning Representations, 2020.
URL
net/ Eorum? (eerrumiiddHlggz22222
Suraj Nair, Silvio Savarese, and Chelsea Finn. Goal-aware prediction: Learning to model what
matters. In Proceedings of the 37th International Conference on Machine Learning.
Lorenzo Natale, Francesco Orabona, Giorgio Metta, and Giulio Sandini. Sensorimotor coordination in a "baby" robot: learning about objects through grasping. Progress in bruin research, 164: 403-424, 2007.
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp- 2050-2053, 2018.
Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari. Rl-cyclegan: Reinforcement learning aware comulationn-t--eall In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Dripta S. Raychaudhuri, Sujoy Paul, Jeroen van Baar, and Amit K. Roy-Chowdhury, Cross-domain imitation from observations. In International Conference on Machine Learning. 2021.
Antoine Richard, Stephanie Aravecchia, Matthieu Geist, and Cédric Pradalier. Learning behaviors through physics-driven latent imagination. In Conference on Robot Learning, 2021.
Josh Roy and George Konidaris. Visual transfer for reinforcement learning via wasserstein domain confusion. In Association for the Advancement of Artificial Intelligence, 2021.
Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Daniliddis, Igor Mordatch, and Sergey Levine. Model-based reinforcement learning via latent-space collocation. International Conference on Machine Learning (ICML), 2021.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Tint-contrattive networks: supe-supe learning from video. In IEEE International Conference on Robotics and Automation, pp. 1 1134-1141,, 2018.
Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. AVID: learning multi-stage tasks via pixel-level translation of human videos. In Robotics: Science and Systems, 2020.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. arXiv preprint arXiv: 1804000645 2018.
S. Tian, F. Ebert, D. Jayaraman, M. Mudigonda, C. Finn, R. Calandra, and S. Levine. Manipulation by feel: Touch-based control with deep predictive models. In 2019 International Conference on Robotics and Automation (ICRA), pp. 818 824, 2019. doi: 10.1 009//CCAA.001.87442219
11

---

Published as a conference paper at ICLR 2022
Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Model-based visual planning with self-supervised functional distances. In International Conference Learning Representations, 2021.
Maxim Tkachenko, Mikhail Malyuk, Nikita Shevchenko, Andrey Holmanyuk, and Nikolai Liu- bimov. Label Studio: Data labeling software, 2020-2021. URL https://github: com/
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference Intelligent Robots and Systems, pp- 5026-5033, 2012.
Angelina Wang, Thanard Kurutach, Kara Liu, Picter Abbeel, and Aviv Tamar. Learning robotic 2019.
manipulation through visual planning and acting. arXiv preprint
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Representations, 2018. URL
https::/epenneeview net firee/foruuee wow.mmidd-Sssiiiiiiii
John S Watson. The development and generalization of" contingency awareness" in early infancy: Some hypotheses. Merrill-Pulmer Quarterly of Behavior and Development, 1232):123-135, 1966.
Brian Yang, Dinesh Jayaraman, Glen Berseth, Alexei Efros, and Sergey Levine. Mavric: Morphology- agnostic visual robotic control. ICRA and RA-L, 2020.
Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, and Chelsea Finn. Unsupervised visuomotor control through distributional planning networks. In Proceedings of Robotics; Science and Systems (RSS), June 2019.
Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. XIRL: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning. 2021.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. SOLAR: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning. 2019.
Yuxiang Zhou, Yusuf Aytar, and Konstantinos Bousmalis. Manipulato-indeeendent representations for visual imitation. Robotics: Science and Systems, 2021.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imagee translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision, 2017.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
We encourage the reader to visit the website at the url: https: /ts:://www... seas. wttp://wwwww.........eee. edu/ Fhued/ rac for an explanation video and additional video visualizations.
A.1 NETWORK DETAILS
The robot-aware dynamics model described in Section 3.1.1 robot-specific dynamics model P, and a learned, visual dynamics world model P.. The robot dynamics model is required to predict the next robot state r' (end effector pose and mask) from the current robot state and action a and feed that as input into the world model. This future robot state and mask is useful for the world model to predict the overall future state, it can infer object displacement from such information.
consists of two modules, an analytical
Neural network architecture for the learned visual dynamics Pa. We extend the stochastic video generation (SVG) architecture (Denton & Fergus 2018) which consists of the convolutional encoder, frame predictor LSTM, and decoder alongside learned prior and posterior.
The encoder consists 4 layers of VGG blocks (convolution, batch norm, leaky RELU) and passes skip connections to the decoder. We use a convolutional LSTM for the learned prior, posterior, and frame predictor. As seen in Figure 2 visual data such as the current RGB observation One with dimension (3,48,64), current mask with dimension (1,48,64), and future mask M,,, M, (1, 48, 64) are concatenated channel-wise and fed into the encoder, which convolves the input into a feature map of dimension (256, 8, 8). Additional data such as the action, current state, and future state are tiled onto the feature map before being passed into the recurrent frame predictor, outputting a feature map of size (256, 8, 8) that gets decoded into the next RGB image o'.
Generating predictions from the dynamics model at test time. During training, the RA-model utilizes ground truth future robot state as input for prediction since all data is recorded in advance. From our prediction results in Table we find that using future robot state and masks to be useful for future prediction. But how might we produce the future input r' for Pa during test time?
We use the analytical dynamics model P. as described in Section 3.1.1 to compute the future inputs. We now describe this process with more detail. The analytical dynamics model P. predicts the future robot state r' as a function of the current state and action a. For example, if the control commands are robot end-effector displacements, then the robot state r is the gripper pose, and next state r' P(r, a)
Then, to obtain the masks, we must compute the full robot pose and its visual projection in two simple steps. First, we compute the joint positions using inverse kinematics q' (r'), and then project the joint positions into mask M, Project(g'). We use the analytical inverse kinematics solver from the PyRobot library to obtain future joint positions for the WidowX200 while for the Franka, we use the Movelt! motion planning ROS package.
Next, we use simulator such as MuJoCo that supports 3D rendering of the robot along with camera calibration to project the image of the virtual robot into a 2D mask. These steps are also important for recursively applying the dynamics models to produce "rollouts". Given a starting image o(t) and actions at + 1), alt + 2),,..,..,.,... (T)), we can predict ö(t+ + 1) as above. To predict a(t + 2), P.. needs as input Mt + 1), Mt - 2), r(t + 1), + 2), all of which are inferred as above through the analytical dynamics P,. Algorithm (arrihmmmppoorr provides pseudocode to generate such rollouts.
RA-loss implementation. In Algorithm 3] we provide a PyTorch implementation of the robot-aware dynamics loss (eq. [ over RGB images. Note that the principle of separating the robot region and world region in the Toss computation is general, and can be applied to other state formats as well.
A.2 VISUAL DYNAMICS EXPERIMENT DETAILS
We now present some details of the visual dynamics experiments in Section I where evaluated models on unseen robots as seen in Figure 4
Training and finetuning details. Models were pretrained for 150,000 gradient steps the RoboNet dataset with the Adam optimizer, learning rate 3e-4 and batch size of 16. We use scheduled sampling to change the prediction loss from 1-step to 5-step future prediction over the course of training. For
13

---

Published as a conference paper at ICLR 2022
Algorithm TRAIN
1: Input: World model W, analytical model R, image sequence o(1)... + 1), mask sequence M,(1)... (T,1))..,,(T"" (T++ + 1), state sequence r(1) r.( + 1), actions ((T).
2: Ow t {}
o(2)..... (1 -M,(T + 1)) o( + -1) 3: Ow - {(1-
4: < (
// Here we abuse the ow notation to be the masked image
5: for to do
6:
0' + W(o,,
1), a(())) ( (1 (, o'
7:
8:
Ow + Ow ( (t)
9:
Ow + MoskRobot M,(t+ + +))
10: end for
Compute world loss (eq.
between Ow and Ow.
12:
return
Algorithm 2 TEST
1: Input: World model W, analytical model R, Start image (11,, start mask M,(1), start state r(1), actions a(1);-.)))) a()).
2: Ow = ft
3: < r(1)
4: < (osk)
// Here we abuse the ou notation to be the masked image
5: for = 1 to T do
6: r' +- a(
7:
ō' + W(ow, r, (7,,''a(()) a(t)) Mp ő'
8:
ów +1)
9:
O, + Ow (1
10:
On + (as M Mr)
r r+r
12:
end for
14:
return Ô
fine-tuning, all models were trained on the fine-tune dataset for 10,000 gradient steps with learning rate 1e-4, batch size of 10, and scheduled sampling. See algorithm for pseudocode.
Evaluation details. To evaluate PSNR and SSIM metrics over the world region of the images instead of the entire image, we preprocess the images by setting the robot region of the images to black. This corresponds to removing all pixel differences between the predicted and target robot region before computing the PSNR and SSIM over the entire region.
For all experiments, we evaluate on sequences of length 5 and calculate the average world PSNR and SSIM metrics over the timestep. Because the models are stochastic, we perform Best-of-3 evaluation where 3 videos are sampled from the model, and report the sample with the best PSNR and SSIM.
2019) conventions, the image dimension 64 by 48 pixels. and the action space of the robot is the displacement in end effector pose (x, y, z, 0, f) where v, y/ z are the cartesian coordinates of the gripper with respect to robot base, 0 is the yaw of the gripper, and ff is the gripper force. The videos are length 31, but sample subsequences of length 6 from the video to train the network to predict images from conditioning image. Due to the workspace bounds varying in size across robots, RoboNet chooses normalize the cartesian coordinates to [0, 1] using the minimum and maximum boundary of each workspace.
Dataset. Following RoboNet (Dasari
The WidowX200 dataset consists of 1800 videos collected using random gaussian action policy that outputs end effector displacements in the action space. For the pushing task, we fix 2, 8, f to constants, and sample æ; y displacements from dimensional gaussian with O mean and 3.5g in
14

---

it Published as a conference paper at ICLR 2022
Algorithm 3 Pytorch code for the Lo loss.
def za 1oas (pred, target, mask): nan
pred and carget are of images of dim (3, H, x)
maak is binary segmentation mask of the zobot of din (1, B, W) L1 narm of world pixels
cutputs the *an
difference - target pred # (3, H, R) channel axis
repeat mask to index the zebet zegion in difference robat_region
mask, rask.rapect (3, ignore the pixel ditterences in the rooot reglon [i [robot_regioo]]]
compute the norm over the world pixels num num_wor world_pixels = (enasttreegonn) .sum() _t_per_pixee difr.abs .un(() sun_world return pi_per_pixel
Ground Truth
Gruund Truth
RA Pred.
VTS Pred. T=5
Zero-shot WidowX
Zero-shol Franka
Figure 7: Qualitative outputs the zero-shot prediction experiments on models that were pretrained on multiple robots. From left right, we show the ground truth start, ground truth future, and predictions for the robot-aware model and VFS model. The baseline blurs the bear in the top row, and predicts minimal movement for the octopus in the bottom row. The proxy robot is visualized in green. Refer to the vidcos on the website for more visual comparisons.
centimeters. Similar to RoboNet, the states are normalized by the workspace minimum and maximum. We also collected 25 trajectories of the modified Widow X200 as seen in 4 for evaluation.
A.3 SIMULATED CONTROL EXPERIMENT DETAILS.
We study the effect of robot transfer on pushing and pick-and-place tasks using the MuloCo simulation (Todorov al., 2012). The models trained on 10k videos of the WidowX200 and evaluated their task performance on the unseen Fetch robot.
Pushing. The robot must push the block from randomized starting pose randomized goal position 10cm away from the start. To initialize the environment, we first set the robot and block spawn locations. The block is initialized with uniform noise of 5cm in the center of the workspace. The robot gripper is then moved behind the block, and then perturbed with uniform noise of lcm. The episode limit 10 steps. We only used the image-based world cost for pushing so we set
The robot must pick an object from a randomized starting pose and place it on an elevated platform a location specified through a goal image. For pick-and-place, we give a goal for picking, lifting, and placing for total of 3 goals per pick-and-plas and-laee episode. We found it helpful to use the robot cost term in Equation 3 specifically by using the L2 distance between the current and goal end effector position. We scaled À so the magnitude of world and robot cost would roughly contribute equal weight to the total cost. Note that using only the robot cost term causes the agent to completely focus arm placement and ignore block placement. To initialize the environment, first set the robot and block spawn locations. The block is initialized with uniform noise 6em next to the platform. The robot gripper is then moved above the block, and then perturbed with uniform
Pick-and-place
15

---

Published as a conference paper at ICLR 2022
noise of 3cm. We give the controller up 5 steps achieve the current goal, before switching the next goal for maximum episode length steps.
A.4 CYCLEGAN BASELINE.
To the best of our knowledge, prior methods perform zero-shot transfer of visuomotor skills between robots. However, we have looked into alternative approaches that might plausibly perform few-shot transfer between robots. In particular, based the human-to-robot transfer approach of(Smith et al.] (2020), we can train a CycleGAN to translate images of the test-time robot to the train-time robot. Then, having trained full robot+world dynamics models the train-time robot, we may apply them directly to the translated images for accomplishing robotic tasks. This is also
similar transfer approaches proposed Rao al. (2200) Raychaudhuri et al. (021]) and Roy & Konidaris (021).
We used the official CycleGAN implementation with default hyperparameters. The training process used 1k videos (12k images) per robot. We trained the CycleGAN until convergence, and the CycleGAN outputs look visually accurate (see website).
A.5 REAL WORLD CONTROL EXPERIMENT DETAILS.
Figure 8: The objects used in the control experiments. The few-shot WidowX250 transfer experiment used all objects, while the zero-shot Franka experiment used the bear and watermelon.
The bear, watermelon, box, octopus, and shark used in the control experiments seen in Figure S and vary in size, texture, color, and deformability.
In the few-shot WidowX200 control experiment, the robot tasked with pushes on five different objects that vary in shape, color, deformability, and size. We chose two directions, a forward and sideways direction, giving total of 10 push tasks. We run trials for each push task for total of 50 pushes method. For the zero-shot Franka experiment, the controllers are evaluated on two different objects pushed three different directions. Each pushing task repeated times for a total of 30 trials per controller.
For the cost function, we only used the image-based world cost function C.p by setting X 0.
CEM action selection. As mentioned in Section 3 the CEM algorithm is used to search for action trajectories that minimize the given cost function. The CEM hyperparameters are constant across controllers to ensure that all methods get the same search budget for action selection. For the few-shot W200wX200 experiment, the CEM action selection generates 300 action trajectories of length 5, selects the top 10 sequences, and optimizes the distribution for 10 iterations. For the zero-shot Franka experiment, the CEM action selection generates 100 action trajectories length 5, selects the top 10 sequences, and optimizes the distribution for iterations.
16

---

Published as a conference paper at ICLR 2022
Final Distance Per Task ID
Dist. to goal pose (cm)
V-C
8
RA-C
0
20
40
60
80
100
Task ID
Figure 9: We evaluate the performance of the pixel cost (V-C) and robot-aware cost (RA-C) by running 100 push tasks where the robot must achieve the object-only goal image. Then we visualize for each task, the final pose distance between the object and the object pose in the goal image. The dashed horizontal line indicates the success threshold of lcm. The robot-aware cost succeeds in most tasks by outputting sensible cost over the world region, while pixel cost distorted by the robot.
A.6 EVALUATING THE ROBOT-AWARE COST ISOLATION
The control experiments in performed in Section evaluate the rohot-aware cost with various choices of dynamics models. Here, we evaluate the cost function more closely in isolation by using ground-truth dynamics of the simulator instead of a learned model.
In section 3.1.2] we propose to separate the conventional pixel-wise cost into a robot-specific and world-specific cost, We report the results of a simulated experiment, where set up a block pushing environment with the Fetch robot. The environment consists of three objects with varying shapes, colors, and physics. We sample the goal image by moving one of the objects from its initial pose to random pose 10cm away.
We then run the visual foresight pipeline using ground truth dynamics and the given cost function for 5 action steps, and record the final distance between the object and its pose in the target image. Success is defined as moving the object within lcm of the goal pose. As seen in Figure 9 The robot-aware cost (RA-C) gets 95% success rate, where as pixel cost (V-C) only gets 16% success rate.
The robot-aware cost is able to disregard the extraneous robot when computing the pixel difference between the current image with the robot and the goal image without the robot. Pixel cost on the other hand, results in the CEM selecting actions that move the robot out of the scene rather than moving the object to the correct pose.
A.7 IMITATING HUMAN GOAL IMAGES.
We evaluate the robot-aware cost's effectiveness achieving goal images with a human arm. Videos are on the project website. We collect five goal images by recording human pushing demonstrations, and use the last image from each video as the goal image seen in Figure 10 Human masks are annotated using Label Studio (Tkachenko et al. 2020-2021). We run RA/RA and RA/Pixel controllers, which differ only in the planning cost. RA/RA achieves all goal images, whereas RA/Pixel fails on all goal images. See the website for videos.
A.8 USING ADDITIONAL CAMERA VIEWS
Next, we analyze the benefit using additional viewpoints for the robot-aware cost function. One potential local minima of the robot-aware cost is occlusion of the object by the gripper, since robot-
17

---

Published as a conference paper at ICLR 2022
Figure 10: Example human goal image.
aware cost does not compute costs over the robot region. However, we did not find this occlusion behavior to be a problem in practice since our CEM search was powerful enough to find the true solution (which has lower cost than the occlusion minima). We can also address this by adding an tested multi-view version of our cost function on simulated pushing in Table 4
additional viewpoint. setup, and found improved performance
Using multiple views gave larger performance gains in the pick-and-place domain. Here, we repeat the pick-and-place experiment from section by training multi-view video prediction model on the WidowX, and then transfer it to multi-view scene of the Fetch robot.
Table 4: Adding an additional view improves pick-and-plan performance.
Fetch
Fetch
Dynamies Model
Cost
Pick-and-Place
Pick-and-place
(1 View)
(2 Views)
VF+-State
Pixel
0/20 (0%)
0/20 (0%)
RA
Pixel
0/20 (0%)
0/20 (0%)
Vtate
RA
4/20 (20%)
8/20 (40%)
RA
RA
8/20 (40%)
14/20 (70%)
A.9 EFFECT OF ROBOT APPEARANCE ON PREDICTION
In addition to the real-world transfer examples reported in the main paper, we experiment with robot transfer in MuloCo simulations Todorov al. 2012) by setting up tabletop manipulation workspace. During training, the model is trained on 10,000 videos of a gray Widow 5-DOF robot performing random exploration. Then, we evaluate the model on various modifications of the WidowX. First, we try changing the color of the entire WidowX from gray to red. Next, we extend the forearm link of the WidowX by 10cm, similar to the real-world modified WidowX experiment in Section 4 Finally, we evaluate on WidowX with longer and red link. As seen in Table the robot-aware model outperforms the vanilla model world PSNR and SSIM metrics.
For the color change, the vanilla model predicts still image for all timesteps, which suggests that the network does not recognize the robot as the training time robot. Our model invariant to the color shift of the robot due to the masking, and can accurately predict the trajectory with little degradation quality.
With the link length change, the vanilla model is able to predict robot movement, but it replaces the longer link with the original short link. In some cases, the longer link allows the robot to contact the object and move it. Our model correctly predicts the object movement, but the vanilla model fails to model the object interaction and movement since the object contact is not possible with the shorter link.
18

---

Published as a conference paper at ICLR 2022
Finally, in the longer and different color link setting, the vanilla model is able recognize and predict movement for the unaltered parts of the robot. It replaces the long red link with the original short gray link, and leaves the long red link in the image artifact. Similar to the previous experiment, the vanilla model has degraded object prediction while our model can still accurately predict the dynamics.
Table 5: Zero-shot video prediction results to a modified WidowX200 in simulation. The modified WidowX200 has different color and/or link lengths.
Color and Link Change
Color Change
Link Change
PSNR t
SSIM t
PSNR
SSIM t
PSNR
SSIM
VF
37.5
0.976
42.3
0.994
40.3
0.987
RA (Ours)
38.7
0.985
45.2
0.997
40.9
0.991
A0 RAC PERFORMANCE ON TRAINING ROBOT
As a samity check, we tested if the baseline and our method is suitable for planning on the original robot. We follow the exact setup as the simulated pushing and pick-and-place experiments, except that we do not change the robot during evaluation. As seen in Table 6 our method is marginally better in performance in prediction and control in comparison with the baselines.
Table 6: Performance on the training robot.
Dynamies Model
Push Success
Pick-and- and--pace Success
Cost
VF+SStte
Pixel
86/100 (86%) 92/100 (92%)
9/20 (45%)
RA
RA
11/20 (55%)
A.11 ROBOT MASKS AND CALIBRATION.
Acquiring robot masks that accurately segment the robot and world is crucial for the robot-aware method, as it depends on the mask to train the dynamics model and evaluate costs as mentioned in Section 1.1 and 3.1.2 The first step acquiring robot masks to get the camera calibration, which is the intrinsics and extrinsics matrix of the camera. If we have physical access to the robot and camera, acquiring the camera calibration is trivial. In our experiment setup with the Franka and WidowX200, we use AprilTag to calibrate the camera extrinsics, i.c. the transformation between camera coordinates and robot coordinates given the camera intrinsics.
Extracting calibration from RoboNet. However, if we do not have access to the robot and camera, as in RoboNet, acquiring camera calibration still possible. RoboNet does not contain the camera extrinsics information, but does contain the camera model information. Therefore, we use the default factory-calibrated camera intrinsics for each corresponding camera model.
Next, RoboNet contains the 3D positions of the robot end-effector in the robot coordinates for each image. For each viewpoint, we hand-annotate a few-dozen 2D image coordinates of the corresponding end-effector, and use OpenCV's camera calibration functionality to regress the camera extrinsics given the camera intrinsics and labeled 3D-2D end-effector point pairs.
Synthesizing masks. Once we have the camera calibration of the robot, we render the 3D model of the robot, and project it to a 2D segmentation map using the camera calibration. We use the MuloCo simulator for this process. Conveniently, the MuJoCo simulator the geometries for a given camera viewpoint. By setting up an empty MuJoCo scene with only the robot geometries,
render segmentation map of
render the geometry segmentation map and use that as the robot mask.
19

---

Published as a conference paper at ICLR 2022
Using depth. Observe that the robot mask (t) above is computed from only the robot state, without any reference to the world state. As a result, it can not account for robot occlusion by objects in the scene. For example. if the robot is pushing large object towards the camera, the part of the object occluding the robot will count as the rohot region. With RGB-D observations oft), which are commonly used in robotics, it is easy refine (t) to remove occluded regions. To do so, may compute the distance (,(r, y) of all robot pixels through the above projection, and zero out pixels in the mask where (x, y) is greater than the observed depth at that pixel. In other words, pixels that are closer to the camera than the computed distance of the robot at that location must correspond occluders. In practice, in our experiments, we find it sufficient to use RGB cameras and ignore occlusions.
20


Reviews:
[{"review_id": "qcQHH-KPznu", "reviewer": "Reviewer_rNxF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " Strengths:\n* A nice example of a method that cleverly utilises \"knowledge we already know\" rather than trying to learn it from data!\n* The overall idea is well-explained with a sufficient number of illustrative figures and experiments, both in simulation and in the real world, across multiple platforms and datasets (e.g. RoboNet).\n* Very interesting few-shot and zero-shot transfer results!\n* Thorough analysis has been performed on the experiments. Both quantitative and qualitative results are presented.\n\nWeaknesses and Suggestions:\n* The method used to project and segment a robot shape into the image plane of a camera is a central component that makes RAC possible but not much is said about it.\n* _Disentangled_ might not be the best term here - in the representation learning literature we could claim two things are disentangled from each other if they are independent and contain little mutual information for each other - e.g. dimensions in a multi-dimensional learned feature space of a GAN, VAE. In the presented method we do separate robot pixels from world pixels when deciding what will P_w see during its training. However, that does not mean that P_w does not contain information about the robot's movement too - it operates over the masks of the robot in the images, which evolve over time. If that is not the case it should be better explained exactly what image information does P_w see and how is the robot information factored out such that P_w is disentangled from the robot's dynamics.\n* The notion of a Proxy Robot is mentioned in Section 3.2 - this was a bit confusing. Aren't you just masking out the robot pixels or _\"...projecting a black \"proxy” robot over the true robot in all images...\"_ refers to something else?\n* Given that the evaluation is over multiple videos, you could provide standard deviation statistics for the results in Table 1, together with the mean estimates that are already there (if it would not clutter the table too much)\n* Choosing goal images that only have objects in them and no robot arms, in the experiments for transferring visual policies across robots, might be putting the baselines at an unfair disadvantage - why not 50/50 - half the images have robot arms and the other half don't?\n* A few words on the the limitations of the method would be very useful - e.g. how much is RAC dependent on proper calibration - both for the camera'a parameters and the relative pose of the camera with respect to the observed robot. How good are camera-to-robot calibration methods for robots that look similar - e.g. a UR3 vs UR5 vs UR10?\n* It's rarely the case that the world stay the same and only the robot would change - objects could change colour and size, camera pose might change, scene illumination might change, etc. Being robust to such perturbations is no less important. However, given the robot-aware nature of the work, keeping the world fixed is a fair assumption for this paper's contributions.", "summary_of_the_paper": "The presented work is can be seen as a more sample efficient improvement/augmentation of the Visual Foresight method (Ebert et al.) in the context of transfer learning for robots. The authors present the notion of robot-aware models - the overall dynamics model is explicitly decomposed onto a known analytical dynamics model for the robot and a learned dynamics model (from pixels) for the objects in the scene. The analytical model, combined with off-the-shelf image plane projection methods, tell us how a robot would look like from the camera's PoV after a set of taken actions. This in turn allows us to exclude any robot-related pixels from the scene observations and provide the learned dynamical model only with object-related pixel information, effectively making the learned model invariant to the robot's appearance. The same logic is applied to a novel planning cost function used for performing visual MPC. As a result cross-robot transfer can be more easily facilitated in the context of table-top manipulation tasks, where the goal is specified through a goal image.", "summary_of_the_review": "Overall, this paper does present a novel solution to the problem of robust transfer learning on robots in a sample efficient fashion. It is a good example of how to successfully leverage information we have about the problem a priori (robot dynamics) but still learn anything else we don't know (object dynamics) from data. This is proposed as an alternative to the otherwise brittle methods which try to learn everything from data, end-to-end. While there are minor details to be elaborated on and fixed, the idea itself and the evidence provided here are sufficient for publication and are a useful contribution to the field.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_TiOZrE9yL_", "reviewer": "Reviewer_BJUg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper works on a quite interesting problem and provides an intuitive approach. The neural network architectue is based on an extension of stochastic video generation architecture using LSTM. The overall design of the system as described in the main text looks interesting and seems to perform well on certain tasks. Overall, it is a bit hard to follow the details of the methodology in the main text, especially in terms of presenting the novel contributions in this paper. The algorithm for testing in Appendix definitely is helpful, is it possible to add one for the training phase to summarize? It would be interesting to see a discussion on why the pick-and-place results are always much lower than pushing results. Real robot experiments seems to be limited, 50 pushesfor few-shot experiments, 30 for zero-shot.\nMinor: Could you clarify what was the final form of Eq 3, how was it used in the final system? How was Lambda chosen? In experiments section, there is a repetition: \"1.8k videos videos\", so please do a thorough proofread. ", "summary_of_the_paper": "This paper addresses the problem of how to leverage data collected from on a robot to reduce/remove the need for robot specific data. The authors propose a model based RL policy based on coupling robot-agnostic and robot-specific dynamcis modules. Experiments with simulated and real robots are demonstrated, showing transferability of visual model based policies. ", "summary_of_the_review": "The paper addresses an interesting problem, presents good results, needs some clarifications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LeOZ0NtOMuV", "reviewer": "Reviewer_2sNR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is an attempt to enhance previous works on visual foresight by adding the information coming from the analytical model of the robot forward kinematics. The basic idea is to use such model to analytically compute a mask to decouple the robot specific visual appearance from the task-relevant world appearance. Decoupling world-specific and robot-specific visual information attempts at avoiding a limitation of non-decoupled visual foresight, which doesn't allow to privilege world specific visuals with respect to robot-specific visuals. \n\nThe paper is well written and easy to read even though some important details (e.g. the definition of the proxy-robot) require the reader to read carefully the supplementary material. \n\nAn important limitation of the paper is that authors fail at explaining if in the world observation the robot information is completely removed from the information related to objects present in the scene (which is what I was expecting as a natural improvement from the original VF paper). Even after reading the paper a few times, it actually seems that some robot information is retained in the world observation (i.e. as mentioned in A.1 \"Proxy-robot. An image o is composed of the robot image or = o ◦ Mr and world image ow = o ◦ (1 − Mr) where Mr is the mask of the robot. We propose to edit all images ow = ProxyRobot(o, M)\".). If that's the case, it seems that even ignoring completely the robot aware cost in Eq. 3 (by setting λ to zero) wouldn't remove completely the robot-specific information. \n\nA minor limitation of the paper is that it is incremental leveraging a significant amount of prior works (e.g. Smith at al. 2020) without being clear about how prior work has been accessed to (e.g. authors' implementation, open-source release or other).\n\n= SOME OTHER COMMENTS = \n- Page 6. \"The current and future mask are concatenated with the image, and the end effector pose is tiled into the latent spatial embedding\".  This sentence isn't clear. Which image and which latent space are authors referring to?\n\n- PSNR and SSIM haven't been defined and it would be better if authors could define them in the paper. ", "summary_of_the_paper": "The paper presents an extension of previous works on training a visual policy to control a robot arm in a manipulation context. The policy is pre-trained on a large robot experience dataset (e.g. different robots interacting with different objects in a table-top setting). The dataset is used to train a model for the visual dynamics (i.e. next visual & robot state given current visual & robot state and applied action). The visual dynamics are used to reach a desired visual state using an existing technique: visual foresight (VF) by Finn & Levine, 2017 and Ebert et al., 2018. \n\nThe main novelties of the paper of the paper are two. First, authors propose a technique to learn a decoupled visual dynamic model: the world model (i.e. a model to propagate forward the world-observations consisting of an image of all objects in the scene excluding the robot) is decoupled from the robot model (i.e. a model to propagate forward the end-effector pose and a mask which represents the robot body in the visual space). Second, the visual foresight planning relies on a newly proposed cost which combines and decouples two costs: a desired robot-state cost and a desired world-state cost.", "summary_of_the_review": "As mentioned above, the major limitation of the paper is its limitation in explaining to which extent the world model is affected by the robot model and by the state of objects in the scene. Ideally, a natural evolution of previous approaches would be to provide a world model completely decoupled from the robot model and mainly influenced by the state of objects in the scene. \n\nEven after reading the paper carefully, it's still unclear to me if the world model proposed by the authors is mainly used to propagate forward the information of the state of objects in the scene. Considering that the authors rely on an analytical model of the robot, this is an important requirement and I would like to be convinced that either this is already a property of the proposed world model or that this is not the case for good/understandable/sound reasons. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CLi4_1cnqBW", "reviewer": "Reviewer_U6rV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method presented in this paper is interesting and seems to fill a gap in the existing literature.\n\nThe paper is written clearly and presents the concepts in a cohesive way. The experimentation is fairly convincing and uses both simulated and real data.\n\nThe papers suffers from the following limitations (sorted by decreasing order of severity):\n\nThe paper starts and ends with a bold claim about the fact that it permits \"zero-shot transfer onto new robots for the first time\".  This is not the case, as the literature review itself shows.  That being said, the paper does seem to fill a gap in the literature.  It would make sense to re-calibrate this claim.\n\nThe paper does not provide enough technical details about the models' implementation.  It is therefore hard to conclude whether the approach is technically sound. Section 3.2 that provides implementation details falls short of providing enough information.\n\nThe complementarity to relevant work in Section 5 should be discussed in greater details.  In particular, the comparison with Chen et al. (2018) should be clarified.\n\nIn Section 2, the paper mentions that \"When parts of this projection may be occluded, we may further use RGB-D observations to easily identify and handle them\". This approach is debatable, as the appearance of the robot and of the world could be very similar on this metric in the real-world.\n\nA number of references must be corrected (for example, reference 7 seems to be an ICML paper but mentions PMLR, reference 2 lacks details, etc.)", "summary_of_the_paper": "This paper proposes a robot-aware RL-based approach to transfer learning from a robot to another. The approach factorizes the model into two models, a robot model and a world model. The method is evaluated against several baselines on simulated and real data. The paper claims that it permits zero-shot transfer onto new robots for the very first time.", "summary_of_the_review": "This paper provides an interesting approach to zero-shot learning but falls short of providing enough technical details about the implementation. The complementarity to existing work is not discussed enough.  For these reasons, I advocate for a reject.\n\nEDIT AFTER DISCUSSION PERIOD: The authors have addressed the majority of my concerns.  Reading their answers and the discussions with other reviewers, I updated several scores upward and my final score to \"marginally above\".\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to improve the transfer of visual control policies between robots.  The method extends a visual foresight approach using a learned robot-agnostic world-dynamics model and a (potentially analytic) robot-specific robot-dynamics module.  A key aspect of the method is to form a blocky mask over the robot's body in the visual image, thus allowing the learned dynamics to depend less on appearance attributes of the robot.  Planning with these dynamics models operates in the visual observation space.  The method is tested for zero-shot transfer on multiple physical robots and also with simulated robots, with positive results across multiple experiments.

The reviews raised multiple concerns on the details of the method and the clarity of the presentation that were largely addressed in the author response.  The authors refined their claims to a demonstration of zero-shot transfer of visual skills across real-world robots.  The evaluation of the proposed transfer method on real-world robots is a notable strength of the paper.  The core limitation, raised by one reviewer, is that the generality of the invariance is only provided by a visual mask on the color and appearance of the robot.  This serves as a limited form of invariance to the robot's dynamics.  

The discussion phase did not yield a consensus on the merits of the paper, with the proposed method seen as useful but still limited.  Three knowledgeable reviewers indicate to accept the paper and one indicates to reject.  The formal description does not make the world-dynamics explicitly robot-agnostic, as the learned model $(P_w(o_w'|o_w, r, r', a)$ in Equation 2) still explicitly depends on the actions of the robot ($a$) and thus the robot's dynamics.  Despite some limitations in the formalism, the practical utility of the method is convincingly demonstrated in multiple robot experiments.  This is a clear contribution to the literature, which is supported by three reviewers.  The paper is therefore accepted.
Decision: Accept (Poster)