Vision-Based Manipulators Need to Also See from Their Hands
Authors: Kyle Hsu
Moo Jin Kim
Rafael Rafailov
Jiajun Wu
Chelsea Finn
Venue: ICLR 2022 Oral (2022)
Keywords: reinforcement learning
observation space
out-of-distribution generalization
visuomotor control
robotics
manipulation

Abstract: We study how the choice of visual perspective affects learning and generalization in the context of physical manipulation from raw sensor observations. Compared with the more commonly used global third-person perspective, a hand-centric (eye-in-hand) perspective affords reduced observability, but we find that it consistently improves training efficiency and out-of-distribution generalization. These benefits hold across a variety of learning algorithms, experimental settings, and distribution shifts, and for both simulated and real robot apparatuses. However, this is only the case when hand-centric observability is sufficient; otherwise, including a third-person perspective is necessary for learning, but also harms out-of-distribution generalization. To mitigate this, we propose to regularize the third-person information stream via a variational information bottleneck. On six representative manipulation tasks with varying hand-centric observability adapted from the Meta-World benchmark, this results in a state-of-the-art reinforcement learning agent operating from both perspectives improving its out-of-distribution generalization on every task. While some practitioners have long put cameras in the hands of robots, our work systematically analyzes the benefits of doing so and provides simple and broadly applicable insights for improving end-to-end learned vision-based robotic manipulation.

Published as a conference paper at ICLR 2022
VISION-BASED MANIPULATORS NEED TO ALSO SEE FROM THEIR HANDS
Kyle Hsu", Moo Jin Kim', Rafael Rafailov, Jiajuun Wu, Chelsea Finn Stanford University
(kyleh eesue fai lov, ji junuu,, cbfi in ecs stanford.edu
ABSTRACT
We study how the choice of visual perspective affects learning and generalization in the context of physical manipulation from raw sensor observations. Compared with the more commonly used global third-person perspective, hand-centric (eye-in-hand) perspective affords reduced observability, but we find that it consis- tently improves training efficiency and out-of-distributiin generalization. These benefits hold across a variety of learning algorithms, experimental settings, and distribution shifts, and for both simulated and real robot apparatuses. However, this is only the case when hand-centric observability is sufficient; otherwise, in- cluding third-person perspective is necessary for learning, but also harms out- of of-distribution generalization. To mitigate this, we propose to regularize the third- person information stream via variational information bottleneck. On six repre- sentative manipulation tasks with varying hand-centric observability adapted from the Meta- Meta-World World benchmark, this results ing agent operating from both perspectives improving its out-of -ff-oii-rrriuutionn gen- eralization on every task. While some practitioners have long put cameras in the hands of robots, our work systematically analyzes the benefits of doing so and provides simple and broadly applicable insights for improving end-to-end learned dision-based robotic manipulation.'
state-of-the-an reinforcement learn-
00
O3
O
3
84
To
Sb
T
T
S
On
On
O
+
Figure 1: Illustration suggesting the role that visual perspective can play facilitating the acquisition of sym- metries with respect to certain transformations on the world state s. To: planar translation of the hnd-effcctor and cube. Ti: vertical translation of the table surface, and-effecton and cube. T2: addition of distractor objects. O3: third-person perspective. On: hand-centric perspective.
INTRODUCTION
Physical manipulation is so fundamental a skill for natural agents that it has been described as "Rosetta Stone for cognition" (Ritter & Haschke, 2015). How can we endow machines with similar
"Co-first authorship. Order determined by coin flip. Projecct website: https: // tes. .oog ee e com www/seeingg. from- hands.

---

Published as a conference paper at ICLR 2022
mastery over their physical environment? One promising avenue is use data-driven approach, in which the mapping from raw sensor observations of the environment (and other readily avail- able signals, e.g. via proprioception) to actions is acquired inductively. Helpful inductive biases in modern machine learning techniques such as over-parameteriin descent have enabled surprising (and poorly understood) generalization capabilities in some appli- cations (Neyshabur et al., 2014; Belkin et al.. 2019; Zhang et al., 2021). Despite this, visuomotor policies learned end-to-end remain brittle relative many common real-world distribution shifts: subtle changes in lighting, texture, and geometry that would not faze a human cause drastic perfor- mance drops (Julian et al., 2020).
models and stochastic gradient
While a wide variety of algorithms have been proposed to improve the learning and generalization of object manipulation skills, in this paper we instead consider the design of the agent's s observation space, facet of the learning pipeline that has been underexplored (Section 5). Indeed, in some applications of machine learning, e.g., image classification or text summarization, the disembodied nature of the task affords relatively little flexibility in this regard. Yet, even in these settings, simple data processing techniques such as normalization and data augmentation can have noticeable effects on learning and generalization (Perez& & Wang, 2017). The role of data only be more profound in an embodied setting: any sensors capable of being practically instrumented will only provide a partial observation of the underlying world state. While partial observability is typically regarded as a challenge that only exacerbates the difficulty of a learning problem (Kaelbling et al., 1998), we may also consider how partial observations can facilitate the acquisition of useful symmetries.
The natural world gives clear examples of this. For instance, because cutaneous touch is inherently restricted sensing portions of the environment in direct contact with the agent, tactile sensing by construction exhibits invariances to many common transformations on the underlying world state; grasping an apple from the checkout counter (without looking at it) is largely the same as doing so from one's 's kitchen table. Due in part to the nascent state of tactile sensing hardware (Yuan et al., 2017) and simulation (Agarwal et al., 2020), in this work we investigate the above insight in vision, the ubiquitous sensory modality in robotic learning. In particular, we focus on the role of perspective as induced from the placement of cameras. To roughly imitate the locality of cutancous touch, we consider the hand-centric (eye-in-hand) perspective arising from mounting camera on robotic manipulator's s wrist. We also consider the more commonly used third-person perspective afforded by fixed camera in the world frame.
The main contribution of this work is an empirical study of the role of visual perspective in learning and generalization in the context of physical manipulation. We first perform head-to-head compar- ison between hand-centric and third-person perspectives in a grasping task that features three kinds of distribution shifts. We find that using the hand-centric perspective, with no other algorithmic modifications, reduces aggregate out-of-distribttion failure rate by 92%, 99%, and 100% (relative) in the imitation learning, reinforcement learning, and adversarial imitation learning settings in sim- ulation, and by 45% (relative) in the imitation learning setting on real robot apparatus.
Despite their apparent superiority, hand-centric perspectives cannot be used alone for tasks in which their limited observability is a liability during training. To realize the benefits of hand-centric per- spectives more generally, we propose using both hand-centric and third-person perspectives in con- junction for full observability while regularizing the latter with a variational information bottle- neck (Alemi et al., 2016) to mitigate the latter's detrimental effects on out-of-distributio general- ization. We instantiate this simple and broadly applicable principle in DrQ-v2 (Yarats et al., 2021), state-of-the-art vision-based reinforcement learning algorithm, and find that it reduces the aggregate out-of-distributte failure rate compared to using both perspectives naively by 64% (relative) across six representative manipulation tasks with varying levels of hand-centric observability adapted from the World benchmark (Yu et al., 2020).
PROBLEM SETUP
Preliminaries: MDPs and POMDPs. We frame the physical manipulation tasks considered in this work as discrete-time infinite-horizon Markov decision processes (MDPs). An MDP M is 6-tuple
AS P. R. 7, ()), where S is a set of states, AA is a set of actions, P : S x A + IISS) is a state- transition (or dynamics) function, R. S A + F is reward function, Y (0, 1) is discount factor, and μ E II(S) is an initial state distribution. An MDP whose state cannot be directly observed can be formalized as partially observable MDP (POMDP), 8-tuple (S,A, P. R, 7, u4 Ω, O)
2

---

Published as a conference paper at ICLR 2022
that extends the underlying MDP with two ingredients: a set of observations Q and an observation
function O- : S x A + [[))). We consider only a restricted class of POMDPs in which the observation function is limited to be O S - Ω. To solve POMDP, we optimize policy æ .2 + (TA)) to maximize the expected return R(M, क o O) = P... iSt= 7 R(s,, az)]. where O maps state an action distribution via composing the policy and observation function.
Observation functions. In this work, we denote the observation functions corresponding to the hand-centric and third-person visual perspectives as On and O3, respectively. We also consider proprioception, denoted as Op. Often, multiple observation functions are used together; for example, we denote using both the hand-centric and proprioceptive observations as OK+P.
Invariances and generalization. We say that function f : X' x V + 2 invariant in domain sub- space X transformation T - X' iff Vr X. E V f(T().)) = f y)- We formalize the notion of generalization by saying that π o 0 generalizes in M to distribution shift caused by transformation T iff R(M. π O) is invariant in . to T. We consider two kinds of generalization: in-distribution and outtr--distiibuttio generalization, also referred to as interpolation and extrapo- lation. The latter corresponds to the agent generalizing in M to some specified transformation, and the former is special case when the transformation is identity. In this work, we limit the scope of the transformations on M we consider to those acting on the initial state distribution μ through the state set S. A few concrete examples of such transformations are illustrated in Figure 1.
3 HAND-CENTRIC vs.
PERSPECTIVES
The first hypothesis we investigate is that using the hand-centric perspective On instead of the third- person perspective O3 can significantly improve the learning and generalization of the agent ToO. In this section, we probe this hypothesis in settings where the hand-centric perspective gives sufficient observability of the scene (we consider when this does not hold in Section 4).
3.1 SIMULATED EXPERIMENTS
We first consider visuomotor grasping task instantiated in the PyBullet physics engine (Coumans & Bai, 2016-2021). A simulated Franka Emika Panda manipulator is tasked with picking up a specific cube that initially rests on a table. The action space is 4-DoF, consisting of 3-DoF end-effector position control and 1-DoF gripper control. Observation functions include O₁, and Oz, which output 84 84 RGB images, and Op, which outputs 3D end-effector position relative to the robot base, 1D gripper width, and Boolean contact flag for each of two gripper "fingers".
We use three learning algorithms: imitation learning with dataset aggregation (DAgger) (Ross al., 201 1), reinforcement learning using data-regularized Q-functions (DrQ) (Kostrikov et al., 2020), and adversarial imitation learning using discriminator--ctor--iiii We defer exposition these algorithms to Appendix A.2. We run DAgger and DrQ on three exper- iment variants that each target a test-time distribution shift in the table height, distractor objects, and table texture. The distribution shifts are detailed and visualized in Appendix A.1. With DAC, assess in-distribution generalization in the training environment and out-of-distribution generaliza- tion between demonstration (demo) collection and the training environment. Details on the model architectures and hyperparameters used can be found in Appendices A.3 and A.4. DAgger and DrQ results are reported in Figure 2 and aggregated in Table 1. DAC results are reported in Figure 3, with experiment variant descriptions in the caption.
(DAC) (Kostrikov et al., 2018).
For DAgger (left two columns of Figure 2), we find that the hand-centric perspective leads to clear improvements in out-of- distribution generalization (test) across all three experiment variants de- spite in-distribution generalization progress (train) being essentially identical between π o Oh+p and Ostp- The only exceptions to т Ontp generalizing better are in some instances of the distractor objects variant. Here, seeing the red, green, and blue distractor objects during training was sufficient for both Or+p and O3+p to learn to ignore these object colors, even under distractor distri- bution shift. Generalization to white distractors was likely facilitated by the RGB representation of white the "sum" of red, green, and blue.
For DrQ (right two columns of Figure 2), the differences between π TOON+p and π Os+p extend into training time. In the table height variant, Ontp exhibits increased sample efficiency for training as well as similar objects variant, Ohtp converges before n Oztp makes any significant progress on success
generalization benefits as seen for DAgger. For the distractor
3

---

Published as a conference paper at ICLR 2022
Table 1: Aggregate cube grasping out-of-distribuiis generalization performance across the three experiment variants computed from taking the three highest success rates achieved in each run of each DAgger or DrQ agent. The hand-centric perspoctive leads to the best aggregate success rate for both algorithms, with its 95% confidence intervals of the interquartilc mean (IQM) not overlapping those of the third-person perspective.
success rate (%)
mean
median
IQM
95% CI of IQM
T O Oh+pp Ostp
92.2
99.4
97.5
997.0, 97.8)
DAgger
64.4
61.1
69.3
[67.0, 71.4]
T O Oh+p
94.7
100.0
99.3
99.44 4, 99.9]
DrQ
π O3+p
46.6
59.4
46.9
[44.3, 49.3]
ι o Ou-p
x o Ostp
Zshift = -0.10
Rubiit
-0.05
Sehiit
Fehih
+0.05
Zubiit +0.10
DAgger train. /1I
DAgerr test, (OOD
DrQ train/ll)
DrQ test /OOD
1.0
1.0
1:00
0.8
SU008S rste
0.8
6.6
0.6
6.6-
0.4
0.4
c.4
6.00
0.0
0.0
10
60
80
-0
40
10
epochs
epochs
enviromment
steps (x10²)
environment stegis (x10*)
1 rod, 1 green, and 1 blue
3 rodl green 3 blue
3 brown
3 white
black
DAgger train//D
DAggee test, (OOD
DrQ train/II
DrQ text /OOD
1.0
1.00
1009
6.8
SUN0SS LAbe
5000000
0.0
Strees
6.6
6.4
0.2
0.0
40
60
10
7001
epocls
eguaadas
environment stepe (x10)
enviconmnnt steps (x10')
5 tabll textures
20 held-out table textures
DAgger maiin/ ITD
DAggerr test /OOD
DrQ train/ID
DrC test/OOD
1.04
1.0g
1.0
100
0.6
8000008 rate:
0.5
stceess na2e
0.8
0.8
0.6
0.6
2.6
0.0
0.4
0.0
2.4
0.4
0.2
0.0
0.0
200
50
200
eguoclas
sporchs
environment sters (x 10
environmentt steps (x10')
Figure 2: DAgger and DrQ results for eube grasping. The first, second, and third rows respectively contain results for the table height (shifted by Zaitt), distractor objects, and table textures experiment variants. See Appendix A.1 for visualizations of the train and test distributions for each experiment variant. Compared to the third-person perspective (dashed lines), the hand-centric perspective (solid lines) leads better out-of- distribution generalization performance across all three distribution shifts for both DAgger and DrQ. For DrQ. we also see appreciable improvements sample efficiency when using the hand-centric perspective. Shaded regions indicate the standard error of the mean over three random seeds.
rate (though we did observe increasing returns). Since DrQ trained π o Ostp to convergence for the other variants within the same interaction budget, follows that the presence of the distractors rendered the training task too hard for и O3+p. but not for π Optp- In the table textures variant, the generalization improvement of π On+p over π o Ostp is less extreme. We attribute this to invariances to image-space transformations learned via the data augmentation built into DrQ. In Appendix A.5, an ablation in which this augmentation is removed further shows its importance.
For DAC, we find stark improvements in the generalization of On over that of π O3. In the first DAC-specific experiment variant (left plot of Figure 3), т o O₂ fully generalizes in-distribution with few as 5 demos, whereas æ O₂ achieves significantly lower success, even with 25 demos and much more online interaction. In the second variant (center plot Figure 3), the distribution shift between demo collection and training barely affects On but severely compromises the
4

---

Published as a conference paper at ICLR 2022
O
25 ditmos
20 decuacs
demoos
100
11.00
25 dezuve
212 demos
DAC train /T>
DAC /OD
DAC train/IID ( distractoos)
SUSCERS ratee 8
auevess ratle
0.8
0.6-
66
0.6
stpeess
0.4
0.4
0.2
012
10 10
10 12
environment steps
environment steps
(x10))
environment steps (x100)
Figure 3: DAC results for cube grasping. Left: base variant (initial object and end-effector position random- ization) with distrihution shift hetween demo collection and training. Center: hase variant with table height shift between collection of demos and training. Right: base variant plus three distractor objects with no distribution shift between demo collection and training. Across the three experiment variants, the hand-centric perspective enables the agent to generalize in- and out-of out-of-distriuution more efficiently and effectively. Shaded regions indicate the standard error of the mean five random seeds.
training of O3. In the third variant (right plot of Figure 3), despite the presence of distractor objects giving the discriminator strong predictive power in distinguishing between demos and agent behavior, π O₁ still achieves significant measure of in-distribution generalization, whereas Og makes little progress even with eight times the number of demos. We remark that, in the context of adversarial imitation learning, On, achieves its sample efficiency and robustness without any special requirements on the training data (Zolna et al., 2020) or modified training objectives (Xu & Denil, 2020).
3.2 REAL ROBOT EXPERIMENTS
We further investigate our hypothesis in a world analogue of the above environment: a Franka Emika Panda manipulator equipped with a parallel-jaw gripper is tasked with grasping a Scotch- Brite sponge amongst distractors (Figure 4). The action space consists of 3-DoF end-effector posi- tion control and I-DoF gripper control. O₁ and O3 output 100 x 100 RGB images, and O, outputs the 3D end-effector position relative to the robot base and the ID gripper width. train π Oh+p and π To0s+p via behavior cloning (BC) on 360 demonstrations collected via teleoperation, obtaining 85% success rate on the training distribution for both. Like above, we consider test-time distribution shifts in the table height, distractor objects, and table texture. Assessment of each distribution shift instance was done using 20 sampled environment initializations. Appendix B presents the setup in full detail well results stratified by distribution shift. Table summarizes the results. Videos available on our project website. These experiments indicate that the hand-centric perspective better facilitates ofiitt-ditti generalization for visuomotor manipulation not only in simulation, but also on a real robot.
Aggregate sponge grasping out-of- distribution generalization performance across the three sets of distribution shifts.
Table 2:
success rate (%) median
mean
IQM
TT O Oh+p
52.0
52.5
53.3
Figure 4: Sample observations from Og (left) O (right) in our real robot apparatus.
Oztp
20.0
12.5
15.8
INTEGRATING HAND-CENTRIC AND THIRD-PERSON PERSPECTIVES
The previous experiments demonstrate how hand-centric perspectives can lead to clear improve- ments in learning and generalization over third-person perspectives. Unfortunately, this does not mean that the use of hand-centric perspectives panacea. The limited observability of hand- centric perspectives is double-edged sword: depending the environment and task. it can enable π o O₁ to establish useful invariances, or confuse म o O, by enforcing harmful ones. In this sec- tion, we focus evaluating across tasks of varying hand-centric observability, including those in
5

---

Published as a conference paper at ICLR 2022
which insufficient observability severely undermines o O₁- How can we realize the benefits of hand-centric perspectives even in such scenarios?
4.1 REGULARIZING THE
INFORMATION STREAM
Insufficient observability arising from using O₁ alone necessitates the inclusion of O3. While using both perspectives should effectively resolve the issue of insufficient observability and enable the agent to train, we know from Section that the use of the third-person perspective can hamper out-of-distribution generalization by allowing the agent to "overfit" to particularities of the training distribution. To mitigate this, we propose to regularize the third-person perspective's representation. While multiple regularization techniques could conceivably be suitable to this end, we choose the variational information bottleneck (VIB) to use in our experiments due to its simplicity, theoretical justification, and empirical performance (Alemi et al., 2016).
For our subsequent experiments, we build on top of the state-of-the-art vision-based actor-critic reinforcement learning algorithm DrQ-v2 (Yarats et al., 2021) (see Appendix C.3 for detailed description). When we use both hand-centric and third-person observations о₁, and 0g, we instantiate two separate image encoders fs. and fea- We denote the corresponding representations as 2 and 23- These are concatenated before being fed to the actor Th and critic networks Qa Q0.-
We apply a VIB to the third-person information stream to regularize the DrQ-v2 critic. This amounts to a variational approximation to maximizing the mutual information between the third-person ob- servations and the critic's 's predictions of the temporal difference targets while minimizing the mutual information between the third-person observations and their representations. We implement this by replacing the deterministic third- hirr-ppesoon encoder fca with a stochastic encoder Pc, (Zs 03), specify- ing a prior p(Zz). and adding a weighted KL divergence term to the critic loss. The VIB VIB-regularized DrQ-v2 critic objective is
C(E,, Es. 02) = ED, [ critic (En, E3, 01, 02)] + ED [Bg DKI. (pt, (za (03) p(Z)))], (1)
where D the replay buffer. We specify Pes (2))03) as a diagonal Gaussian and p(za) as a standard Gaussian, which enables analytical computation of the KL divergence. We use the reparameteriza- tion trick to enable optimization of the first term via pathwise derivatives. We do not need to modify the actor objective as only gradients from the critic are used update the encoder(s) in DrQ-v2. We remark that (variational) information bottleneck can be applied to many imitation learning and reinforcement learning algorithms (Peng et al., 2018; Goyal et al., 2019; Igl et al., 2019; Kumar et al., 2021).
4.2 META- WORLD EXPERIMENTAL SETUP
We evaluate the learning and generalization performance of seven DrQ-v2 agents: т o ORtP (hand-centric perspective), o Oa+p (third-person perspective), O8+3+p (both perspectives), т o On-3-p + VIB(z3) (both perspectives with a VIB on the third-person information stream), and three ablation agents introduced later. We evaluate the agents on six tasks adapted from the Meta-World benchmark (Yu et al., 2020). We design the task set to exhibit three levels of hand- centric observability (high, moderate, and low) with tasks per level. In each task, simulated Sawyer robot manipulates objects resting on a table. The action space is 4-DoF, consisting of 3-DoF end-effector position control and 1-DoF gripper control. We do not use the original Meta- Meta-World ob- servation space as it contains low-dimensional pose information about task-pertinent objects instead of images. Rather, we configure the observations so that On and O₂ output 84 x 84 RGB images, and Op outputs 3D end-effector position and 1D gripper width. See Figure 5 for a visualization of each task through the lens of On, and O3. Experiments in Appendix C.5 establish that propriocep- tion alone is not sufficient to reliably solve any of the tasks. Experiments in Appendix C.6 consider variations of reg-insert-side that require an additional 1-DoF end-effector orientation control.
While the distribution shifts in the experiments of the previous section arise from transformations on the table height, distractor objects, and table textures, in this section we focus on distribution shifts arising from transformations on the initial object positions. All object positions have disjoint initial train and test distributions such that the latter's support "surrounds" that of the former (see Table in Appendix C.2 for details).
Aside from adapting the DrQ-v2 algorithm to our setting as described above, we use the original DrQ-v2 model and hyperparameters with some minor exceptions (see Appendix C.7 for details).

---

Published as a conference paper at ICLR 2022
hundle-press-side
butlon-press
socuer
pey-inserl-side
reach-burd
peg-insert-side-mae
Figure The Meta- World tasks used the experiments in Section top row contains third-person obser- vations Os, and the bottom row contains corresponding hand-centric observations On Initial object positions are randomized, The last two tasks, reach-hard and peg- insert-side siddhd, are custom custom-made, made; there, the green goal and the green peg randomly initialized either the left the right of the gripper with equal probability, and they are initially visible the hand-centric perspective. Because of the severely limited hand-centric observability, the third-person perspective crucial for learning to direct This is especially the case for the reach-hard task, which we modified to prohibit any vertical end-effector movement. See Section C.I in the Appendix for more details about each task.
gripper to correct location.
Hyperparameters that are common to agents are shared for a given task. With agents that include regularization, we tune the regularization weight(s) on validation sample from the test distribution. Test success rate is computed on a separate sample of 20 environments from the test distribution.
4.3 META- META-WORLD WORLD RESULTS AND DISCUSSION
Main experimental results in Meta- World are summarized in Table 3. Figure 6 provides detailed comparisons between the four DrQ-v2 agents introduced above. When using both perspectives, regularizing the third-person perspective's representation via a VIB reduces the interquartile mean of the out-of-distribution failure rate across all six tasks by 64% (relative). We also note that this method achieves the best performance in each individual task, albeit sometimes with less sample efficiency. To properly explain these phenomena, we embark on more stratified analysis and discussion of the results.
Table 3: Aggregate out-of- di-iiitribttion generalization performance across all six Meta- tasks computed from taking the three highest success rates achieved in each of each DrQ-v2 agent. Using both perspectives with VIB the third-person perspective's representation results in the best aggregate success The last three agents are ablations above are presented discussed Appendix C.4.
success rate (%)
mean
median
IQM
95%
CI of IQM
OLtP
69.6
73.1
73.7
72.1, 75.4
π o Oz4p
55.6
51.7
56.6
53.4, 59.1
O443+P
67.4
68.1
66.3
62..9.9.53
π O O843+p + VIB(z3)
84.4
82.5
87.7
85.2,90.0
Ou+3+p VIB Zn VIB
51.8
54.7
58.2
50.3,65.99
o Oy43tp + VIB(Zs)
40.6
32.8
34.5
32.7,36.4
Oh43+p E2(23)
70.3
70.0
69.8
667.7, 72.3
Characterization of hand-centric observability via training performance. When the world state is sufficiently observable via the hand-centric perspective, we expect the convergence during training t to match or surpass that of To0 We find that this is indeed the case for handle-press- side, button-press, soccer, and peg-insert-side (high and moderate hand-centric observability), and not the case for reach-hard or peg-insert-sidee hard (low hand-centric observability). This validates selection and framing of the tasks at different levels of hand-centre observability. Interestingly, we observe that in veg-insert-sse side-hard, т Ontp eventually achieves some success during training by "zooming out" improve its observability
of ToO

---

Published as a conference paper at ICLR 2022
train/ID
Lest/ OOO
Oh-p
π o Oztp
On+3+p
On-3pt VIB(zz)
handle-press
button-press
1.0
1.0)
1.0
1.0
SU0000 1008
Sucess 1008
Stecette Dite
0.6
9.0
0.6
0.6
0.4
0.4
0.2
0.0
0.01
0.0
0.0
100
250
100
150
200
100
2X0
250
100
150 260
anvirommmtt (x10f)
enviromment steps (x10))
envirommet steps (x10')
enviromment (x10')
soccer
peg-insert-side
5000006 1800
0.8-
SHA00R rake
9.
succese rate
0.8-
SHuc00e belee
0.5
0.6
66
0.6
0.4
9.4-
0.2
0.22
0.2
200
COO
800
300
010
mnvirnmmmet stegis (x10*)
enviroomment slegss (x10))
Cnvinnmmee stepss (x10')
onvinomment slegus (x10*)
reach-hard
peg insert-side hard
1.0
1.0
1.0
1.00
810008 rate
Stives cole
0.8
E1OCSS rale
0.6
0.5
8.6
0.4
0.4
0.4
0.0
0.0
0.0
0.0
2000
400
6000
200
G0O
201
400
630
500
2011
400
800
envirommmnnt steps (x10')
steps (x10))
stepe (x1)0)
steps (x10')
Figure 6: DrQ-v2 results for Seta-World. Each row contains results for two manipulation tasks that roughly exhibit the same level of hand-centric observability. which decreases from top bottom (high, moderate, low). Using the proposed approach (both perspectives with VIB the third-person perspective's representa- tion) leads to the best out-of-distrrbution generalization performance for levels hand-centric observability (though it is matched by the hand-centric perspective when hand-centric observability is high, as expected). Shaded regions indicate the standard error of the mean over three random seeds.
Hand-centric perspective vs. third-person perspective. When hand-centric observability is high or moderate, ToOn+p generalizes better out-of-distribution than ToO3-p. corroborating results from Section 3 with another form of distribution shift. When hand-centric observability is low, OR+p both trains and generalizes worse than π Os-p- This supports our motivation for considering using both perspectives in conjunction.
Effect of combining the hand-centric and third-person perspectives. When hand-centric observ- ability is high moderate, including the third-person perspective harm generalization. We that for button-press, peg-insert-side, handle-press, and soccer, æ o Oh+3+p is sandwiched between Or4p and o O34p on the test distribution. The drop from On+p 004334p is significant for the former two tasks, and marginal for the latter two. This validates our hypothesis that includ- ing Og enables the agent to "overfit" training conditions. When hand-centric observability is low, combining both perspectives results in π Oh+3+p matching or surpassing the training performance of π o Ohtp and т T00+pp and greatly outperforming both test time. This validates our hypothesis that, when necessary, including third-person observations helps resolve training difficulties arising from insufficient hand-centric observability.
Effect of regularizing the third-person information stream via a VIB. π Ok+3+p + VIB(za) consistently improves upon π On+3+p in out-of-distribution generalization for tasks except handle-press-side, in which the two are about equal. This directly indicates the benefit of the VIB regularization. These gains come at the cost of slightly delaying the convergence of training. How- ever, it is arguable that this is inevitable and even desirable. A known phenomenon in neural network training that spurious correlations or "shortcuts" the data are sometimes easier to lear than causal relationships (Sagawa et al., 2019). Slower training and higher generalization may indicate
8

---

Published as a conference paper at ICLR 2022
the avoidance of such behavior. Additionally, in button-press, π Oh+3+p + VIB (z3) recovers the -usto--disttiuutionn generalization exhibited by т o Oltpp and when hand-centric observability is moderate, π Oh+3+p + VIB(zz) improves upon o OhtP"
Ablations on π o Ox+3+p + vIB(zs). We conduct three ablations on this best-performing agent to better understand the design decisions underlying its gains. See Appendix C.4 for description, results, and discussion.
5 RELATED WORK
Learning for vision- based object manipulation. A wide range of works have focused on algorith- mic development for end-to-end learning of vision-based object manipulation skills (Levine et al., 2016; Agrawal et al., 2016; Finn et al., 2016; 2017; Kalashnikov al., 2018; Srinivas et al., 2018; Ebert et al., 2018; Zhu et al., 2018; Jayaraman et al., 2018; Rafailov et al., 2021). Some works on learned visuomotor control use eye-in-hand cameras for tasks such grasping (Song et al., 2020) and insertion (Zhao et al., 2020; Puang et al., 2020; Luo et al., 2021; Valassakis et al., 2021), and oth- which pre-date end-to-end visuomotor learning use both sye-in-hand and third-person cameras for visual servoing (Flandin et al., 2000; Lippiello et al., 2005). Very few works consider the design of camera placements (Zaky al., 2020) conduct any controlled comparisons on different combi- nations of visual perspectives (Zhan et al., 2020; Mandlekar et al., 2021; Wu et al., 2021). Unlike all of these works, we propose specific hypotheses regarding the benefits of different choices of visual perspective and perform a systematic empirical validation of these hypotheses with evaluation on multiple families of learning algorithms, manipulation tasks, and distribution shifts. Concurrently with our work, Jangir et al. (2022) investigate fusing information from hand-centric and third-person perspectives using a cross-view attention mechanism and demonstrate impressive sim2real transfer.
The role of perspective on generalization. Hill et al. (2019) assess an agent learning to execute language instructions in simulated environments using high-level actions and find that using an egocentric observation space results in better systematic generalization to new instruction noun- verb combinations. Szot et al. (2021) find that an agent tasked to pick up a certain object (using abstracted grasping) in cluttered room generalizes better to unseen objects and room layouts when using wrist- and head-mounted cameras in conjunction. Our work provides complementary evidence for the effect of perspective on the generalization of learned agents in markedly different setting: we consider vision-based physical manipulation. Also, the aforementioned works rely on memory- augmented agents resolve partial observability is common in navigation tasks, whereas observations as is standard in tabletop manipulation and demonstrate the importance of regularizing their representation.
Invariances through data augmentation in reinforcement learning, Several works have inves- tigated ways to apply standard data augmentation techniques from computer vision in the rein- forcement learning setting (Laskin et al., 2020; Kostrikov et al., 2020; Yarats et al., 2021). These works consider data augmentation as a means to prescribe invariances to image-space transfor- mations, whereas we are concerned with how different observation functions facilitate generaliza- tion to envirommental transformations. To emphasize that these directions are orthogonal, we use DrQ (Kostrikov et al., 2020) and DrQ-v2 (Yarats et al., 2021) in experiments.
CONCLUSION
In this work, we abstain from algorithm development and focus on studying an underexplored de- sign choice in the embodied learning pipeline: the observation function. While sand-centrie robotic perception is more traditionally instrumented with tactile sensing, our findings using vision affirm that perspective, even when controlling for modality, can play an important role in learning and generalization. This insight may very well apply robotic systems that leverage tactile sensing- Overall, in the context of end-to-end learning for visuomotor manipulation policies, our findings lead us to recommend using hand-centric perspectives when their limited observability is sufficient, and otherwise defaulting to using both hand-centric and -erron perspectives while regulariz- ing the representation of the latter. The breadth of the learning algorithms, manipulation tasks, and distribution shifts that we base these conclusions on, coupled with their simplicity and lack of re- strictive assumptions, suggests that these recommendations should be broadly applicable, even to more complex, longer-horizon tasks that feature sub-tasks analogous to those experiment with.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank Kaylee Burns, Ashvin Nair, Eric Mitchell, Rohan Taori, Suraj Nair, Ruohan Zhang, Michael Lingelbach, Qian Huang, Ahmed Ahmed, and Tengyu Ma for insightful discussions and feedback on early drafts. We also thank our anonymous ICLR reviewers for their constructive com- ments. This work was in part supported by Google, Apple, Stanford Institute for Human-Centered AI (HAD, Amazon Research Award (ARA), Autodesk, Bosch, Salesforce, and ONR grant N00014- 21-1-2685. KH was supported by Sequoia Capital Stanford Graduate Fellowship. CF is fellow in the CIFAR Learning in Machines and Brains program.
REPRODUCIBILITY STATEMENT
Appendices A, B, and C flesh out the full experimental protocol in stringent detail. We expect this to be sufficient for independent replication of our main findings. Separately, we have included links to code used for our simulation experiments our project website.
REFERENCES
Arpit Agarwal, Tim Man, and Wenzhen Yuan. Simulation of vision-based tactile sensors using physics based rendering. arXiv preprint arXiv:2012.13184, 2020.
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. arXiv preprint (.Xiv::606.074400 2016.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.004100 2016.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical biri-variance trade-off. Proceedings of the National Academy of Sciences, (5642)):5549-11855 2019.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http:://pybullet.rr, 000 2016-2021.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint 1rXiv:1812.00568,, 2018
ChelseaFini Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512-519. IEEE, 2016.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imita- tion learning via meta-learning In Conference on Robot Learning, pp. 357-368. PMLR, 2017.
Grégory Flandin, François Chaumette, and Eric Marchand. Eye-in-hand/eee-too visual servoing. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Confer- ence on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 3, pp. 2741-2746. IEEE, 2000.
cooperation for
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the in- formation bottleneck. arXiv preprint arXiv: 1901. 10902, 2019.
10

---

Published as a conference paper at ICLR 2022
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning. pp. 1861-1870. PMLR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L Mc- Clelland, and Adam Santoro. Envirommental drivers of systematicity and generalization in situated agent. arXiv preprint arXiv: 2019.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances Neural Information Processing Systems, 2016.
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. arXiv preprint (et../.11111111 2019.
Rishabh Jangir, Nicklas Hansen, Sambaran Ghosal, Mohit Jain, and Xiaolong Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 2022.
Dinesh Jayaraman, Frederik Ebert, Alexei A Efros, and Sergey Levine. Time-agnostic prediction: Predicting predictable video frames. arXiv preprint arXiv: 1808.07784, 2018.
Ryan Julian, Benjamin Swanson, Gaurav Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. arXiv preprint arXiv:2004. /0/90, 2020.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprini arXiv: 10293, 2018.
Ilya Kostríkov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- Diiseriminator-actor-cooi Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv::809.00. 1809.02955, 2018.
llya Kostríkov, Denis Yarats, and Rob Fergus. Image augmentation all you need: Regularizing
deep reinforcement learning from pixels. arXiv preprint arXiv: 13649, 2020.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. arXiv preprint 1rXiv:2109.1081:: 2021.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. arXiv preprint arXiv:2004. 14990, 2020.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1377 2016.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 02971, 2015.
Vincenzo Lippiello, Bruno Siciliano, and Luigi Villani. Eye-in-hand/eye- han multi-camera vi- sual servoing. In Proceedings of the 44th IEEE Conference on Decision and Control, pp. 5354- 5359. IEEF, 2005.
11

---

Published as a conference paper at ICLR 2022
Jianlan Luo, Oleg Sushkov, Rugile Pevceviciute, Wenzhao Lian, Chang Su, Mel Vecerik, Ning Ye, Stefan Schaal, and Jon Scholz. Robust multi-modal policies for industrial assembly via rein- forcement learning and demonstrations: A large-scale study. arXiv preprint arXiv:2103.1151, 2021.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint 10Xiv:2108.032000 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 11Xiv:1312.5602, 2013.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6611 2014.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational dis- criminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining infor- mation flow. arXiv preprint arXiv:/880.00e 2018.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04622) 2017.
En Yen Puang, Keng Peng Tee, and Wei Jing. Kovis: Keypoint- visual servoing with zero-shot sim-to-real transfer for robotics manipulation. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 7527-7533. IEEE, 2020.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154 1168. PMLR, 2021.
Helge Ritter and Robert Haschke. Hands, dexterity, and the brain. Humanoid robotics and neuro- science: science, engineering and society, 2015.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv: 1.00873,, 2019.
Shuran Song, Andy Zeng, Johnny Lee, and Thomas Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters, 5(3):4978 49855 2020.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal plan- ning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning, pp. 4732-4741. PMLR, 2018.
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, al. Habitat 2.0: Training home assistants to rearrange their habitat. arXiv preprint arXiv:2106. 14405, 2021.
Eugene Valassakis, Norman Di Palo, and Edward Johns. Coarse-to-fine for sim-to-real: Sub- millimetre precision across the workspace. arXiv preprint arXiv:2105.1 /283, 2021.
Bohan Wu, Suraj Nair, Li Fei-Fei, and Chelsea Finn. Example-driven model-based reinforcement learning for solving long-horizon visuomotor tasks. arXiv preprint 109iv:2109.100 10312. 2021.
Danfci Xu and Misha Denil. Dositive-unlabeled reward learning. Conference on Robot Learning, 2020.
12

---

Published as a conference paper at ICLR 2022
Denis Yarats, Rob Fergus, Alessandro Lazarie, and Lerrel Pinto. Mastering visual continuous trol: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.0964. 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094-1100. PMLR, 2020.
Wenzhen Yuan, Siyuan Dong, and Edward H Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
Youssef Zaky, Gaurav Paruthi, Bryan Tripp. and James Bergstra. Active perception and representa- tion for robotic manipulation. arXiv preprint arXiv:2003.06734 2020.
Albert Zhan, Philip Zhao, Lerrel Pinto, Pieter Abbeel, and Michael Laskin. A framework for effi- cient robotic manipulation. arXiv preprint arXiv:2012.07975, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communicutions of the ACM, 64(3):107- 115, 2021.
Tony Z Zhao, Anusha Nagabandi, Kate Rakelly, Chelsea Finn, and Sergey Levine. Meld: Meta- reinforcement learning from images via latent state models. arXiv preprint arXiv:2010.13957 2020.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arxhv:180..09564, 2018
Konrad Zolna, Scott Reed, Alexander Novikov, Sergio Gomez Colmenarejo, David Budden, Serkan Cabi, Misha Denil, Nando de Freitas, and Ziyu Wang. Task-relevant adversarial imitation learn- ing. Conference on Robot Learning, 2020.
13

---

Published as a conference paper at ICLR 2022
A CUBE GRASPING EXPERIMENT DETAILS
A. ENVIRONMENT DETAILS
For the cube grasping experiments Section 3, we investigate three types of distribution shifts. The experiment variants for DAgger and DrQ are summarized in Table 4. The DAC experiments featured a subset of these conditions explained the caption of Figure 3. Figures 7, 8, and visualize each type of distribution shift.
Table 4: Variants of the cube grasping environment with distribution shifts used for the DAgger and DrQ experiments. All task variants include initial object position and initial end-effector position randomization that is consistent across train and test. Table textures are from the describable textures datasct (DTD) (Cimpoi et al., 2014).
train
test
table height distractor objects table texture
Sshift. = 0 1 red, green, blue texture E 5 DTD textures
Sshift E {-0.10,-0.. -0.05, +0.05, +0.10} 3 of color {red, green, blue, brown, white, black) texture E 20 held-out DTD textures
Figure 7: Visualization of the table height distribution shift used in the cube grasping experiments. From left to right, Shit is -0.10, -0.05, 0, +0.05, +0.10, +0.10. The top and bottom rows contain the third-person and hand- randomized in this visualization
centric perspectives. respectively. Positions of the cube and end-effector for the sake of elarity.
Figure 8: Visualization of the distractor objects distribution shift used in the cube grasping experiments. From left to right, we have "mix" (1 red, 1 green, 1 blue), 3 red, 3 green, 3 blue, 3 brown, 3 white. and 3 black. The top and bottom rows contain the third-person and hand-centric perspectives, respectively. Positions of the cubes and end-effector
randomized in this visualization for the sake of clarity.
14

---

Published as a conference paper at ICLR 2022
table textures distribution shift used in cube grasping experiments. top two rows contain the third-person and hand-centric perspectives of the five table textures used during training for DAgger and DrQ, and the bottom two rows contain the perspectives of five held-out textures used at test time (out of twenty total held-out textures). Positions of the eube and end-effector not randomized in this visualization for the sake of clarity. The textures acquired from the describable textures dataset (DTD) (Cimpoi al., 2014).
Figure 9: Visualization
15

---

Published as a conference paper at ICLR 2022
A.2 ALGORIHTM
The dataset aggregation (DAgger) algorithm proposed by Ross et al. (2011) is an iterative online algorithm for training imitation learning policy. In each iteration i (which call a "DAgger round"), the current policy т; is run to sample a set of trajectories, and an expert policy x' used label each of the visited states with an optimal action. These labeled trajectories are aggregated into a dataset D that grows in size over the DAgger rounds, and the imitation learning policy कं is trained on the entire 'D for some number of epochs before repeating the above procedure in the next iteration. The geeeettrr-ggnnniingggg policy is often modified such that in earlier DAgger rounds the expert policy x is utilized more heavily than the imitation learning policy म. when collecting new trajectories, i.e. B, + (1 3.) where B; is typically annealed over time (e.g., linearly from 1 to 0 over the DAgger rounds).
The Data-regularized Q (DrQ) algorithm proposed by Kostrikov al. (2020) is a model-free, off- policy, actor-critic reinforcement learning algorithm that applies image augmentation techniques commonly used in computer vision (primarily random shifts) to input images, along with regular- izations of the Q target and function, such that deep neural network-based agents can be trained effectively from pixels. The original DrQ paper uses soft actor-critic (Haarnoja et al., 2018) and DQN (Mnih et al., 2013) as backbones; we use the soft actor-critic version in our experiments be- cause the cube grasping action space continuous.
The discriminator actor-critic algorithm (DAC) was proposed Kostrikov et al. (2018) 8) and is an off- policy version of the generative adversarial imitation learning (GAIL) method (Ho & Ermon, 2016). Unlike Kostrikov et al. (2018) 8) we use a deterministic reinforcement learning algorithm similar to that of Fujimoto et al. (2018), as we find this helps stability. To scale the method image observations, we apply similar augmentation techniques as in Kostrikov et al. (2020).
A.3 MODEL ARCHITECTURES
For DAgger in the cube grasping experiments discussed Section 3, we feed the 84 x 84 images into ResNet- 18 convolutional image encoder (He al., 2016) trained from scratch, with the final classification layer replaced by a linear layer that outputs a 64-dimensional representation. We con- catenate proprioceptive information (3D end-effector position relative the robot base, ID gripper width, and Boolean contact flag for each of two gripper "fingers") to the image representation, and the result passed into feedforward policy and value networks with two hidden layers of 32 units each.
For DrQ, we use the original actor-critic DrQ model proposed by Kostrikov al. (2020), except for one modification: we concatenate proprioceptive information (3D end-effector position relative to the robot base, 1D gripper width, and Boolean contact flag for each of two gripper "fingers") to the flattened image representation before feeding it into the actor and critic networks.
For the DAC algorithm we use the same convolutional architectures as Kostrikov et (2018). The convolutional encoder is shared between the discriminator, actor and critic. We use additional MLP heads with capacities 128, 256 and 256 respectively for those components, as we empirically found that lower-capacity networks decrease the likelihood of overfitting to spurious features.
16

---

Published as a conference paper at ICLR 2022
A.4 HYPERPARAMETERS
The DAgger, DrQ, and DAC hyperparameters used in the cube grasping experiments are listed in Tables 5, 6, and respectively.
Table The default hyperparameters used DAgger in cube grasping environment.
parameter
setting
num. DAgger rounds
6
num. new episodes per round
200
num. epochs per round
batch size
256
3 schedule (% time expert policy is used)
linear anneal from to 0 over rounds 10-3
learning rate
Table The default hyperparameters used DrQ in cube grasping environment.
parameter
setting
replay buffer capacity
10a
action repeat
2
seed steps
1000
n-step returns
3
mini-batch size
128
discount y
0.99
optimizer
Adam
learning rate
10-3
critic target update frequency
2
critic Q-function soft-update rate t
0.01
actor update frequency
2
actor log stddev. bounds
[-10,
init. temperature features dim. hidden dim.
0.1
50
1024
Table The default hyperparameters used for DAC the cube grasping environment.
parameter
setting
replay buffer capacity
15
10x
action repeat
1
seed steps
200
mini-batch size
512
discount y
0.99
optimizer
Adam
learning rate
10-3
actor
deterministic
critic Q Q-function soft-update rate
0.01
features dim.
64
discriminator hidden dim.
128
actor hidden dim.
256
critic hidden dim.
256
17

---

Published as a conference paper at ICLR 2022
A.5 ABLATION STUDY: REMOVING THE DATA AUGMENTATION DRQ
In this experiment, we investigate the effect of the data augmentation component of the DrQ al- gorithm by ablating it. The motivation is see whether data augmentation is still necessary for policy using the hand-centric perspective, which already leads to lower overfitting and better gen- cralization. The results in Figure 10 reveal that the augmentation is indeed still crucial because without it, training does not converge even with much more environment interaction. However, the hand-centric perspective does still enable the agent to make greater progress.
Oh-p
Ostp
Zolilt 0.10
Zshill
0.05
Zohilt
Zshult
+.05
Fallii +0.10
TiQ train/10
DrQ test /O0n
DiQ angg) train/IID
DrQ (no ang] test/OOD
St00000 1ate
SHOSSE
110
201
100
200
100
200
30
000
300
101
2000
3.0
(0)
500
environment stepe (x10*)
eavironment steps (x10")
envisomment steps
(x10²)
euvironment steps (x10')
red, green, blue
3 red or green or 3 blue
3 brown
3 white
black
TeQ train/lD ID
DrQ test /OOD
DrQ [no ang) train/ID
DrQ (no ang) OODD
1.0
1.0
1/0-
1.04
5000096 1929
0.5
0.6
0.8
0.6
0.0
0.61
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
022
6.0
250
250
500
200
300
environment steps (x10*)
cavironment steps (x110)
enviromment steps (x109)
euvironment steps (x))
5 table textures
20 held-out table texctrres
DeQ IID
lest/OOD
DrQ aug) Irain/ID
aug) lest/OOD
sale:
0.5
80000588 zate
stccoss rate
0.6-
8.6
0.4
0.0
0.4
0.2
0.2
0.00
0.0
200
200
200
200
400
500)
800
200
000
500
arvizonaccal steps (x10')
envirecnnnn: steps (x10P)
cnvironmnnnn stergs (x10)))
enviromnent stepe (x10')
Figure 10: DrQ results for cube grasping with (left) and without (right) image augmentation. Note that the left half of this figure is exact replica of the right half of Figure 2. Ablating the image augmentation component of DrQ reveals its importance; without training fails to converge of environment interaction. However, hand-centric perspective still facilitates faster training than third- person perspective for the first two experiment variants. Shaded regions indicate the standard error of the mean over three random seeds.
with much larger amounts
A.6 MINOR DISCREPANCIES BETWEEN ALGORITHMS
Due to implementation idiosyncrasies, there are minor discrepancies in how each algorithm pro- cesses environment observations. Following Kostrikov et al. (2020), for DrQ and DAC-DrQ ob- servations are "frame-stacked" with three time steps' observations. This was not done for DAgger. Proprioceptives are used for DAgger and DrQ but not for DAC-DrQ. take the position that these differences increase the generalizability of the trends observe. We emphasize that the target effect under consideration is O₁ vs. Og in each setting.
18

---

Published as a conference paper at ICLR 2022
B REAL ROBOT EXPERIMENTS
In this section, we discuss real robot experiments resembling the simulated experiments in Section 3, which presented head-to-head comparison between the hand-centric and third-person perspectives. A few minor differences exist between the simulated and real experiments, which are delineated in Section I. However, the key findings discussed in Section B.2 match those from the simulated experiments, validating the improved generalization performance that the hand-centric perspective provides over the third-pere perdopperoonno perspective vision- based manipulation tasks.
B.I EXPERIMENTAL SETUP
As in the simulated experiments in Section 3, we conduct the real robot experiments with Franka Emika Panda robot arm. The robot tasked with grasping and lifting sponge from a gray bin while other distractor objects present. The action space 4-DoF, consisting of 3-DoF end- effector position control and 1-DoF gripper control. Observation functions include O₁, and Os. which output 100 100 RGB images, and Oş. which outputs 3D end-effector position relative to the robot base and ID gripper width. As before, we perform a head-to-head comparison between OktD and и O3+p. i.e. the policies using hand-centric and third sarrd-peroon visual perspectives (and proprioceptive observations), respectively.
During the training phase, we train behavioral cloning policy until convergence using the same set of 360 demonstrations for both o Ohtp and O3tp- collected via robot teleoperation using vir- tual reality headset and controller. This is roughly the quantity of demonstrations needed achieve reliable grasping performance on the training distribution (85% success rate over 20 episodes) due to randomized initial object positions as well as randomized initial gripper position. Unlike Sec- tion 3, we do not use dataset aggregation (DAgger) here. The target object to grasp is mootch-Britt sponge, with the green side always facing upwards. In addition, at training time, three distractor objects present: folded red washcloth, folded blue washcloth, and yellow sponge decorated with spots.
At test time, introduce three categories of distribution shifts, similar to those in Section 3: unseen table heights, unseen distractor objects, and unseen table textures. Figures 11, 12, and 13 illustrate these distribution shifts. When testing against unseen table heights and table textures, the scene contains the same set of target object and distractor objects that we used at training time.
Figure 11: Table height distribution shifts. Columns from left to right: -0.05 m, -0.025 train, +0.025 m,
+0005 m. Top (bottom): observations from Os (O₂).
19

---

Published as a conference paper at ICLR 2022
Distractor object distribution shifts. Columns from left to right: train, distractor test set 1, distractor Figure 12:
test set 2, distractor test set 3. Top (bottom): observations from Og (On).
Figure 13: Table texture distribution shifts. Columns from left to right: train, blue floral, green watercolor garden, rainbow floral. Top (hottom): observations from O₂ (Ox).
20

---

Published as a conference paper at ICLR 2022
B.2 EXPERIMENTAL RESULTS AND DISCUSSION
The real robot behavioral cloning results are reported in Table 8. We find that the hand-centric perspective leads to significantly greater out-of-distribttion generalization performance across all three experiment variants despite both hand-centric and third-person policies achieving the same performance on the training distribution (85% success rate over 20 episodes), validating the results we sec in simulation.
Table -itto---iittitttionn generalization performance comparisons between hehavior cloning (BC) policy using a hand-centric visual perspective and a BC policy using third-person perspective. The distribution shifts include unseen table heights (where indicates the change from the base table height used during training, meters), unseen distractor objects, and unseen table textures. Compared to the third-person perspective, the hand-centric perspective leads to better -ut-of-generalization distribution performance across all three distribu- tion shifts. In-distribution results are provided to give an idea of a performance ceiling. Each success rate computed over 20 evaluation episodes.
success rate (%)
I Oh+p
O3+p
Sshift 0.05 m
50
Shhift -0.025 m
80
60
unseen table heights
Shilt = +0.025 m
85
35
Stift +0.05
65
distractor test set
55
40
unseen distractor objects
distractor test set
55
10
distractor test set
45
20
blue floral
50
unseen table textures
green watercolor garden
25
15
rainbow floral
10
in-distribution
85
85
21

---

low because Published as a conference paper at ICLR 2022
C META-WORLD EXPERIMENT DETAILS
C.I INDIVIDUAL TASK DESCRIPTIONS
In this section, we explain the tasks that the agents must learn accomplish in the six Mera-World environments discussed in Section 4.2 and visualized in Figure 5. We also explain why each task falls under certain level of hand-centric observability. For details regarding the train and test distributions, see Appendix C.2.
handle- presss--eess--de:: The goal is to press the handle fully downwards. Hand-centric observ- ability is high because the handle is well aligned with the hand-centric camera's 's field of view.
The goal is to push the button fully inwards. Hand-centric observability is high because the button well in view of the hand-centric camera, and the button remains largely in view the gripper approaches and presses it.
button-pree preso:
soccer: The goal is to push or pick-and-piae -n-andl--lce the ball into the center of the goal net. Hand- centric observability is moderate because when the gripper approaches the ball, the observ- ability of the goal net is appreciably reduced.
peg-insert-side: The goal is to lift the peg and insert it into the hole in the target box. Hand-centric observability moderate because when the gripper approaches the peg, the observability of the target box is appreciably reduced.
reach-hard: The goal is to move the gripper to the green goal site, which is initialized either the left right side of the gripper with equal probability (see Figure 14). Hand-centric observability is low because the gripper is initialized at the same height as the goal, and we restrain the gripper from moving vertically. Effectively, if given just the hand-centric perspective's 's observations, the agent does not know in which direction to move the gripper in the beginning of an episode.
peg-insert-side-han The goal is the same in peg-insert-side, but like the green goal site in reach-hard, the peg in this environment is initialized either to the left or right side of the gripper with equal probability (see Figure 14). Hand-centric observability the gripper is initialized at the same height as the peg such that the peg is not initially visible the hand-centric view (though we do not prohibit vertical movement of the gripper as in reach-hard, since this would make the peg insertion part of the task impossible), and also because the peg and target box are initialized much farther apart than they are peg-insert- side (thus, the target box is completely out of view as the agent approaches and grasps the peg).
Figure 14: Visualizations of the sides that the green goal site or peg can he initialized in the reach-hard
and peg-insert-side- tasks in Meta-Worle World, respectively. One the two sides chosen via "coin flip" at the heginning of each episode.
22

---

Published as a conference paper at ICLR 2022
C.2 TRAIN AND TEST DISTRIBUTIONS
At training time, initial positions of the objects in the Meta-World tasks are uniformly sampled within some support. At test time, initial positions are sampled from uniform distribution that is completely disjoint from the training distribution, such that we test on out-of-distribution initial object positions. To implement this, test time we resample the set of initial object positions if any of the positions overlaps with its train-time distribution. The full set of train-time and test-time initial object positions is shown in Table For visualizations, see Figure 15.
Table 9: The distributions of the initial object positions used in the six Meta-World environments. At training time, we use the "narrow" initial object positions. At test time, we use the set difference hetween the "narrow" and "wide" initial object positions, i.e. we repeatedly resample from "wide" until none of the objects in the environment lie in the "narrow" distribution. As result, the train- and test-time distributions are disjoint. The range of positions for each object is given pair of coordinates: (Ziow, yow Pim) ) and (2high, Mhight 2high).
task
"narrow" " init. obj. positions (xr, 2) 2)
"wide" init. obj. positions (x, 2) handlelow : (-0.55, 0.4, -0.001)
handle (- 0.33,, 0.55, -0.001)
handle- pressl---ees-ssiee
handlenigh (-0.15, 0.65, 0.001)
handleaigh
(-0.15, 0.8, 0.001)
button
1-.2,0.85, 0.115)
button
(-0.4, 0.75, 0.115)
button-prec press
buttonhigh
(0, 0.115)
buttonnigh
(0.2,0.9, 0.115)
ballow
0-222 0.6, 0.03)
ballow
0.6, 0.03) (0.1,0.7,0.03)
ballnigh
0.7, 0.03)
ballhigh
soccer
goal
(02, 0.8, 0.0)
goal
0.8,
goah
(0., 0.9,
goal
00.1, 0.0)
peg
(.05, 0.55, 0.02)
pegtow
(.0,0.5,0.02
peg-insert-side
pegnigh (.15, 0.65, 0.02)
geghh
(2,0.7,0.02)
goal (-0.325, 0.5, 0.5,-0.001)
goal : (-0.35, 00.400.001)
goal high 0.227, 0.6, 0.001)
goal high (-225, 0.7,
if goal is on left of gripper:
if goal is on left of gripper:
goal low (-0.2, 0.85, 0.05)
goal low (-0.5, 0.9, 0.05)
goal (-0.2, 0.85, 0.05)
goal (0.1, 0.9, 0.05)
reach-hard
if goal is on right of gripper:
if goal is on right of gripper:
goal : (-0.2, 0.35, 0.05)
goal low (-0.5, 0.3, 0.05)
goal (-0.2, 0.35, 0.05)
goal high (0.1, 0.3, 0.05)
if peg starts on left of gripper:
if peg starts on left of gripper: peg (-0.3, 0.85, 0.02)
pegrow (-0.025, 0.85, 0.02)
gighh (0.025, 0.85, 0.02)
peghh 0.81, 0.02)
goal (-0.525, 0.5, -0.001)
goal low (-0.525, 0.3, -0.001)
goal (-0.525, 0.6, 0.001)
goalnggh (-0.525, 0.75, 0.001)
peg-insert-side-hard
if peg starts right of gripper:
if peg starts on right of gripper:
pegiow : (-0.025, 0.35, 0.02)
pegtow (-0.3, 0.35, 0.02)
pegnigh (0.025, 0.35, 0.02)
pegmigh 0.35, 0.02)
goal
(-0.525, 0.5, -0.001)
goal low (-0.525, 0.3, -0.001)
goal
(-0.525, 0.6, 0.001)
goal (-0.525, 0.75, 0.001)
23

---

Published as a conference paper at ICLR 2022
Figure 15: Visualization of the train and test distributions in the six Meta-Word environments used in the experiments in Section 4. The three columns in the left half of the figure show three sets of initial object positions randomly sampled from the training distribution; the three columns in the right half correspond to the test distribution. From top to bottom are handle-pree preee-ppeesseeee button button-press, soccer, peg-inse reg-insert-side side, reach- hard, and peg mideer--sidehaaeee Both the third-person and hand-centrie perspectives are shown for each random initialization.
24

---

Published as a conference paper at ICLR 2022
C.3 DRQ-V2
DrQ-v2 (Yarats et al., 2021) is state-of-the-art vision-based actor-critic reinforcement learning algorithm that uses deep deterministic policy gradients (DDPG) (Lillicrap et al., 2015) as backbone (whereas DrQ-v1 by Kostrikov et al. (2020) uses soft actor-critie). The DrQ-v2 model includes:
convolutional image encoder f& that outputs representation stacked image observations o and a data augmentation function aug,
fg (aug(o)) given frame-
two critic networks Qo, that output Q-values Qo, (z, a), k 1,2, à clipped double Q- learning (Fujimoto et al., 2018),
and an actor network Th that outputs action (=(2))+,r N(0, a with o² annealed over the course of training.
The individual critic losses are given by
Ci = ETWD (Oo, (z,a) - -n² k == 1,2
where (o,, ay; ft:t+n-11 1, Octn) is sample from replay buffer D and is the temporal difference target estimated via n-step returns:
Y Te
min Qā, (Ze+n, actn) kE(1,2)
0=0
for slow-moving critic weights 01. 02. We omit presentation of the actor loss do not need to modify it; DrQ-v2, only gradients from the critic loss are used to update the weights of the encoder(s).
In terms of the model architecture used in the experiments discussed in Section 4, we use the original DrQ-v2 architecture, except for two modifications: first, concatenate proprioceptive information (3D end-effector position and 1D gripper width) to the flattened image representation before feeding into the actor and critic networks. Second, when using two perspectives the same time (e.g., hand-centric and third-person), we use two separate image encoders that do not share weights. The two representations are concatenated together (along with the proprioceptive information) and fed into the actor and critic networks. The dimensionality of each encoder's output representation is preserved, thereby doubling the dimensionality of the final combined image representation.
25

---

Published as a conference paper at ICLR 2022
C.4 ABLATIONS ON Oh+3+p + VIB(Zs)
To better understand what makes т o 3 + (iB(zz) work the best, we conduct the following ablations. Figure 17 presents the train and test curves of the ablation experiments.
What if both perspectives are regularized? The ablation agent л o Ou+3+p + (z,( + VIB (2)) adds separate VIB to the hand-centric information stream in an analogous manner how the third-person perspective's s representation is regularized (detailed in Section 4.1). We use the same 3g for both and tune Br. Note that setting 3, 0 for о O1+3+p + VIB + VIB recovers O++3+2 VIB (z)) modulo stochasticity in 2n, so we limit the lowest value 5A can take to 0.01. We find that in no task does o Oh+3+p + VIB + (IB outperform o Oh+3+p + VIB(zs), validating our choice of only regularizing the third-person perspective's representation.
Assessing the importance of the hand-centric perspective. +3+3+p + (IB( uses a second third-person perspective Oy instead of the hand-centric perspective On- Visualizations from this additional huird-pesoo perspective are shown in Figure 16. We re-tune 83 for this agent. We find that и 03'33+ + (IB(2a) performs significantly worse than OR+3+p VIB(zz), affirming the benefit of using the hand-centric perspective in the multi- perspective setting.
Zn dependent regularization of Zz. VIB(2 reduces the information contained in zy without di- rectly considering ž₁.- With the ablation agent π Oh+3+pp + (z(z),, we consider a simple form of Zy de-eenendnt regularization of Z3 in which push Z3 towards Zú by adding weighted regulariza- tion term Ora |23 stopgrad (2,)) [ |3 to the DrQ-v2 critic objective. This approach seems promising O3+p across all six tasks, suggesting that even in the midst of substantial partial observability, Zn may represent information in useful and gen- eralizable way. We tune atg. We find that π Oh+3+p + (z(za) marginally improves over vanilla O 3++ but still comes far short of π o On+3+p + VIB(za), suggesting that the two perspectives contain important complementary information that better represented separately.
given that и Oh+3+p consistently outperforms
panndle-reess-idee
button-press
soccer
peg-insert-side
reach-hard
peg-insert-side-hard
Figure 16: Visualizations of the two third-person person perspectives used in the ablation agent र T000 43+ + VIB (za)
discussed in Section 4.3. The top contains the original third-person perspective; the bottom row contains the second third-person perspective used only in the ablation.
26

---

Published as a conference paper at ICLR 2022
train/ID
lest/OOD
x. a Oh+3-pp VIB
a Os-Sep
VIBizs) + VIB(zs)
x Op43ep + VIB
Chesup (1(2s)
handle press-side
button-press
1.0
1.0
1.0
1.0
800008 1ete
00A0 pale
8820055 rate
tole
0.0
9.
0.6
0.0
0.2
0.0
0.00
0.0
0.01
150
250
envirommmnt
(x10')
enviromment steps (x10))
enviremment
(x10')
envirommennt (x100))
SOccer
peg- -ngetssrt-
8000996 130e
11.8
H0LOSe tale
0.8
succsss rate
11.8
stouebb Iile
0.5
0.6
0.4
0.2
200
100
600
8.0
2.0
300
800
200
0.0
600
800
200
600 800
envirommment steps (x10f)
envirommennt steps (x10*)
envirstmeet steps (x10')
envirommeent stegs (x10")
reach-hard
peg- pegginsert-sidee hard
1.0
1.0
1.0-
1.0
810008 cale
SUNOSS cale
0.0
0.6
0.6
0.4
0.4
0.4
8.4
0.0
0.0
0.0
0.0
200
010
0.01
200
GUD
200
800
(x10')
steps
[x 10")
steps (x.10)
Figure 17: Ablation studies on æ Oh+a+p + (IB(za) in World. Shaded regions indicate the standard стror of the mean three random seeds. Note that for Oh+344p VIB(za) ( + VIB VIB(z:) (z:) in peg- peg-insert-siie insert-side, DrQ-v2 training did converge within the specified number of training steps for one of three random seeds (hence larger shaded regions the lower -fiittiiittiitttion generalization performance). In addition, π Oh+3+p + VIB(z,) + (IB(za) in peg-insert-side-hard. DrQ-v2 training did not converge for any of the
three seeds.
27

---

Published as a conference paper at ICLR 2022
C.5 PROPRIOCEPTION-ONEE ABLATION
In this ablation experiment, we demonstrate that visual observations are necessary component of the observation space, i.e. that the tasks we experiment with cannot be consistently solved with pro- prioceptive observations alone. We run DrQ-v2 on all six World tasks introduced Section 4.2 without image observations and show the results in Figure 18. Unlike policies that are afforded vision, these proprioception-only policies do not approach 100% success rate on the training distri- butions.
train IID
test/ /OOD
bandle-press
button-press
1.0
1.0
5000096 1908
0.8
SHORCR EMC
0..8
500000S ItLe
0.8
Suceess
0.4
0.4
0.4
0.4
0.2
0.0
0.0
0.0
0.0
50
100
200
200
50
100
150
200
200
50
100
100
200
200
50
100
15n 200 200
envirnomment stoges (x10*)
enciinomennt stegss (x1)))
euviromment stepus (x10*)
enviromment slegus (x100)
S0c0eT
peg- -ngg-ssrt-eede
1.00
1.0
1.00
1.01
SU0088 rate
SHOOO rale
SUOves rate
SHICOOO rale
0.0
S..
0.5
0.0
0.2
0.2
0.2
0.0
environment stoprs (P))
mnvinomment steps (x 10P)
environment steps (x (P)
mnvimoment stegus 10")
reach-hard
peg- regginsert-side- hard
1.0
810000 2028
0.8
SHR0RR 2000
0.5
0.8
0.8
0.4
0.4
0.7
0.7
0.0
0,00
200
400
600
200
-00
600
200
400
600
800
3000
400
00
800
anvirocmmnnt stops (x (P))
ensiromment steps
environment stope [x 102)
environment steps (x1)0)
Figure 18: DrQ-v2 results for Meta-World for Op. i... agent that operates solely from proprioception. These results establish necessity of vision for these tasks, as the policies are unable consistently solve task instances without it. For the reach- hard task, in which the goal is randomly initialized to the left or right of the end-effector, the policy learns to always direct the end-effector towards one side. Shaded regions indicate the standard error of the mean over three random seeds.
28

---

Published as a conference paper at ICLR 2022
C.6 EXPERIMENTS WITH END- EEDEETOROO ORIENTATION CONTROL
The experiments in Section 4 involved 4-DoF action space consisting of 3-Dof' end-effector posi- tion control and I-DoF gripper control, which was sufficient for solving all of the Meta-World tasks. In this section, we add one more degree of freedom for end-effector orientation control (allow- ing the parallel-jaw gripper to swivel) and then construct and experiment on two modified versions of the peg- inseessrt--sie task that cannot be solved without end-effector rotations. The train and test distributions of initial object center-of-masse are the same as those in the original peg-insert-side task.
In the first modified version, the end-effector is initially rotated 90 degrees from its original orien- tation, forcing the agent to rotate the end-effector before grasping the peg (see the center column of Figure for visualization). The second modified version of the task includes the following changes: (1) the proprioceptive observations also include the end-effector's orientation (as quater- nion), and (2) the peg not the end-effector- is initially rotated by 90 degrees (see the rightmost column of Figure 19 for visualization), Not only does (2) force the agent to rotate the end-effector before grasping the peg, but also requires the agent to re-orient the peg correctly before inserting it into the box. The experimental results for DrQ-v2 in these two environments shown in Figure 20.
Figure 19: Visualizations of the two modified versions of the peg ge-insert--sid task in Meta World, where the first and second rows contain the third-person and hand-centric perspectives of the initial configurations, re- spectively. Left: original peg-insert-side setup. Center: end-effector initially rotated 90 degrees about the vertical axis (corresponding
left half of Figure 20). Right: peg initially rotated 90 degrees about vertical axis (corresponding to the right half of Figure 20).
train/ID
OOst
O Oa-p
JT o Ostp
Oh+a+p
a O + VIB(zz)
pog-insert-sidee (end-eflector initially rolated 90 deg)
peg-inser--side (peg initially rotated 90 deg)
1,0F
0.8
0.8
atccess rate
0.5
0.8
Add
0.6
06
0.4
0.4
0.4
04
0.2
0.2
0.0
0.0
0.0g
0.0g
200
400
600
900
200
600
600
S00
400
200
1200
1606
20000
4001
800
1210 1000 2000
mnviromeent stape [x
envirooment stops (x 10*)
crviromeet steps (x10*)
envirommen steps (x10²)
Figure 20: DrQ-v2 results for the modified versions of the Meta-Wosd World reg-insert-side task (visualized Figure 19). For the first modified version, we see generalization performance trends similar to those in the original rotation-less peg-insert-side task (second row, second column Figure 6). In the second modified version, π Ou+3=p + VIB (z3) outperforms Ox+atp in terms of sample complexity and generalization. and π Ox in terms of sample complexity. However, test performance begins droop after 1.6M steps the training distribution, which would likely occur the other agents well if
attribute this to overfitting they were trained post-convergence similar extent. Shaded regions indicate the standard error of mean over three random seeds.
29

---

Published as a conference paper at ICLR 2022
C.7 HYPERPARAMETERS
We present the DrQ-v2 hyperparameters used in the Meta-Works experiments in Table 10. The configuration is largely identical the used in the original DrQ-v2 algorithm (Yarats et al., 2021).
Table 10: The hyperparameters used for DrQ-v2 in the Meta-Worid tasks. The lower half of the table presents the hyperparameters that specific to the regularization methods and ablation agents.
parameter
setting
replay buffer capacity
100,000 for thandle-press-sidee button-press, reach-hard);
400,000 for all other environments
action repeat
2
frame stack
seed frames
4000
exploration steps
2000
n-step returns
mini-batch size
256
discount y
0.99
optimizer
Adam
learning rate
10- 4
agent update frequency
2
critic Q-function soft-update rate
0.01
features dim.
50
hidden dim.
1024
exploration stddev. clip exploration stddev. schedule
0.3
0.1,
weight 33 of KL div. term in π O143+p (2B(2s)
500 for {soccer, peg-insert-side-hardy
for peg-insert-side w/ rotated initial gripper (for Appendix C.6)
for peg-insert-side w/ rotated initial peg (for Appendix C.6)
10 for all other environments
weights Sh, 33 of KL div. terms in
Bx:
On43tp (IB( + (IB(zs)
0.1 for (button-pree peatoon-pres,, reach- soccer);
0.001 for all other environments
8g:
same as 33 of KL div. term in o Oa+3+p + VIB(zs)
weight 33 of KL div. term in
for bandle-press-sidee press-sid;; 500 for soccer; for all other environments
π o Oy + VIB(za)
weight 0g of &2 reg. term in π O143tp + (z(a))
1 for all environments
D MISCELLANEOUS DETAILS
applied an exponentially weighted moving average filter on the data for DrQ Figure 2 (a 0.6), for DAC in Figure (a 0.3), and for DrQ-v2 in Figures and 17 (a 0.5) to smoothen the train and test curves for increased readability. The smoothing factor a lies in the range [0, 1]. where values closer to 0 correspond to more smoothing.
30


Reviews:
[{"review_id": "el9rpcf4T6v", "reviewer": "Reviewer_z6Z4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well motivated, clearly written and coherently structured. The exposition of the main ideas is linear and easy to follow. The contribution is clear, well presented and well motivated. The notation and the formulation of the proposed method are clearly presented. Claims are supported by thorough experimental results. Figures and tables are presented in a nice and easily readable way, and help grasping the contribution of the study. Videos are also helpful in understanding the experimental setup and the results in a qualitative way.\nYou mention tactile signals as a good example in robot manipulation of local streams of information. What is the applicability of your study to the tactile sensor modality?\nIt may be that some of the tasks could be solved using proprioceptive information only - do you have results using proprioception only as your observation space? Comparing the results presented in the paper with results obtained with proprioception only would be instrumental to understand the effect/contribution of vision (especially in the case of O_{h+p})\n", "summary_of_the_paper": "This paper presents an interesting empirical evaluation of the role of visual perspective in learning and generalization in the context of physical manipulation. The paper compares performances using third-person and in-hand visual perspectives, showing that hand-centric vision consistently improves training and out-of-distribution generalization. Authors also explore a combination of the two perspectives by proposing to regularize the third-person information stream to maintain generalization performance.", "summary_of_the_review": "The paper is well written, the contribution is clearly presented and the experimental results are thoroughly executed. An evaluation comparing the presented results with the proprioception-only case could further improve the analysis.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dKGpZTvTXkk", "reviewer": "Reviewer_u4WJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Specifics about the training \n- 84x84 RGB input images. \n- outputs 3D end-effector position relative to the robot base, 1D gripper width, and a boolean contact flag for each of two gripper fingers. No rotation.\n- Three learning algorithms: Dagger, DrQ, DAC. \n\nThe experimental set up to show that hand perspective is better than third perspective involves picking up a cube (which is somewhat trivial task and naturally favours hand perspective). My first impression was that it's certainly helpful to have hand perspective as it always gets a close up / zoomed-in view of the object free of any occlusions etc. so it makes sense that the hand perspective performs better. However, if you have a more complicated manipulator e.g. hands the self occlusions from fingers and as the hand starts to cage the object, you'd need a third person view to get a better view of the object. Secondly, I feel adding rotation in the end-effector target (which the current output parameterisation doesn't include) could make things even for third perspective too. Is there a reason why the network only outputs just 3D end-effector position? For this particular task I believe you don't need rotation but any general task you'd need to also parameterise rotations as rotations will bring occlusions for hand perspective too. \n\nIn the 6 meta-world tasks, they show that combining a third perspective together with information bottleneck regularisation leads to better generalisation. Though most (if not all) of these tasks involve decision making that has more to do with the object they are reaching to and less about doing any long horizon interaction with scene or other objects in the scene after interaction with the object they are reaching to. \n\nThe paper seems to suggest that a zoomed-in view of the object by using a hand perspective almost always helps which I agree with. It seems to highlight the design choices that we regularly make when doing manipulation with raw observations need to be carefully looked into. Although this is not the first paper to show that e.g. https://arxiv.org/pdf/2012.07975.pdf have also shown that adding hand perspective helps. This paper was also not cited. \n\n\n", "summary_of_the_paper": "The paper studies the effect of visual perspective by using the images coming from a camera installed on the hand of the robot in the specific context of robot manipulation from raw observations.  Results demonstrate that such a choice of visual perspective requires no algorithmic changes but can improve OOD generalisation and training efficiency. Of course this does not mean that we abandon altogether the traditional third person perspective as in many cases just having a close-up view of one object may not be enough when making decisions about the scene as a whole. Therefore, they also show that combined with third person perspective with information bottleneck regularisation can improve the OOD generalisation. They show results on six different manipulation tasks adapted from Meta-World and their choice of perspective improves OOD generalisation. ", "summary_of_the_review": "The paper is a good case study. The ideas presented in the paper are not something that's unknown but this paper does a good investigation on the choice of perspectives.  \n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_JvWE22AH8B", "reviewer": "Reviewer_RmQW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- Well motivated idea\n- Compelling results\n- Overall clearly written\n- Good ablations and the idea is shown for multiple algorithms and in a wide variety of settings\n- Interesting approach to generalize the hand camera to settings with a higher degree of partial observability\n\n### Weakness\n\n- $$z_\\text{shift}$$ is only explained in the appendix, it should be explained in the main paper.\n- The information bottleneck technique harms performance initially\n\n\n\n### Suggestions for improvement\n\n- In concurrent work, Szot et al (NuerIPS 2021) also used a hand/arm camera to learn manipulation policies and found similar trends. It may be worth citing as additional support for these finding.\n\n- How useful is the data-aug in DrQ for the hand camera? Part of the argument for DrQ is to reduce overfitting of the Q function during training. Perhaps with the hand camera you no longer need aug?\n\n- What about recurrent policies instead of the third person camera?\n\n\nSzot et al: https://arxiv.org/abs/2106.14405", "summary_of_the_paper": "This paper presents an analysis of camera placement for vision-based manipulators. Specifically it compares the performance of a disembodied third person camera vs. place the camera on the robot's hand/gripper.\n\nThe authors find that the hand camera improves generalization and training performance in the cases where a hand camera still reveals enough information to complete the task.\n\nWhen the hand camera does not reveal enough information to perform the task, the third person camera is still needed and the authors propose to use an information bottleneck to reduce the amount of information used from the third person camera, thereby improving generalization even when it's needed.\n", "summary_of_the_review": "This paper presents thorough analysis of using a 3rd person camera vs. a hand camera and finds that hand cameras generalize better. I believe this work presents a useful contribution.\n\n\n### Post Rebuttal Update\n\nI thank the authors for their response. I continue to think this is a good paper that should be accepted for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers consistently agree on the high quality of the research presented in this paper, such that it the paper clearly is significantly above the acceptance threshold of ICLR.
Decision: Accept (Oral)