Year: 2021

Abstract:
We study how the choice of visual perspective affects learning and generalization in the context of physical manipulation from raw sensor observations. Compared with the more commonly used global third-person perspective, a hand-centric (eye-in-hand) perspective affords reduced observability, but we find that it consistently improves training efficiency and out-of-distribution generalization. These benefits require no algorithmic changes other than the perspective and hold across a variety of learning algorithms, experimental settings, and distribution shifts, albeit only when hand-centric observability is sufficient. When this is not the case, including a third-person perspective is necessary for learning, but also harms out-of-distribution generalization. To mitigate this, we propose to regularize the third-person information stream via a variational information bottleneck. On six representative manipulation tasks with varying hand-centric observability adapted from the Meta-World benchmark, this results in a state-of-the-art reinforcement learning agent operating from both perspectives improving its out-of-distribution generalization on every task. While some practitioners have long put cameras in the hands of robots, our work systematically analyzes the benefits of doing so and provides simple and broadly applicable insights for improving vision-based robotic manipulation.1 Figure 1: Illustration suggesting the role that visual perspective can play in facilitating the acquisition of symmetries with respect to certain transformations on the world state s. T0: planar translation of the end-effector and cube. T1: vertical translation of the table surface, end-effector, and cube. T2: addition of distractor objects. O3: third-person perspective. Oh: hand-centric perspective.

heading: 1 INTRODUCTION
text: Physical manipulation is so fundamental a skill for natural agents that it has been described as a “Rosetta Stone for cognition” (Ritter & Haschke, 2015). How can we endow machines with similar mastery over their physical environment? One promising avenue is to use a data-driven approach,
1Project website: https://sites.google.com/view/seeing-from-hands
in which the mapping from raw sensor observations of the environment (and other readily available signals, e.g. via proprioception) to actions is acquired inductively. Helpful inductive biases in modern machine learning techniques such as over-parameterized models and stochastic gradient descent have enabled surprising (and poorly understood) generalization capabilities in some applications (Neyshabur et al., 2014; Belkin et al., 2019; Zhang et al., 2021). Despite this, visuomotor policies learned end-to-end remain brittle relative to many common real-world distribution shifts: subtle changes in lighting, texture, and geometry that would not faze a human cause drastic performance drops (Julian et al., 2020).
While a wide variety of algorithms have been proposed to improve the learning and generalization of object manipulation skills, in this paper we instead consider the design of the agent’s observation space, a facet of the learning pipeline that has been underexplored (Section 5). Indeed, in some applications of machine learning, e.g., image classification or text summarization, the disembodied nature of the task affords relatively little flexibility in this regard. Yet, even in these settings, simple data processing techniques such as normalization and data augmentation can have noticeable effects on learning and generalization (Perez & Wang, 2017). The role of data can only be more profound in an embodied setting: any sensors capable of being practically instrumented will only provide a partial observation of the underlying world state. While partial observability is typically regarded as a challenge that only exacerbates the difficulty of a learning problem (Kaelbling et al., 1998), we may also consider how partial observations can facilitate the acquisition of useful symmetries.
The natural world gives clear examples of this. For instance, because cutaneous touch is inherently restricted to sensing portions of the environment in direct contact with the agent, tactile sensing by construction exhibits invariances to many common transformations on the underlying world state; grasping an apple from the checkout counter (without looking at it) is largely the same as doing so from one’s kitchen table. Due in part to the nascent state of tactile sensing hardware (Yuan et al., 2017) and simulation (Agarwal et al., 2020), in this work we investigate the above insight in vision, the ubiquitous robotic sensory modality. In particular, we focus on the role of perspective as induced from the placement of cameras. To roughly imitate the locality of cutaneous touch, we consider the hand-centric (eye-in-hand) perspective arising from mounting a camera on a robotic manipulator’s wrist. We also consider the more commonly used third-person perspective afforded by a fixed camera in the world frame.
The main contribution of this work is an empirical study of the role of visual perspective in learning and generalization in the context of physical manipulation. We first perform a head-to-head comparison between hand-centric and third-person perspectives in a custom simulated grasping environment that features three kinds of distribution shifts. We find that using the hand-centric perspective, with no other algorithmic modifications, reduces aggregate out-of-distribution failure rate by 92%, 99%, and 100% (relative) in the imitation learning, reinforcement learning, and adversarial imitation learning settings, respectively. We also observe appreciable gains in training efficiency in the latter two settings.
Despite their apparent superiority, hand-centric perspectives cannot be used alone for tasks in which their limited observability is a liability during training. To realize the benefits of hand-centric perspectives more generally, we propose using both hand-centric and third-person perspectives in conjunction for full observability while regularizing the latter with a variational information bottleneck (Alemi et al., 2016) to mitigate the latter’s detrimental effects on out-of-distribution generalization. We instantiate this simple and broadly applicable principle in DrQ-v2 (Yarats et al., 2021), a state-of-the-art vision-based reinforcement learning algorithm, and find that it reduces the aggregate out-of-distribution failure rate compared to using both perspectives naively by 64% (relative) across six representative manipulation tasks with varying levels of hand-centric observability adapted from the Meta-World benchmark (Yu et al., 2020).

heading: 2 PROBLEM SETUP
text: Preliminaries: MDPs and POMDPs. We frame the physical manipulation tasks considered in this work as discrete-time infinite-horizon Markov decision processes (MDPs). An MDPM is a 6-tuple (S,A, P,R, γ, µ), where S is a set of states, A is a set of actions, P : S × A → Π(S) is a statetransition (or dynamics) function, R : S × A → R is a reward function, γ ∈ (0, 1) is a discount factor, and µ ∈ Π(S) is an initial state distribution. An MDP whose state cannot be directly observed can be formalized as a partially observable MDP (POMDP), an 8-tuple (S,A, P,R, γ, µ,Ω, O)
that extends the underlying MDP with two ingredients: a set of observations Ω and an observation function O : S × A → Π(Ω). We consider only a restricted class of POMDPs in which the observation function is limited to be O : S → Ω. To solve a POMDP, we optimize a policy π : Ω→ Π(A) to maximize the expected return R(M, π ◦O) = Eµ,P,π [ ∑∞ t=0 γ
tR(st, at)], where π ◦O maps a state to an action distribution via composing the policy and observation function. Observation functions. In this work, we denote the observation functions corresponding to the hand-centric and third-person perspectives as Oh and O3, respectively. We also consider proprioception via forward kinematics, denoting this as Op. Often, multiple observation functions are used together; for example, we denote using both the hand-centric and proprioceptive observations as Oh+p.
Invariances and generalization. We say that a function f : X ×Y → Z is invariant in domain subspace X to a transformation T : X → X iff ∀x ∈ X , y ∈ Y. f(T (x), y) = f(x, y). We formalize the notion of generalization by saying that π ◦ O generalizes inM to a distribution shift caused by transformation T iff R(M, π ◦O) is invariant inM to T . We consider two kinds of generalization: in-distribution and out-of-distribution generalization, also referred to as interpolation and extrapolation. The latter corresponds to the agent generalizing inM to some specified transformation, and the former is a special case when the transformation is identity. In this work, we limit the scope of the transformations onM we consider to those acting on the initial state distribution µ through the state set S. A few concrete examples of such transformations are illustrated in Figure 1.

heading: 3 HAND-CENTRIC VS. THIRD-PERSON PERSPECTIVES
text: The first hypothesis we investigate is that using the hand-centric perspective Oh instead of the thirdperson perspectiveO3 can significantly improve the learning and generalization of the agent π◦O. In this section, we probe this hypothesis in a setting where the hand-centric perspective gives sufficient observability of the scene (we consider when this does not hold in Section 4). See project website https://sites.google.com/view/seeing-from-hands for animated visualizations.

heading: 3.1 EXPERIMENTAL SETUP
text: We consider a visuomotor grasping task instantiated in PyBullet (Coumans & Bai, 2016–2021). A Franka Emika Panda manipulator is tasked with picking up a specific cube that initially rests on a table. The action space is 4-DoF, consisting of 3-DoF end-effector position control and 1-DoF gripper control. Observation functions include Oh and O3, which output 84 × 84 RGB images, and Op, which outputs 3D end-effector position relative to the robot base, 1D gripper width, and a Boolean contact flag for each of two gripper “fingers“.
We use three learning algorithms: imitation learning with dataset aggregation (DAgger) (Ross et al., 2011), reinforcement learning using data-regularized Q-functions (DrQ) (Kostrikov et al., 2020), and adversarial imitation learning using discriminator-actor-critic (DAC) (Kostrikov et al., 2018). We defer exposition on these algorithms to Appendix A.2. We run DAgger and DrQ on three experiment variants each targeting a different distribution shift (detailed and visualized in Appendix A.1). With DAC, we assess in-distribution generalization in the training environment and out-of-distribution generalization between demonstration (demo) collection and the training environment. Details on the model architectures and hyperparameters used can be found in Appendices A.3 and A.4.

heading: 3.2 EXPERIMENTAL RESULTS AND DISCUSSION
text: DAgger and DrQ results are reported in Figure 2 and aggregated in Table 1. DAC results are reported in Figure 3, with experiment variant descriptions in the caption.
For DAgger (left two columns of Figure 2), we find that the hand-centric perspective leads to clear improvements in out-of-distribution generalization (test) across all three experiment variants despite in-distribution generalization progress (train) being essentially identical between π ◦Oh+p and π◦O3+p. The only exceptions to π◦Oh+p generalizing better are in some instances of the distractor objects variant. Here, seeing the red, green, and blue distractor objects during training was sufficient for both π ◦ Oh+p and π ◦ O3+p to learn to ignore these object colors, even under distractor distribution shift. Generalization to white distractors was likely facilitated by the RGB representation of white as the “sum” of red, green, and blue.
For DrQ (right two columns of Figure 2), the differences between π◦Oh+p and π◦O3+p extend into training time. In the table height variant, π ◦Oh+p exhibits increased sample efficiency for training as well as similar out-of-distribution generalization benefits as seen for DAgger. For the distractor objects variant, π ◦ Oh+p converges before π ◦ O3+p makes any significant progress on success rate (though we did observe increasing returns). Since DrQ trained π ◦ O3+p to convergence for the other variants within the same interaction budget, it follows that the presence of the distractors rendered the training task too hard for π ◦O3+p, but not for π ◦Oh+p. In the table textures variant, the generalization improvement of π ◦ Oh+p over π ◦ O3+p is less extreme. We attribute this to invariances to image-space transformations learned via the data augmentation built into DrQ. In Appendix A.5, an ablation in which this augmentation is removed further shows its importance.
For DAC, we find stark improvements in the generalization of π ◦ Oh over that of π ◦ O3. In the first DAC-specific experiment variant (left plot of Figure 3), π ◦Oh fully generalizes in-distribution with as few as 5 demos, whereas π ◦ O3 achieves significantly lower success, even with 25 demos and much more online interaction. In the second variant (center plot of Figure 3), the distribution shift between demo collection and training barely affects π ◦ Oh, but severely compromises the training of π ◦ O3. In the third variant (right plot of Figure 3), despite the presence of distractor objects giving the discriminator strong predictive power in distinguishing between demos and agent behavior, π◦Oh still achieves a significant measure of in-distribution generalization, whereas π◦O3 makes little progress even with eight times the number of demos. We remark that, in the context of adversarial imitation learning, π ◦ Oh achieves its sample efficiency and robustness without any special requirements on the training data (Zolna et al., 2020) or modified training objectives (Xu & Denil, 2020).

heading: 4 COMBINING HAND-CENTRIC AND THIRD-PERSON PERSPECTIVES
text: The previous set of experiments demonstrate how hand-centric perspectives can lead to clear improvements in learning and generalization over third-person perspectives. Unfortunately, this does not mean that the use of hand-centric perspectives is a panacea. The limited observability of handcentric perspectives is a double-edged sword: depending on the environment and task, it can enable π ◦ Oh to establish useful invariances, or confuse π ◦ Oh by enforcing harmful ones. In this section, we focus on evaluating across tasks of varying hand-centric observability, including those in which insufficient observability severely undermines π ◦ Oh. How can we realize the benefits of hand-centric perspectives even in such scenarios?

heading: 4.1 REGULARIZING THE THIRD-PERSON INFORMATION STREAM
text: Insufficient observability arising from using Oh alone necessitates the inclusion of O3. While using both perspectives should effectively resolve the issue of insufficient observability and enable the agent to train, we know from Section 3.2 that the use of the third-person perspective can hamper out-of-distribution generalization by allowing the agent to “overfit” to particularities of the training distribution. To mitigate this, we propose to regularize the third-person perspective’s representation. While multiple regularization techniques could conceivably be suitable to this end, we choose the variational information bottleneck (VIB) to use in our experiments due to its simplicity, theoretical justification, and empirical performance (Alemi et al., 2016).
For our subsequent experiments, we build on top of the state-of-the-art vision-based actor-critic reinforcement learning algorithm DrQ-v2 (Yarats et al., 2021) (see Appendix B.3 for a detailed description). When we use both hand-centric and third-person observations oh and o3, we instantiate two separate image encoders fξh and fξ3 . We denote the corresponding representations as zh and z3. These are concatenated before being fed to the actor πφ and critic networks Qθ1 , Qθ2 .
We apply a VIB to the third-person information stream to regularize the DrQ-v2 critic. This amounts to a variational approximation to maximizing the mutual information between the third-person ob-
servations and the critic’s predictions of the temporal difference targets while minimizing the mutual information between the third-person observations and their representations. We implement this by replacing the deterministic third-person encoder fξ3 with a stochastic encoder pξ3(z3|o3), specifying a prior p(z3), and adding a weighted KL divergence term to the critic loss. The VIB-regularized DrQ-v2 critic objective is
L(ξh, ξ3, θ1, θ2) = ED, pξ3 [LDrQ-v2 critic(ξh, ξ3, θ1, θ2)] + ED [β3DKL(pξ3(z3|o3) ‖ p(z3))] , (1) where D is the replay buffer. We specify pξ3(z3|o3) as a diagonal Gaussian and p(z3) as a standard Gaussian, which enables analytical computation of the KL divergence. We use the reparameterization trick to enable optimization of the first term via pathwise derivatives. We do not need to modify the actor objective as only gradients from the critic are used to update the encoder(s) in DrQ-v2. We remark that a (variational) information bottleneck can be applied to many imitation learning and reinforcement learning algorithms (Peng et al., 2018; Goyal et al., 2019; Igl et al., 2019; Kumar et al., 2021).

heading: 4.2 EXPERIMENTAL SETUP
text: We evaluate the learning and generalization performance of seven DrQ-v2 agents: π ◦ Oh+p (hand-centric perspective), π ◦ O3+p (third-person perspective), π ◦ Oh+3+p (both perspectives), π ◦ Oh+3+p + VIB(z3) (both perspectives with a VIB on the third-person information stream), and three ablation agents introduced later. We evaluate the agents on six tasks adapted from the Meta-World benchmark (Yu et al., 2020). We design the task set to exhibit three levels of handcentric observability (high, moderate, and low) with two tasks per level. In each task, a simulated Sawyer robot manipulates objects resting on a table. The action space is 4-DoF, consisting of 3-DoF end-effector position control and 1-DoF gripper control. We do not use the original Meta-World observation space as it contains low-dimensional pose information about task-pertinent objects instead of images. Rather, we configure the observations so that Oh and O3 output 84 × 84 RGB images, and Op outputs 3D end-effector position and 1D gripper width. See Figure 4 for a visualization of each task through the lens of Oh and O3. Experiments in Appendix B.5 establish that proprioception alone is not sufficient to reliably solve any of the tasks. Experiments in Appendix B.6 consider variations of peg-insert-side that require an additional 1-DoF end-effector orientation control.
While the distribution shifts in the experiments of the previous section arise from transformations on the table height, distractor objects, and table textures, in this section we focus on distribution shifts arising from transformations on the initial object positions. All object positions have disjoint initial train and test distributions such that the latter’s support “surrounds” that of the former (see Table 7 in Appendix B.2 for details).
Aside from adapting the DrQ-v2 algorithm to our setting as described above, we use the original DrQ-v2 model and hyperparameters with some minor exceptions (see Appendix B.7 for details). Hyperparameters that are common to all agents are shared for a given task. With agents that include regularization, we tune the regularization weight(s) on a validation sample from the test distribution. Test success rate is computed on a separate sample of 20 environments from the test distribution.

heading: 4.3 EXPERIMENTAL RESULTS AND DISCUSSION
text: Main experimental results in Meta-World are presented in Figure 5. Results for ablation studies on π ◦ Oh+3+p + VIB(z3) are presented in Figure 12 in Appendix B.4. Table 2 summarizes both figures. When using both perspectives, regularizing the third-person perspective’s representation via a VIB reduces the interquartile mean of the out-of-distribution failure rate across all six tasks by 64% (relative). We also note that this method achieves the best performance in each individual task, albeit sometimes with less sample efficiency. To properly explain these phenomena, we now embark on a more stratified analysis and discussion of the results.
Characterization of hand-centric observability via training performance. When the world state is sufficiently observable via the hand-centric perspective, we expect the convergence during training of π◦Oh+p to match or surpass that of π◦O3+p. We find that this is indeed the case for handle-pressside, button-press, soccer, and peg-insert-side (high and moderate hand-centric observability), and not the case for reach-hard or peg-insert-side-hard (low hand-centric observability). This validates our selection and framing of the tasks at different levels of hand-centric observability. Interestingly, we observe that in peg-insert-side-hard, π ◦Oh+p eventually achieves some success during training by “zooming out” to improve its observability.
Hand-centric perspective vs. third-person perspective. When hand-centric observability is high or moderate, π◦Oh+p generalizes better out-of-distribution than π◦O3+p, corroborating results from Section 3.2 with another form of distribution shift. When hand-centric observability is low, π◦Oh+p both trains and generalizes worse than π ◦O3+p. This supports our motivation for considering using both perspectives in conjunction.
Effect of combining the hand-centric and third-person perspectives. When hand-centric observability is high or moderate, including the third-person perspective can harm generalization. We see that for button-press, peg-insert-side, handle-press-side, and soccer, π ◦ Oh+3+p is sandwiched between π ◦ Oh+p and π ◦ O3+p on the test distribution. The drop from π ◦ Oh+p to π ◦ Oh+3+p is significant for the former two tasks, and marginal for the latter two. This validates our hypothesis that including O3 enables the agent to “overfit” to training conditions. When hand-centric observability is low, combining both perspectives results in π◦Oh+3+p matching or surpassing the training performance of π ◦Oh+p and π ◦O3+p, and greatly outperforming both at test time. This validates our hypothesis that, when necessary, including third-person observations helps resolve training difficulties arising from insufficient hand-centric observability.
Effect of regularizing the third-person information stream via a VIB. π ◦ Oh+3+p + VIB(z3) consistently improves upon π ◦ Oh+3+p in out-of-distribution generalization for all tasks except handle-press-side, in which the two are about equal. This directly indicates the benefit of the VIB regularization. These gains come at the cost of slightly delaying the convergence of training. However, it is arguable that this is inevitable and even desirable. A known phenomenon in neural network training is that spurious correlations or “shortcuts” in the data are sometimes easier to learn than causal relationships (Sagawa et al., 2019). Slower training and higher generalization may indicate the avoidance of such behavior. Additionally, in button-press, π ◦ Oh+3+p + VIB(z3) recovers the out-of-distribution generalization exhibited by π ◦ Oh+p, and when hand-centric observability is moderate, π ◦Oh+3+p + VIB(z3) improves upon π ◦Oh+p. To better understand what makes π ◦ Oh+3+p + VIB(z3) work the best, we also conduct three ablation studies. Curves for the ablation studies are presented in Figure 12 in the Appendix.
What if both perspectives are regularized? The ablation agent π ◦Oh+3+p+ VIB(zh)+ VIB(z3) adds a separate VIB to the hand-centric information stream in an analogous manner to how the third-person perspective’s representation is regularized (detailed in Section 4.1). We use the same β3 for both and tune βh. Note that setting βh = 0 for π ◦ Oh+3+p + VIB(zh) + VIB(z3) recovers π ◦Oh+3+p + VIB(z3) modulo stochasticity in zh, so we limit the lowest value βh can take to 0.01. We find that in no task does π ◦Oh+3+p + VIB(zh) + VIB(z3) outperform π ◦Oh+3+p + VIB(z3), validating our choice of only regularizing the third-person perspective’s representation.
Assessing the importance of the hand-centric perspective. π ◦O3′+3+p + VIB(z3) uses a second third-person perspective O3′ instead of the hand-centric perspective Oh. Visualizations from this additional third-person perspective are shown in Figure 13 in Appendix B.4. We re-tune β3 for this agent. We find that π◦O3′+3+p+VIB(z3) performs significantly worse than π◦Oh+3+p+VIB(z3), affirming the benefit of using the hand-centric perspective in the multi-perspective setting.
zh-dependent regularization of z3. VIB(z3) reduces the information contained in z3 without directly considering zh. With the ablation agent π ◦ Oh+3+p + `2(z3), we consider a simple form of zh-dependent regularization of z3 in which we push z3 towards zh by adding a weighted regularization term α3‖z3 − stopgrad(zh)‖22 to the DrQ-v2 critic objective. This approach seems promising given that π ◦ Oh+3+p consistently outperforms π ◦ O3+p across all six tasks, suggesting that even in the midst of substantial partial observability, zh may represent information in a useful and generalizable way. We tune α3. We find that π ◦ Oh+3+p + `2(z3) marginally improves over vanilla π ◦Oh+3+p but still comes far short of π ◦Oh+3+p + VIB(z3), suggesting that the two perspectives contain important complementary information that is better represented separately.

heading: 5 RELATED WORK
text: Learning for vision-based object manipulation. A wide range of works have focused on algorithmic development for end-to-end learning of vision-based object manipulation skills (Levine et al., 2016; Agrawal et al., 2016; Finn et al., 2016; 2017; Kalashnikov et al., 2018; Srinivas et al., 2018; Ebert et al., 2018; Zhu et al., 2018; Jayaraman et al., 2018; Rafailov et al., 2021). Some works on learned visuomotor control use eye-in-hand cameras for tasks such as grasping (Song et al., 2020) and insertion (Zhao et al., 2020; Puang et al., 2020; Luo et al., 2021; Valassakis et al., 2021), and others which pre-date end-to-end visuomotor learning use both eye-in-hand and third-person cameras for visual servoing (Flandin et al., 2000; Lippiello et al., 2005). Very few works consider the design of camera placements (Zaky et al., 2020) or conduct any controlled comparisons on different combinations of visual perspectives (Zhan et al., 2020; Mandlekar et al., 2021; Wu et al., 2021). Unlike all of these works, we propose specific hypotheses regarding the benefits of different choices of visual perspective and perform a systematic empirical validation of these hypotheses with evaluation on multiple families of learning algorithms, manipulation tasks, and distribution shifts.
The role of perspective on generalization. Hill et al. (2019) assess an agent learning to execute language instructions in simulated environments using high-level actions and find that using an egocentric observation space results in better systematic generalization to new instruction nounverb combinations. Szot et al. (2021) find that an agent tasked to pick up a certain object (using abstracted grasping) in a cluttered room generalizes better to unseen objects and room layouts when using wrist- and head-mounted cameras in conjunction. Our work provides complementary evidence for the effect of perspective on the generalization of learned agents in a markedly different setting: we consider vision-based physical manipulation. Also, the aforementioned works rely on memoryaugmented agents to resolve partial observability as is common in navigation tasks, whereas we use third-person observations as is standard in tabletop manipulation and demonstrate the importance of regularizing their representation.
Invariances through data augmentation in reinforcement learning. Several works have investigated ways to apply standard data augmentation techniques from computer vision in the reinforcement learning setting (Laskin et al., 2020; Kostrikov et al., 2020; Yarats et al., 2021). These works consider data augmentation as a means to prescribe invariances to image-space transformations, whereas we are concerned with how different observation functions facilitate generalization to environmental transformations. To emphasize that these directions are orthogonal, we use DrQ (Kostrikov et al., 2020) and DrQ-v2 (Yarats et al., 2021) in our experiments.

heading: 6 CONCLUSION
text: In this work, we focus on empirically studying an underexplored design choice in the embodied learning pipeline: the observation function. While hand-centric robotic perception is more traditionally instrumented with tactile sensing, our findings using vision affirm that perspective, even when controlling for modality, can play an important role in learning and generalization. This insight may very well apply to robotic systems that leverage tactile sensing. Overall, in the context of end-to-end learning for visuomotor manipulation policies, our findings lead us to recommend using hand-centric perspectives when their limited observability is sufficient, and otherwise defaulting to using both hand-centric and third-person perspectives while regularizing the representation of the latter. The breadth of the learning algorithms, manipulation tasks, and distribution shifts that we base these conclusions on, coupled with their simplicity and lack of restrictive assumptions, suggests that these recommendations should be broadly applicable, even to more complex, longer-horizon tasks that feature sub-tasks analogous to those we experiment with.

heading: REPRODUCIBILITY STATEMENT
text: Sections 3.1 and 4.2 give overviews of the experimental setup, and Appendices A and B flesh out the full experimental protocol in stringent detail. We expect this to be sufficient for independent replication of our main findings. Separately, for the sake of reproduction, we will also release code for all experiments upon publication.

heading: A CUBE GRASPING EXPERIMENT DETAILS

heading: A.1 ENVIRONMENT DETAILS
text: For the cube grasping experiments in Section 3, we investigate three types of distribution shifts. The experiment variants for DAgger and DrQ are summarized in Table 3. The DAC experiments featured a subset of these conditions explained in the caption of Figure 3. Figures 6, 7, and 8 visualize each type of distribution shift.
table height zshift = 0 zshift ∈ {−0.10,−0.05,+0.05,+0.10} distractor objects 1 red, 1 green, 1 blue 3 of color ∈ {red, green, blue, brown,white, black} table texture texture ∈ 5 DTD textures texture ∈ 20 held-out DTD textures

heading: A.2 ALGORITHMS
text: The dataset aggregation (DAgger) algorithm proposed by Ross et al. (2011) is an iterative online algorithm for training an imitation learning policy. In each iteration i (which we call a “DAgger round”), the current policy πi is run to sample a set of trajectories, and an expert policy π∗ is used to label each of the visited states with an optimal action. These labeled trajectories are aggregated into a dataset D that grows in size over the DAgger rounds, and the imitation learning policy π̂i is trained on the entire D for some number of epochs before repeating the above procedure in the next iteration. The trajectory-generating policy πi is often modified such that in earlier DAgger rounds the expert policy π∗ is utilized more heavily than the imitation learning policy π̂i when collecting new trajectories, i.e. πi = βiπ∗+ (1−βi)π̂i, where βi is typically annealed over time (e.g., linearly from 1 to 0 over the DAgger rounds).
The Data-regularized Q (DrQ) algorithm proposed by Kostrikov et al. (2020) is a model-free, offpolicy, actor-critic reinforcement learning algorithm that applies image augmentation techniques commonly used in computer vision (primarily random shifts) to input images, along with regularizations of the Q target and function, such that deep neural network-based agents can be trained effectively from pixels. The original DrQ paper uses soft actor-critic (Haarnoja et al., 2018) and DQN (Mnih et al., 2013) as backbones; we use the soft actor-critic version in our experiments because the cube grasping action space is continuous.
The discriminator actor-critic algorithm (DAC) was proposed in Kostrikov et al. (2018) and is an offpolicy version of the generative adversarial imitation learning (GAIL) method (Ho & Ermon, 2016). Unlike Kostrikov et al. (2018) we use a deterministic reinforcement learning algorithm similar to that of Fujimoto et al. (2018), as we find this helps stability. To scale the method to image observations, we apply similar augmentation techniques as in Kostrikov et al. (2020).

heading: A.3 MODEL ARCHITECTURES
text: For DAgger in the cube grasping experiments discussed in Section 3, we feed the 84 × 84 images into a ResNet-18 convolutional image encoder (He et al., 2016) trained from scratch, with the final classification layer replaced by a linear layer that outputs a 64-dimensional representation. We concatenate proprioceptive information (3D end-effector position relative to the robot base, 1D gripper width, and a Boolean contact flag for each of two gripper “fingers”) to the image representation, and the result is passed into feedforward policy and value networks with two hidden layers of 32 units each.
For DrQ, we use the original actor-critic DrQ model proposed by Kostrikov et al. (2020), except for one modification: we concatenate proprioceptive information (3D end-effector position relative to the robot base, 1D gripper width, and a Boolean contact flag for each of two gripper “fingers”) to the flattened image representation before feeding it into the actor and critic networks.
For the DAC algorithm we use the same convolutional architectures as Kostrikov et al. (2018). The convolutional encoder is shared between the discriminator, actor and critic. We use additional MLP heads with capacities 128, 256 and 256 respectively for those components, as we empirically found that lower-capacity networks decrease the likelihood of overfitting to spurious features.

heading: A.4 HYPERPARAMETERS
text: The DAgger, DrQ, and DAC hyperparameters used in the cube grasping experiments are listed in Tables 4, 5, and 6, respectively.

heading: A.5 ABLATION STUDY: REMOVING THE DATA AUGMENTATION IN DRQ
text: In this experiment, we investigate the effect of the data augmentation component of the DrQ algorithm by ablating it. The motivation is to see whether data augmentation is still necessary for a policy using the hand-centric perspective, which already leads to lower overfitting and better generalization. The results in Figure 9 reveal that the augmentation is indeed still crucial because without it, training does not converge even with much more environment interaction. However, the handcentric perspective does still enable the agent to make greater progress.

heading: A.6 MINOR DISCREPANCIES BETWEEN ALGORITHMS
text: Due to implementation idiosyncrasies, there are minor discrepancies in how each algorithm processes environment observations. Following Kostrikov et al. (2020), for DrQ and DAC-DrQ observations are “frame-stacked” with three time steps’ observations. This was not done for DAgger. Proprioceptives are used for DAgger and DrQ but not for DAC-DrQ. We take the position that these differences increase the generalizability of the trends we observe. We emphasize that the target effect under consideration is Oh vs. O3 in each setting.

heading: B META-WORLD EXPERIMENT DETAILS
text: B.1 INDIVIDUAL TASK DESCRIPTIONS
In this section, we explain the tasks that the agents must learn to accomplish in the six Meta-World environments discussed in Section 4.2 and visualized in Figure 4. We also explain why each task falls under a certain level of hand-centric observability. For details regarding the train and test distributions, see Appendix B.2.
• handle-press-side: The goal is to press the handle fully downwards. Hand-centric observability is high because the handle is well aligned with the hand-centric camera’s field of view.
• button-press: The goal is to push the button fully inwards. Hand-centric observability is high because the button is well in view of the hand-centric camera, and the button remains largely in view as the gripper approaches and presses it.
• soccer: The goal is to push or pick-and-place the ball into the center of the goal net. Handcentric observability is moderate because when the gripper approaches the ball, the observability of the goal net is appreciably reduced.
• peg-insert-side: The goal is to lift the peg and insert it into the hole in the target box. Hand-centric observability is moderate because when the gripper approaches the peg, the observability of the target box is appreciably reduced.
• reach-hard: The goal is to move the gripper to the green goal site, which is initialized either to the left or right side of the gripper with equal probability (see Figure 10). Hand-centric observability is low because the gripper is initialized at the same height as the goal, and we restrain the gripper from moving vertically. Effectively, if given just the hand-centric perspective’s observations, the agent does not know in which direction to move the gripper in the beginning of an episode.
• peg-insert-side-hard: The goal is the same as in peg-insert-side, but like the green goal site in reach-hard, the peg in this environment is initialized either to the left or right side of the gripper with equal probability (see Figure 10). Hand-centric observability is low because the gripper is initialized at the same height as the peg such that the peg is not initially visible to the hand-centric view (though we do not prohibit vertical movement of the gripper as in reach-hard, since this would make the peg insertion part of the task impossible), and also because the peg and target box are initialized much farther apart than they are in peg-insertside (thus, the target box is completely out of view as the agent approaches and grasps the peg).

heading: B.2 TRAIN AND TEST DISTRIBUTIONS
text: At training time, initial positions of the objects in the Meta-World tasks are uniformly sampled within some support. At test time, initial positions are sampled from a uniform distribution that is completely disjoint from the training distribution, such that we test on out-of-distribution initial object positions. To implement this, at test time we resample the set of initial object positions if any of the positions overlaps with its train-time distribution. The full set of train-time and test-time initial object positions is shown in Table 7. For visualizations, see Figure 11.
B.3 DRQ-V2
DrQ-v2 (Yarats et al., 2021) is a state-of-the-art vision-based actor-critic reinforcement learning algorithm that uses deep deterministic policy gradients (DDPG) (Lillicrap et al., 2015) as a backbone (whereas DrQ-v1 by Kostrikov et al. (2020) uses soft actor-critic). The DrQ-v2 model includes:
• a convolutional image encoder fξ that outputs representation z = fξ(aug(o)) given framestacked image observations o and a data augmentation function aug,
• two critic networks Qθk that output Q-values Qθk(z,a), k = 1, 2, à la clipped double Qlearning (Fujimoto et al., 2018),
• and an actor network πφ that outputs action a = πφ(z)+ , ∼ N (0, σ2), with σ2 annealed over the course of training.
The individual critic losses are given by Lk = Eτ∼D [ (Qθk(z,a)− y)2 ] , k = 1, 2 (2)
where τ = (ot,at, rt:t+n−1,ot+n) is a sample from replay bufferD and y is the temporal difference target estimated via n-step returns:
y = n−1∑ i=0 γirt+i + γ n min k∈{1,2} Qθ̄k(zt+n,at+n) (3)
for slow-moving critic weights θ̄1, θ̄2. We omit presentation of the actor loss as we do not need to modify it; in DrQ-v2, only gradients from the critic loss are used to update the weights of the encoder(s).
In terms of the model architecture used in the experiments discussed in Section 4, we use the original DrQ-v2 architecture, except for two modifications: first, we concatenate proprioceptive information (3D end-effector position and 1D gripper width) to the flattened image representation before feeding it into the actor and critic networks. Second, when using two perspectives at the same time (e.g., hand-centric and third-person), we use two separate image encoders that do not share weights. The two representations are concatenated together (along with the proprioceptive information) and fed into the actor and critic networks. The dimensionality of each encoder’s output representation is preserved, thereby doubling the dimensionality of the final combined image representation.
B.4 ABLATIONS ON π ◦Oh+3+p + VIB(z3)
Figure 12 presents the train and test curves of the ablation experiments presented in Section 4.3. Figure 13 contains illustrations of the second third-person perspective used in π◦O3′+3+p+VIB(z3).

heading: B.5 PROPRIOCEPTION-ONLY ABLATION
text: In this ablation experiment, we demonstrate that visual observations are a necessary component of the observation space, i.e. the tasks we experiment with cannot be consistently solved with proprioceptive observations alone. We run DrQ-v2 on all six Meta-World tasks introduced in Section 4.2 without image observations and show the results in Figure 14. Unlike policies that are afforded vision, these proprioception-only policies do not approach 100% success rate on the training distributions.

heading: B.6 EXPERIMENTS WITH END-EFFECTOR ORIENTATION CONTROL
text: The experiments in Section 4 involved a 4-DoF action space consisting of 3-DoF end-effector position control and 1-DoF gripper control, which was sufficient for solving all of the Meta-World tasks. In this section, we add one more degree of freedom for end-effector orientation control (allowing the parallel-jaw gripper to swivel) and then construct and experiment on two modified versions of the peg-insert-side task that cannot be solved without end-effector rotations. The train and test distributions of initial object center-of-masses are the same as those in the original peg-insert-side task.
In the first modified version, the end-effector is initially rotated 90 degrees from its original orientation, forcing the agent to rotate the end-effector before grasping the peg (see the center column of Figure 15 for a visualization). The second modified version of the task includes the following changes: (1) the proprioceptive observations also include the end-effector’s orientation (as a quaternion), and (2) the peg—not the end-effector—is initially rotated by 90 degrees (see the rightmost column of Figure 15 for a visualization). Not only does (2) force the agent to rotate the end-effector before grasping the peg, but it also requires the agent to re-orient the peg correctly before inserting it into the box. The experimental results for DrQ-v2 in these two new environments are shown in Figure 16.

heading: B.7 HYPERPARAMETERS
text: We present the DrQ-v2 hyperparameters used in the Meta-World experiments in Table 8. The configuration is largely identical to the one used in the original DrQ-v2 algorithm (Yarats et al., 2021).

heading: C MISCELLANEOUS DETAILS
text: We applied an exponentially weighted moving average filter on the data for DrQ in Figure 2 (α = 0.6), for DAC in Figure 3 (α = 0.3), and for DrQ-v2 in Figures 5 and 12 (α = 0.5) to smoothen the train and test curves for increased readability. The smoothing factor α lies in the range [0, 1], where values closer to 0 correspond to more smoothing.

Review summaries:
This paper presents an analysis of camera placement for vision-based manipulators. Specifically it compares the performance of a disembodied third person camera vs. place the camera on the robot's hand/gripper.  The authors find that the hand camera improves generalization and training performance in the cases where a hand camera still reveals enough information to complete the task.  When the hand camera does not reveal enough information to perform the task, the third person camera is still needed and the authors propose to use an information bottleneck to reduce the amount of information used from the third person camera, thereby improving generalization even when it's needed. 

The paper studies the effect of visual perspective by using the images coming from a camera installed on the hand of the robot in the specific context of robot manipulation from raw observations.  Results demonstrate that such a choice of visual perspective requires no algorithmic changes but can improve OOD generalisation and training efficiency. Of course this does not mean that we abandon altogether the traditional third person perspective as in many cases just having a close-up view of one object may not be enough when making decisions about the scene as a whole. Therefore, they also show that combined with third person perspective with information bottleneck regularisation can improve the OOD generalisation. They show results on six different manipulation tasks adapted from Meta-World and their choice of perspective improves OOD generalisation. 

This paper presents an interesting empirical evaluation of the role of visual perspective in learning and generalization in the context of physical manipulation. The paper compares performances using third-person and in-hand visual perspectives, showing that hand-centric vision consistently improves training and out-of-distribution generalization. Authors also explore a combination of the two perspectives by proposing to regularize the third-person information stream to maintain generalization performance.