Vision-Based Manipulators Need to Also See from Their Hands

1 INTRODUCTION . Physical manipulation is so fundamental a skill for natural agents that it has been described as a “ Rosetta Stone for cognition ” ( Ritter & Haschke , 2015 ) . How can we endow machines with similar mastery over their physical environment ? One promising avenue is to use a data-driven approach , 1Project website : https : //sites.google.com/view/seeing-from-hands in which the mapping from raw sensor observations of the environment ( and other readily available signals , e.g . via proprioception ) to actions is acquired inductively . Helpful inductive biases in modern machine learning techniques such as over-parameterized models and stochastic gradient descent have enabled surprising ( and poorly understood ) generalization capabilities in some applications ( Neyshabur et al. , 2014 ; Belkin et al. , 2019 ; Zhang et al. , 2021 ) . Despite this , visuomotor policies learned end-to-end remain brittle relative to many common real-world distribution shifts : subtle changes in lighting , texture , and geometry that would not faze a human cause drastic performance drops ( Julian et al. , 2020 ) . While a wide variety of algorithms have been proposed to improve the learning and generalization of object manipulation skills , in this paper we instead consider the design of the agent ’ s observation space , a facet of the learning pipeline that has been underexplored ( Section 5 ) . Indeed , in some applications of machine learning , e.g. , image classification or text summarization , the disembodied nature of the task affords relatively little flexibility in this regard . Yet , even in these settings , simple data processing techniques such as normalization and data augmentation can have noticeable effects on learning and generalization ( Perez & Wang , 2017 ) . The role of data can only be more profound in an embodied setting : any sensors capable of being practically instrumented will only provide a partial observation of the underlying world state . While partial observability is typically regarded as a challenge that only exacerbates the difficulty of a learning problem ( Kaelbling et al. , 1998 ) , we may also consider how partial observations can facilitate the acquisition of useful symmetries . The natural world gives clear examples of this . For instance , because cutaneous touch is inherently restricted to sensing portions of the environment in direct contact with the agent , tactile sensing by construction exhibits invariances to many common transformations on the underlying world state ; grasping an apple from the checkout counter ( without looking at it ) is largely the same as doing so from one ’ s kitchen table . Due in part to the nascent state of tactile sensing hardware ( Yuan et al. , 2017 ) and simulation ( Agarwal et al. , 2020 ) , in this work we investigate the above insight in vision , the ubiquitous robotic sensory modality . In particular , we focus on the role of perspective as induced from the placement of cameras . To roughly imitate the locality of cutaneous touch , we consider the hand-centric ( eye-in-hand ) perspective arising from mounting a camera on a robotic manipulator ’ s wrist . We also consider the more commonly used third-person perspective afforded by a fixed camera in the world frame . The main contribution of this work is an empirical study of the role of visual perspective in learning and generalization in the context of physical manipulation . We first perform a head-to-head comparison between hand-centric and third-person perspectives in a custom simulated grasping environment that features three kinds of distribution shifts . We find that using the hand-centric perspective , with no other algorithmic modifications , reduces aggregate out-of-distribution failure rate by 92 % , 99 % , and 100 % ( relative ) in the imitation learning , reinforcement learning , and adversarial imitation learning settings , respectively . We also observe appreciable gains in training efficiency in the latter two settings . Despite their apparent superiority , hand-centric perspectives can not be used alone for tasks in which their limited observability is a liability during training . To realize the benefits of hand-centric perspectives more generally , we propose using both hand-centric and third-person perspectives in conjunction for full observability while regularizing the latter with a variational information bottleneck ( Alemi et al. , 2016 ) to mitigate the latter ’ s detrimental effects on out-of-distribution generalization . We instantiate this simple and broadly applicable principle in DrQ-v2 ( Yarats et al. , 2021 ) , a state-of-the-art vision-based reinforcement learning algorithm , and find that it reduces the aggregate out-of-distribution failure rate compared to using both perspectives naively by 64 % ( relative ) across six representative manipulation tasks with varying levels of hand-centric observability adapted from the Meta-World benchmark ( Yu et al. , 2020 ) . 2 PROBLEM SETUP . Preliminaries : MDPs and POMDPs . We frame the physical manipulation tasks considered in this work as discrete-time infinite-horizon Markov decision processes ( MDPs ) . An MDPM is a 6-tuple ( S , A , P , R , γ , µ ) , where S is a set of states , A is a set of actions , P : S × A → Π ( S ) is a statetransition ( or dynamics ) function , R : S × A → R is a reward function , γ ∈ ( 0 , 1 ) is a discount factor , and µ ∈ Π ( S ) is an initial state distribution . An MDP whose state can not be directly observed can be formalized as a partially observable MDP ( POMDP ) , an 8-tuple ( S , A , P , R , γ , µ , Ω , O ) that extends the underlying MDP with two ingredients : a set of observations Ω and an observation function O : S × A → Π ( Ω ) . We consider only a restricted class of POMDPs in which the observation function is limited to be O : S → Ω . To solve a POMDP , we optimize a policy π : Ω→ Π ( A ) to maximize the expected return R ( M , π ◦O ) = Eµ , P , π [ ∑∞ t=0 γ tR ( st , at ) ] , where π ◦O maps a state to an action distribution via composing the policy and observation function . Observation functions . In this work , we denote the observation functions corresponding to the hand-centric and third-person perspectives as Oh and O3 , respectively . We also consider proprioception via forward kinematics , denoting this as Op . Often , multiple observation functions are used together ; for example , we denote using both the hand-centric and proprioceptive observations as Oh+p . Invariances and generalization . We say that a function f : X ×Y → Z is invariant in domain subspace X to a transformation T : X → X iff ∀x ∈ X , y ∈ Y. f ( T ( x ) , y ) = f ( x , y ) . We formalize the notion of generalization by saying that π ◦ O generalizes inM to a distribution shift caused by transformation T iff R ( M , π ◦O ) is invariant inM to T . We consider two kinds of generalization : in-distribution and out-of-distribution generalization , also referred to as interpolation and extrapolation . The latter corresponds to the agent generalizing inM to some specified transformation , and the former is a special case when the transformation is identity . In this work , we limit the scope of the transformations onM we consider to those acting on the initial state distribution µ through the state set S. A few concrete examples of such transformations are illustrated in Figure 1 . 3 HAND-CENTRIC VS. THIRD-PERSON PERSPECTIVES . The first hypothesis we investigate is that using the hand-centric perspective Oh instead of the thirdperson perspectiveO3 can significantly improve the learning and generalization of the agent π◦O . In this section , we probe this hypothesis in a setting where the hand-centric perspective gives sufficient observability of the scene ( we consider when this does not hold in Section 4 ) . See project website https : //sites.google.com/view/seeing-from-hands for animated visualizations . 3.1 EXPERIMENTAL SETUP . We consider a visuomotor grasping task instantiated in PyBullet ( Coumans & Bai , 2016–2021 ) . A Franka Emika Panda manipulator is tasked with picking up a specific cube that initially rests on a table . The action space is 4-DoF , consisting of 3-DoF end-effector position control and 1-DoF gripper control . Observation functions include Oh and O3 , which output 84 × 84 RGB images , and Op , which outputs 3D end-effector position relative to the robot base , 1D gripper width , and a Boolean contact flag for each of two gripper “ fingers “ . We use three learning algorithms : imitation learning with dataset aggregation ( DAgger ) ( Ross et al. , 2011 ) , reinforcement learning using data-regularized Q-functions ( DrQ ) ( Kostrikov et al. , 2020 ) , and adversarial imitation learning using discriminator-actor-critic ( DAC ) ( Kostrikov et al. , 2018 ) . We defer exposition on these algorithms to Appendix A.2 . We run DAgger and DrQ on three experiment variants each targeting a different distribution shift ( detailed and visualized in Appendix A.1 ) . With DAC , we assess in-distribution generalization in the training environment and out-of-distribution generalization between demonstration ( demo ) collection and the training environment . Details on the model architectures and hyperparameters used can be found in Appendices A.3 and A.4 . 3.2 EXPERIMENTAL RESULTS AND DISCUSSION . DAgger and DrQ results are reported in Figure 2 and aggregated in Table 1 . DAC results are reported in Figure 3 , with experiment variant descriptions in the caption . For DAgger ( left two columns of Figure 2 ) , we find that the hand-centric perspective leads to clear improvements in out-of-distribution generalization ( test ) across all three experiment variants despite in-distribution generalization progress ( train ) being essentially identical between π ◦Oh+p and π◦O3+p . The only exceptions to π◦Oh+p generalizing better are in some instances of the distractor objects variant . Here , seeing the red , green , and blue distractor objects during training was sufficient for both π ◦ Oh+p and π ◦ O3+p to learn to ignore these object colors , even under distractor distribution shift . Generalization to white distractors was likely facilitated by the RGB representation of white as the “ sum ” of red , green , and blue . For DrQ ( right two columns of Figure 2 ) , the differences between π◦Oh+p and π◦O3+p extend into training time . In the table height variant , π ◦Oh+p exhibits increased sample efficiency for training as well as similar out-of-distribution generalization benefits as seen for DAgger . For the distractor objects variant , π ◦ Oh+p converges before π ◦ O3+p makes any significant progress on success rate ( though we did observe increasing returns ) . Since DrQ trained π ◦ O3+p to convergence for the other variants within the same interaction budget , it follows that the presence of the distractors rendered the training task too hard for π ◦O3+p , but not for π ◦Oh+p . In the table textures variant , the generalization improvement of π ◦ Oh+p over π ◦ O3+p is less extreme . We attribute this to invariances to image-space transformations learned via the data augmentation built into DrQ . In Appendix A.5 , an ablation in which this augmentation is removed further shows its importance . For DAC , we find stark improvements in the generalization of π ◦ Oh over that of π ◦ O3 . In the first DAC-specific experiment variant ( left plot of Figure 3 ) , π ◦Oh fully generalizes in-distribution with as few as 5 demos , whereas π ◦ O3 achieves significantly lower success , even with 25 demos and much more online interaction . In the second variant ( center plot of Figure 3 ) , the distribution shift between demo collection and training barely affects π ◦ Oh , but severely compromises the training of π ◦ O3 . In the third variant ( right plot of Figure 3 ) , despite the presence of distractor objects giving the discriminator strong predictive power in distinguishing between demos and agent behavior , π◦Oh still achieves a significant measure of in-distribution generalization , whereas π◦O3 makes little progress even with eight times the number of demos . We remark that , in the context of adversarial imitation learning , π ◦ Oh achieves its sample efficiency and robustness without any special requirements on the training data ( Zolna et al. , 2020 ) or modified training objectives ( Xu & Denil , 2020 ) .

Summary: The paper studies the effect of visual perspective by using the images coming from a camera installed on the hand of the robot in the specific context of robot manipulation from raw observations.  Results demonstrate that such a choice of visual perspective requires no algorithmic changes but can improve OOD generalisation and training efficiency. Of course this does not mean that we abandon altogether the traditional third person perspective as in many cases just having a close-up view of one object may not be enough when making decisions about the scene as a whole. Therefore, they also show that combined with third person perspective with information bottleneck regularisation can improve the OOD generalisation. They show results on six different manipulation tasks adapted from Meta-World and their choice of perspective improves OOD generalisation.