Vision-Based Manipulators Need to Also See from Their Hands

1 INTRODUCTION . Physical manipulation is so fundamental a skill for natural agents that it has been described as a “ Rosetta Stone for cognition ” ( Ritter & Haschke , 2015 ) . How can we endow machines with similar mastery over their physical environment ? One promising avenue is to use a data-driven approach , 1Project website : https : //sites.google.com/view/seeing-from-hands in which the mapping from raw sensor observations of the environment ( and other readily available signals , e.g . via proprioception ) to actions is acquired inductively . Helpful inductive biases in modern machine learning techniques such as over-parameterized models and stochastic gradient descent have enabled surprising ( and poorly understood ) generalization capabilities in some applications ( Neyshabur et al. , 2014 ; Belkin et al. , 2019 ; Zhang et al. , 2021 ) . Despite this , visuomotor policies learned end-to-end remain brittle relative to many common real-world distribution shifts : subtle changes in lighting , texture , and geometry that would not faze a human cause drastic performance drops ( Julian et al. , 2020 ) . While a wide variety of algorithms have been proposed to improve the learning and generalization of object manipulation skills , in this paper we instead consider the design of the agent ’ s observation space , a facet of the learning pipeline that has been underexplored ( Section 5 ) . Indeed , in some applications of machine learning , e.g. , image classification or text summarization , the disembodied nature of the task affords relatively little flexibility in this regard . Yet , even in these settings , simple data processing techniques such as normalization and data augmentation can have noticeable effects on learning and generalization ( Perez & Wang , 2017 ) . The role of data can only be more profound in an embodied setting : any sensors capable of being practically instrumented will only provide a partial observation of the underlying world state . While partial observability is typically regarded as a challenge that only exacerbates the difficulty of a learning problem ( Kaelbling et al. , 1998 ) , we may also consider how partial observations can facilitate the acquisition of useful symmetries . The natural world gives clear examples of this . For instance , because cutaneous touch is inherently restricted to sensing portions of the environment in direct contact with the agent , tactile sensing by construction exhibits invariances to many common transformations on the underlying world state ; grasping an apple from the checkout counter ( without looking at it ) is largely the same as doing so from one ’ s kitchen table . Due in part to the nascent state of tactile sensing hardware ( Yuan et al. , 2017 ) and simulation ( Agarwal et al. , 2020 ) , in this work we investigate the above insight in vision , the ubiquitous robotic sensory modality . In particular , we focus on the role of perspective as induced from the placement of cameras . To roughly imitate the locality of cutaneous touch , we consider the hand-centric ( eye-in-hand ) perspective arising from mounting a camera on a robotic manipulator ’ s wrist . We also consider the more commonly used third-person perspective afforded by a fixed camera in the world frame . The main contribution of this work is an empirical study of the role of visual perspective in learning and generalization in the context of physical manipulation . We first perform a head-to-head comparison between hand-centric and third-person perspectives in a custom simulated grasping environment that features three kinds of distribution shifts . We find that using the hand-centric perspective , with no other algorithmic modifications , reduces aggregate out-of-distribution failure rate by 92 % , 99 % , and 100 % ( relative ) in the imitation learning , reinforcement learning , and adversarial imitation learning settings , respectively . We also observe appreciable gains in training efficiency in the latter two settings . Despite their apparent superiority , hand-centric perspectives can not be used alone for tasks in which their limited observability is a liability during training . To realize the benefits of hand-centric perspectives more generally , we propose using both hand-centric and third-person perspectives in conjunction for full observability while regularizing the latter with a variational information bottleneck ( Alemi et al. , 2016 ) to mitigate the latter ’ s detrimental effects on out-of-distribution generalization . We instantiate this simple and broadly applicable principle in DrQ-v2 ( Yarats et al. , 2021 ) , a state-of-the-art vision-based reinforcement learning algorithm , and find that it reduces the aggregate out-of-distribution failure rate compared to using both perspectives naively by 64 % ( relative ) across six representative manipulation tasks with varying levels of hand-centric observability adapted from the Meta-World benchmark ( Yu et al. , 2020 ) . 2 PROBLEM SETUP . Preliminaries : MDPs and POMDPs . We frame the physical manipulation tasks considered in this work as discrete-time infinite-horizon Markov decision processes ( MDPs ) . An MDPM is a 6-tuple ( S , A , P , R , γ , µ ) , where S is a set of states , A is a set of actions , P : S × A → Π ( S ) is a statetransition ( or dynamics ) function , R : S × A → R is a reward function , γ ∈ ( 0 , 1 ) is a discount factor , and µ ∈ Π ( S ) is an initial state distribution . An MDP whose state can not be directly observed can be formalized as a partially observable MDP ( POMDP ) , an 8-tuple ( S , A , P , R , γ , µ , Ω , O ) that extends the underlying MDP with two ingredients : a set of observations Ω and an observation function O : S × A → Π ( Ω ) . We consider only a restricted class of POMDPs in which the observation function is limited to be O : S → Ω . To solve a POMDP , we optimize a policy π : Ω→ Π ( A ) to maximize the expected return R ( M , π ◦O ) = Eµ , P , π [ ∑∞ t=0 γ tR ( st , at ) ] , where π ◦O maps a state to an action distribution via composing the policy and observation function . Observation functions . In this work , we denote the observation functions corresponding to the hand-centric and third-person perspectives as Oh and O3 , respectively . We also consider proprioception via forward kinematics , denoting this as Op . Often , multiple observation functions are used together ; for example , we denote using both the hand-centric and proprioceptive observations as Oh+p . Invariances and generalization . We say that a function f : X ×Y → Z is invariant in domain subspace X to a transformation T : X → X iff ∀x ∈ X , y ∈ Y. f ( T ( x ) , y ) = f ( x , y ) . We formalize the notion of generalization by saying that π ◦ O generalizes inM to a distribution shift caused by transformation T iff R ( M , π ◦O ) is invariant inM to T . We consider two kinds of generalization : in-distribution and out-of-distribution generalization , also referred to as interpolation and extrapolation . The latter corresponds to the agent generalizing inM to some specified transformation , and the former is a special case when the transformation is identity . In this work , we limit the scope of the transformations onM we consider to those acting on the initial state distribution µ through the state set S. A few concrete examples of such transformations are illustrated in Figure 1 . 3 HAND-CENTRIC VS. THIRD-PERSON PERSPECTIVES . The first hypothesis we investigate is that using the hand-centric perspective Oh instead of the thirdperson perspectiveO3 can significantly improve the learning and generalization of the agent π◦O . In this section , we probe this hypothesis in a setting where the hand-centric perspective gives sufficient observability of the scene ( we consider when this does not hold in Section 4 ) . See project website https : //sites.google.com/view/seeing-from-hands for animated visualizations . 3.1 EXPERIMENTAL SETUP . We consider a visuomotor grasping task instantiated in PyBullet ( Coumans & Bai , 2016–2021 ) . A Franka Emika Panda manipulator is tasked with picking up a specific cube that initially rests on a table . The action space is 4-DoF , consisting of 3-DoF end-effector position control and 1-DoF gripper control . Observation functions include Oh and O3 , which output 84 × 84 RGB images , and Op , which outputs 3D end-effector position relative to the robot base , 1D gripper width , and a Boolean contact flag for each of two gripper “ fingers “ . We use three learning algorithms : imitation learning with dataset aggregation ( DAgger ) ( Ross et al. , 2011 ) , reinforcement learning using data-regularized Q-functions ( DrQ ) ( Kostrikov et al. , 2020 ) , and adversarial imitation learning using discriminator-actor-critic ( DAC ) ( Kostrikov et al. , 2018 ) . We defer exposition on these algorithms to Appendix A.2 . We run DAgger and DrQ on three experiment variants each targeting a different distribution shift ( detailed and visualized in Appendix A.1 ) . With DAC , we assess in-distribution generalization in the training environment and out-of-distribution generalization between demonstration ( demo ) collection and the training environment . Details on the model architectures and hyperparameters used can be found in Appendices A.3 and A.4 . 3.2 EXPERIMENTAL RESULTS AND DISCUSSION . DAgger and DrQ results are reported in Figure 2 and aggregated in Table 1 . DAC results are reported in Figure 3 , with experiment variant descriptions in the caption . For DAgger ( left two columns of Figure 2 ) , we find that the hand-centric perspective leads to clear improvements in out-of-distribution generalization ( test ) across all three experiment variants despite in-distribution generalization progress ( train ) being essentially identical between π ◦Oh+p and π◦O3+p . The only exceptions to π◦Oh+p generalizing better are in some instances of the distractor objects variant . Here , seeing the red , green , and blue distractor objects during training was sufficient for both π ◦ Oh+p and π ◦ O3+p to learn to ignore these object colors , even under distractor distribution shift . Generalization to white distractors was likely facilitated by the RGB representation of white as the “ sum ” of red , green , and blue . For DrQ ( right two columns of Figure 2 ) , the differences between π◦Oh+p and π◦O3+p extend into training time . In the table height variant , π ◦Oh+p exhibits increased sample efficiency for training as well as similar out-of-distribution generalization benefits as seen for DAgger . For the distractor objects variant , π ◦ Oh+p converges before π ◦ O3+p makes any significant progress on success rate ( though we did observe increasing returns ) . Since DrQ trained π ◦ O3+p to convergence for the other variants within the same interaction budget , it follows that the presence of the distractors rendered the training task too hard for π ◦O3+p , but not for π ◦Oh+p . In the table textures variant , the generalization improvement of π ◦ Oh+p over π ◦ O3+p is less extreme . We attribute this to invariances to image-space transformations learned via the data augmentation built into DrQ . In Appendix A.5 , an ablation in which this augmentation is removed further shows its importance . For DAC , we find stark improvements in the generalization of π ◦ Oh over that of π ◦ O3 . In the first DAC-specific experiment variant ( left plot of Figure 3 ) , π ◦Oh fully generalizes in-distribution with as few as 5 demos , whereas π ◦ O3 achieves significantly lower success , even with 25 demos and much more online interaction . In the second variant ( center plot of Figure 3 ) , the distribution shift between demo collection and training barely affects π ◦ Oh , but severely compromises the training of π ◦ O3 . In the third variant ( right plot of Figure 3 ) , despite the presence of distractor objects giving the discriminator strong predictive power in distinguishing between demos and agent behavior , π◦Oh still achieves a significant measure of in-distribution generalization , whereas π◦O3 makes little progress even with eight times the number of demos . We remark that , in the context of adversarial imitation learning , π ◦ Oh achieves its sample efficiency and robustness without any special requirements on the training data ( Zolna et al. , 2020 ) or modified training objectives ( Xu & Denil , 2020 ) .

Summary: This paper presents an interesting empirical evaluation of the role of visual perspective in learning and generalization in the context of physical manipulation. The paper compares performances using third-person and in-hand visual perspectives, showing that hand-centric vision consistently improves training and out-of-distribution generalization. Authors also explore a combination of the two perspectives by proposing to regularize the third-person information stream to maintain generalization performance.