Learning Latent Representations for Inverse Dynamics using Generalized Experiences

1 INTRODUCTION . In reinforcement learning ( RL ) , an agent optimizes its behaviour to maximize a specific reward function that encodes tasks such as moving forward or reaching a target . After training , the agent simply executes the learned policy from its initial state until termination . In practical settings in robotics , however , control policies are invoked at the lowest level of a larger system by higher-level components such as perception and planning units . In such systems , agents have to follow a dynamic sequence of intermediate waypoints , instead of following a single policy until the goal is achieved . A typical approach to achieving goal-directed motion using RL involves learning goal-conditioned policies or value functions ( Schaul et al . ( 2015 ) ) . The key idea is to learn a function conditioned on a combination of the state and goal by sampling goals during the training process . However , this approach requires a large number of training samples , and does not leverage waypoints provided by efficient planning algorithms . Thus , it is desirable to learn models that can compute actions to transition effectively between waypoints . A popular class of such models is called Inverse Dynamics Model ( IDM ) ( Christiano et al . ( 2016 ) ; Pathak et al . ( 2017 ) ) . IDMs typically map the current state ( or a history of states and actions ) and the goal state , to the action . In this paper , we address the need of an efficient control module by learning a generalized IDM that can achieve goal-direction motion by leveraging data collected while training a state-of-the-art RL algorithm . We do not require full information of the goal state , or a history of previous states to learn the IDM . We learn on a reduced goal space , such as 3-D positions to which the agent must learn to navigate . Thus , given just the intermediate 3-D positions , or waypoints , our agent can navigate to the goal , without requiring any additional information about the intermediate states . The basic framework of the IDM is shown in Fig . 1 . The unique aspect of our algorithm is that we eliminate the need to randomly sample goals during training . Instead , we exploit the known symmetries/equivalences of the system ( as is common in many robotics settings ) to guide the collection of generalized experiences during training . We propose a class of algorithms that utilize the property of equivalence between transitions modulo the difference in a fixed set of attributes . In the locomotion setting , the agent ’ s transitions are symmetric under translations and rotations . We capture this symmetry by defining equivalence modulo orientation among experiences . We use this notion of equivalence to guide the training of latent representations shared by these experiences and provide them as input to the IDM to produce the desired actions , as shown in Fig . 4 . A common challenge faced by agents trained using RL techniques is lack of generalization capability . The standard way of training produces policies that work very well on the states encountered by the agent during training , but often fail on unseen states . Achieving good performance using IDMs requires both these components : collecting generalized experiences , and learning these latent representations , as we demonstrate in Section 6 . Our model exhibits high sample efficiency and superior performance , in comparison to other methods involving sampling goals during training . We demonstrate the effectiveness of our approach in the Mujoco Ant environment ( Todorov et al . ( 2012 ) ) in OpenAI Gym ( Brockman et al . ( 2016 ) ) , and the Minitaur and Humanoid environments in PyBullet ( Coumans & Bai ( 2016 ) ) . From a limited number of experiences collected during training under a single reward function of going in one direction , our generalized IDM succeeds at navigating to arbitrary goal positions in the 3-D space . We measure performance by calculating the closest distance to the goal an agent achieves . We perform ablation experiments to show that ( 1 ) collecting generalized experience in the form of equivalent input pairs boosts performance over all baselines , ( 2 ) these equivalent input pairs can be condensed into a latent representation that encodes relevant information , and ( 3 ) learning this latent representation is in fact critical to success of our algorithm . Details of experiments and analysis of results can be found in Sections 5 and 6 . 2 RELATED WORK . Several recent works learn policies and value functions that are conditioned over not just the state space , but also the goal space ( Andrychowicz et al . ( 2017 ) ; Schaul et al . ( 2015 ) ; Kulkarni et al . ( 2016 ) ) and then generalize those functions to unseen goals . Goal-conditioned value functions are also largely used in hierarchical reinforcement learning algorithms ( Kulkarni et al . ( 2016 ) ) , where the higher level module learns over intrinsic goals and the lower level control modules learn subpolicies to reach those goals , or the lower level control modules can efficiently execute goals proposed by the higher-level policy ( Nachum et al . ( 2018 ) ) . Ghosh et al . ( 2018 ) use trained goalconditioned policies to learn actionable latent representations that extract relevant information from the state , and use these pretrained representations to train the agent to excel at other tasks . Pong * et al . ( 2018 ) learn goal-conditioned value functions , and use them in a model-based control setting . IDMs are functions that typically map the current state of the agent and the goal state that the agent aims to achieve , to the desired action . They have been used in a wide variety of contexts in existing literature . Christiano et al . ( 2016 ) train IDMs using a history of states and actions and full goal state information for transferring models trained in simulation to real robots . Pathak et al . ( 2017 ) and Agrawal et al . ( 2016 ) use IDMs in combination with Forward Dynamics Models ( FDMs ) to predict actions from compressed representations of high-dimensional inputs like RGB images generated by the FDM . Specifically , Pathak et al . ( 2017 ) use IDMs to provide a curiosity-based reward signal in the general RL framework to encourage exploration ; Agrawal et al . ( 2016 ) use IDMs to provide supervision for the learning of visual features relevant to the task assigned to the robot . We circumvent the need to learn goal-conditioned policies or value functions by combining IDMs with known symmetric properties of the robot . We train an IDM conditioned on the state space and a reduced goal space , using data collected while training any state-of-the-art RL algorithm . Our data collection is unique in that we exploit equivalences in experiences observed during training and learn a latent representation space shared between such equivalent experiences . Our IDM produces the action given this latent representation as an input , leading to generalization over parts of the state and goal spaces unobserved during training . 3 PRELIMINARIES . In the general RL framework ( Sutton et al . ( 1998 ) ) , a learning agent interacts with an environment modeled as a Markov Decision Process consisting of : 1 ) a state space S , 2 ) an action space A , 3 ) a probability distribution P : S × S × A → [ 0 , 1 ] , where P ( s′|s , a ) is the probability of transitioning into state s′ by taking action a in state s , 4 ) a reward function R : S × A × S → R that gives the reward for this transition , and 5 ) a discount factor Γ . The agent learns a policy πθ : S → A , parameterized by θ while trying to maximize the discounted expected return J ( θ ) = Es0 , a0 , ... [ ∑∞ t=0 Γ tR ( st , at , st+1 ) ] . Goal-conditioned RL optimizes for learning a policy that maximizes the return under a goal-specific reward function Rg . On-policy RL algorithms , such as Policy Gradient methods ( Williams ( 1992 ) ; Mnih et al . ( 2016 ) ) and Trust Region methods ( Schulman et al . ( 2015 ) ; Wu et al . ( 2017 ) ) use deep neural networks to estimate policy gradients , or maximize a surrogate objective function subject to certain constraints . Off-policy RL algorithms ( Lillicrap et al . ( 2016 ) ; Haarnoja et al . ( 2018 ) ) incorporate elements of deep Q-learning ( Mnih et al . ( 2013 ) ) into the actor-critic formulation . Hindsight Experience Replay ( HER ) ( Andrychowicz et al . ( 2017 ) ) is a popular technique used in conjunction with an off-policy RL algorithm to learn policies in a sample-efficient way from sparse rewards in goal-based environments . 4 LEARNING GENERALIZED INVERSE DYNAMICS . Our method leverages samples collected while training a state-of-the-art RL algorithm to train an IDM that maps the current state and desired goal position to the action required to reach the goal . There are four major components involved in this process : 1 ) collecting data while training the RL algorithm , 2 ) learning a basic IDM that maps the current state and the desired goal to the required action , 3 ) collecting experiences that are equivalent to those observed , and using them to train the IDM , and 4 ) learning a latent representation that generalizes this model to unseen parts of the state space by utilizing the equivalent experiences collected in step 3 . We elaborate on each of these in the following sections . 4.1 INITIAL TRAINING AND COLLECTING EXPERIENCE . Our goal in this steps is to collect data for our IDM in the process of learning a policy under a single reward function . Recall the motivation for learning IDMs : we want a model that can take in the current state of the agent and the next command , in the form of a location in the space that the agent should travel to . Thus , we collect state , action , and position data from the transitions observed during the training process . We emphasize the difference between the state space S and the goal space O . The state space S is high-dimensional , consisting of information related to joint angles , velocities , torques , etc . The goal space , O , is low-dimensional , consisting , in this case , of the 3-D coordinates of the goal position that the agent is supposed to navigate to . Definition 1 ( Experiences ) . We define experiences as tuples ( s , o , o′ , a ) , where s is the current state of the agent , o is its current 3-D position , and a is the action that the agent performed to move from the state s and position o to the next position , or intermediate goal , o′ . We write Eτ = { ( s , o , o′ , a ) i } i=1 , ... , T to denote all the experience tuples collected from a trajectory τ of length T . 4.2 LEARNING THE INVERSE DYNAMICS MODEL . Given a set of experiences E , we can train the IDM using supervised learning techniques . Definition 2 ( Inverse Dynamics Model ) . We define the Inverse Dynamics Model ( IDM ) as φ : S ×O ×O → A , φ ( s , o , o′ ) → a ( 1 ) where s , o , o′ and a represent the current state , current position , desired goal , and action respectively . The IDM can reproduce seen actions on state and observation tuples that have appeared in the training data . However , it can not generalize good behaviour to states and observations that have not appeared in the initial training ( see Fig . 6 for qualitative evidence ) . Our aim in the next steps is to generalize the observed experiences so that they can be used over previously unseen inputs from the S ×O ×O space .

Summary: The paper proposes a method for exploiting structure in locomotive tasks for efficiently learning low-level control policies that pass through waypoints while achieving some goal (typically 3D Cartesian position). This is in contrast to goal-conditioned RL policies that sample random goals during training and are thus sample inefficient, which are trained to execute one policy at a time. In particular, the paper proposes the notion of generalized experiences, where new trajectories are generated from existing trajectories, in such as a way that they are equivalent to each other (in this case translation and orientation invariant) with respect to actions.