Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based robotics

Scaling end-to-end reinforcement learning to control real robots from vision presents a series of challenges , in particular in terms of sample efficiency . Against end-to-end learning , state representation learning can help learn a compact , efficient and relevant representation of states that speeds up policy learning , reducing the number of samples needed , and that is easier to interpret . We evaluate several state representation learning methods on goal based robotics tasks and propose a new unsupervised model that stacks representations and combines strengths of several of these approaches . This method encodes all the relevant features , performs on par or better than end-to-end learning , and is robust to hyper-parameters change . 1 INTRODUCTION . A common strategy to learn controllers in robotics is to design a reward function that defines the task and search for a policy that maximizes the collected rewards with a Reinforcement Learning ( RL ) approach . In RL , the controlled system ( environment and robot ) is defined by a state st , i.e. , the relevant variables for a controller , often of low dimension ( e.g. , positions of a robot and a target ) . At a given state st , the agent will receive an observation ot from the environment and a reward rt . In some applications , the observation may be directly the state , but in the general case , the observation is raw sensor data ( e.g. , images from the robot camera ) . RL must then learn a policy that takes observations as input and returns the action at that maximizes expected return . When the state is not directly accessible , RL should recover it from the observation to learn a good control policy . This could be learned implicitly by an end-to-end approach ( cf Fig . 1 ) , i.e . by learning a policy from observation to action , or explicitly by , at first , extracting a representation of this state from the observation and then learning the policy from it . State representation learning ( SRL ) ( Lesort et al. , 2018 ) aims at learning those states as a compact representation from raw observations and without explicit supervision . One key goal of learning state representation separately from learning the policy is to improve the sample efficiency of the full process by reducing the search space . Indeed , end-to-end approaches , even if adequate for simulation settings , are often not sample efficient enough for real life learning as sampling observations from the environment is particularly costly and time consuming in robotics . Another crucial advantage of reducing the search space is to improve stability of policy learning , a common issue in RL ( Henderson et al. , 2017 ) . Although SRL is not restricted to robotics , in this paper , we demonstrate its utility in goal-based robotics tasks , i.e . the controlled agent is a robot , the reward is sparse and only depends on the previous state and taken action , not on a succession of states ( therefore excluding tasks like walking or running ) . Several approaches exist for SRL that differ in the information they can encode . This paper aims at investigating the benefit of different ways of combining state of the art SRL approaches on policy learning for various goal based robotics tasks . The contributions of this paper are : • we show the usefulness of decoupling feature extraction from policy learning ( Section 5.4 ) and that random features provide a good baseline • we propose a new way of combining approaches by stacking state representations instead of mixing them , that allows to mitigate the problem of conflicting objectives and favors disentanglement ( Section 4.4 ) • we investigate the influence of hyper-parameters when learning a state representation ( Section 5.5 ) This paper is organized the following way : we first introduce the state of the art in SRL for robotics ( Section 2 ) , and clarify how we define an appropriate state representation ( Section 3 ) . Then , we explain how we designed our SRL combination approach ( Section 4 ) . Finally , we justify and illustrate our approach with experiments in various simulated robotics tasks ( Section 5 ) . 2 RELATED WORK . In reinforcement learning , a classic preparatory approach is to design some features by hand , in order to facilitate policy learning . However the manual design may be difficult , laborious and requires domain knowledge . Hence , this process can be automated using methods that are able to learn these features ( also called representations ) ( Böhmer et al. , 2015 ; Singh et al. , 2012 ; Boots et al. , 2011 ) . This problem is commonly called State Representation Learning ( SRL ) . We can define it more precisely as a particular kind of representation learning where the learned features are in low dimension , evolve through time , and are influenced by the actions of an agent ( Lesort et al. , 2018 ) . SRL is used as a preliminary step for learning a control policy . The representation is learned on data gathered in the environment by an exploration policy . One particular advantage of this step is that it reduces the search space and gives to reinforcement learning an informative representation , instead of raw data ( e.g . pixels ) . This allows to solve tasks more efficiently ( Lange et al. , 2012 ; Wahlström et al. , 2015 ; Munk et al. , 2016 ) . We refer to ( Lesort et al. , 2018 ) for a complete review of SRL for control . In robotics , SRL is particularly interesting as the learning process is very slow and data hungry . With real robots , learning happens in real time and can not be accelerated or easily multiprocessed as in simulated environments . However , since the learning process remains time consuming and the availability of robots is limited by cost and maintenance constraints , several approaches prefer to iterate at first in simulation to learn robotics tasks ( Jonschkowski & Brock , 2015 ; Watter et al. , 2015 ; Curran et al. , 2016 ; Lesort et al. , 2017 ; Jonschkowski et al. , 2017 ) . Our proposal is based along this line . The robotics environment provides us with rewards , observations and actions , that can be used to define SRL loss functions . Forward models ( Munk et al. , 2016 ) , inverse models ( Shelhamer et al. , 2017 ) , datareconstruction models ( Mattner et al. , 2012 ; Curran et al. , 2016 ) or priors knowledge ( Jonschkowski & Brock , 2015 ; Lesort et al. , 2017 ) are several approaches that exploit those environments data to learn meaningful representations . These methods can also be combined to improve the quality of the learned representations . Some examples include mixing a data-reconstruction objective and a forward model loss ( Watter et al. , 2015 ; Krishnan et al. , 2015 ; Ha & Schmidhuber , 2018 ) , coupling a forward model together with an inverse model ( Pathak et al. , 2017 ) , and using both data-reconstruction and priors loss functions ( Finn et al. , 2015 ) . The goal of this paper is thus to compare decoupling feature extraction ( SRL ) from end-to-end policy learning , and to explore various possible combinations to learn these features . Our setting is similar to the one used in Hindsight Experience Replay ( Andrychowicz et al. , 2017 ) that tackles the problem of solving goal-based robotics tasks with sparse reward . In their experiments , the agent has a direct access to the positions of the controlled robot and target . Our work , on the contrary , uses the raw pixels as input . The extraction of relevant positions must be learned by the different methods . 3 STATE REPRESENTATION REQUIREMENTS . SRL aims at extracting relevant information from raw sensor data . This ability is not the only substantial characteristic of a SRL model . In this section , we provide additional important facets of a good state space . From a high-level point of view , the state representation should retain useful information from the observation in order to solve the task and filter out irrelevant parts . More precisely , the state space should be : Compact : a good state representation should have a low dimension compared to the raw sensor data . It should only keep relevant information , ignoring distractors ( irrelevant parts of the observation ) . This will reduce the search space for RL , leading to a more stable and sample-efficient policy learning . A lowdimensional space is also easier to interpret . Sufficient : all the important information to solve the task should be encoded into the state space . Otherwise , the agent will under-perform ( can not reach maximal performance ) or even fail . Disentangled : the state representation should untangle factors of variation independently , i.e. , each dimension of the feature space should represent a particular factor of variation . A disentangled state representation should facilitate policy learning ( because the policy network does not have to learn how to decipher the raw data ) . The non-redundancy of representation is preferable for disentanglement but not necessary , as long as the representation dimension stays low . Generalizable : As for any machine learning algorithm , representations learned by SRL should be able to generalize to unseen situations . In the SRL context , we can define 2 different degrees of generalization : 1 ) generalization to new environment states and 2 ) generalization to different environments . In the context of a goal-based robotics task , a sufficient state representation should extract the position of the robot , and the position of the goal . If velocities are also needed , they can be approximated using finite differences between two consecutive positions , as in ( Jonschkowski et al. , 2017 ) . A disentangled feature space should encode only one coordinate per dimension , i.e. , one dimension should encode the x-coordinate of the robot position , another one the y-coordinate , etc . 4 INCREMENTALLY BUILDING A POTENTIAL ADEQUATE METHOD . Given the general objectives defined in the previous section , we now propose a way to combine several approaches by tackling one objective at a time , using a particular context for a concrete illustration . This part aims at giving insights on the different SRL methods , taking advantage of goal-based robotics tasks as an application example . 4.1 ENCODING STATE OF THE AGENT : ROBOT POSITION . One important aspect to encode for RL is the state of the controlled agent . In the context of goal-based robotics tasks , it corresponds to the robot position . A simple method consists of using an inverse dynamics objective : given the current st and next state st+1 , the task is to predict the taken action at . The type of dynamics learned is constrained by the network architecture . For instance , using a linear model imposes linear dynamics . The state representation learned encodes only controllable elements of the environment . Here , the robot is part of them . However , the features extracted by an inverse model are not always sufficient : in our case , they do not encode the position of the target since the agent can not act on it . 4.2 ENCODING ADDITIONAL INFORMATION : TARGET POSITION . Since learning to extract the robot position is not enough to solve goal-based tasks , we need to add extra objective functions in order to encode the position of the target object . In this section , we consider two of them : minimizing a reconstruction error ( auto-encoder model ) or a reward prediction loss . Auto-encoder : Thanks to their reconstruction objective , auto-encoders compress information in their latent space . Auto-encoders tend to encode only aspects of the environment that are salient in the input image . This means they are not task-specific : relevant elements can be ignored and distractors ( unnecessary information ) can be encoded into the state representation . They usually need more dimensions that apparently required to encode a scene ( e.g . in our experiments , it requires more than 10 dimensions to encode a 2D position ) . Reward prediction : The objective of a reward prediction module leads to state representations that are specialized in a task . However , this does not constrain the feature space to be disentangled or to have any particular structure .

Summary: This paper discusses State Representation Learning for RL from camera images. Specifically, it proposes to use a state representation consisting of 2 (or 3) parts that are trained separately on different aspects of the relevant state: reward prediction, image reconstruction and (inverse) model learning. The paper is easy to read, and seems technically sound. However, the conclusions do not directly follow from the results, so should be made more precise. The contribution is minor, and the reasoning behind it could be better motivated.