On the Importance of Distraction-Robust Representations for Robot Learning

1 INTRODUCTION . Representation Learning techniques form an integral part in many Reinforcement Learning ( RL ) robot control applications ( Lesort et al. , 2018 ) . Utilising low-dimensional representations can allow for a faster and more efficient learning of tasks than when using high-dimensional sensor information ( Munk et al. , 2016 ) . This is particularly useful in vision-based learning when high-dimensional images of the robot ’ s environment are the principal source of information available to the learning algorithm ( Zhu et al. , 2020 ) . Most commonly , representations are learned by applying dimensionality reduction techniques such as Autoencoders ( AEs ) ( Hinton & Salakhutdinov , 2006 ) or Variational Autoencoders ( VAEs ) ( Kingma & Welling , 2014 ) to the robot ’ s sensory data ( Lange et al. , 2012 ; Zhu et al. , 2020 ) . Generally , an AE consists of two Neural Networks , an Encoder E and a Decoder D. The Encoder attempts to condense all available information in the input data x into a latent representation z from which a reconstruction of the inputs D ( E ( x ) ) is generated by the Decoder . When the dimensionality of the representation is smaller than that of the input data , some information is lost when creating the representations . An AE is typically trained to shrink the magnitude of this information loss by minimising a reconstruction error . This error is commonly given by the squared norm of differences , LAE = ||x−D ( E ( x ) ) ||22 . ( 1 ) However , the optimisation of the reconstruction error in Eq . 1 does not necessarily result in the generation of representations that are optimal for the use in robot learning algorithms . For example , an accurate reconstruction of the decorative patterns on a dinner plate is less important than the plate ’ s dimensions to a robot learning to place it into a cabinet . It can therefore be desirable to control which aspects of the information contained in the inputs are most critical to be preserved in the representations . For instance , Pathak et al . ( 2017 ) design a Neural Network that learns representations from visual inputs by using them to predict the action taken by the RL agent in its state transition . By asking the network to predict the action , the authors eliminate the requirement for the representations to retain any state information that is unrelated to the agent ’ s behaviour . A focus on the learned representations ’ preservation of task-relevant information becomes even more crucial in the presence of distracting influences ( DIs ) in the environment . These DIs can materialise in the presence of additional environment objects which exhibit dynamics that are not only uncorrelated with the robot ’ s behaviour but additionally misleading . For instance , a robot that is learning to move objects to different positions in a room can find the observation of a moving autonomous vacuum cleaner misleading . Alternatively , DIs can impact the dynamics of existing objects in the room . For instance , after the robot has moved a box to a certain position in the room , further movements of the box due to external forces can be distracting to the robot ’ s learning process . In this paper , we introduce the concept of Distraction-Robust Representation Learning . We investigate the learning outcomes of robot control tasks when DIs are present in the environment . We show that in the presence of DIs , representations learned exclusively from environment observations can mislead the robot ’ s perceptions of its control over the environment . This finding demonstrates that Distraction-Robust Representation Learning needs to be afforded increased attention . In particular , works in the strand of research that aim to make RL algorithms more applicable to real-world scenarios largely concentrate on improving algorithm attributes such as the data efficiency ( Zhu et al. , 2020 ) . However , few works acknowledge the challenges posed by the inherently stochastic nature of real-world environments and the presence of DIs ( Forestier et al. , 2017 ) . Furthermore , we introduce a Robot Action Encoder-Decoder architecture ( RAED ) which successfully produces representations that are robust to DIs in the environment . RAED follows the simple but effective approach of using only the values that parameterise the robot ’ s actions as the input to the Encoder . Such a set of parameters defines a robot controller for instance . The representations produced by the Encoder are used by RAED ’ s Decoder to generate predictions of the environment observations . RAED ’ s design allows for static environment elements to be learned by the Decoder while concentrating the information in the representations on the observable consequences of the robot ’ s behaviour . Moreover , when environment observations are distorted by the presence of DIs , RAED produces representations that capture the expected consequences of the robot ’ s environment interactions . This is not the case when training representation learning methods such as AEs to reconstruct the full content of the robot ’ s visual perceptions . We can therefore draw some parallels between RAED ’ s design and the concept of a forward model ( Jordan & Rumelhart , 1992 ) . Given the simplicity of the approach , we expect the applicability of RAED to generalise to various different learning algorithms . 2 RELATED WORK . Several works have investigated mechanisms to preserve only task-relevant information in learned representations . Pathak et al . ( 2017 ) propose a Neural Network architecture that learns representations of visual inputs by predicting the action taken by the RL agent in its state transition . This design allows the representations to dedicate their information capacity to the observable consequences of the robot ’ s actions . Finn et al . ( 2016 ) propose a spatial AE to learn representations that aim to preserve only the configuration of objects in the environment rather than all aspects of the information contained in the camera images . However , in both approaches , the representations are learned from visual inputs which will be distorted if DIs are present in the environment . Without an explicit correction mechanism , these representation learning techniques therefore remain susceptible to distractions . The concept of affordance learning formulates a similar goal in discovering the consequences of the robot ’ s actions on its environment ( Cakmak et al. , 2007 ; Şahin et al. , 2007 ) . However , the works in this strand of research rarely consider the problem of DIs in the environment . Instead , they mainly concentrate on the robot ’ s ability to infer how an object in the environment would behave in response to its actions when no prior interaction experience with that particular object is available ( Dehban et al. , 2016 ; Mar et al. , 2015 ) . A work that uses learned representations and investigates robot interactions in the presence of DIs is presented in the Intrinsically Motivated Goal Exploration Processes ( IMGEP , Laversanne-Finot et al . ( 2018 ) ) . IMGEP aims to enable robots to explore the possible interactions with various tools in an environment that also features distractor objects . These objects either can not be interacted with or move independently of the robot . The authors show that a variant of their proposed algorithm remains unaffected by the presence of these distractors . This robustness is demonstrated by the robot ’ s lack of interaction with the distractor objects . However , the DIs we consider in this paper pose an arguably larger challenge for two main reasons . First , we evaluate distractor objects which exhibit dynamics that are not only independent of the robot ’ s behaviour but also misleading to the robot ’ s perceptions of its interactions with the actual objects of interest . For instance , a distractor object can collide with one of the objects manipulated by the robot . Second , we consider the case of misleading external DIs on the interesting objects themselves . In this case , the object interaction can not be avoided . Furthermore , in IMGEP the distractor robustness is not learned by the VAE that generates the representations . Instead , the algorithm identifies the components in the representations that correspond to the distractors during the robot ’ s interaction phase . Indeed , the learned representations remain static in this phase as the VAE is pre-trained on a manually generated dataset that covers all possible environment observations by spawning the various environment objects , including the distractors , in different positions . Alternative exploration-based approaches have been proposed recently ( Forestier et al. , 2017 ; Pathak et al. , 2017 ; Eysenbach et al. , 2019 ; Sharma et al. , 2019 ) . A particularly interesting framework is presented in AURORA ( Cully , 2019 ) , which we use in our experimental evaluations . Instead of decoupling the representation learning phase from the interaction phase as done in IMGEP , AURORA proposes a joint approach to discover a diverse set of behaviours of any robot in any given environment without the need to specify a task objective or reward function . More precisely , AURORA applies a dimensionality reduction algorithm such as an AE to the robot ’ s sensory data that are collected during the execution of the discovered controllers . The learned latent representation associated to a controller ’ s collected sensory data defines the Behavioural Descriptor ( BD ) for this controller . By measuring the distance between different controllers ’ BDs , the novelty of a given behaviour can be determined . The BD can therefore be used to build a collection of controllers that each exhibit different behaviours . AURORA uses this collection to further explore the space of possible behaviours via stochastic mutations , and continuously adds additional novel controllers to the collection . Periodically , the AE is trained using the sensory data that are extracted from all controllers in the collection . The AURORA framework is well-suited to judge the impact of DIs on the learned representations . Indeed , AURORA ’ s decision to accept newly generated controllers into the collection is exclusively based on the distance between their BDs . We can therefore analyse the influence of DIs using metrics that are formulated directly in the latent representation space . These metrics are more sensitive to the impact of DIs than measures of the overall learning outcomes . This is because the learning outcomes are typically also influenced by components of the learning algorithm other than the representations . Furthermore , AURORA ’ s training of the dimensionality reduction technique provides a particularly challenging setting for RAED . The training dataset composition is continually evolving as the data is collected online . Additionally , AURORA rejects controllers that generate behaviours very similar to that of existing controllers . This complexifies the task of learning the expected consequences of the robot ’ s actions when the collected environment observations are distorted by DIs . 3 METHOD . In this work , we use a single vector a to parameterise the set of actions taken by the robot from the beginning until the end of the simulation . This vector of the robot ’ s actions can store the parameter values of a controller for instance . We use a vector x to parameterise the single environment observation that is returned by each simulation . This vector can store the pixel values of an image for example . RAED consists of an Encoder-Decoder structure that uses a and x to produce a representation z that is distraction-robust . More precisely , the latent representation z is generated from RAED ’ s Encoder networkE which takes the robot ’ s actions a as its inputs . This conditioning of the representations on the robot ’ s actions is the crucial element in RAED ’ s design . The Decoder uses the representations to produce a prediction of the environment observation D ( E ( a ) ) . The actual environment observation x is used as the prediction target in the training of RAED . This yields the prediction error given in Eq . 2 . Note , the difference in the Encoder input compared to an AE ’ s reconstruction error in Eq . 1 . C = ||x−D ( E ( a ) ) ||22 . ( 2 ) Depending on the specifics of the robot design and the task , a set of controllers with different parameters can yield the same observed behaviour . For instance , when steering a robot arm with redundancies in its joints , different controllers can achieve the same position in a gripper connected to the arm . For such a set of controllers , it is possible that the differences in their parameters lead to the generation of different representations when using RAED , even if the observation predictions are exactly the same . This is less likely to occur in a dimensionality reduction technique that takes the visual perceptions as its inputs . For these methods , the principal sources of variation in the representations are the differences in the observations rather than the differences in the robot ’ s actions . To reduce RAED ’ s freedom to disperse the representations of similar observed behaviours , we investigate the addition of a penalty term P . The overall training loss is then given by LRAED = C+P . One possible instantiation of P is the Kullback-Leibler divergence ( KL ) term used in a VAE ( Kingma & Welling , 2014 ) . This KL term implicitly imposes a penalty on the magnitude of the means produced by the VAE ’ s Encoder network . The consequence is a crowding of the representations into a smaller latent subspace . This can have the desired effect of reducing RAED ’ s freedom to produce dissimilar representations for similar observed behaviours . However , the practical implementation of the KL term also imposes a penalty on any Encoder variance magnitudes different than 1 . Large Encoder variances result in an increased dispersion of representations when these are sampled from the Encoder . The presence of these variances can therefore be interpreted as a further source of stochasticity when learning representations . Our experimental results in Section 4.4 demonstrate that the application of the KL term benefits from leaving the Encoder variance outputs unused . The derived penalty can be summarised in a simple linear term given by Plinear = ||z||22 . In the AURORA framework , RAED can be used as a modular representation learning component to produce the BDs for the discovered controllers . We present an illustration of AURORA in Fig . 1 and a pseudo-code in Algorithm 1 in Appendix B.1 . AURORA largely follows a standard QualityDiversity algorithm ( Cully & Demiris , 2018a ) . At the start of each iteration , a predefined number of controllers is selected from the collection . This group of controllers forms the set of parent controllers . Each pair of parents undergoes cross-over and mutation operations . Cross-over operations produce new controllers that bear similarities to both parents . Mutation operations introduce some stochastic changes to these new controllers . If the collection is empty at the start of the iteration , we instead generate new controllers by randomly sampling in the controller space . Each controller is executed in the simulated environment and the associated observations are captured . These observations are used to produce the BD of each controller . When using an AE in AURORA , each BD is the latent representation generated using the environment observation x as input . In RAED , the robot ’ s actions a are the inputs instead . AURORA , therefore , stores for each controller a vector of robot action parameters , an environment observation and a BD . In the decision to accept a given new controller into the collection , AURORA considers the distance between its BD to the descriptor of the nearest controller contained in the collection . If this distance exceeds a minimum distance threshold d , the controller is deemed sufficiently novel to be added to the collection ( Cully & Demiris , 2018b ) . Otherwise , it is discarded . d is initialised at the start of the algorithm using the BDs of the initial set of controllers contained in the collection . AURORA then updates the value of d after each iteration to steer the size of the collection towards the target size . Every N iterations , AURORA extracts the environment observations and robot actions from all controllers contained in the collection to train the AE or RAED respectively . Once the training of the network concludes , all controllers are removed from the collection and assigned new BDs . This set of controllers is finally re-added to the collection using the acceptance mechanism described in the previous paragraph . This concludes one iteration of the AURORA algorithm . The hyperparameters used in AURORA are given in Appendix B.1 .

Summary: This paper approaches the problem of representation learning for robotic learning, with a focus on being robust to potentially misleading distractor objects. Concretely, they propose a method which learns a representation of sequences of actions by reconstructing a future state, and uses this representation of actions to learn skills. They show that this leads to some improvement in identifying more diverse "skills" in settings where there are external distractors.