Unsupervised Emergence of Spatial Structure from Sensorimotor Prediction

Despite its omnipresence in robotics application , the nature of spatial knowledge and the mechanisms that underlie its emergence in autonomous agents are still poorly understood . Recent theoretical work suggests that the concept of space can be grounded by capturing invariants induced by the structure of space in an agent ’ s raw sensorimotor experience . Moreover , it is hypothesized that capturing these invariants is beneficial for a naive agent trying to predict its sensorimotor experience . Under certain exploratory conditions , spatial representations should thus emerge as a byproduct of learning to predict . We propose a simple sensorimotor predictive scheme , apply it to different agents and types of exploration , and evaluate the pertinence of this hypothesis . We show that a naive agent can capture the topology and metric regularity of its spatial configuration without any a priori knowledge , nor extraneous supervision . 1 INTRODUCTION . Space appears to be a pervasive concept in our perception of the world , and as such plays a central role in most artificial perception systems , in particular in computer vision and robotics applications . Yet its fundamental nature and the mechanisms that could lead to its emergence in an artificial system still remain poorly understood , despite some noteworthy contributions ( Kant , 1781 ; Poincaré , 1895 ; Nicod , 1924 ) . In most cases , the problem is circumvented by implementing prior knowledge in the system regarding the structure of space , and how motor and sensory information convey spatial properties ( for instance through a kinematics model ( Siciliano & Khatib , 2016 ) , or a sensor model ( Cadena et al. , 2016 ) ) . In more recent years , and with the developments of machine learning techniques , approaches with less hand-engineered priors developed to solve spatial tasks ( ( Kahn et al. , 2017 ; Quillen et al. , 2018 ; Levine et al. , 2018 ; Smolyanskiy et al. , 2017 ) to name a few ) . However they tend to set aside the specificity of spatial experiences in favor of a global assessment of the agent ’ s performance , leaving the question of the origin and structure of spatial knowledge largely open . Nevertheless , recent work presented in ( Laflaquière et al. , 2018 ) , directly addresses the grounding of spatial knowledge . It gives a theoretical perspective on how a naive agent could build a notion of spatial configuration , in an unsupervised way . The approach takes inspiration from philosophical ideas formulated more than a century ago by H.Poincaré ( Poincaré , 1895 ) , and from the more recent Sensori-Motor Contingencies Theory ( SMCT ) which puts forward an unsupervised sensorimotor grounding of perception ( O ’ Regan & Noë , 2001 ) . It describes how space naturally induces specific invariants in any situated agent ’ s sensorimotor experience , and how these invariants can be autonomously captured to improve the compactness of internal representations and the prediction of sensorimotor experiences . In this paper , we propose a practical evaluation of this theoretical work . We introduce a simple self-supervised learning scheme in which an agent learns to predict its sensory experience , given its motor state . We analyze the encoding of the motor experience produced this way , and in particular how it is impacted by different types of exploration of the environment . More precisely , we show that by experiencing consistent sensorimotor transitions in a moving environment , a naive agent can learn to encode its motor state in such a way as to capture the topological structure and metric regularity of its external position , without the need for additional a priori knowledge . 2 RELATED WORK . The problem of building spatial representations of some sort is addressed in a multitude of machine learning papers . Due to space constraints , we can not give a complete overview , but will only highlight the most relevant to this work . The problem of spatial knowledge acquisition is often conceptualized in a supervised or reinforcement learning ( RL ) framework . The goal is to build an internal representation of an agent ’ s position and displacement in an allocentric frame . Many approaches have been proposed to represent localization in the environment as place cells ( Arleo et al. , 2001 ; Weiller et al. , 2010 ; Stachenfeld et al. , 2017 ) . These representations are typically grounded in the sensory space , making them brittle to environmental changes . They also rely on specifically designed priors about how to form and compare prototypical sensory states . An extreme case would of course be SLAM-type approaches where the whole spatial structure of the problem is hard-coded a priori in the robot ( Cadena et al. , 2016 ) . Taking inspiration from biology , other approaches rather focus on representing displacements via grid cells ( Banino et al. , 2018 ; Cueva & Wei , 2018 ) . They typically rely on the prior definition of place cells , or on the direct access to the agent ’ s position in the world during training . Many works focus on building state representation for RL . Although not necessarily spatial in nature , these representations are often optimized to solve spatial tasks . They typically rely on priors such as physical properties of the world ( Jonschkowski & Brock , 2015 ) , controllability of the representation ( Watter et al. , 2015 ) , or disentanglement of the representation ( Thomas et al. , 2017 ) . A large class of RL-based approach also aims at solving intrinsically spatial tasks in an end-to-end fashion . This is for instance the case of the influential Atari playing agent ( Mnih et al. , 2013 ) , or of ( Mirowski et al. , 2016 ) where auxiliary tasks are used to enforce some spatial properties ( loop closing ) . Recently , ( Eslami et al. , 2018 ) proposed a network able to generate perspective-conditioned images of 3D scenes , given explicit spatial perspective inputs . These work are however focused on solving a task , and do not investigate how spatial knowledge might be internally encoded and used by the agent . A last class of work rely more on unsupervised or self-supervised mechanisms to build spatial representations or solve spatial tasks . This is for instance the case in ( Ha & Schmidhuber , 2018 ; Wayne et al. , 2018 ) , where compression and self-prediction are used to build a world model helping an agent to navigate in its environment . Intrinsic curiosity has also been used to learn to control an agent moving in an environment , even in the absence of reward ( Pathak et al. , 2017 ) . More related to this work , sensorimotor prediction has also been considered as a mechanism to integrate motor sequences into displacement-like representations ( Ortiz & Laflaquière , 2018 ) . The work presented in this paper differs from these related works in that we study the emergence of the concept of egocentric position . A naive agent with only access to its sensorimotor flow indeed has to discover that the motor configurations it produces correspond to external spatial configurations , with their own topology and ( Euclidean ) metric . We cast the problem outside of the typical reinforcement learning framework , by not considering any extraneous reward , and by not assuming the existence of any spatial prior . We instead consider learning of a sensorimotor predictive model as the only driving force in building spatial representations . Moreover , we explicitly study how representations with spatial properties can be derived from the motor space . As such , this paper is more in line with theoretical work addressing the problem of space perception from the SMCT perspective ( Laflaquiere et al. , 2012 ; Laflaquière et al. , 2015 ; Terekhov & ORegan , 2016 ) . 3 PROBLEM SETUP . In this work , we study how a naive agent can build an internal representation of its external position in an egocentric frame . Inspired by Poincaré ’ s insight ( Poincaré , 1895 ) , and borrowing from differential geometry ( Philipona et al. , 2003 ) , we look at the agent ’ s experience as a sensorimotor mapping parametrized by the state of the environment . We assume that an agent ’ s instantaneous sensory and motor states are respectively defined by vectors s ∈ RNs and m ∈ RNm . The motor states can be seen as static joint configurations , or as proprioceptive readings in the case the agent is not controlled in position . No additional assumption is made regarding the way sensorimotor information is encoded . The unknown sensorimotor mapping induced by the environment is assumed continuous and denoted φ , such that s = φ ( m ) , where ∈ RN is the state of the environment . The mapping φ can be seen as describing how “ the world ” transforms changes in motor states into changes in sensory states , and thus encapsulates the structure of the external space in which the agent is embedded . Finally we denote p ∈ RNp the unknown spatial position of the agent ’ s sensor in an egocentric frame of reference . The problem we address in this work is the one of building an internal representation h which captures the properties of p. Indeed , p has some topological and metric properties that the naive agent can not directly access . They need to be derived from sensorimotor experiences . It has been shown that space induces , through φ , specific sensorimotor invariants that an agent can capture to discover the structure of p ( Laflaquière et al. , 2018 ; Laflaquiere et al. , 2013 ) . A first kind of invariants concerns the topology of the sensor position p. Assuming a rich enough environment and sensory apparatus , there exists a homeomorphic relation between the sensor position and the sensory manifold generated by exploring the motor space . Intuitively , for all environment states , motor changes which generate small sensor displacements are associated with small sensory changes : ∀ , if ( mi , mi′ ) is such that |−−−→pipi′ | µ then | −−−−−−−−−−→ φ ( mi ) φ ( mi′ ) | = |−−→sisi′ | µ , ( 1 ) where |.| denotes the Euclidean norm , and µ is a small value . This means that the topology of the sensor ’ s position is accessible to the agent via the sensorimotor flow . Note that in the case of a redundant motor system , multiple motor states m lead to the same position p , and thus to identical sensory states ( |−−→sisj | = 0 ) , for any state of the environment . Based on this sensory equivalence , the agent can discover that the manifold of positions p has a lower dimension than its motor space . This space-induced topological invariant should be accessible to the agent under condition I : the agent should experience consistent sensorimotor transitions such that the state of the environment does not change during the transition ( see Eq . ( 1 ) ) . A second kind of invariant concerns the metric regularity of the sensor position p. Assuming the environment can move relative to the agent , multiple motor changes associated with the same external displacement of the sensor can generate the same sensory changes , depending on the position of the environment . Consider for instance that the agent looks at a bottle that can be at different locations . Then the same sensory change , corresponding to moving the sensor ( camera ) from the bottle ’ s top to its bottom , can be generated by different motor changes depending on where the bottle is located relatively to the agent . All these motor changes actually correspond to the same external displacement ( change of position ) of the sensor . This sensory equivalence of motor changes associated with the same displacement holds for all environmental states ( the bottle in the example can be replaced with any other object ) : ∀ , if ( mi , mi′ , mj , mj′ ) is such that |−−−→pipi′ −−−−→pjpj′ | µ then ∃δ ( ) : | −−−−−−−−−−→ φ ( mi ) φ ( mi′ ) − −−−−−−−−−−−−−−−−−−→ φ +δ ( ) ( mj ) φ +δ ( ) ( mj′ ) | = |−−→sisi′ −−−→sjsj′ | µ , ( 2 ) where δ ( ) represents a displacement of the environment . This means that the spatial equivalence of motor changes , and thus the metric regularity of the sensor position p are accessible to the agent via the sensorimotor flow . This space-induced metric invariant should be accessible to the agent under condition II : the agent should experience displacements of the environment in-between sensorimotor transitions ( see Eq . ( 2 ) ) . This ensures the agent has the opportunity to experience the same sensory changes with different motor changes . It has been shown ( Laflaquière et al. , 2018 ) that it is theoretically possible to build a motor representation h which respects space-induced invariants by : i ) having similar representations h for motor states m associated with similar sensory states s , and ii ) having similar representation changes −−→ hh′ for motor changes −−−→ mm′ associated with similar sensory changes −→ ss′ . We hypothesize that building such a representation is beneficial to a naive agent learning to predict its own sensorimotor experience . Indeed , these invariants hold for all states of the environment . They should thus act as a prior over sensorimotor transitions not yet experienced , improving this way the predictive capacity of the system . As a consequence , sensorimotor prediction should naturally lead to the emergence of a representation h capturing the topology and metric regularity of the egocentric spatial configuration p. Given the topological and metric relations we described , any h which is an affine transformation of the ground-truth position p would be an optimal solution , as such a transformation preserves topology and distance ratios .

Summary: This very interesting paper is based on the sensorimotor contingency theory, which grounds the perception of the agent (motor perception and sensory perception) in the capacity to learn to predict future sensory experience and to build a compact internal representation of proprioception and motor states. They show that through active experience (exploration) of an environment, an agent can encode its motor state in a way that captures both the topology and the relative distances. The authors show that in the case of an environment consistent with sensorimotor transitions and with some changes in the environment that are not inconsistent with the sensorimotor transitions (resets?), the network can learn the representation h of motor that project to the actual position of the agent in the environment (albeit in very contrived 2D toy tasks).