Learning Semantically Meaningful Representations Through Embodiment

1 INTRODUCTION . When the way supervised neural networks learn is compared to the way humans learn one can easily make out some major differences . Two of those differences are supervision and embodiment . Taking the example of object recognition from visual observations , a neural network will be presented with thousands of images of the object in question , each of them accompanied by a class label . A toddler in comparison will also collect many observations of the object of interest , however , will do so by interacting with the object , looking at it from different perspectives by moving the head or even moving the object ( Bambach et al. , 2018 ) . This law-governed change in observations conditioned on the movements of the toddler emphasises the importance of embodied cognition ( Engel et al. , 2013 ) . It will make it possible to recognize the object as a distinct entity , separate from its surroundings and learn a general concept of it . This allows it to robustly recognize the object again even when seen from new perspectives or under different lighting conditions ( Smith & Slone , 2017 ) . When the toddler is now told the name of the object , an almost instantaneous association between label and object can be made without the need of thousands of labeled examples ( Samuelson & Smith , 2005 ) . This therefore makes a very efficient strategy for learning stable representations of objects . Fully supervised neural networks have been shown to suffer from shortcomings that humans usually do not exhibit . Szegedy et al . ( 2013 ) showed how very small perturbations to an image , undetectable to the human eye , can drastically change the classification accuracy of a neural network . Even simply holding such adversarial examples in front of a camera ( Kurakin et al. , 2016 ) or specific natural images ( Hendrycks et al. , 2019 ) can have this effect . The networks seem to possess an over-reliance on local image features such as texture and do not consider global features such as the overall shape and outline of an object ( Baker et al. , 2018 ) . Considering the training circumstances , this effect is unsurprising . The networks are expected to learn the concept of objects solely from pixel values . Without being able to interact with objects or even just looking at them from slightly different perspectives , it is very difficult to figure out basic knowledge such as object and background relationships . We expect that an active exploration of the world would make it possible to learn a more general and robust concept of objects . Already in 2001 O ’ Regan & No ( 2001 ) argued that even though it is clear that action requires perception , this relation also reverses . Perception and the understanding of what is perceived requires action ( Noë , 2005 ) . According to O ’ Regan & Noë ( 2001 ) , ” experience is not something that happens in us but is something we do ” ( p.99 ) . They argue that an important part of perception is to learn how actions affect sensations . These sensory motor contingencies help us make sense of our perceptions , predict them and efficiently sample the environment for information ( Engel et al. , 2013 ) . In humans , perception is hugely influenced by how we interact with the world ( Witt , 2011 ) . Goals and the expected cost to perform actions to achieve a goal influence our perception of physical entities ( Proffitt , 2006 ) . Also more abstract processes such as language comprehension are linked to action systems in the brain ( Pulvermüller & Fadiga , 2010 ) . We therefore postulate that in order to teach an artificial agent a true understanding of its ( simulated ) world it needs to be able to interact with the world . This paper will present results from an embodied agent acting in a virtual 3D world and learning an internal representation of its sensory input . The framework of learning by interacting with the world produced a meaningful and action-oriented internal representation of the agents observations , even though no semantic labels were provided . 2 RELATED WORK . There is a strong research interest in learning visual structure in an unsupervised way which can for example be approached by using auto-encoders ( Tschannen et al. , 2018 ) . In its simplest form , relevant structure is supposed to change slowly ( Körding & König , 2001 ; Wiskott & Sejnowski , 2002 ) facilitating learning of invariant representations . To further incorporate a time component and learn visual structure and changes over time , future frame prediction is a commonly used task ( Villegas et al. , 2017 ; Srivastava et al. , 2015 ; Patraucean et al. , 2015 ; Oliu et al. , 2017 ; Mahjourian et al. , 2016 ; Finn et al. , 2016 ; Denton & Birodkar , 2017 ) . However , only a few of the papers dealing with time series prediction actually investigate the learned representations in the network ( Lotter et al. , 2015 ; Qiao et al. , 2018 ) . An alternative concept for unsupervised representation learning is the use of predictive coding ( Rao & Ballard , 1999 ) which can be applied to train ANNs ( van den Oord et al. , 2018 ; Wen et al. , 2018 ; Han et al. , 2018 ; Lotter et al. , 2016 ) . The idea of predicting future observations based on current actions can also be used in a reinforcement learning setting to inject agents with some sense of curiosity ( Pathak et al. , 2017 ) . Ha & Schmidhuber ( 2018 ) has shown that training a recurrent world model using a variational auto-encoder can increase the performance of agents in several games . Chaplot et al . ( 2019 ) show how jointly training semantic goal navigation and embodied question answering can improve performance on both of these tasks . Also , simply seeing a visual scene from different angles can get a network to learn disentangled representations of individual objects ( Eslami et al. , 2018 ) and get it to imagine the scene from a previously unseen viewpoint ( Eslami et al. , 2018 ; Rosenbaum et al. , 2018 ) . Researchers who investigate representation learning in reinforcement learning agents often use additional regularization or losses to enforce a certain representation in the latent space ( Nachum et al. , 2018 ; Lesort et al. , 2018 ; de Bruin et al. , 2018 ) . Shang et al . ( 2019 ) even gets agents to explicitly learn world graphs . As such explicit constraints are biologically implausible , we will investigate what kind of representations arise naturally within an embodied training setup . Lillicrap et al . ( 2015 ) have already shown results from a simple deep reinforcement learning agent which indicate that perceptually similar observations are mapped close to each other in the latent space . We will further investigate this and look explicitly at the type of representation encoding that is learned as well as the meaningfulness of the representation and the type of information that is encoded . 3 EXPERIMENTS . 3.1 TRAINING A DEEP REINFORCEMENT LEARNING AGENT . The representation under investigation in this paper is the activation in the hidden layer of a deep neural network trained in a reinforcement learning environment . Figure 1 shows the network structure of the agent . As input , it receives visual and vector observations from the simulated environment with size 168x168x3 and size 8 respectively . The visual and vector observations are first processed separately by two convolutional layers and two dense layers for the visual input and two dense layers for the vector input , until they are concatenated into one encoded state . This encoded state has dimensionality 512 where 256 of its activations come from the visual encoding pathway and 256 from the vector encoding pathway . The encoded state of the high dimensional visual input and its properties will be the main focus of this paper . Based on the encoded state , two dense layers output action probabilities and a value estimate . The action probabilities are translated into actions and sent to the environment to obtain the next set of observations . They are also used , together with the value estimate and the reward received from the environment to optimize the neural network and thereby , train the agent to perform better actions . For training of the network proximal policy optimization ( PPO ) is used which is an efficient policy gradient method for deep reinforcement learning ( Schulman et al. , 2017 ) . More specifically , the PPO implementation and reinforcement learning framework of the Unity ML-Agents toolkit is used ( Juliani et al. , 2018 ) . The overall setup and training procedure are chosen such that a neural network is trained in an embodied way with a closed loop between perception and action . As setting for training , the environment proposed in the Unity obstacle tower challenge ( Juliani et al. , 2019 ) is used . This environment was introduced as a new benchmark in reinforcement learning for pixel-based learning in a procedurally-generated 3D environment using a sparse reward signal . The agent needs to learn to navigate through a 3-dimensional maze environment , solving successively harder tasks . Every level of the tower consists of several rooms , connected by doors . When reaching the final door on a floor the agent receives a reward of one and is placed on the next , randomly generated floor of the tower . Starting from level five the agent needs to learn picking up a key which unlocks a key door and gives the agent access to further rooms that lead to the next level door . The key can be placed in one of the rooms on the ground or on a static or moving platform . Starting from level ten a new type of door is introduced which opens only after a puzzle is solved . The puzzle requires the agent to push a block onto a colored spot on the ground . The randomly generated floors can be illuminated in different color variations and the visual theme of the environment can vary . As overfitting to specific color values in the input or floor layouts is not useful , this makes it important for the agent to learn general and stable representations . The agent observations are collected from a third person view RGB camera . Additionally , a small vector observation is provided indicating the number of keys the agent is holding as well as the time remaining and the current level . Rewards are very sparse as a reward ( of value 1 ) is only received when walking through a final level door or when picking up a key . A small reward of value 0.1 is given when walking through normal doors and when picking up small blue orbs that provide additional time . One episode ends when the agent runs out of time , which means that the better the agent gets the longer exploring the tower is possible as more time-orbs can be collected and extra time received by going through level doors . The actions of the agent are discrete and divided into four action branches . One for moving forward or backwards , one to control the camera rotation , one for jumping and one for moving left or right . The distribution of these actions in a trained agent is show in the appendix ( figure 10 ) . 3.2 AGENT PERFORMANCE After training the agent for 30 million steps using the parameters specified in the appendix A.1 level 8 is reached on average . As can be expected from the network structure , the agent never exceeds level ten . The network structure used here incorporates no concept of time such that the agent is unable to solve the puzzles introduced at level 10 as they require some more elaborate planning of a long action sequence . Figure 2 shows the agent performance during one inference run . This particular run lasted 4000 frames which means the agent saw 4000 observations and performed 4000 actions . As the rewards are very sparse , the agent only received a reward ( 1 or 0.1 ) in 76 of these frames which is about 1.9 % of all frames ( only 15 of those are the full reward of one , the other 61 frames contain a 0.1 reward ) . One can see how the value estimate , which expresses the reward the agent expects in the future , drops off significantly after the agent reaches level ten as the agent does not expect to solve the puzzle and receive any more rewards . One can also observe how the value estimate rises in the frames leading up to the agent entering a new level . This indicates that the agent recognizes the door to the next level and already anticipates the upcoming reward . The results of the network trained in the embodied setting will be compared to an untrained network ( called random network ) as well as the results from an autoencoder . The autoencoder has the exact same structure up to the encoding layer as the trained and random network ( see Appendix , figure 8 ) . The autoencoder is trained on a classic frame reconstruction task where its loss is the difference between the network input and output . This training setup separates the factor of embodiment in the training without introducing semantic labels . In figure 9 some example input , outputs and activations in the encoding layer are shown . When comparing the hidden layer activations of the three networks the same 4000 observations of one agent run are used . 3.3 SPARSE ACTIVATION PATTERNS We will take a look at how the activations in the hidden layer of the agent network look like1 . Figure 3 shows in how many of the 4000 frames of one run each of the 256 neurons in the visual encoding were active . For visualization purposes , the 256x1 vector was reshaped into a grid pattern . The location of a neuron in this grid has no further meaning . In the visual encoding only 4.64 % ( mean= 11.88 , min=3 max=35 var=17.891 ) of the neurons are activated in each frame , making it a very sparse activation pattern . Over the investigated total time , 173 of the 256 neurons ( =67.58 % ) are active in at least 1 frame , but only 7 of them ( =2.73 % ) are active in more than 40 % of the frames . The most active neuron of the visual embedding is active in 74.15 % of all frames . This shows that there is a wide variety in the activation patterns . The agent utilizes most but not all of the available neurons in the 1You can find a frame by frame visualization of the activations as well as other interactive displays of our results and our code here : https : //embodiedlearning.github.io/ICLR-Submission-2020/ visual embedding at some point but only activates a small part in each frame . In the random network , there are on average 130 active neurons per frame in the visual embedding ( min=112 max=152 var=27.352 ) , which is a much denser activity ( 50.62 % ) . The trained autoencoder has an average of 251 active neurons in each frame ( min=227 max=254 var=15.994 ) and only two of the neurons are never active . Thus , in comparison the trained agent has learned a very sparse representation of the input using selective spikes in activation compared to the continuous activations in the autoencoder encoding information with varying activation strengths . The sparse representations in the trained agent match observations of sparse encodings for sensory input in insects ( Perez-Orive et al. , 2002 ; Laurent , 2002 ) as well as in the mammalian brain ( Young & Yamane , 1992 ; Brecht & Sakmann , 2002 ) . These form efficient and stable representations of high dimensional sensory input ( Olshausen & Field , 2004 ) . The agent picks up on this strategy to efficiently encode input without any explicit regularization being applied . Even though there is no cost associated to using more neurons than needed to encode information , the agent learns to use sparse activation patterns and even leaves some of the available neurons completely unused . The trained agent seems to discover on its own that a sparse representation of the high dimensional image input is more robust and stable to noise ( Ahmad & Scheinkman , 2019 ) which helps to make better action decisions based on this sparse and selective representation . The training setup of the autoencoder enforces a different , more dense way of representating information which is of advantage to optimize its objective but has less resemblance with what we observe in nature .

Summary: This is an empirical study of the representations learned by a reinforcement learning agent. An agent is trained, using a standard RL algorithm, to solve puzzles by navigating through a 3D visual environment (Unity obstacle tower challenge). The analyses in the paper show that the visual representations of the trained agents are sparse and cluster according to the actions performed by the agent. The goal of the paper is to show that these features are due to the embodied nature of the agent. Specifically, the paper states that “the quality of the representations learned shows the strength of embodied learning and its advantages over fully supervised approaches with regards to robustness and generalizability”.