Divergent representations of ethological visual inputs emerge from supervised, unsupervised, and reinforcement learning

Artificial neural systems trained using reinforcement , supervised , and unsupervised learning all acquire internal representations of high dimensional input . To what extent these representations depend on the different learning objectives is largely unknown . Here we compare the representations learned by eight different convolutional neural networks , each with identical ResNet architectures and trained on the same family of egocentric images , but embedded within different learning systems . Specifically , the representations are trained to guide action in a compound reinforcement learning task ; to predict one or a combination of three task-related targets with supervision ; or using one of three different unsupervised objectives . Using representational similarity analysis , we find that the network trained with reinforcement learning differs most from the other networks . Through further analysis using metrics inspired by the neuroscience literature , we find that the model trained with reinforcement learning has a high-dimensional representation wherein individual images are represented with very different patterns of neural activity . These representations seem to arise in order to guide long-term behavior and goal-seeking in the RL agent . Our results provide insights into how the properties of neural representations are influenced by objective functions and can inform transfer learning approaches . 1 INTRODUCTION . Many studies in machine learning aim to improve model quality or data efficiency by training the model with multiple objectives or transferring representations trained on other data . While leveraging other data or objectives may lead to higher data efficiency or increased generalization outside the primary task , relying too heavily on tasks other than the primary one can undercut performance if the other tasks are insufficiently well aligned . Humans and animals do in fact confront the full complexity of this continual learning problem of building representations , reusing what is worth transferring to new settings as they learn new things . The potential utility of transferring representations is particularly salient in deep reinforcement learning , where rich sensory inputs must be transformed into actions . While landmark developments in deep RL have enabled the end-to-end training of agents capable of leveraging pixel-level visual information in complex environments to guide behavior ( Mnih et al. , 2015 ) , there is a strong demand for methods that improve the sample efficiency of these training-intensive agents through pre- or joint-training of the representations they rely on . Indeed several studies aim to accelerate representation learning in vision-based RL through the use of additional objectives ( e.g. , Jaderberg et al. , 2017 ; Wayne et al. , 2018 ; Schwarzer et al. , 2021 ) . In the present work , we examine a specific instance of a high-dimensional embodied control problem , the virtual rodent of Merel et al . ( 2020 ) , which is also included in the RL Unplugged datasets for offline learning ( Gulcehre et al. , 2020 ) . This problem involves visuomotor control of a highdimensional body to solve multiple tasks , and it is solvable using a visuomotor policy trained through deep RL . In particular , we will compare multi-layer visual representations that arise in this model to those resulting from different training objectives on the same architecture . Transfer learning approaches tend to be evaluated empirically , demonstrating through performance comparisons circumstances under which pretrained or transferred representations add value . While useful and pragmatic , this approach yields limited general insights into new problems . Instead of focusing on performance comparisons , we will focus on analysis of the representations at multiple layers of the same architecture trained with unsupervised , supervised , and reinforcement learning methods . To perform this analysis , we exploit tools that have been commonly applied in the interpretation of biological neural data . Our hope is to develop intuition for the properties of these different representations in order to support future efforts to develop training procedures . 1.1 RELATED WORK . Transfer learning in the supervised context has been widely applied , studied and reviewed , including within the narrower context of deep learning ( Bengio , 2012 ) . Thus , we limit the scope of our discussion to connections to the most relevant literature on transfer learning in the context of reinforcement learning , as well as literature focusing on the analysis of representations learned . Transfer learning for reinforcement learning Within RL specifically , transfer learning has been of interest since before the recent deep RL boom ( Taylor & Stone , 2009 ; Lazaric , 2012 ) . There have been diverse attempts to learn representations from experience that would support generalization across other tasks , including a focus on predicting future states or rewards ( Rafols et al. , 2005 ; Lehnert et al. , 2020 ) . Relevant to our present setting , Hill et al . ( 2019 ) found specifically that generalization is facilitated by egocentric observations . In addition , transfer learning for reinforcement learning has been one way to operationalize the broader and more natural problem of continual learning , wherein representations must be learned , transferred , reused , and adapted repeatedly over the lifetime of an agent ( Hadsell et al. , 2020 ) . Among more recent deep RL approaches , it is also possible to leverage multiple objectives for learning representations concurrently . That is , rather than first learning a representation from previously logged data and then learning a policy from that pretrained representation , it is possible to perform deep RL with auxiliary objectives ( Jaderberg et al. , 2017 ) including self-supervised tasks such as predicting past or future experience ( Wayne et al. , 2018 ; Schwarzer et al. , 2021 ) . Critically , concurrent learning with unsupervised or self-supervised objectives as well as RL offers the advantage that as RL proceeds and the data distribution changes , the auxiliary tasks continue to train on the shifting ( and increasingly relevant ) data . As constrastive approaches have become popular , they have also been explored as auxiliary losses , such that policies rely on features shaped both by the contrastive and RL objectives ( Oord et al. , 2018 ; Laskin et al. , 2020 ) . While intuitively it seems reasonable that joint training of a representation with RL and an additional loss may help performance , new results by Stooke et al . ( 2021 ) keep alive the possibility that exclusively pre-training a representation from unsupervised objectives could outperform end-to-end RL . Nevertheless , it presently remains far from resolved which objectives used for pretraining or as auxiliary tasks ( i.e . concurrent with RL ) will work best . Indeed , in one of the larger empirical studies to date , Yang & Nachum ( 2021 ) sweep over many pre-training objectives . Their results “ suggest that the ideal representation learning objective may depend on the nature of the downstream task , and no single objective appears to dominate generally ” ( with the caveat for our setting that their environments are relatively simple control environments and they focus on state rather than image observations ) . Analysis of neural representations In general , the present state of the literature on transfer of representations for RL leaves us with meaningful leads as to what objectives to try , but still incomplete insight into why various objectives add value or how to anticipate what will work well on a new problem . However , there is precedent both in the machine learning literature as well as biological neural analysis literature for attempting to analyze learned representations . Taskonomy ( Zamir et al. , 2018 ) determined transfer learning performance across the same architecture trained with several supervised and unsupervised tasks . While this work did not compare representations directly , one relevant finding was that autoencoders tended to be an outlier in the revealed task structures . Furthermore , the Taskonomy networks were also used to predict fMRI activity ( Wang et al. , 2019 ) , and it was found that several tasks related to 3D image processing made very similar predictions , while the autoencoder was not highly similar to any other models . In Zhuang et al . ( 2021 ) , several different unsupervised networks and one supervised network are assessed as models of primate visual processing . This work found that contrastive unsupervised methods generally had good transfer performance and predicted primate neural activity well . Note that neural predictivity is only an indirect way of comparing components of network representation ; we perform direct comparisons across different networks , including an RL-trained model ( which is to our knowledge the first time such a full comparison has been done ) . Neural analysis metrics We use representation similarity analysis ( RSA ) to compare representations across networks . RSA has been used extensively in neural network analysis and has connections to many other metrics used in neuroscience ( Kriegeskorte & Wei , 2021 ) . Historically , the concept of sparsity has also been important in neural analysis . Sparsity can be defined several different ways ( Willmore & Tolhurst , 2001 ) but in general sparse representations have been associated with efficient encoding of natural stimuli statistics , effective associative memory , and enhanced downstream classification ( Rolls & Treves , 1990 ; Olshausen & Field , 1996 ; Babadi & Sompolinsky , 2014 ; Olshausen & Field , 2004 ) . Another commonly-measured property of neural representations is dimensionality , which is used as a way to probe the number of latent variables encoded in a neural population as well as understand how they are embedded ( Jazayeri & Ostojic , 2021 ) . Higher dimensionality is also associated with better downstream classification ( Rigotti et al. , 2013 ) . 2 METHODS . 2.1 MODELS AND TRAINING . Table 1 summarizes the models trained for this study ( for all but RLRod , we train three instantiations of each ) . For more details on these networks and training see Appendix A . The ResNet architecture shared by all models is shown in Figure 1A . The number of feature channels is 16 for the first meta-layer and 32 thereafter . The final layer is a 128-unit fully connected layer . The images used are all 64x64 RGB egocentric images generated from the virtual rodent environments Merel et al . ( 2020 ) built in MuJoCo ( Todorov et al. , 2012 ) using dm control ( Tunyasuvunakool et al. , 2020 ) . For the RL model , the agent creates its own training images through exploration in the environment ( see Appendix A.0.1 ) . Other models were trained using images generated by the agent ’ s exploration – these images , along with some other features of the rodent ’ s state and action output , have been made publicly available and documented for offline RL in Gulcehre et al . ( 2020 ) . Example images can be seen in Figure 1B . The rodent engages in one of four possible tasks at a time : bowl escape ( where the rodent must crawl out of an uneven valley ) , gaps run ( the rodent must run and jump over gaps ) , two-tap task ( the rodent must tap an orb twice with a set amount of time in between ) , and maze forage ( the rodent has to find orbs in a maze structure ) . 2.2 ANALYSIS METHODS . Activity was recorded from four different layers ( marked as R1-4 in Figure 1A ) in response to 2048 test images drawn equally from the four tasks . Here we briefly describe the analyses applied , with more detailed descriptions available in Appendix A. Representational Similarity Analysis RSA was used to determine the extent to which different models represent visual inputs similarly . First , dissimilarity matrices were made for each layer in each network ( see Eqn 2 ; example dissimilarity matrices in Figure 4A ) . RSA matrices ( resulting from two different correlation metrics ) indicate how similar these dissimilarity matrices are across networks . Separately we also performed RSA across layers within networks and included pixel and action spaces . Sparsity To measure the sparsity of representations in these networks we use a lifetime sparsity metric which determines the extent to which a neuron responds selectively to different images ( Vinje & Gallant , 2000 ) . s = 1− ( 1n ) ( ( ∑ ri ) 2∑ r2i ) 1− 1n ( 1 ) where n is the number of images and ri is the response of the neuron to image i . A sparsity value of 1 indicates very selective responses , with 0 indicating equal responses to all inputs . Dimensionality We perform PCA on population activity and then estimate embedding dimensionality using both the participation ratio ( see Eqn 3 ) and the number of PCs needed to reach 85 % variance explained .

Summary: The paper explores how similar the representations learned by supervised, unsupervised, and RL techniques are in an egocentric virtual rodent environment. The main claim is that representations in RL-trained networks are most different from supervised and unsupervised networks, which is tested with an RSA metric on 4 layers of the networks. This is explored further by analyzing the sparsity, dimensionality (RL has much higher dimensionality) and action prediction (proprioception alone without visual features is sufficient here).