Divergent representations of ethological visual inputs emerge from supervised, unsupervised, and reinforcement learning

Artificial neural systems trained using reinforcement , supervised , and unsupervised learning all acquire internal representations of high dimensional input . To what extent these representations depend on the different learning objectives is largely unknown . Here we compare the representations learned by eight different convolutional neural networks , each with identical ResNet architectures and trained on the same family of egocentric images , but embedded within different learning systems . Specifically , the representations are trained to guide action in a compound reinforcement learning task ; to predict one or a combination of three task-related targets with supervision ; or using one of three different unsupervised objectives . Using representational similarity analysis , we find that the network trained with reinforcement learning differs most from the other networks . Through further analysis using metrics inspired by the neuroscience literature , we find that the model trained with reinforcement learning has a high-dimensional representation wherein individual images are represented with very different patterns of neural activity . These representations seem to arise in order to guide long-term behavior and goal-seeking in the RL agent . Our results provide insights into how the properties of neural representations are influenced by objective functions and can inform transfer learning approaches . 1 INTRODUCTION . Many studies in machine learning aim to improve model quality or data efficiency by training the model with multiple objectives or transferring representations trained on other data . While leveraging other data or objectives may lead to higher data efficiency or increased generalization outside the primary task , relying too heavily on tasks other than the primary one can undercut performance if the other tasks are insufficiently well aligned . Humans and animals do in fact confront the full complexity of this continual learning problem of building representations , reusing what is worth transferring to new settings as they learn new things . The potential utility of transferring representations is particularly salient in deep reinforcement learning , where rich sensory inputs must be transformed into actions . While landmark developments in deep RL have enabled the end-to-end training of agents capable of leveraging pixel-level visual information in complex environments to guide behavior ( Mnih et al. , 2015 ) , there is a strong demand for methods that improve the sample efficiency of these training-intensive agents through pre- or joint-training of the representations they rely on . Indeed several studies aim to accelerate representation learning in vision-based RL through the use of additional objectives ( e.g. , Jaderberg et al. , 2017 ; Wayne et al. , 2018 ; Schwarzer et al. , 2021 ) . In the present work , we examine a specific instance of a high-dimensional embodied control problem , the virtual rodent of Merel et al . ( 2020 ) , which is also included in the RL Unplugged datasets for offline learning ( Gulcehre et al. , 2020 ) . This problem involves visuomotor control of a highdimensional body to solve multiple tasks , and it is solvable using a visuomotor policy trained through deep RL . In particular , we will compare multi-layer visual representations that arise in this model to those resulting from different training objectives on the same architecture . Transfer learning approaches tend to be evaluated empirically , demonstrating through performance comparisons circumstances under which pretrained or transferred representations add value . While useful and pragmatic , this approach yields limited general insights into new problems . Instead of focusing on performance comparisons , we will focus on analysis of the representations at multiple layers of the same architecture trained with unsupervised , supervised , and reinforcement learning methods . To perform this analysis , we exploit tools that have been commonly applied in the interpretation of biological neural data . Our hope is to develop intuition for the properties of these different representations in order to support future efforts to develop training procedures . 1.1 RELATED WORK . Transfer learning in the supervised context has been widely applied , studied and reviewed , including within the narrower context of deep learning ( Bengio , 2012 ) . Thus , we limit the scope of our discussion to connections to the most relevant literature on transfer learning in the context of reinforcement learning , as well as literature focusing on the analysis of representations learned . Transfer learning for reinforcement learning Within RL specifically , transfer learning has been of interest since before the recent deep RL boom ( Taylor & Stone , 2009 ; Lazaric , 2012 ) . There have been diverse attempts to learn representations from experience that would support generalization across other tasks , including a focus on predicting future states or rewards ( Rafols et al. , 2005 ; Lehnert et al. , 2020 ) . Relevant to our present setting , Hill et al . ( 2019 ) found specifically that generalization is facilitated by egocentric observations . In addition , transfer learning for reinforcement learning has been one way to operationalize the broader and more natural problem of continual learning , wherein representations must be learned , transferred , reused , and adapted repeatedly over the lifetime of an agent ( Hadsell et al. , 2020 ) . Among more recent deep RL approaches , it is also possible to leverage multiple objectives for learning representations concurrently . That is , rather than first learning a representation from previously logged data and then learning a policy from that pretrained representation , it is possible to perform deep RL with auxiliary objectives ( Jaderberg et al. , 2017 ) including self-supervised tasks such as predicting past or future experience ( Wayne et al. , 2018 ; Schwarzer et al. , 2021 ) . Critically , concurrent learning with unsupervised or self-supervised objectives as well as RL offers the advantage that as RL proceeds and the data distribution changes , the auxiliary tasks continue to train on the shifting ( and increasingly relevant ) data . As constrastive approaches have become popular , they have also been explored as auxiliary losses , such that policies rely on features shaped both by the contrastive and RL objectives ( Oord et al. , 2018 ; Laskin et al. , 2020 ) . While intuitively it seems reasonable that joint training of a representation with RL and an additional loss may help performance , new results by Stooke et al . ( 2021 ) keep alive the possibility that exclusively pre-training a representation from unsupervised objectives could outperform end-to-end RL . Nevertheless , it presently remains far from resolved which objectives used for pretraining or as auxiliary tasks ( i.e . concurrent with RL ) will work best . Indeed , in one of the larger empirical studies to date , Yang & Nachum ( 2021 ) sweep over many pre-training objectives . Their results “ suggest that the ideal representation learning objective may depend on the nature of the downstream task , and no single objective appears to dominate generally ” ( with the caveat for our setting that their environments are relatively simple control environments and they focus on state rather than image observations ) . Analysis of neural representations In general , the present state of the literature on transfer of representations for RL leaves us with meaningful leads as to what objectives to try , but still incomplete insight into why various objectives add value or how to anticipate what will work well on a new problem . However , there is precedent both in the machine learning literature as well as biological neural analysis literature for attempting to analyze learned representations . Taskonomy ( Zamir et al. , 2018 ) determined transfer learning performance across the same architecture trained with several supervised and unsupervised tasks . While this work did not compare representations directly , one relevant finding was that autoencoders tended to be an outlier in the revealed task structures . Furthermore , the Taskonomy networks were also used to predict fMRI activity ( Wang et al. , 2019 ) , and it was found that several tasks related to 3D image processing made very similar predictions , while the autoencoder was not highly similar to any other models . In Zhuang et al . ( 2021 ) , several different unsupervised networks and one supervised network are assessed as models of primate visual processing . This work found that contrastive unsupervised methods generally had good transfer performance and predicted primate neural activity well . Note that neural predictivity is only an indirect way of comparing components of network representation ; we perform direct comparisons across different networks , including an RL-trained model ( which is to our knowledge the first time such a full comparison has been done ) . Neural analysis metrics We use representation similarity analysis ( RSA ) to compare representations across networks . RSA has been used extensively in neural network analysis and has connections to many other metrics used in neuroscience ( Kriegeskorte & Wei , 2021 ) . Historically , the concept of sparsity has also been important in neural analysis . Sparsity can be defined several different ways ( Willmore & Tolhurst , 2001 ) but in general sparse representations have been associated with efficient encoding of natural stimuli statistics , effective associative memory , and enhanced downstream classification ( Rolls & Treves , 1990 ; Olshausen & Field , 1996 ; Babadi & Sompolinsky , 2014 ; Olshausen & Field , 2004 ) . Another commonly-measured property of neural representations is dimensionality , which is used as a way to probe the number of latent variables encoded in a neural population as well as understand how they are embedded ( Jazayeri & Ostojic , 2021 ) . Higher dimensionality is also associated with better downstream classification ( Rigotti et al. , 2013 ) . 2 METHODS . 2.1 MODELS AND TRAINING . Table 1 summarizes the models trained for this study ( for all but RLRod , we train three instantiations of each ) . For more details on these networks and training see Appendix A . The ResNet architecture shared by all models is shown in Figure 1A . The number of feature channels is 16 for the first meta-layer and 32 thereafter . The final layer is a 128-unit fully connected layer . The images used are all 64x64 RGB egocentric images generated from the virtual rodent environments Merel et al . ( 2020 ) built in MuJoCo ( Todorov et al. , 2012 ) using dm control ( Tunyasuvunakool et al. , 2020 ) . For the RL model , the agent creates its own training images through exploration in the environment ( see Appendix A.0.1 ) . Other models were trained using images generated by the agent ’ s exploration – these images , along with some other features of the rodent ’ s state and action output , have been made publicly available and documented for offline RL in Gulcehre et al . ( 2020 ) . Example images can be seen in Figure 1B . The rodent engages in one of four possible tasks at a time : bowl escape ( where the rodent must crawl out of an uneven valley ) , gaps run ( the rodent must run and jump over gaps ) , two-tap task ( the rodent must tap an orb twice with a set amount of time in between ) , and maze forage ( the rodent has to find orbs in a maze structure ) . 2.2 ANALYSIS METHODS . Activity was recorded from four different layers ( marked as R1-4 in Figure 1A ) in response to 2048 test images drawn equally from the four tasks . Here we briefly describe the analyses applied , with more detailed descriptions available in Appendix A. Representational Similarity Analysis RSA was used to determine the extent to which different models represent visual inputs similarly . First , dissimilarity matrices were made for each layer in each network ( see Eqn 2 ; example dissimilarity matrices in Figure 4A ) . RSA matrices ( resulting from two different correlation metrics ) indicate how similar these dissimilarity matrices are across networks . Separately we also performed RSA across layers within networks and included pixel and action spaces . Sparsity To measure the sparsity of representations in these networks we use a lifetime sparsity metric which determines the extent to which a neuron responds selectively to different images ( Vinje & Gallant , 2000 ) . s = 1− ( 1n ) ( ( ∑ ri ) 2∑ r2i ) 1− 1n ( 1 ) where n is the number of images and ri is the response of the neuron to image i . A sparsity value of 1 indicates very selective responses , with 0 indicating equal responses to all inputs . Dimensionality We perform PCA on population activity and then estimate embedding dimensionality using both the participation ratio ( see Eqn 3 ) and the number of PCs needed to reach 85 % variance explained .

Summary: This work identifies the important problem of transfer learning for problems that require deep learning due to high-dimensional inputs. It notes that transfer learning approaches often cannot transfer very far across learning objectives and are judged empirically as black boxes, making for brittle approaches. It proposes to study internal representations to assess potential for transfer.   This study trains several supervised and unsupervised, and one reinforcement, tasks with the same data and ResNet architecture. It uses representational similarity analysis (inter-task similarity similarity of intra-task dissimilarity), sparsity analysis (neuroscience-sourced measure of response similarity between neurons), and dimensionality analysis (PCA) to assess the representations; these metric choices are taken from neuroscience literature.   RSA shows that representations across different tasks diverge with layer depth (except in a few cases, though the paper doesn't call these out as clearly); the same is seen within different instantiations of the same task in all but three cases. RLRod differs the most. Sparseness analysis shows that RLRod has the most internal dissimilarity in responses to various inputs, and dimensionality analysis shows that it is also the highest dimensional. The paper speculates that this might be due to a one-image-per-neuron representation emerging even in early layers. To test whether the high dimensionality in RLRod is purely due to a higher-dimensional output (38-dim action), the paper trains encoders to predict action and finds that the visual inputs are pretty useless and proprioceptions are somewhat helpful, indicating the value of proprioceptive data.   The discussion lists some main claims: representations become less similar at later layers, unsupervised objectives can capture some but not all of the representations of RL, the RL-trained visual encoder seems to drive long-term behavioral planning; then suggests the potential for more study to determine promising transfers. It also postulates value for the neuroscience community, since sparse representations are prevalent in biological intelligence and the similarity may be indicative of similar objectives in artificial and biological intelligence.