Divergent representations of ethological visual inputs emerge from supervised, unsupervised, and reinforcement learning
Authors: Grace W Lindsay
Josh Merel
Thomas D. Mrsic-Flogel
Maneesh Sahani
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
transfer learning
representations
dimensionality
sparsity
RSA

Abstract: Artificial neural systems trained using reinforcement, supervised, and unsupervised learning all acquire internal representations of high dimensional input. To what extent these representations depend on the different learning objectives is largely unknown. Here we compare the representations learned by eight different convolutional neural networks, each with identical ResNet architectures and trained on the same family of egocentric images, but embedded within different learning systems. Specifically, the representations are trained to guide action in a compound reinforcement learning task; to predict one or a combination of three task-related targets with supervision; or using one of three different unsupervised objectives. Using representational similarity analysis, we find that the network trained with reinforcement learning differs most from the other networks. Through further analysis using metrics inspired by the neuroscience literature, we find that the model trained with reinforcement learning has a high-dimensional representation wherein individual images are represented with very different patterns of neural activity. These representations seem to arise in order to guide long-term behavior and goal-seeking in the RL agent. Our results provide insights into how the properties of neural representations are influenced by objective functions and can inform transfer learning approaches.

Under review as a conference paper at ICLR 2022
DIVERGENT REPRESENTATIONS OF ETHOLOGICAL VI- SUAL INPUTS EMERGE FROM SUPERVISED, UNSUPER- VISED, AND REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Artificial neural systems trained using reinforcement, supervised, and unsuper- vised learning all acquire internal representations of high dimensional input. To what extent these representations depend on the different learning objectives is largely unknown. Here we compare the representations learned by eight differ- ent convolutional neural networks, each with identical ResNet architectures and trained on the same family of egocentrie images, but embedded within different learning systems. Specifically, the representations are trained to guide action in a compound reinforcement learning task; to predict one or a combination of three task-related targets with supervision; or using one of three different unsupervised objectives. Using representational similarity analysis, we find that the network trained with reinforcement learning differs most from the other networks. Through further analysis using metrics inspired by the neuroscience literature, we find that the model trained with reinforcement learning has high-dimensional represen- tation wherein individual images are represented with very different patterns neural activity. These representations seem arise in order guide long-term behavior and goal-seeking in the RL agent. Our results provide insights into how the properties of neural representations are influenced by objective functions and can inform transfer learning approaches.
INTRODUCTION
Many studies in machine learning aim to improve model quality or data efficiency by training the model with multiple objectives transferring representations trained on other data. While leverag- ing other data or objectives may lead to higher data efficiency or increased generalization outside the primary task, relying too heavily tasks other than the primary can undercut performance the other tasks are insufficiently well aligned. Humans and animals do in fact confront the full complexity of this continual learning problem of building representations, reusing what is worth transferring to new settings as they learn new things.
The potential utility of transferring representations is particularly salient in deep reinforcement learning, where rich sensory inputs must be transformed into actions. While landmark develop- ments deep RL have enabled the end-to-end training of agents capable of leveraging pixel-level visual information in complex environments to guide behavior (Mnih et al. 2015),, there is strong demand for methods that improve the sample efficiency of these pre- or joint-training of the representations they rely on. Indeed several studies aim to accelerate representation learning in vision-based RI. through the use of additional objectives (e.g., Jaderberg et al. 2017 Wayne et al. 2018 Schwarzer et al. 202I).
agents through
In the present work, we examine specific instance of a high-dimensional embodied control prob- lem, the virtual rodent of Merel al. (2020), which is also included in the RL Unplugged datasets for offline learning (Gulcehre et al.) 2020). This problem involves visuomotor control of a high- dimensional body to solve multiple tasks, and it is solvable using visuomotor policy trained through deep RL. In particular, we will compare multi-layer visual representations that arise in this model to those resulting from different training objectives on the same architecture.

---

Under review as a conference paper at ICLR 2022
Transfer learning approaches tend to be evaluated empirically, demonstrating through performance comparisons circumstances under which pretrained or transferred representations add value. While useful and pragmatic, this approach yields limited general insights into new problems. Instead of focusing on performance comparisons, we will focus on analysis of the representations at multiple layers of the same architecture trained with unsupervised, supervised, and reinforcement learning methods. To perform this analysis, we exploit tools that have been commonly applied in the in- terpretation of biological neural data. Our hope to develop intuition for the properties of these different representations in order to support future efforts to develop training procedures.
1.1 RELATED WORK
Transfer learning in the supervised context has been widely applied, studied and reviewed, including within the narrower context of deep learning (Bengio 2012). Thus, we limit the scope of our discus- sion to connections to the most relevant literature on transfer learning in the context of reinforcement learning, as well as literature focusing on the analysis of representations learned.
Transfer learning for reinforcement learning Within RL specifically, transfer learning has been of interest since before the recent deep RL. boom (Taylor & Stone 2009 Lazaric 2012). There have been diverse attempts to learn representations from experience that would support generaliza- tion across other tasks, including focus on predicting future states or rewards (Rafols et al. 2005
Lehnert al. 2020). Relevant our present setting, Hill al. (019) found specifically that gen- eralization is facilitated by egocentric observations. In addition. transfer learning for reinforcement learning has been one way to operationalize the broader and more natural problem of continual learning, wherein representations must be learned, transferred, reused, and adapted repeatedly over the lifetime of an agent (Hadsell et al., 2020).
Among more recent deep RL approaches, it is also possible to leverage multiple objectives for learn- ing representations concurrently. That is, rather than first learning a representation from previously logged data and then learning policy from that pretrained representation, it is possible to perform deep RL with auxiliary objectives (Jaderberg et al., (017) including self self-supervised tasks such as predicting past future experience (Wayne et al. 2018 Schwarzer et al. 2021). Critically, concur- rent learning with unsupervised or self-supervised objectives as well as RL offers the advantage that as RL proceeds and the data distribution changes, the auxiliary tasks continue train the shifting (and increasingly relevant) data.
As constrastive approaches have become popular, they have also been explored auxiliary losses, such that policies rely on features shaped both by the contrastive and RL objectives (Oord et al.] 2018 Loskin et al. 20200). While intuitively it seems reasonable joint training of a representation with RL and an additional loss may help performance, new results by (Stokke et al. (202I) keep alive the possibility that exclusively pre-training a representation from unsupervised objectives could outperform end-to-end RL. Nevertheless, it presently remains far from resolved which objectives used for pretraining or as auxiliary tasks (i.e. concurrent with RL) will work best. Indeed, in one of the larger empirical studies to date, Yang & Nachum (2021) sweep over many pre-training objectives. Their results "suggest that the ideal representation learning objective may depend on the nature of the downstream task, and no single objective appears to dominate generally" (with the caveat for our setting that their environments are relatively simple control environments and they focus on state rather than image observations).
Analysis of neural representations In general, the present state of the literature on transfer of representations for RL leaves us with meaningful leads as to what objectives try. but still incom- plete insight into why various objectives add value or how to anticipate what will work well on new problem. However, there is precedent both in the machine learning literature as well as biological neural analysis literature for attempting to analyze learned representations.
Taskonomy (Zamir et al., 2018) determined transfer learning performance across the same archi- tecture trained with several supervised and unsupervised tasks. While this work did not compare representations directly, one relevant finding was that autoencoders tended to be an outlier in the revealed task structures. Furthermore, the Taskonomy networks were also used to predict fMRI activity (Wang et al., 2019), and it was found that several tasks related to 3D image processing made very similar predictions, while the autoencoder was not highly similar to any other models.
2

---

Under review as a conference paper at ICLR 2022
In Zhuang al. (2021), several different unsupervised networks and one supervised network assessed as models of primate visual processing. This work found that contrastive unsupervised methods generally had good transfer performance and predicted primate neural activity well. Note that neural predictivity is only an indirect way of comparing components of network representation; perform direct comparisons across different networks, including RL-trained model (which is to our knowledge the first time such a full comparison has been done).
Neural analysis metrics We use representation similarity analysis (RSA) to compare represen- tations across networks. RSA has been used extensively in neural network analysis and has con- nections to many other metrics used neuroscience (Kriegeskorte & Weil 2021). Historically, the concept of sparsity has also been important in neural analysis. Sparsity be defined several different ways (Willmore & Tolhurst 2001) but in general sparse representations have been associ- ated with efficient encoding of natural stimuli statistics, effective associative memory, and enhanced downstream classification (Rolls & Treves 1990 Olshausen & Field 1996. [Babadi & Sompolinsky, 2014 Olshausen & Field 2004). Another commonly meaaurred property of neural representations dimensionality, which is used as way to probe the number of latent variables encoded in a neu- ral population as well as understand how they are embedded (Jazayeri Ostojic (02T). Higher dimensionality is also associated with better downstream classification (Rigotti et al. 2013).
METHODS
2.1 MODELS AND TRAINING
the models trained for this study (for all but RLRod, we train three instantiations of each). For more details on these networks and training see Appendix [A] The ResNet architecture shared by all models is shown in Figure MA. The number of feature channels is 16 for the first meta-layer and 32 thereafter. The final layer is a 128-unit fully connected layer.
The images used are all 64x64 RGB egocentric images generated from the virtual rodent environ- ments Merel et al. (2020) built in MuJoCo (Todorov et al. 2012)) using cim control (Tunyasuvu- nakool et al. 2020)) For the RL model, the agent creates its own training images through exploration in the environment (see Appendix J.. Other models were trained using images generated by the agent's exploration these images, along with some other features of the rodent's state and action output, have been made publicly available and documented for offline RL in Gulcehre et al. (2020).
The rodent engages in one of four possible tasks time: bowl escape (where the rodent must crawl out of an uneven valley). gaps run (the rodent must run and jump over gaps), two-tap task (the rodent must tap an orb twice with set amount of time in between), and maze forage (the rodent has to find orbs in a maze structure).
Example images be in Figure B..
A.
R2
R4
Corry
Res
Relu
Relu
Bowl escape
Gaps
Twu lsp
Maze forage
Figure 1: The network architecture (A) and images used for training (B), which were drawn equally from the four different virtual rodent tasks. The layers from which activity was recorded for analysis are labeled A. For RI, 2 and the Relu layer after the Pooling was recorded.
3

---

Under review as a conference paper at ICLR 2022
Thiining style
Name (Abrv.)
Description
Outpor Dimension
Supervised
Reward model (SupRwd)
Predict reward received after this frame
Supervised
Tack model (SupTak)
Classity which task the rodent is
Supervised
Orientation Mudel (SupOri)
Predict egucentric onentation in space
Supervised
(SupAll)
the ahove
Unsupervised
Autoencoder (UnsAut)
Reconstruct the image
6dx64x3
Unsupervised
Variational Autocnooder (Uns
Reconstruct the image
64x64x3
Unsupervised
Prodictive Coding Model (UnsCpo)
Classity pairs images
Untrained
Kandom (Rand.)
Kandom weight initialization
Reinforcement
Rodent Model (RLRod)
Achieve high reward on embedied tasks
38
Table I: Description of models compared. For details, see Appendix
A.
1.00
1.0
Clessifiation Accurac)
1.00
Binrry Accuracy
0.99
0.9
0.8
0.90
0.007
0.7
0.006
0.85
Sup7sk
SupRed
SupOri
UnsAut
UnsVar
UnsCPC
Autoencoder
Variational Autoencoder
Onlg
Recon ORRO
Recon
Figure 2: Performance of trained networks: supervised models in A, left (classification accuracy for the task model; 72 values for the reward and orientation models) and unsupervised models in A, right (mean squared error for image reconstructions for the standard and variational autoencoders are shown; binary accuracy for the contrastive predictive coding model.) Each point represents one of three random instantiations. Chance for task classification is .25 and for CPC is 55. For SupOri, x² is averaged over all three dimensions. Performance for the SupAll networks is shown in black for each task. Example reconstructions from the two types of autoencoders are shown in B.
2.2 ANALYSIS METHODS
Activity recorded from four different layers (marked R1-4 in Figure in response to 2048 test images drawn equally from the four tasks. Here we brietly describe the analyses applied, with more detailed descriptions available in Appendix
Representational Similarity Analysis RSA was used determine the extent to which different models represent visual inputs similarly. First, dissimilarity matrices were made for each layer in each network (see Eqn 2 example dissimilarity matrices in Figure 4A). RSA matrices (resulting from two different correlation metries) indicate how similar these dissimilarity matrices across networks. Separately we also performed RSA across layers within networks and included pixel and action spaces.
Sparsity
To measure the sparsity of representations in these networks we use lifetime sparsity metric which determines the extent to which neuron responds selectively to different images (Vinje Gallant 2000).
1 1-
(1)
E B
5
where n is the number of images and r; is 1 indicates very selective responses, with 0 indicating equal responses all inputs.
the response of the neuron to image i. A sparsity value of
4

---

Under review as a conference paper at ICLR 2022
We perform PCA population activity and then estimate embedding dimen- sionality using both the participation ratio (see Eqn and the number of PCs needed to reach 85% variance explained.
Dimensionality
RESULTS
3.1 MODEL PERFORMANCE
We show the test performance of our supervised and unsupervised models here (for more informa- tion on the performance of the RL agent see Merel et al. (2020).
All models reach an acceptable level performance, though some are clearly better than others, which may be relevant for how to interpret their different representations. For example, the varia- tional autoencoder performs much better than the standard autoencoder as can be seen both in the MSE measure and in the example reconstructions shown in Figure@p: The joint training of SupAll also seems to provide performance boost compared to the individually trained networks.
Orientation Prediction
22
0
Reward Prediction
12
Task Classification
Acc.
SupTtwd
SupiThe
SuppOi
SupAAl
UroCpp
Dartt
Figure 3: Transfer learning performance using final layer (R4) activity for supervised, unsupervised, random, and RL networks.
3.2 TRANSFER LEARNING PERFORMANCE
To understand what kind of information is present in the outputs of each trained network (layer R4 in Figure [T] and to compare to previous transfer learning work, we trained classifiers (linear or logistic regression) to readout the three variables used for supervised training: task, reward, and orientation. The test performance of these classifiers (across 20 test-train splits) shown in Figure
3
As expected, the supervised networks perform well on the tasks they were directly trained on. Many of the networks also perform well task classification, including the random untrained ones. Uns Var and UnsCpe perform fairly well on all three tasks as well. The RL model performs well on task classification and relatively well on orientation and reward prediction. These results align with previous claims about the generalizability of certain unsupervised representations and adds that RL representations may generalize well.
3.3 REPRESENTATIONAL SIMILARITY ANALYSES
The RSA matrices in Figure 4 compare the representations of the different networks at each layer individually.
First, we can see there little variability across different instantiations networks trained the same way (this can be seen the large 3x3 blocks on the diagonal). Another strong finding is that networks trained with different algorithms become less similar across the layers in the network. Specifically. in R4, many similarity values are near 0, whereas in R1 many are closer to (this can be seen most clearly in the summary figure provided in Figure 4B). This may a result the loss function having stronger influence later layers.
5

---

Under review as a conference paper at ICLR 2022
A.
Averaa 8m Lante
SAp
SucToc
Suon
Unchut
Unww
Iage
Iuge
Teag
Ing:2
SupRord
SupTsk
SupAl
UnsAut
Unsver
Rend
RL Rod
SupRvd
SupTsk
SupOri
SupAl
UnsAur
UnsCpe
Rand
RLRod
SupReed
SupOr
Supda
Unsitr
Undtze
Ravil
RLRod
supRed
suptes
Sippi
sppil
Unse
suote
SupOrt
Sip/A
Limoout
Unstor
UnAGr
Ravi
RL Rod
Figure 4: A. Example dissimilarity matrices for the four recorded layers (R1-4) indicating the extent to which a pair of images is represented similarly or not (higher values are less similar). Images are grouped according to task type. Color range: 0-1.2 Dissimilarity matrices for all networks can be seen in Supp. Fig B. The correlation between these dissimilarity matrices for pairs of networks is given the RSA matrices. On the top (bottom), we the whitened cosine metric (Spearman's 's rho rank correlation) to measure this correlation. Higher values are more similar. C. Summary of the Spearman RSA results. Each line shows the mean + standard deviation of how similar the network type is to all other network types, for each layer.
Comparing representations across networks, we see in FCgure that on average the RLRod model is the least similar to the others, particularly at layers R1-3. Looking what models do have similarity to RLRod, at layers R1, R3, and R4, the SupT'sk networks show some of the best matches to RLRod (followed mainly by SupAll). This suggests that identifying, possible tasks the rodent is in is part of what the RL agent has learned to use its visual encoder for. task classification shown by RLRod in Figure
least implicitly, which of the four
This aligns with the high performance
Interestingly, throughout the layers the random (untrained) networks have fairly high similarity to the trained networks. This suggests that the architecture alone has large influence over the repre- sentations generated by these networks in response to these particular images. While for many of the trained networks, this similarity fades by R4, it is still somewhat high for the UnsAut, Uns Var and SupOri networks.
We also created RSA matrices that represent how information is transformed within a single net- work. Here, in addition to using layer activity we also compare activity to pixel space and to the 38-dimensional action space of the rodent, which is the output the full rodent agent model (see dissimilarity matrices for these spaces in Supp. Fig. [T). Examples of these matrices are shown in Figure 5A, with all shown in Supp. Fig. 10 Looking at the RSA matrix for the RLRod network, we can see this network fades somewhat gradually from representing pixel space to starting repre- sent action space, whereas other networks, like the untrained ones, retain a strong similarity to pixel space. In some networks, such RLRod and Uns R2 is more similar pixel space than R1. This odd effect seems to be the result of the maze task being quite distinct in pixel space, but this distinction doesn't emerge until later in trained networks (see Supp Figs and 17).

---

Under review as a conference paper at ICLR 2022
Sup
UnsVar
Suptsk
Pood
Supfieed
0000
SupOri
SupAil
Action
UnsVar
Unscpo
Rand.
Rod
RLRoc
supt
Sup
Undkur
Pand
Musod
C.
Figure 5: How networks transform their representations across layers. Here we do RSA on the activity of all recorded layers (R1-4) along with pixel inputs and the 38-dimension action space generated by the RL agent. In A, example RSA matrices are shown for different networks. In B, the Spearman rank correlation between these RSA matrices is shown. In is summary of the similarities in B, as in Figure
Looking at the comparison across all networks in Figure 5B. find bit more diversity within networks trained the same way than suggested by Figure 4 However, here still the RLRod network on average the least similar the other networks (with UnsAut close second), as be seen in Figure
3.4 SPARSENESS
Noting that the RLRod network is different from the others, wanted to look further into how its representations differ. We started by looking more closely at the dissimilarity matrices for the RLRod model. Figure 4/ shows the dissimilarity matrices for RLRod and for of the best matches to RLRod, the SupTssk network. Of note is that starting at layer R2 the RLRod network has much higher dissimilarity values than SupTsk (and indeed higher than other networks, can be in Supp Fig.). That is, pairs of images that are represented with somewhat similar patterns of neural activity in the SupTsk network have very disparate representations in the RLRod network. Indeed, the correlation between activity patterns is near zero or even negative for most pairs of images in the RLRod model.
One possible way of creating such dissimilarity is to have sparse representation wherein individual units are only responsive to small and specific subset of images. the extreme, having each neuron respond to one and only one image would make the responses to any pair of images highly dissimilar. As can be seen in Figure 7 the RL.oo representations are indeed sparser than the other networks, particularly at layers R3 and R4. This suggests that reinforcement learning may naturally generate sparse neural activity as way differentiating different visual states.
3.5 DIMENSIONALITY
Another metric commonly used in the neuroscience literature is that of the embedding dimension (Jazayeri & Ostojic 202T). Embedding dimension is commonly estimated using linear dimension- ality reduction. Here, we show the estimated embedding dimensions according to two metrics (see Nethods/A.00.4) for all layers and networks in Figure
One striking finding is the high dimensionality of the RLRod model, particularly past layer R1. Dimensionality neural representation can also be related to sparsity in that an extremely sparse population wherein there is one-to-one correspondence between images and neurons will have dimensionality determined by the number of images. In this way, these results align with the high sparsity of the RI Rld model.

---

Under review as a conference paper at ICLR 2022
Supervised
Unsupervised
Random and RL
R1
R2
R3
R4
SupRai SupTsk SupOn SupAll
UnsAut
UnsVor
UnsCpp
Random
RLRod
Figure 6: Sparsity distributions across layers and networks (median values marked). Higher sparsity values means individual units respond selectively to a smaller number images.
Conteeneee nentpessniate 8 2 E
Figure 7: Dimensionality representations. Top row shows participation ratio each layer all networks (same y-axis across layers); Bottom row shows the average number of principal compo- nents needed to reach 85% variance explained. the (R4) plot, the dimensionality of the pixel space (action space) is indicated with black dashed line.
Looking at the number of PCs needed to explain 85% variance at R4, the supervised models have average dimensionality roughly equivalent the dimensionality of the tasks they were trained to perform wdd 1.33; SupTsk, 4.33; SupOri, 3.67; SupAll, 7). Extrapolating to the other models we can say that UnsCPC model is using 5.33 dimensions to perform its binary classification; UnsAut and Uns Var are using 10.67 and 8 dimensions respectively for their reproductions; and the RlRod visual encoder performing a 22-dimensional task. Such a high dimensional representation may be crucial for the RL model perform high-dimensional control over range of environments and timescales. The high dimensionality of the untrained networks at R4 suggests that low dimension- ality is learned by the supervised and unsupervised networks. Ways to retain high dimensionality through supervised or unsupervised training may thus make them more beneficial as pre-training for RL.
3.6 SUPERVISED TRAINING OF ACTION SELECTION
is conceivable that the high dimensionality of RLRod due to its use in predicting the 38- dimensional action space (whose dimensionality according to these metries is indicated by the black
8

---

Under review as a conference paper at ICLR 2022
dashed line in Figure 8 To test this, trained models to predict actions (i.e., behavioral cloning). Interestingly, the visual encoder alone performs very poorly on this task (Figure 8 implying that proprioception is necessary for reliable action modeling. We thus tried to add a portion of the orig- inal RI. architecture by training a parallel MLP that encodes proproceptive information. Proprio- and visual network outputs are concatenated, fed into 128-unit layer then used jointly to predict action. This network performs much better (Figure [ predicted versus real values for each output dimension can be seen in Supp. Fig. I2p. but surprisingly does not use the visual inputs all for this performance. In fact, it learns to zero-out the activity in the visual encoder, and performance while zero-ing visual inputs to the network is nearly identical. The fact that proprioceptive information alone is sufficient to reach this level of action prediction is also supported by the performance of network trained with only the proprio-MLP.
Note that vision is necessary both for task identification and good performance on these RI tasks due to their design. What these results suggest is that RLRod is not using visual inputs to control im- mediate actions, single-step action selection cannot be done with vision. However, propriceptive information alone still only achieves r² of around .4. Therefore visual information is still playing role, but likely longer timescales: via interaction with the downstream LSTMs visual input is used to identify goals and plan longer term actions.
0.3
0.2
0.1
VisProp
Misrrp-NoVis
Prop
Figure 8: Action prediction performance (averaged over all 38 dimensions) for a network trained with just visual encoder, visual and proprioceptive encoder, visual and proprioceptive encoder but visual inputs set to during test, and a proprioceptive network alone (this network predicts actions immediately from the output of the 3-layer proprio MLP). Three networks shown for each but performance is overlapping. Performance broken down by action dimension in Supp. Fig 12
4 DISCUSSION
In this work, we've ve sought to systematically characterize the population level properties of the neural representations that emerge when training neural networks egocentric visual observations with different objectives. While some works actually do provide a few anecdotal visualizations of the image features emphasized by RL well unsupervised objectives (e.g.. Stooke et al.) (02T), (e.g.,,Stooke
are not aware of efforts to more comprehensively compare the properties of representations learned by RL with those that arise from other objectives. To this end, have leveraged analysis techniques adapted from neuroscience. which have rather consistently shown the RL-derived representation to be an outlier. In particular we find that representations become less similar across networks later layers, unsupervised objectives can capture some features of the RL-trained network such as sparsity and high dimensionality certain layers but no single method captures all RL features, and the RL- trained visual encoder seems to drive long-tern term-termmo behavioral planning.
We want to clarify that representations which differ from those that arise from end-to-end RL may be useful for pre- or joint-training of some tasks in some environments. But insofar as the repre- sentations differ critical ways, enriched understanding of the properties of the different resulting representations may help better anticipate which objectives support transfer in which settings and why. For example, high representational dimensionality and representational diversity across inputs may serve good stand-ins for how easy it will be learn controllers for wide range of tasks.
Finally, these results may also be of interest to the neuroscience community, which has studied sparsity of neural representations for decades. While simple learning procedures have been shown to create sparsity (Földiak] 1990), the fact that it arises here from reinforcement learning may further contextualize its origins and role in the brain.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBLITY STATEMENT
Here we used images openly available through RL Unplugged (Gulcchre et al. 2020). Code for analysis and training of the supervised and unsupervised networks is included in uploaded supple- mentary materials.
REFERENCES
Baktash Babadi and Haim Sompolinsky. Sparseness and expansion in sensory representations. Neu- ron, 83(5):1213-1225 2014.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Pro- ceedings of ICML workshop on unsupervised and transfer learning, pp. 17-36. JMLR Workshop and Conference Proceedings, 2012.
F. Chollet. Vae.
https://github. http:///www...//eeeeeeeeeeeaaaa- exampless/geeeratiiveeeee..
2020.
Peter Földiak. Forming sparse representations by local anti-hebbian learning. Biological cybernet- ics, 64(2):165-170, 1990.
Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. BioRxiv, pp- 214262, 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1.
MIT Press, 2016.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: collection of benehmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends cognitive sciences, 2020.
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L. Mc- Clelland, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent. International Conference on Learning Representations, 2019.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Jocl Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations, 2017.
Mehrdad Jazayeri and Srdjan Ostojic. Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity. Current Opinion in Neurobiology. 70:113-120 2021.
Diederik Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Nikolaus Kriegeskorte and Xue-Xin Wei. Neural tuning and representational geometry. arXiv preprint arXhv:2104.097433 2021.
Michael Laskin, Aravind Srinivas, and Picter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference Machine Learning, pp. 5639- 5650. PMLR, 2020.
Alessandro Lazaric. Transfer reinforcement learning: a framework and a survey. In Reinforcement Learning, pp. 143-173. Springer, 2012.
10

---

Under review as a conference paper at ICLR 2022
Lucas Lehnert, Michael L Littman, and Michael Frank. Reward-predietie representations gen- eralize across tasks in reinforcement learning. PLoS computational biology, 16(10):e108337, 2020.
Josh Merel, Diego Aldarondo, Jesse Marshall, Yuval Tassa, Greg Wayne, and Bence Olveczky, Deep neuroethology of a virtual rodent. In International Conference on Learning Representations, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
Hamed Nili, Cai Wingfield, Alexander Walther, Li Su, William Marslen-Wild and Nikolaus Kriegeskorte. A toolbox for representational similarity analysis. PLoS computational biology, 1004))e 2014.
Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 (6583))66677, 1996.
Bruno Olshausen and David Field. Sparse coding of sensory inputs. Current opinion in neuro- biology, 14(4))481-48) 2004.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1rXiv:1807.0374.. 2018
Eddie Rafols, Mark B Ring, Richard S Sutton, and Brian Tanner. Using predictive representations to improve generalization in reinforcement learning. In IJCAI, pp. 835- 840, 2005.
Mattia Rigotti, Omri Barak, Melissa R Warden, Xiao-Jing Wang, Nathaniel D Daw, Earl K Miller, and Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature, 497 (7451):585-590, 2013.
Edmund T Rolls and Alessandro Treves. The relative advantages of sparse versus distributed encod- ing for associative neuronal networks in the brain. Network: computation in neural systems, 144): 407, 1990.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learming with gelf-predictive representations. In International Conference on Learning Representations, 2021.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pp. 9870-9879. PMLR, 2021.
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
D. Tellez. Contrastive predictive coding.
https://gitttbb.cmm///ddddl
coottaassiiee-ppalletiee-ooooooo 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm controlo::: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
William Vinje and Jack Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. Science, 287(5456) 2000.
Aria Wang, Michael Tarr, and Leila Wehbe. Neural taskonomy: Inferring the similarity of task- derived representations from brain activity. Advances Neural Information Processing Systems, 32: 15501111155 2019.
11

---

Under review as a conference paper at ICLR 2022
Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska- Barwinska, Jack Rae, Piotr Mirowski, Joel Z Leibo, Adam Samtoro, et al. Unsupervised predictive memory in goal-directed agent. arXiv preprint 1rXfv:1803.1076 2018.
predicccii
Benjamin Willmore and David Tolhurst. Characterizing the sparseness of neural codes. Network:
Computation in Neural Systems, 12(3):255, 2001.
Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential deci- sion making. arXiv preprint arXiv:2102.058/5. 2021.
Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Pro- ceedings of the National Academy of Sciences, 118(3), 2021.
APPENDIX
A EXTENDED METHODS
In all but the RL model, networks were trained end-to-end with ADAM using batch size of 128, 1.2 weight regularization of .001 and learning rate of .001 (dropping three times during training). These networks were trained for 75 epochs (this value was chosen through preliminary explorations that in- dicated validation curves tended to plateau around this stage). Three different random instantiations of each type of network were trained.
A.0.1 REINFORCEMENT LEARNING MODEL.
For example of ResNet trained through reinforcement learning complex visual task, use the visual encoder from the virtual rodent model introduced in Merel et al.) (2020). This agent takes in egocentric visual and proprioceptive information control realistic rodent-like body to perform the four above-mentioned tasks. After the visual encoder, visual outputs are combined with encoded proprioceptive information, which then feeds into recurrent neural network that is trained to estimate value. These networks also feed into a recurrent policy network, which defines the movement of the rodent body two-tap tasks are sparse upon orb hit, whereas bowl escape and gaps have denser rewards. Here we use weights only from the visual encoder of the trained model, provided by the authors of Merel et al. (2220).
18-dimensional continuous space. Rewards for the maze and
A.0.2 SUPERVISED, UNSUPERVISED. AND RANDOM MODELS
While the images generated from the agent's actions don't 't have obvious labels, there is some asso- ciated information that can be used for supervised learning. Here, we train four different models: task model (SupTsk), a reward model (SupRwd), an orientation model (SupOri), and a model that performs all three tasks All..ll).. Weights for supervised models were initialized with a truncated normal distribution with standard deviation .05.
Task Model: In SupTssk, the final 128 unit layer of the ResNet followed by 4 unit layer trained with categorical cross entropy loss to classify image as coming from one of the 4 different tasks described above that the RI. agent was trained to perform. Because the RI. agent is not given an explicit cue about which task it is in, it must deduce it from visual information; therefore we can assume that task identity is determinable from visual inputs.
In Sup the final layer of the ResNet is followed by single sigmoidal unit which is trained with a mean squared error loss to predict the reward the agent received after the action it took in response the image (the performance of this model is therefore constrained by the extent to which a single frame is sufficient to predict upcoming reward). To make the distribution of rewards similar across tasks, all tasks were normalized to have max reward of and the sparse
Reward Model:
12

---

Under review as a conference paper at ICLR 2022
rewards of the maze and two-tap tasks were made dense by assigning small reward to several frames leading up to the reward.
Orientation Model:
In SupOri, the final ResNet layer is followed by 3 unit tanh layer that is trained with mean squared error loss to predict the rodent's body position in 3-D egocentric space (i.e., equivalent the yaw, roll, and pitch).
All Task Model: SupAll, three parallel layers are added after the 128-unit layer that perform each of the above tasks.
For unsupervised learning. we trained an autoencoder (UnsAut), a variational autoencoder (UnsVar) and contrastive predictive coding model (UnsCPC). Unsupervised network weights were initialized with the Glorot uniform distribution Glorot & Bengio (2010).
Autoencoder: For UnsAut, the ResNet architecture is rebuilt in reverse after the 128 unit layer. It is trained on image construction using the binary cross entropy loss. (i.e., undercomplete autoencoder; Goodfellow al. (2016)
Variational Autoencoder: For UnsVar, after the final 128-unit layer, mean and variance layers are created which feed into a sampling layer followed by the reverse ResNet architecture and the network is trained using reconstruction loss plus KL divergence loss (Kingma & Welling, (013) (we used code provided by Chollet (2020).
For UnsCpe, we implement the architecture and learning procedure from Oord et al. (2018). Briefly, the encoding network is trained predict whether two images are of consecutive frames from the rodent's exploration or not. We use code adapted from Tellez 2018).
Contrastive Predictive Coding Model:
Random Model: To assess the impact of architecture alone on representations we include untrained or random (Rand.) models. These were initialized using the Glorot uniform distribution.
A.0.3 REPRESENTATIONAL SIMILARITY ANALYSES
Representational similarity analysis (RSA) is two-step analysis used determine the extent to which different models represent visual inputs similarly. First, dissimilarity matrices were made for each layer in each network. Each entry the dissimilarity matrix defined as
dej = corr (r:r()))))))
(2)
where is the population activity in response to image i and corr is the Pearson correlation coeffi- cient.
Example dissimilarity matrices from two networks can be see in Figure 4A.
These dissimilarity matrices are then compared to determine how similar two networks are. We use the RSA toolbox (Nili et al.) 2014) to perform these comparisons. Specifically, we use two metrics to ensure our findings are robust: the whitened cosine metric, which is equivalent to linear centered kernel alignment, and Spearman's ''s tho rank correlation.
We also use RSA to compare different layer representations within a network. For this we create the same dissimilarity matrices for layer activity, and also for pixel values of the input images and the 38-dimensional output space of the virtual rodent. These dissimilarity matrices were compared using the Pearson correlation coefficient create RSA matrices. These RSA matrices were then compared across networks using the Spearman rank correlation.
A.0.4 DIMENSIONALITY REDUCTION
Embedding dimension is defined as the number dimensions (in ambient Euclidean space) explored by the neural activity (Jazayeri & Ostojic, 2021). To estimate the embedding dimension of the activity at different layers in these networks, we perform principal components analysis (PCA) on the population responses. We then calculate the participation ratio:
14)
PR =
(3)
Σ
13

---

Under review as a conference paper at ICLR 2022
where μ4; is the eigenvalue associated with the ith PC. This value correspond to the number of PCs needed to reach 80-90% variance explained under certain distributions of the eigenvalues of the covariance matrix (Gao al. 2017). As a secondary measure we also calculate empirically the number PCs needed to reach 85 % variance explained.
14

---

Under review as a conference paper at ICLR 2022
SupRwd
Sup
SupOri
SupAll
UnsAut
UnsVar
UnsCpc
Rand.
RLRod
Figure 9: Supplementary Figure: Dissimilarity matrices for all networks. Color range: 0-1.2

---

Under review as a conference paper at ICLR 2022
Poxel
SupRwd
SupTak
MOdns
Action
UmsAut
UnsVar
SupAll
Uns
RoRdd
Rand
Figure 10: Supplementary Figure: Within Network RSA matrices for all networks.
Pixels
Action
1.
Image #
Image #
0.2
Image #
Image #
Figure 11: Supplementary Figure: Dissimilarity Matrices for pixel and action spaces (note these on a different color scale than those Figure 9
16

---

Under review as a conference paper at ICLR 2022
Vis
VisProp
OINOLS
OHAILE
NPL
NGLA
GUIDL
IPLL
WIOK
VisPrup-No Vis
Prop
kmbbra
NOLSS
NALA
Lebbaa
NALLS
predicted
real
Figure 12: Action prediction from different supervised networks, as labeled in Figure g = bend, t
twist, extend, abduct, supinate
17


Reviews:
[{"review_id": "qqo8c099RjX", "reviewer": "Reviewer_vGGV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed a lot about this paper. It's very topical and I agree that studying representations is a more powerful and robust way to set up transfer learning than speculation/trial/empirical black-box analysis. \n\n**Strengths**\n1. I like the idea of taking metrics inspiration from neuroscience, and while many works in this space make only a tenuous connection between neuroscience and ML that doesn’t justify their claims, this one does a good job of not overstating its implications or forcing its hand. The metrics lend themselves intuitively and elegantly to the representation analysis outlined here\n2. The experiments are thoughtful and interesting, and I think each piece of work in this paper is useful. Rather than doing several different experiments and simply listing them all, this paper justifies each one. Nice work, and (imo) not as common as it should be in our field! \n3. The figures in this paper are incredibly informative. Particularly figures 6 and 7, but also the similarity matrices in 4 and 5: they're really informative, and 6 and 7 in particular are also easy to read and aesthetic.\n4. For the most part, this paper is well-written. It's easy to follow while delivering lots of information. Compliments to the authors on writing a paper that lets you sit with the science instead of forcing you to puzzle over sentence structure. \n\n**My critiques:**\nThis paper has great experiments and is one of the few where I liked the results section more than the prior sections (though that makes sense with an analytical paper). However, I don't think it makes or supports clear claims. This is both a content and a presentation issue, and it is the reason I do not recommend acceptance at this moment. I realize that this paper is meant to be exploratory and open new avenues, but even an introductory paper should come to one or a few concrete, robust points and that doesn't quite happen here. That said, I think this work could be turned into two or three (still introductory, exploratory) papers with some more work done on each of its preliminary findings and I highly encourage this direction. I also note that this may be a matter of my research taste and other reviewers may feel differently. \n\nSpecifics:\n1. This paper doesn't make clear claims. Until section 3, I know what problem it is working on but I have no idea what answer it is going to come up with. Even if it is exploratory, I should know which findings the exploration yielded once I'm done with the introduction because otherwise I have no idea what to look for and have to piece together the unifying threads of analysis myself. Points where this shows up:\n- The first sections only talk about exploration and problem, but the conclusion talks about several findings (e.g. unsupervised objectives can capture some but not all the representations of RL) that I learned about halfway through section 3 or later - more than halfway through the paper\n- The claims made in the conclusion appear to be that RL is a general outlier, all objectives have representations that become less similar at later layers, unsupervised objectives can capture some but not all of the representations of RL, and the RL-trained visual encoder seems to drive long-term behavioral planning. However, so much of section 3.4 and 3.5 is about how RL representations are sparse that I would expect more discussion of it in Section 4 than \"RL is an outlier\". So I don't know what to expect, I form expectations based on the distribution of detail in section 3, and then those expectations are violated at the end. \n- There are interesting details thrown at the reader, e.g. unsupervised tasks representing pixel space and supervised tasks reaching action space. But this doesn't get enough follow-up experimentation, and it's hard to tell if it's surprising given the nontrivial difference in performance. Another example is the fact that RLRod and UnsVar have R2 look more like pixel space than R1. \n- The paper seems to focus on RL and its distinctiveness from supervised and unsupervised learning, but the title and introductory sections don't really suggest this. The problem setup talks about RL in particular, but we aren't told that the results will focus on it; only that it's a good motivator. \n\n2. By the end, we have some claims (what I listed above). However, they seem somewhat speculative and I don't think there is enough evidence behind most of them. Examples:\n- As I said, much of the attention is on RL, but there's only one RL objective and several supervised and unsupervised objectives. I would recommend more RL experimentation with alternative tasks, even if not as high-dimensional (in fact this might strengthen the promising results about RL representation dimensionality). \n- The claim about RL having neurons that respond to one and only one image is major, especially since it's true in early layers and becomes less true with later layers. I think the evidence supports this hypothesis, but I am not sure if it's enough when all we know is that the representation is sparse and high dimensional. I may not be knowledgeable enough to know what's required to convince of this claim, but it would help to have more exposition and I think more evidence\n- It's interesting that RLRod and UnsVar have R2 look more like pixel space than R1, but the analysis is \"this odd effect seems to be the result of the maze task being quite distinct in pixel space, but this distinction doesn't emerge until later.\" That's a strong causal claim for a surprising result, and we don't have nearly enough evidence to suspect that because it's not clear mathematically how it would happen. \n- The claim that proprioception is necessary for reliable action modeling is definitely suggested, but not quite demonstrated with only one experiment. The offshoot, that not only is proprioception important but visual information becomes useful longer-term, is *very* speculative. The explanation brings up objectives that haven't been tested. The discussion then reports this as one of ~four primary findings even though it's a one line suggestion in the analysis, which once again feels insufficient.\n\n**Presentational notes:**\n- Figure 4 and figure 5 are very useful and informative. That said, I find correlation matrices a little difficult to visually parse, so better signposting (callouts, highlights, more labelling) could be helpful \n- I would appreciate visual examples if possible. For example, renders of the activations to show pixel vs. action representation could be helpful to give the reader an intuitive sense for the claims that these are surprising results and not just matters of less learning. \n\n**Nit:**\nSection 3.4 refers to figure 7 but I think it's meant to refer to figure 6. This was quite a confusing error (and I may be wrong). \n\nThis paper delivers analysis as though it is happening in real-time, rather than before writing. Even if that is what happened (as it does), the paper still shouldn't sound like that because it's confusing and forces the reader to do work that 1) they shouldn't have to do and 2) they will not be as good at as the authors. This paper has interesting, promising content and with more work could yield two or three interesting papers about the long-term use of visual information, the sparsity of of RL representations, or the comparative expressiveness of RL and unsupervised learning representations. These themselves would open avenues of investigation while still being complete papers; as it stands, the work currently feels more like research notes and less like a checkpoint, let alone finished work. I highly recommend expanding and submitting to a later venue, where I expect multiple works stemming from this to do well. ", "summary_of_the_paper": "This work identifies the important problem of transfer learning for problems that require deep learning due to high-dimensional inputs. It notes that transfer learning approaches often cannot transfer very far across learning objectives and are judged empirically as black boxes, making for brittle approaches. It proposes to study internal representations to assess potential for transfer. \n\nThis study trains several supervised and unsupervised, and one reinforcement, tasks with the same data and ResNet architecture. It uses representational similarity analysis (inter-task similarity similarity of intra-task dissimilarity), sparsity analysis (neuroscience-sourced measure of response similarity between neurons), and dimensionality analysis (PCA) to assess the representations; these metric choices are taken from neuroscience literature. \n\nRSA shows that representations across different tasks diverge with layer depth (except in a few cases, though the paper doesn't call these out as clearly); the same is seen within different instantiations of the same task in all but three cases. RLRod differs the most. Sparseness analysis shows that RLRod has the most internal dissimilarity in responses to various inputs, and dimensionality analysis shows that it is also the highest dimensional. The paper speculates that this might be due to a one-image-per-neuron representation emerging even in early layers. To test whether the high dimensionality in RLRod is purely due to a higher-dimensional output (38-dim action), the paper trains encoders to predict action and finds that the visual inputs are pretty useless and proprioceptions are somewhat helpful, indicating the value of proprioceptive data. \n\nThe discussion lists some main claims: representations become less similar at later layers, unsupervised objectives can capture some but not all of the representations of RL, the RL-trained visual encoder seems to drive long-term behavioral planning; then suggests the potential for more study to determine promising transfers. It also postulates value for the neuroscience community, since sparse representations are prevalent in biological intelligence and the similarity may be indicative of similar objectives in artificial and biological intelligence. ", "summary_of_the_review": "Strengths:\n1. Good, well-scoped inspiration from neuroscience \n2. Thoughtful and interesting experiments \n3. Very informative figures, several of which are also aesthetically appealing and clear \n4. Overall very interesting problem and motivation, plus a well-written paper \n\nMy critiques:\n1. Lack of clear claims, especially at the beginning, making it difficult to have a good lens with which to view the results - see examples\n2. Claims appear by the end, but they are somewhat speculative and disparate - see examples\n\nPresentation notes: \n- The figures are very rich and useful. Better signposting in them would help the reader really get the value of these experiments \n- Visual examples of activations for certain points would help build intuition for why these might be surprising results ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "l8V0NP1aw_H", "reviewer": "Reviewer_WoDx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros:\n* comparing representations of networks trained with different objectives is a very timely question, and I am frankly really glad someone is working on this direction. Results here will inform not just efficient models in reinforcement learning, but, as the authors also point out, neuroscience as well (which is where I'm coming from).\n* the setup is well done: the same architecture is used for all trainings, with 3 different variants per objective, and everything is trained in the same environment, i.e. very similar/the same images.\n* figures generally well support the claims, and comparisons are done thoroughly with additional investigation into the underlying reasons for the RL-trained model's dissimilarity to other models\n\ncons: \n* simplistic setting: all models are trained in a rather simple environment without much variation in the images (Figure 1). Because of that, supervised models are not trained in the more common object classification paradigm, but instead to e.g. predict reward and classify tasks. This makes me worried that results might not generalize to more challenging environments and classification tasks\n* potential confounds to determine similarity between objectives: what if RLRod simply has not learned the task well enough? I.e. if you were to continue training, its representations would become more similar to the representations learned by other objectives? Better establishing the \"comparability\" of networks at the stages of training they were captured at would be good here. For instance, in Figure 3, the RLRod model is the one that does not perform as well, so I wonder if its representations just have not converged. This could also explain the high dimensionality (lack of invariance) in Figure 7\n* figures are difficult to read: while I appreciate the showing of all 3 models and all sub-tasks within a category (e.g. SupRwd, SupTsk, SupOri, SupAll for supervised), it does make the figures difficult to interpret, especially without additional aids such as different colors and symbols. Perhaps you could add condensed plots next to the more detailed versions that average over the 3 models or even the sub-tasks?\n\nminor:\n* When computing similarity to other models, how is each condition weighted? Are supervised approaches just \"more similar\" because there are more supervised models compared to e.g. the *single* RLRod network?\n* I didn't get how \"neural predictivity is only an indirect way of comparing components of network representation\" -- it evaluates one aspect of representations, just like RSAs evaluate a different aspect. None of them are complete. Neural predictivity has an additional linear mapping step, RSA has an additional image-wise similarity step. https://arxiv.org/abs/1905.00414 is a useful paper discussing the differences\n* order between SupRwd and SupTsk is different in table 1 and figure 2A\n* the influence of architecture on representations (2nd-to-last paragraph page 6) has recently been the attention of some related work in neuroscience as well, e.g. https://www.biorxiv.org/content/10.1101/2020.06.08.140111v1 and https://openreview.net/forum?id=rkxcXmtUUS\n* typos: figure 4 caption is missing a \".\" before \"B.\"; page 7 \"As can be seen in Figure 7\" should be 6 I believe; and page 9 \"dashed line in Figure 8\" should be 7", "summary_of_the_paper": "The paper explores how similar the representations learned by supervised, unsupervised, and RL techniques are in an egocentric virtual rodent environment. The main claim is that representations in RL-trained networks are most different from supervised and unsupervised networks, which is tested with an RSA metric on 4 layers of the networks. This is explored further by analyzing the sparsity, dimensionality (RL has much higher dimensionality) and action prediction (proprioception alone without visual features is sufficient here).", "summary_of_the_review": "This paper has definitely made me think about some questions in the similarity of representations from different objectives, so I am inclined to accept. However, it also has limited applicability due to the simple environment and objectives, and could use some improvements in presenting a more general finding rather than detailed individual models (see main review for suggestions).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "V43T2DK0FY", "reviewer": "Reviewer_RDPx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nWhile I am not particularly familiar with this subarea, they claim that this is the first time that an RL network is compared with the other types, so there is some novelty.\nAlso, the comparisons are fairly thorough, and the methodology is sound.\nThe results are fairly clear, if not totally unexpected. For example, they analyze different modules of the resnet backbone and find similarity between layers decreases with depth. This makes sense, as later layers will be more task oriented (like an information bottleneck analysis).\n\nWeaknesses, with concrete, actionable feedback\nA main weakness is that the results are pretty much what one might expect. For example, the RL network has to decide between 38 different actions, while the outputs of the other networks are low dimensional classification or regression problems. Thus it is not surprising that it creates high dimensional representations.\n \nAnother result is that networks trained the same way have similar representations. Another is (as mentioned above) that in networks trained for different tasks, the similarity between their layers decreases the deeper the layer.\n\nAnother result is that the RL network transfers fairly well to the objective of predicting the task, which would also be expected.\n\nSo, all of these results are not particularly surprising. \n\nIn the cases where the results are not completely expected, the authors remark on them, but do not perform any additional analysis to see why these results held. For example, for the RL network and the variational autoencoder, the second module's representations are more similar to pixel space than the first module's. I.e., the representation diverges from pixel space in the first layer, and then becomes more similar in the next layer up. They try to explain this in one sentence as having to do with the maze task (which I don't find convincing). This doesn't explain why the variational autoencoder would do this. This is one result that perhaps could be analyzed more closely as to why it happens.\n\nAnother somewhat puzzling result that isn't remarked on is in Figure 7, where the number of principal components needed to explain 85% of the variance within representations over the dataset is measured. This gives a measure of the dimensionality of the representation. What I find puzzling is that the dimensionality of, for example, the random network makes a very large jump between the 3rd module and the 4th. Maybe I'm just bad at math, but it is unclear to me how the layers' dimensionality jumps by about 4X between these two layers. However, it does seem to be a property of many of the networks, that the intrinsic dimensionality increases gradually from layer 2 to layer 4. Why is this? \n\nIn order to test whether it is the 38-dimensional output of the RL network that causes the high dimensionality, they train a network to predict the actions from the input images (behavioral cloning). This doesn't work well, but adding proprioceptive input does succeed (what proprioceptive inputs are here should be explained for those of us not familiar with the rat RL task). But then they don't measure the dimensionality of the representation, which I thought was the point of doing this. There is also the puzzling result that the network basically turns off the image input in favor of the proprioceptive input. This makes sense if the proprioception indicates what the rat is about to do - e.g., if there is the equivalent of a muscle tension proprioception, this certainly could be indicative of the actions. \n\nMinor points/typos:\nFigure 2A is not referred to in the text. It should be in the paragraph under section 3.1.\n\nThere are several places where the text refers to the wrong Figure - e.g., mixing up Figures 7 and 8.\n\nPage 6, second paragraph, you might mention the deep image prior paper here. (https://arxiv.org/abs/1711.10925).\nPage 6, third paragraph, you might mention the information bottleneck here.\n\nSection 3.4, second paragraph, 3rd line from the bottom, it should be Figure 6, not 7, I believe.\n\nTop of page 9: this should be Figure 7, not 8.", "summary_of_the_paper": "This paper compares the representations learned by otherwise identical networks (up to the output layer) for different tasks: supervised (4 tasks), unsupervised (autoencoders, vanilla and variational), self-supervised predictive coding, untrained (randomly initialized), and one RL policy network. They are all trained on the same images from the RL task, and the supervised tasks are related to the RL task (e.g., what task is this image from). \n\nThe representations are compared by using relatively standard neuroscience measures: RSA (with two distance measures), a kind of meta-RSA that compares the correlations between the RSA matrices of each network to each other (again with two correlation measures), two sparsity measures, and two measures of dimensionality of the representations. \n\nThey find that the RL network stands alone, treating images in complex ways, being very uncorrelated to the other networks, being more sparse, and more high dimensional. The unsupervised networks also stand out, although not as much, from the rest.\n\nThey also investigate transfer learning between the networks, training single-layer networks from the output of the penultimate layer of each network (frozen) on each other network's tasks.\n\nI have read the authors' response, and given that they are not responding to individual reviews or changing the paper, I am keeping my score the same.", "summary_of_the_review": "This paper does a nice job of comparing representations learned by an identical network architecture, given different training objectives. The results are interesting, but not completely unexpected. The idea of comparing an RL-trained network to other networks doing image classification and regression is a good idea, but the results are about what one would expect. Where they are not what one would expect, they are not explained or analyzed in any depth. Hence I don't believe this paper is ICLR-worthy.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4-EMiO__CQ", "reviewer": "Reviewer_mWVA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n\n- A thorough and novel characterization of various properties of representations learned via various training objectives\n\nweaknesses:\n\n- Logic of motivation for the work unclear. It's not clear what question this characterization helps to answer or why this analysis is most appropriate. Hypotheses are not clearly described nor are the impacts or take-home messages.\n- Poorly written in places.\n- Related work is missing and irrelevant work is cited. Similarity analysis, as well as analyses of dimensionality and sparsity are common in machine learning research, not just in neuroscience. It's not clear why work on neural predictivity is relevant here. Broad swaths of research are cited rather than a targeted background related to a specific question.\n    - **Morcos AS**, **Raghu M**, **Bengio S**. Insights on representational similarity in neural networks with canonical correlation. .\n    - **Leavitt ML**, **Morcos AS**. Selectivity considered harmful : evaluating the causal impact of class selectivity in DNNs. In: *International Conference for Learning Representations (ICLR)*. 2021.\n    - **Leavitt ML**, **Morcos AS**. Linking average- and worst-case perturbation robustness via class selectivity and dimensionality. In: *arXiv:2010.07693*. 2021.\n    - **Morcos AS**, **Barrett DGT**, **Rabinowitz NC**, **Botvinick M**. On the Importance of Single Directions for Generalization. In: *International Conference for Learning Representations (ICLR)*. 2018.\n    - **Kornblith S**, **Norouzi M**, **Lee H**, **Hinton G**. Similarity of Neural Network Representations Revisited. In: *ICML*. 2019.\n    - **Thompson JAF**, **Yoshua Bengio**, **Schönwiesner M**. The effect of task and training on intermediate representations in convolutional neural networks revealed with modified RV similarity analysis. In: *Cognitive Computational Neuroscience*. 2019.\n    - **Raghu M**, **Gilmer J**, **Yosinski J**, **Sohl-Dickstein J**. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Understanding and Improvement. .\n    - **Ansuini A**, **Laio A**, **Macke JH**, **Zoccolan D**. Intrinsic dimension of data representations in deep neural networks. In: *Advances in Neural Information Processing Systems*. 2019.\n    - **Recanatesi S**, **Farrell M**, **Advani M**, **Moore T**, **Lajoie G**, **Shea-Brown E**. Dimensionality compression and expansion in Deep Neural Networks . http://arxiv.org/abs/1906.00443.", "summary_of_the_paper": "The paper includes a comparison of the representations learned according to a variety of training objectives, including reinforcement, supervised and unsupervised objectives, in a common architecture and with similar training data. The comparison looks at dimensionality, sparsity and representational similarity and finds representations learned via RL to be outliers relative to the other kinds of objectives. ", "summary_of_the_review": "The paper presents several interesting characterizations but I'm not sure what they are for. It could be that the work is not well motivated or just that it's not written to make the motivation and implications clear. Either way, I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the representations that are learned by the same network, on the same data, but with different objectives/tasks (RL, supervised, unsupervised). Though the reviewers were positive about some aspects of the paper, the reviews were generally low (3,3,3,6) and indicated rejection. The principle recurring theme in the reviews as to why this was a rejection was the lack of clear motivations/implications. The authors decided not to submit an updated version of the paper. As such, this was a reject decision.
Decision: Reject