Learning Semantically Meaningful Representations Through Embodiment
Authors: Viviane Clay
Peter König
Kai-Uwe Kühnberger
Gordon Pipa
Venue:  (2020)
Keywords: reinforcement learning
deep learning
embodied
embodiment
embodied cognition
representation learning
representations
sparse coding

Abstract: How do humans acquire a meaningful understanding of the world with little to no supervision or semantic labels provided by the environment? Here we investigate embodiment and a closed loop between action and perception as one key component in this process. We take a close look at the representations learned by a deep reinforcement learning agent that is trained with visual and vector observations collected in a 3D environment with sparse rewards. We show that this agent learns semantically meaningful and stable representations of its environment without receiving any semantic labels. Our results show that the agent learns to represent the action relevant information extracted from pixel input in a wide variety of sparse activation patterns. The quality of the representations learned shows the strength of embodied learning and its advantages over fully supervised approaches with regards to robustness and generalizability.

Under review as a conference paper at ICLR 2020
LEARNING SEMANTICALLY MEANINGFUL REPRESENTATIONS THROUGH EMBODIMENT
Anonymous authors Paper under double-blind review
ABSTRACT
How do humans acquire a meaningful understanding of the world with little to no supervision or semantic labels provided by the environment? Here we investigate embodiment and a closed loop between action and perception as one key compo- nent in this process. We take close look at the representations learned by a deep reinforcement learning agent that is trained with visual and vector observations collected in a 3D environment with sparse rewards. We show that this agent learns semantically meaningful and stable representations of its environment without re- ceiving any semantic labels. Our results show that the agent learns to represent the action relevant information extracted from pixel input in wide variety of sparse activation patterns. The quality of the representations learned shows the strength of embodied learning and its advantages over fully supervised approaches with regards to robustness and generalizability.
INTRODUCTION
When the way supervised neural networks learn is compared to the way humans learn one can easily make out some major differences. Two of those differences supervision and embodiment. Taking the example of object recognition from visual observations, neural network will be presented with thousands of images of the object in question, each of them accompanied by class label. A toddler in comparison will also collect many observations of the object of interest, however, will do so by interacting with the object, looking at it from different perspectives by moving the head or even moving the object (Bambach et al. 2018). This law governed change in observations conditioned on the movements of the toddler emphasises the importance of embodied cognition (Engel al. 2013). It will make it possible to recognize the object as distinct entity, separate from its surroundings and learn a general concept of it. This allows it to robustly recognize the object again even when seen from new perspectives or under different lighting conditions (Smith & Slone, 2017). When the toddler is now told the name of the object, an almost instantaneous association between label and object can be made without the need of thousands of labeled examples (Samuelson & Smith (205). This therefore makes a very efficient strategy for learning stable representations of objects.
Fully supervised neural networks have been shown suffer from shortcomings that humans usually do not exhibit. SSzgedy et al. (2013) showed how very small perturbations to an image, undetectable the human eye, can drastically change the classification accuracy of a neural network. Even simply holding such adversarial examples in front of camera (Kurakin et al., 2016) or specific natural images (Hendrycks et al., 2019) can have this effect. The networks seem to possess an over-reliance on local image features such as texture and do not consider global features such as the overall shape and outline of an object (Baker et al., 2018). Considering the training cireumstances, this effect is unsurprising. The networks are expected to learn the concept of objects solely from pixel values. Without being able to interact with objects or even just looking at them from slightly different perspectives, is very difficuul to figure out basic knowledge such object and background relationships. We expect that an active exploration of the world would make it possible to learn a more general and robust concept of objects.
Already in 2001 O RRegan & No (000) argued that even though is clear that action requires per- ception, this relation also reverses. Perception and the understanding of what is perceived requires action (Noë 2005). According to O Regan & Noë (2001). "experience is not something that hap- pens in but is something we do" (p.99). They argue that an important part of perception is to learn

---

Under review as a conference paper at ICLR 2020
how actions affect sensations. These sensory motor contingencies help us make sense of our per- ceptions, predict them and efficiently sample the environment for information (Engel et al. 2013).
In humans, perception is hugely influenced by how we interact with the world (Witt 2011 Goals and the expected cost to perform actions to achieve a goal influence our perception of physical entities (Proffit) 2006). Also more abstract processes such language comprehension are linked to action systems in the brain (Pulvermüller & Fadigal (010). We therefore postulate that in order to teach an artificial agent true understanding of its (simulated) world it needs to be able interact with the world. This paper will present results from an embodied agent acting in a virtual 3D world and learning an internal representation of its sensory imput. The framework of learning by interacting with the world produced a meaningful and agents observations, even though no semantic labels were provided.
internal representation of the
2 RELATED WORK
There is a strong research interest in learning visual structure in unsupervised way which can for example be approached by using auto-encoders (T'shannenn et al., (0018). In its simplest form, relevant structure supposed change (Körding & König 2001 Wiskott & Sejnowski, 2002) facilitating learning of invariant representations. To further incorporate a time component and learn visual structure and changes over time, future frame prediction is a commonly used task
Villegas et al., 2017: Srivastava et al., 20155: Patraucean et al., 2015 Oliu et al. 2017]] Mahjourian ct al. 2016) Finn et al. 2016;]Denton & Birodkar (2017). However, only a few of the papers dealing
with time series prediction actually investigate the learned representations in the network (Lotter
et al. 2013 Qiao al. 2018).
An alternative concept for unsupervised representation learning is the use of predictive coding (Rao & Ballard 1999) which can be applied to train ANNs (van den Oord et al. 2018 Wen et al. 2018 Han et al. 2018 Lotter et al. 2016). The idea of predicting future observations based current
actions can also be used in a reinforcement learning setting to inject agents with some sense of curiosity (Pathak et al. 2017). Ha & Schmidhuber (2018) has shown that training a recurrent world model using a variational auto-encoder can increase the performance of agents in several games. Chaplot et al. (209)) show how jointly training semantic goal navigation and embodied question answering can improve performance on both of these tasks. Also, simply seeing a visual scene from different angles can get network learn disentangled representations of individual objects (Eslami et al. (0188) and get it to imagine the scene from a previously unseen viewpoint (Eslami et al. 2018 Rosenbaum al. 2018).
Researchers who investigate representation learning in reinforcement learning agents often use addi- tional regularization or losses to enforce a certain representation the latent space (Nachum al.) 2018 Lesort et al. 2018 de Bruin et al.] 2018). [Shan et al. (2019) even gets agents to explic- itly learn world graphs. As such explicit constraints are biologically implausible, we will investigate what kind of representations arise naturally within embodied training setup. Lillicrap et al. (2015) have already shown results from a simple deep reinforcement learning agent which indicate that per- ceptually similar observations are mapped close to each other in the latent space. We will further investigate this and look explicitly at the type of representation encoding that is learned as well as the meaningfulness of the representation and the type of information that is encoded.
EXPERIMENTS
3.1 TRAINING A DEEP REINFORCEMENT LEARNING AGENT
The representation under investigation in this paper is the activation in the hidden layer of deep neural network trained in a reinforcement learning environment. Figure Figure[ssoww the network struc- ture of the agent. As input, it receives visual and vector observations from the simulated environment with size 168x 168x3 and size respectively. The visual and vector observations are first processed separately by two convolutional layers and two dense layers for the visual input and two dense lay- ers for the vector input, until they are concatenated into one encoded state. This encoded state has dimensionality 512 where 256 of its activations come from the visual encoding pathway and 256
2

---

Under review as a conference paper at ICLR 2020
from the vector encoding pathway. The encoded state of the high dimensional visual input and its properties will be the main focus of this paper.
Based on the encoded state, two dense layers output action probabilities and value estimate. The action probabilities are translated into actions and sent to the environment to obtain the next set of observations. They are also used, together with the value estimate and the reward received from the environment to optimize the neural network and thereby, train the agent to perform better actions. For training of the network proximal policy optimization (PPO) is used which is an efficient policy gradient method for deep reinforcement learning 4Schulman et al.) 2017). More specifically, the PPO implementation and reinforcement learning framework of the Unity ML-Agents toolkit used (Juliani et al. 2018). The overall setup and training procedure are chosen such that a neural network is trained in an embodied way with a closed loop between perception and action.
As setting for training. the environment proposed in the Unity obstacle tower challenge (Juliani et al., 2019) is used. This environment was introduced as a new benchmark in reinforcement learning for pixel-based learning in procedurally-generaae agent needs to learn to navigate through a 3-dimensional maze environment, solving successively harder tasks. Every level of the tower consists of several rooms, connected by doors. When reaching the final door on floor the agent receives a reward of one and is placed on the next, randomly generated floor of the tower. Starting from level five the agent needs to learn picking up key which unlocks a key door and gives the agent access to further rooms that lead to the next level door. The key can be placed in level ten a new type of door is introduced which opens only after a puzzle is solved. The puzzle requires the agent to push block onto colored spot on the ground. The randomly generated floors can be illuminated in different color variations and the visual theme of the environment can vary. As overfitting to specific color values in the input or floor layouts is not useful, this makes it important for the agent to learn general and stable representations.
3D environment using sparse reward signal. The
of the rooms the ground or static or moving platform. Starting from
Agent Structure
Agent Observations Visual
WILK
Visual
Incur
Old Fribe- DESRA Cp
25C
Villinn
Loit
Adha Taggs
Vector
d o
Kilus
Figure 1: (Left) Network structure of the deep reinforcement learning agent. The environment and everything coming from (Yellow). The deep neural network, optimized with gradient descent, facilitated by PPO (Blue). Parts of the PPO algorithm used to optimize the neural network (Or- ange). (Right) Agent observations of one frame, one can see a normal door to the right of the agent. The agent receives visual and vector observations at each frame. Visual observations are of size 168x168x3. Vector observations are of size 8.
The agent observations are collected from a third person view RGB camera. Additionally, a small vector observation is provided indicating the number of keys the agent is holding as well as the time remaining and the current level. Rewards are very sparse as reward (of value 1) is only received when walking through final level door or when picking up key. A small reward of value 0.1 given when walking through normal doors and when picking up small blue orbs that provide additional time. One episode ends when the agent runs out of time, which means that the better the agent gets the longer exploring the tower is possible as more time-orbs can be collected and extra time received by going through level doors. The actions of the agent four action branches. One for moving forward or backwards, one to control the camera rotation, one for jumping and one for moving left right. The distribution of these actions in a trained agent is show in the appendix (figure 10
discrete and divided into
3

---

Under review as a conference paper at ICLR 2020
3.2 AGENT PERFORMANCE
Inference Run Agent Performance
After training the agent for 30 million steps us- ing the parameters specified in the appendix A.1 level is reached on average. As can be expected from the network structure, the agent never exceeds level ten. The network structure used here incorporates no concept of time such that the agent is unable to solve the puzzles in- troduced at level 10 as they require some more elaborate planning of long action sequence.
Level
Value Estimate
Reward
Figure 2 shows the agent performance during one inference run. This particular run lasted 4000 frames which means the agent saw 4000 observations and performed 4000 actions. As the rewards are very sparse, the agent only re- ceived reward (1 or 0.1) in 76 of these frames which is about 1.9% of all frames (only 15 of those are the full reward of one, the other 61 frames contain 0.1 reward). One can see how the value estimate, which expresses the reward the agent expects in the future, drops off significantly after the agent reaches level as the agent does not expect to solve the puzzle and receive any more rewards. One can also observe how the value estimate rises in the frames leading up to the agent entering new level. This indicates that the agent recognizes the door to the next level and already anticipates the upcoming reward.
Steps
Figure 2: Statistics of one inference run through
the tower. The agent reaches level 10 (blue curve) and the run ends when the time up, here after 4000 frames.
The results of the network trained the embodied setting will be compared to an untrained network (called random network) well the results from autoencoder. The autoencoder has the exact same structure up to the encoding layer as the trained and random network (see Appendix, figure (8. The autoencoder is trained between the network input and output. This training setup separates the factor of embodiment the training without introducing semantic labels. In figure Iguuee[sooe example input, outputs and activations in the encoding layer are shown. When comparing the hidden layer activations of the three networks the same 4000 observations of agent run are used.
classic frame reconstruction task where its loss is the difference
3.3 SPARSE ACTIVATION PATTERNS
Visual Encoding
We will take look at how the activations in the hidden layer the agent network look like") Figure in how many of the 4000 frames of one run each of the 256 neurons in the vi- sual encoding were active. For visualization purposes, the 256x1 vector was reshaped into grid pattern. The location of neuron in this grid has no further meaning.
100%
Trained Agent
-80%
Active in % of Frames
Autoencoder
-60%
In the visual encoding only 4.64% (mean= 11.88, min=3 max=35 var=17.891) of the neu- rons are activated each frame, making very sparse activation pattern. Over the in- vestigated total time, 173 of the 256 neurons (=67.58%) are active in at least 1 frame, but only them (=2.73%) are active in more than 40% of the frames. The most active neuron of the visual embedding is active 74.15% 15% of frames. This shows that there is wide vari- cty in the activation patterns. The agent utilizes most but not all of the available neurons in the
40%
Random Network
20%
Figure 3: Activation pattern comparison between trained agent (top), trained autoencoder (mid- dle) and random network (bottom). Colors indi- cate the percentage of frames in which the neuron in the encoded state is active (activation>-O).
You can find a frame by frame visualization of the activations as well as other interactive displays of our results and code here:

---

Under review as a conference paper at ICLR 2020
visual embedding some point but only activates small part in each frame. In the random net- work, there are on average 130 active neurons per frame in the visual embedding (min= 12 max=152 var=27.352), which is much denser activity (50.62%). The trained autoencoder has average of 251 active neurons in each frame (min=227 max=254 var=15.994) and only two the neurons are never active. Thus, in comparison the trained agent has learned very sparse representation of the input using selective spikes in activation compared to the continuous activations in the autoencoder encoding information with varying activation strengths.
The sparse representations in the trained agent match observations of sparse encodings for sensory input in insects (Perez- Orive al. 2002 Laurent 2002) as well as in the mammalian brain (Young & Yamane [1922] Brecht & Sakmann (2002). .. These form efficient and stable representations of high dimensional sensory input (Olshausen & Field 2004). The agent picks up on this strategy to effi- ciently encode input without any explicit regularization being applied. Even though there is no cost associated to using more neurons than needed to encode information, the agent learns to use sparse activation patterns and even leaves some of the available neurons completely unused. The trained agent seems to discover on its own that a sparse representation of the high dimensional image input more robust and stable to noise (Ahmad & Scheinkman, 2019) which helps make better action decisions based on this sparse and selective representation. The training setup of the autoencoder en- forces different,
dense way of representating information which of advantage to optimize its objective but has less resemblance with what we observe in nature.
3.4 DISTINCT ACTIVATION PATTERNS
Correlation in 20 Frame Window Around Action for Action Forward, Camera Left, Turn Right
Trained Agent
Autoencoder
Random Network
Cluster
Correlation Coeffiiiet
0.25
0.00
-025
-0.50
-0.7
-1.00
10-10
10-10
Sliding Window Offset [Frames]
Sliding Window Offset [Frames]
Sliding Window Offset [Frames]
Avg. Correlation in 10 Frame Window Around Action with Cluster
Trained Agent
Autoencoder
Random Network
0.4
Action Combination
Fonoed, Tim Right
0.3
Foraad, Camaro Tam Right
Furwad, Camoro
Correlation
0.2
Farwar,
Camea Right, light
Everything
0.1
-0.1
2
0
2
1
Cluster
Cluster
Cluster
Figure 4: Correlation between the cluster assignment of frames and corresponding action combi- nations. Clustering performed on the visual embedding of a trained agent (left), an autoencoder (middle) and random network (right). (Top) Example of correlations for action combination "For- ward + Turn Right' with clusters within a 20-frame window. (Bottom) Bar height represents the average correlation in 10-frame window around O in the top plots for all actions and cluster. It shows distinct correlations between clusters and actions in the trained agent.
5

---

Under review as a conference paper at ICLR 2020
To find out if we can discover general meaningful structure in the activations of the hidden layer, we first perform k-means clustering of their activation patterns. The time series of 4000 data points, each 256 dimensional vector, are clustered into six cluste
Ideally these clusters should group the encodings into meaningful and distinct classes. In order to test this, we now correlate the six clusters with the six most common action combinations As actions and the visual execution of them do not always match up exactly (i.e. after pressing the jump button the agent is in the air for several frames and only reaches the highest point several frames after the action was selected) we perform the correlations for a 20-frame window. The vectors which are being correlated are both binary. For the six clusters, the binary vector encodes for each frame if it belongs to a specific cluster and for the actions the vector encodes for each frame if it belongs to a specific action combination or not. This means that the correlation values represent the correlation of two binary vectors of length 4000. For the offset correlations, we shift the action vector either to the left or the right such that the cluster assignment at frame now matches up with action at frame t+1 or t-1 respectively. This gives the correlation between cluster assignments and actions and therefore informs us if there is structure in the visual encoding that correlates with the actions selected.
In figure 4 on the top left one can see the corre- lations of the action Forward + Turn Right with the encodings of the six clusters. The highest magnitude of correlation here is zero offset. However, also the cluster association of the ob- servations a few frames before and after show an increase in correlation/negative correlation.
Sum of Squares of Correlations Between Actions and Cluster
Sum of Squares of Cluster Correlations
Trained Agent
0.12
Autoen.coder
Random Network
0.10
When comparing this with the encodings of an autoencoder or the random untrained network one can see that there a clear association be- tween the learned image encoding and the ac- tions. The trained agent has with correlation between -0.36 and 0.8 a much bigger range than the autoencoder (-0.14, 0.2) and the random network (-0.12 0.14). These stronger posi- tive and negative correlations show that frames that are assigned to one cluster are more or less likely to be associated with a certain action. As the clusters are created and assigned based on the activations in the visual encoding, this means that there is a connection between distinct ac- tivation patterns and actions. Also, the activation patterns in the embodied agent preceding and following an action contain some information about it. The same correlation increase of clusters in a trained agent can be seen when looking at the correlation of clusters with semantic image content such as level doors (see figure 6). These results show that the learned representations of the visual input encode semantically meaningful and action relevant information.
2.06
LLLL
2.04
0.02
3.00
3
Cluster
Figure 5: Sum of squares of the correlations of action combinations for each cluster. One can see that the trained agent has more expressive and dis- tinct correlations with actions in the different clus- ters than a random network or the autoencoder.
The bottom part of figure 4 shows the average correlation in a 10-frame window for all six action combinations. One can see that every cluster has a unique combination of correlations or anti- correlations with the different action combinations. Some action combinations have a specific cluster which seems to mostly represent this action. In the random network in contrast, we can not find correlations of high magnitude or distinctive features between the cluster. The autoencoder has some clusters with higher correlations. However, when looking at the the correlations over time one can see that these are more long time correlations as opposed the precisely times correlations in the embodied agent. This may be a side effect of the trained agents policy who for example tends to perform the action 'Forward, Camera Left, Turn Right' (grey) more often in puzzle rooms which might be represented uniquely in the autoencoder due to the purple colors but isn' 't necessarily any kind of action encoding. Due to the more long term correlations this side effect seems be the more likely explanation.
"The choice for the number of cluster was made after comparing the inter cluster variance and silhouette score for different numbers of cluster (see appendix, figure TT)
Forr the selection of action combinations and their distribution see appendix (figure 10 right)

---

Under review as a conference paper at ICLR 2020
Figure makes this difference in correlation and therefore meaningfulness of clusters more apparent. When calculating the sum of squares of the correlations in each cluster the trained agent outperforms the random network in every cluster and the the autoencoder only marginally outperforms the trained agent in one of the clusters. The overall sum of squared correlations for the trained agent (0.41) is much higher than the one for the autoencoder (0.13) and the random network (0.07). This shows that the learned encoding has a structure which correlates with the actions as well as the image content which is impressive given the dimensionality of the observations (~84,000).
Correlation with Level Doors in the Visual Input
Trained Agent
Autoencader
Random Network
1.60
Cluster
Correittoo Cosmicent
9.50
9.00
3.22
-1.7
-10
10-10
10
Sliding Window Offset [Frames]
Sliding Window Offset [Frames]
Sliding Window Offset [Frames]
Figure 6: Correlation of clusters with Doors. Comparison of trained agent (max=0.42, min=-0.11), autoencoder (max=0. 15, mi0=-0.09) and random network (max=0.09, min=.0.08).
3.5 CONCEPTUAL SIMILARITIES, GENERALIZATION AND ROBUSTNESS
To visualize the encodings and to investigate how conceptually similar inputs are represented, we project the activations in the visual part of the embedding layer into two-dimensional space using t-SNE (Maaten & Hinton 2008). Figure shows the 4000 encoding activations projected into this 2D space, colored by the corresponding action combination Additionally, all data points where the visual observation contained a level door are circled in red. This makes it possible to look at the spatial arrangement of encodings in high dimensional space with respect to semantic and action- oriented content.
Even in the rroddmmensioaa projection of the data, one can very good separation between points associated with the different actions. Also, the frames showing level doors tend to be posi- tioned close to each other within their respective action cluster even though they show doors under very different illumination conditions (see the three example pictures in the top left of figure (7). As the network's task not to recognize doors or other objects. but navigate in a 3D world, it is important to encode the visual information in this way. A door in the right part of the frame needs to be encoded differently than a door in the center or left part of the frame. However, two doors in the right part of the frame under different illumination should be encoded very similarly. This meaningful and action relevant way of encoding the input can be seen in figure 7 shows that in the visual encoding of the input conceptually similar images are positioned close to each other, giving low importance to perceptual similarities. This means that the network encodes the input in an action-oriented way (Clark) 1998) and is rather invariant towards irrelevant parts of the input such as illumination or texture.
To investigate if conceptually similar input images also lay close to each other in high dimensional space, we can calculate the inter-class distance and variance of encodings associated with the differ- ent action combinations. Table [ lists the distances and variances between encodings belonging to the same action combination divided by the overall distances and variance in the data.
In the trained agent, both distance and variance reduce strongly between the first four action combi- nations (0.78 and 0.58 respectively). In the autoencoder and the random network, there is no change
1 For an interactive visualization explore the observations associated with each point see here As control that the structure does not simply originate from the statistics of the input images, t-SNE on the activations of the autoencoder and the untrained agent on the same input images are shown in figure 12 in the appendix.
7

---

Under review as a conference paper at ICLR 2020
T-SNE on the Visual Embedding
Trained Agent
1310
Action Combination
Forward, Right
Forward, Camera Left, Tum Right
Forward, Camera Left, Tum Left
Forward, Camera Right, Turn Right Right, Right
Everything Else
Observation Content
Level
1197
Figure 7: T-SNE performed the visual embedding of trained agent colored by action com- bination associated with each frame. Images show example agent observations which created the encoding activation associated with the point it is connected to. Encodings associated with frames containing level doors are circled in red.
in the distance variance when comparing all data points points belonging to action combi- nation. The last two action combinations which represent backwards motion and all other rare action combinations actually have increased variance in the trained agent. This may be due to a very variable use of backwards motion when seeing possibly confusing visual input and the accumula- tion of multiple action combinations (also including jumping) the last action category. However, we can see that at least for the first four action combinations the encodings of conceptually similar input frames are also closer together in the high dimensional space (256 dimensions) of the visual encoding. Also when comparing the absolute change in variance and distance for each action cluster (last row table [ the trained agent outperforms the two other networks strongly indicating a very distinct action encoding in the network trained in an embodied setup.
Average distance and variance within points belonging to the same action combination divided by the overall distance or variance between all encodings respectively.
Table 1:
Trained Agent
Autoencoder
Random Network
Fraction of overall dist/var
Distance
Variance
Distance
Variance
Distance
Variance
Average
0.96
0.94
1.06
1.12
1.07
1.16
Action 1-4
0.78
0.58
1.02
1.04
1.03
1.06
Action 5-6
1.32
1,66
1.12
1.28
1.15
1.35
Mcan(Abs(I-de) (I-vv///v a))
0.25
0.5
0.09
0.18
0.1
0.23
CONCLUSION
The results presented in this paper show that a neural network, trained in embodied framework, can learn stable and meaningful representations of its high dimensional input. Compared to an autoencoder, the representations of the embodied agent are encoded in a sparse and efficient way and better reflect information encoding found in animals. The information encoded in the latent representation of the network is mainly action focused, but also contains general concepts of action relevant objects such doors, disregarding irrelevant information such illumination. Overall these results suggest deep reinforcement learning as a promising method for investigating stable representation learning similar to what is known from biological findings.
8

---

Under review as a conference paper at ICLR 2020
AUTHOR CONTRIBUTIONS
X.X. conceived the idea investigate latent representations in deep reinforcement learning agents, trained the agents, designed the experiments, analyzed the data and took the lead in writing the manuscript. All authors discussed the experiment design and results and contributed the final manuscript. X.X, X.X. and X.X. supervised the project.
ACKNOWLEDGMENTS
The project financed by the funds of a research training group provided by the XXX (ID XXX).
REFERENCES
Subutai Ahmad and Luiz Scheinkman. How can we be so dense? the benefits of using highly sparse
representations. CoRR, abs/1903.11225 1257, 2019. URL http::/aaxiv..om httgg//bs/1133
11257
Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip J. Kellman. Deep convolutional networks do not classify based on global object shape. PLOS Computational Biology. 14(12):
1-43, 12 2018. doi: Ijouralliau......bbb. 100.611....163 URL
http://cc...rgg//// 10.711
journal pobi 1006613
Sven Bambach, David Crandall, Linda Smith, and Chen Yu. Toddler-inspired visual ob- ject learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp- 1201-1210. Curran Associates, Inc., 2018. URL http::/pappe:: nips. cc/paper/ oo
pdf
Michael Brecht and Bert Sakmann. Dynamic representation of whisker deflection by synaptic po- tentials in spiny stellate and pyramidal cells in the barrels and septa of layer 4 rat somatosen- sory cortex. The Journal of Physiology, 543(1):49-70, 2002. doi: 10. 10.11113jphysiol.2002
018465. https physoo..///pppppppcpp/aosssccc www.eennnneeeee lbbrarr wi wiley wwt.c//wccccccccccocccoo wtt.e//cc..ooooa////////0/111111 jphysiol 2002 018465
Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh, and Dhruv Batra. Embodied multimodal multitask learning. CoRR, abs/1902.01385, 2019. URL http:://rxiv. orr/
abs/ 3bbs/1902.0 01385
Andy Clark. Being there: Putting brain, body, and world together again. MIT press, 1998.
T. de Bruin, J. Kober, K. Tuyls, and R. Babuka. Integrating state representation learning into deep reinforcement learning. IEEE Robotics and Automation Letters, 33)):19941141,, July 2018. doi:
10.110991RA.2008,280000
Emily Denton and Vighnesh Birodkar. Unsupervised learning disentangled representations from 1l051111119915 10915
video. CoRR, Abs/1705,.0915. 2017.
Andreas K. Engel, Alexander Maye, Martin Kurthen, and Peter König. Where's the action? the pragmatic turn in cognitive science. Trends in Cognitive Sciences, 17:202-209, 2013.
M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 360 1210, 2018. ISSN 0036-8075, doi: 10.1126/science.aard 6394/1204
URL /ttpttpsss//cceeeeecccceeeeeeee /tt.a///too.ooooooooootot/////
Chelsea Finn, Ian J. Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. CoRR, abs/1605.07157, 2016. URL /ttt:///axivy org/ abs/ 1605. 071557

---

Under review as a conference paper at ICLR 2020
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 1-50-2462, 2018.
Kuan Han, Haiguang Wen, Yizhen Zhang. Di Fu, Eugenio Culurciello, and Zhongming Liu. Deep predictive coding network with local recurrent processing for object recognition. CoRR, abs/1 abs/1805.015 805.07526, 2018. htt:tt www.v..ivvvv..///vv///.xa wix. org/ abs 805. 07526
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CoRR. abs/1907.07774, 2019. CtLhttt::// http::/aaxiiiie org abs abtpa/as./199 07174
Arthur Juliani, Vincent-Pierra Berges, Esh Vekay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. CoRR, abs/1809.02627, 2018.
URL http://arxiv org wwtggabs//89...00 02627
Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vi- sion, control, and planning. CoRR, abs/1902.01137, 2019. URL httt:://aaxii..oot abs/
1902. 01378
Konrad P. Körding and Peter König. Neurons with two sites of synaptic integration learn invariant representations. Neural Computation, 13012)2823-2249, 2001. doi: 10.1 189976601317098547,,
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial exampees in the physical world.
CoRR, abs/1607.02533, 2016. URL htttp:://axxd...og//ab..s
Gilles Laurent. Olfactory network dynamics and the coding of multidimensional signals. Nature Reviews Neuroscience, 3:884 895, 2002.
Timothée Lesort, Natalia Díaz Rodríguez, Jean-François Goudou, and David Filliat. State rep- resentation learning for control: An overview. CoRR, abs/1802.04181, 2018. URL http:
//arxiv.orr//bb/11002 04181
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:/509.02977 2015.
William Lotter, Gabriel Kreiman, and David D. Cox. Unsupervised learning of visual structure using predictive generative networks. CoRR, abs/1511.003800, 2015. URL http: /tttp://arxiv.
arg/abs/1511. 06380
William Lotter, Gabriel Kreiman, and David D. Cox. Deep predictive coding networks for video
prediction and unsupervised learning. CoRR, abs/1605.08104, 2016. URL http://arxiv.
orq/abs/1605.00 08104
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov):2579-2605, 2008
Reza Mahjourian, Martin Wicke, and Anelia Angelova. Geometry-based next frame prediction from monocular video. CoRR, abs/1600.063777, 2016. URL http://arriiv.or//ab/////0
06377
Ofrr Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation leaning for hierarchical reinforcement learning. CoRR, abs/1810.01257, 2018. URL httthttpp//axxii
org aa////ba///110 01257
Alva Noč. Action in Perception. MIT Press, 2005.
Mare Oliu, Javier Selva, and Sergio Escalera. Folded recurrent neural networks for future video prediction. CoRR, abs/1712.00311. 2017. URL htllhttp://axiiv http:/vv..ggaabb//b.eeee 171110011111111111
Bruno Olshausen and David Field. Sparse coding of sensory inputs. Current opinion in neuro- biology, 14(4):481-487, 2004.
10

---

Under review as a conference paper at ICLR 2020
J. Kevin O'Regan and Alva Noë. What it is like to see: A sensorimotor theory of perceptual expe-
rience. Synthese, 129(1):79-103, Oct 2001. ISSN 1573-0964. doi: 10.1023/A:1011669944777.
Kevin O RRgann and Alva No. A sensorimotor account of vision and visual consciousness. Behav- ioral and Brain Sciences, 24(5):939973, 2001. doi:
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. CoRR, abs/1705.05336 2017. URL http://arxiv..o /bs//
Viorica Patraucean, Ankur Handa, and Roberto Cipolla. Spatio-temporal video autoencoder with differentiable memory. CoRR, abs/1511.06309, 2015. URL http://arxiv. wtt.rxiiiorgg.a///ooeoooo
1511. 06309
Javier Perez-Orive, Ofer Mazor, Glenn C. Turner, Stijn Cassenaer, Rachel Wilson, and Gilles Laurent. Oscillations and sparsening of odor representations in the mushroom body. Science, 29765580) 365, 2002. ISSN 0036-8075. doi: 10.1 10.1/c//ceecce. 00.116//ceence.1000055222 1070502. //science.scienneeee.......eeeee wwwpee/eweem..ooooooooeeeeetetee hatp:/coteette
URL https:
Dennis R. Proffitt. Embodied perception and the economy action. Perspectives on Psychological Science, 1(2): 10-112, 2006. doi: 10. 10.1111/j. 01.5--616.200000000
URL https://doi.
org/10.1111/ 745- 6916 2006. 00008.
x PMID: 26151466.
Friedemann Pulvermüller and Luciano Fadiga. Active perception: Sensorimotor circuits cortical basis for language. Nature reviews. Neuroscience, 1 11:351-60, 04 2010. doi: 10. 10.1888/nm281
Huihui Qiao, Taiyong Wang, Peng Wang, Shibin Qiao, and Lan Zhang. A time-distributed spa- tiotemporal feature learning method for machine health monitoring with multi-sensor time se- ries. Sensors, 18(9). 2018. ISSN 1424-8220. doi: 10.3390/s8002293.. URL https:
7/www /www. wdw.. com 02222422222/11///22
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpreta- tion of some extra-classical receptive-field fiell effects. Nature neuroscience, 2(1):79, 1999.
Dan Rosenbaum, Frederic Besse, Fabio Viola, Danilo J. Rezende, and S. M. Ali Eslami. Learning models for visual 3d localization with implicit mapping. CoRR, abs/1807.05 04149, 2018. URL
httt:://axixs org/ abs 8078807.00 03149
Larissa K. Samuelson and Linda B. Smith. They call it like they see it: spontaneous naming and attention shape. Developmental Science, 8(2):182 198, 2005. doi: 10. 10.1111/..466-7668. 2005.00405.x. URL hattp:://onnnneebrraayy
1467-7687 2005 00405.x
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 006407,,7 2017. URL http://arxiv.oro//
1707. 06347
Wenling Shang, Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Learning world graphs to accelerate hierarchical reinforcement learning. CoRR, 00040000,,, 2019.
URL http::ttttt/t/tt org/ abs 1907. 00664
Linda B. Smith and Lauren K. Slone. A developmental approach to machine learning? Frontiers in Psychology, 8:2124, 2017. ISSN 1664-1078 doi: 10.3889/fssy.2217700222 URL https:
www. front ieeteteriiii www.rrrraieerrriio.o.rraaaaaaaaa /ttp:///ra..aee...333333////2220
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised leaming of video
representations using Istms. CoRR, abs/1502.04681, 2015. URI http:///rxxv..rrg/eee
1502. 04681
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6/2226.. /999 2013.

---

Under review as a conference paper at ICLR 2020
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based rep- resentation learning. CoRR, htbs/1812.0509,, 2018.
05069
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1807.00 087700774. 2018. htttt:://axxi..or//gaaaa
Ruben Villegas, Jimei Yang. Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. CoRR, abs/1 00111706....33 2017. URL http:
//arxiv.org/abs//1 00000000010
Haiguang Wen, Kuan Han, Junxing Shi, Yizhen Zhang, Eugenio Culurciello, and Zhongming Liu. Deep predictive coding network for object recognition. CoRR, abs/1802.04762, 2018. URL
abs 04762
Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: iea of invari-
ances. Neural Computation, 14(4):715-770, 2002. doi: 10. 10.1620899760233311998 URL
pettps:/ddoi.ooe http:///......oo......./////7712
Jessica K. Witt. Actions effect on perception. Current Directions Psychological Science, 20
(3):001-206, 2011. doi: 10.1177096372114114887777 UtL/atpps:://e...cc//011117777 0963721411408770
MP Young and Yamane. Sparse population coding of faces in the inferotepporal cortex. Science,
256(5061):1327-133,, 1992. ISSN 0036-8075. doi: 10.1126/science.
11t.22//sccennc.5.55777 1,595777. URL https:
llscience.scienceeee http://aa....goooootoootoottee ottp://cwc...tooooooooooooooo///
A APPENDIX
A.1
TRAINING SPECIFICS
Batch Size
256
Beta
5.0e-3
Buffer Size
1024
Epsilon
0.2
Gamma
0.999
Lambd
0.9
Learning Rate
1.0c-4
Normalize
True
Use Recurrency
False
# Parallel Environments
16
12

---

Under review as a conference paper at ICLR 2020
A.2
AUTOENCODER
Auto Encoder Structure
Encoded State
Dense 12800
Dense 256
Dense 256
Visual Input
Conv
15
Conv
32
ST e300
gnu
Deco
Visual Input
Loss (MSE)
Figure 8: Network structure of the autoencoder.
Input
Output
Encod mg
Inpult
Outpat
ng
Imput
Output
Encod mg
Figure 9: Autoencoder example input, encoding and output.
13

---

Under review as a conference paper at ICLR 2020
A.3 ACTIONS
Distribution of Actions
Move Forward/Back
Camera
Distribution
9530
3030
1234
Framin
1590
922
602
435
340
Stand
Forward
Back
Straight
Left
Right
Actien Comberstioo
Jump
Move LefU/Right
0030
4030
Rarns
2500
2520
2030
1530
4.0
500
Agght
Stand
Jump
Stand
Right
Left
Taclua Condenatoon
Figure 10: Distribution of actions over one inference run of 4000 frames. (Left) Distribution in the four action branches. (Top Right) Distribution of existing action combinations. (Bottom Right) Distribution of all action combinations used in the paper. All actions marked in grey in the plot above are aggregated the last action combination.
14

---

Under review as a conference paper at ICLR 2020
A.4
K-MEANS CLUSTERINGS
Mean Within Class Variance Comparison
Silhouette Score Comparison
Mean Within Class Variance
0.24
28
Silhouette Score
0.22
26
0.18
22
20
0.12
# of Cluster
of Cluster
Distribution of Frames in Cluster
Within Class Variance
1009
Vatience
30-
Variance
25-
600
15-
#
10-
5-
Cluster
Cluster
Figure 11: (Top) Analysis used for the choice of number of clusters in section 3.4 Both mean within class variance well as the silhouette score for different number of clusters have their minimum at six clusters. (Bottom) Statistics of the k-means clustering with six clusters.
15

---

Under review as a conference paper at ICLR 2020
A.5 T-SNE
T-SN the Visua Embedding Actoensader
T-KE Embeddiing
Random Ageni
Candl Now Tentiigs
Cortinaton
Oharvele
Oberrvtion Contert
Figure 12: T-SNE performed on the visual embedding of trained autoencoder (left) and ran- dom network (right) colored by action combination associated with each frame. Frames where observation contained level door are circled red.
T-SNE Visual Embedding
T-SNE on the Visual Embedding Autoencader
Trained Agent
Feage Eonteve
Cantert
vothing
horm 1ooo
Pastle Doo
Figure 13: T-SNE performed on the visual embedding of trained Agent (left) and trained autoen- coder (right) colored by semantic content of the visual observation associated with the encoding.
A.6 ANALYSIS OF HIGH-D SPACE
Varlance Encodings Corresponding to one
Distance Within Encodings Corresponding to one Action
Trained
Tratred Agam
Autorendarr
karden Retwnik
2.0
2.75
Figure 14: Distances and variances within the encodings for individual actions in the original high dimensions encoding space (dim=256). Data numbers also given in accumulated over actions.
16


Reviews:
[{"review_id": "rkgKdL1RtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\n\nThis is an empirical study of the representations learned by a reinforcement learning agent. An agent is trained, using a standard RL algorithm, to solve puzzles by navigating through a 3D visual environment (Unity obstacle tower challenge). The analyses in the paper show that the visual representations of the trained agents are sparse and cluster according to the actions performed by the agent. The goal of the paper is to show that these features are due to the embodied nature of the agent. Specifically, the paper states that “the quality of the representations learned shows the strength of embodied learning and its advantages over fully supervised approaches with regards to robustness and generalizability”.\n\nDecision:\n\nI suggest to reject this paper. While the topic is interesting and the paper is clearly written, there is a lack of control/comparison experiments, such that the paper’s conclusions are not backed up by the analyses. However, with more experiments, I think this line of research has large potential.\n\nFurther justification for the decision:\n\nMy main criticism is that the paper claims to show that embodiment is important for representation learning, but never actually compares representations learned by an embodied agent to representations learned in some other way.\n\nComparing to a random network is a sanity check, but not sufficient to support the paper’s claims.\n\nOne way to experimentally dissociate embodiment/agency from supervised learning would be to train two separate models, one that is “embodied”/active, and another that gets the same sensory input, but without embodiment (“passive”). The passive network could be trained using the sensory inputs recorded while training the active network. The passive network could be trained to predict the value and/or the actions of the active network. Thus, the passive network would be trained in a supervised way, whereas the active network would be trained by RL (i.e. in an embodied way).\n\nThe representations of the two networks could then be compared using the analyses used in the paper. This setup would experimentally isolate the effect of embodiment. Without such comparisons, it is unclear whether representations learned in a supervised way would be any different from those learned by RL.\n\nIn addition to the descriptive analyses presented in the paper, a transfer learning approach could test whether there is actually a functional difference between the “embodied” and the supervised representations: Take the representations of the “active agent” and the “passive agent” and freeze the weights. Then re-initialize and re-train only the dense layer before the action probabilities on the RL task, leaving everything else frozen. Does the model from the “active agent” do better on the RL task? This would suggest that the embodied agent learned better representations.\n\nMinor comment: The website contains a link to a YouTube profile that is not completely anonymous (it contains the first name and a profile photo). While I did not identify the authors when visiting the paper website, I recommend removing links to personal YouTube profiles and create submission-specific anonymous accounts.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJgrTBn1KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nPaper Summary: The goal of the paper is to analyze what information is encoded in the representation learned using RL for a specific game. The paper shows that the activations are sparse and the activation patterns are distinct and shows that the conceptually similar images are clustered together in t-sne visualization.\n\nPaper Strengths:\n\nThe paper starts with a nice introduction that embodiment is useful for perception. However, the main content of the paper is very different from the introduction.\n\nPaper Weaknesses:\n\nThe conclusions of the paper are either already known or very trivial. So there is nothing new for the community to benefit from. Please refer to my comments below for more information.\n\nThe conclusion of section 3.3 is that \"the agent learns to use sparse activation patterns and even leaves some of the available neurons completely unused\". This has nothing to do with embodiment. The same patterns are observed in non-dynamic tasks such as image classification. The CNNs are usually over-parameterized.\n\nThe conclusion of section 3.4 is that \"When comparing this with the encodings of a random untrained agent one can see that there is a clear association between the learned image encoding and the actions\". That is the whole point of training. We train the models to find correlations between actions and observations. It is obvious that there is more correlation compared to a random agent.\n\nIt is shown in section 3.5 that similar images will be next to each other in the t-sne visualization. It is obvious that this happens.\n\nDue to the issues mentioned above, I do not think there is anything new in the paper and I vote for rejection.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hylx6OChtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work builds on the embodied cognition literature, hypothesizing that representations learned in embodied agents will be of improved “quality” compared to non-embodied models, such as neural networks trained on static supervised datasets. \n\nThe authors provide an excellent motivation to the work, with the introduction nicely laying the groundwork for their hypothesis. In general the motivation is quite strong, and research along these directions will no doubt be of value to the field.\n\nTo assess their hypothesis, the authors compare the representations learned in trained vs. random agents on the Unity Obstacle Tower Challenge, and demonstrate that trained agents develop semantically meaningful, sparse representations, without explicit regularization. \n\nUnfortunately, the work doesn’t provide adequate baselines to properly assess the hypothesis. With the data presented, we can only make claims about *trained* vs. *untrained* agents (both of which are embodied!). In other words, an embodied agent with a random policy is not equivalent to a non-embodied agent. Thus, the data only support the conclusion that performance and task-relevant policies drive good representation learning in embodied agents, which is altogether not surprising, as one wouldn’t expect representations to be good in a randomly initialized network. \n\nTo assess wither *embodiment* is a critical factor for driving good representations, the authors should compare to a model that learns from a static supervised dataset. Curiously, this idea is alluded to in the introduction, but not followed up on.\n\nOverall, there are some nicely presented ideas but the work is unfortunately incomplete, and the results cannot support the hypothesis laid out.\n\nAs a final note, the authors are encouraged to remove any assignments of gender to the agent (“he/him” is unnecessarily used throughout).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: What is investigated is what kind of representations are formed by embodied agents; it is argued that these are different than from non-embodied arguments. This is an interesting question related to foundational AI and Alife questions, such as the symbol grounding problem. Unfortunately, the empirical investigations are insufficient. In particular, there is no comparison with a non-embodied control condition. The reviewers point this out, and the authors propose a different control condition, which unfortunately is not sufficient to test the hypothesis.

This paper should be rejected in its current form, but the question is interesting and hopefully the authors will do the missing experiments and submit a new version of the paper.
Decision: Reject