Unified neural representation model for physical and conceptual spaces
Authors: Tatsuya Haga
Yohei Oseki
Tomoki Fukai
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neuroscience
Grid cell
Concept cell
Spatial navigation
Reinforcement learning
Word embedding

Abstract: The spatial processing system of the brain uses grid-like neural representations (grid cells) for supporting vector-based navigation. Experiments also suggest that neural representations for concepts (concept cells) exist in the human brain, and conceptual inference relies on navigation in conceptual spaces. We propose a unified model called ``disentangled successor information (DSI)'' that explains neural representations for both physical and conceptual spaces. DSI generates grid-like representations in a 2-dimensional space that highly resemble those observed in the brain. Moreover, the same model creates concept-specific representations from linguistic inputs, corresponding to concept cells. Mathematically, DSI vectors approximate value functions for navigation and word vectors obtained by word embedding methods, thus enabling both spatial navigation and conceptual inference based on vector-based calculation. Our results suggest that a single principle can explain computation of physical and conceptual spaces in the human brain.

Under review as a conference paper at ICLR 2023
UNIFIED NEURAL REPRESENTATION MODEL FOR PHYSICAL SPACE AND LINGUISTIC CONCEPTS
Anonymous authors Paper under double-blind review
ABSTRACT
The spatial processing system of the brain uses grid-like neural representations (grid cells) for supporting rector-based navigation. Experiments also suggest that neural representations for concepts (concept cells) exist in the human brain, and conceptual inference relies on navigation conceptual spaces. We propose a uni- fied model called "disentangled successor information (DSI)" that explains neural representations for physical space and linguistic concepts. DSI generates grid-like representations in 2-dimensional space that highly resemble those observed in the brain. Moreover, the same model creates concept-specific representations from linguistic inputs, corresponding to concept cells. Mathematically, DSI vectors ap- proximate value functions for navigation and word vectors obtained by word em- bedding methods, thus enabling both spatial navigation and conceptual inference based on vector-based calculation. Our results suggest that representations for space and concepts can emerge from a shared mechanism in the human brain.
INTRODUCTION
In the brain, grid cells in the entorhinal cortex (EC) represent the space by grid-like representa- tions (Hafting et al., 20055: Doeller et al. 2010 Jacobs et al., 2013). This neural representation is often related vector-based spatial navigation because grid cells provide global metric over the space. Theoretically, an animal can estimate the direction to a goal when representations of cur- rent position and a goal position are given (Fiete et al., 2008: Bush al.] 2015). Furthermore, self-position can be estimated by integrating self-motions when sensory information is not available (McNaughton al. 2006). These functions are the basis of robust spatial navigation by animals.
There are not only spatial but also conceptual representations in EC. Neurons called "concept cells" have been found in human medial temporal lobe including EC (Quiroga 2012] Reber al. 2019). Concept cells respond to specific concepts, namely, stimuli related to specific person, famous place, or a specific category like "foods" and "clothes". Furthermore, recent experiments also suggest that grid-like representations appear not only for physical space but also for conceptual space there is 2-dimensional structure (c.g. lengths of a neck and legs, intensity two odors), and those representations are the basis of vector-based conceptual inference (Bao et al. 2019 Con- stantinescu et af. 2016: Park et al., 2021). Thus, it is expected that there is a shared processing mechanism for physical and conceptual spaces in EC. Existence of shared neural mechanism may also explain why humans use sense of physical space (such as directionality) to communicate ab- stract concepts (conceptual metaphor (Lakotf & Johnson 1980)) ). However, a principle behind such universal computation in the brain is still unclear.
In this paper, we propose representation model which we call disentangled successor information (DSI) model. DSI is an extension of successor representation (SR), which stems from a theory of reinforcement learning and became one of promising computational models of the hippocam- pus and EC (Dayan 1993, Stachenflld et al. 2017) Momennejad et al., 2017] Momennejad (220)). Like eigenvectors of SR, DSI forms grid codes 2-D space, and those representations sup- port vector-based spatial navigation because DSI approximates value functions for navigation in the framework of linear reinforcement learning [Todorov] 2006 2009 Piray & Daw 2021). Remark- ably, when apply DSI to text data by regarding sequence of words sequence of states, DSI forms concept-specific representations like concept cells. Furthermore. we show mathematical correspondence between DSI and word embedding models in natural language processing (NIP)

---

Under review as a conference paper at ICLR 2023
Mikolov et al., 2013a bj Pennington et al.] 2014 Levy & Goldberg 2014), thus can perform intuitive vector- based conceptual inference as in those models. Our model reveals a new theoretical relationship between spatial and linguistic representation learning, and suggests a hypothesis that there is a shared computational principle hehind grid-like and concept-specific representations in the hippocampal system.
2 CONTRIBUTIONS AND RELATED WORKS
We summarize contributions of this work as follows. (1) We extended SR to successor information (SI), by which we theoretically connected reinforcement learning and word embedding. thus spatial navigation and conceptual inference. (2) We found that dimension reduction with constraints for grid-like representations (decorrelative NMF) generates disentangled word vectors with concept- specific units, which has not been found previously. (3) Combining these results, that a computational model for grid cells can be extended to represent and compute linguistic con- cepts in an intuitive and biologically plausible manner, which has not been shown in previous studies.
demonstrated
Our model is an extension of successor representation (SR), which is recently viewed as a plausible model of hippocampus and EC (Dayan 1993] (Stachenfeld al., 2017: Momennejad et al.) 2017] Momennejad 2020). Furthermore, default representation (DR), which is based on linear reinforce- ment learning theory, has been also proposed model of EC (Piray & Daw 2021). We show that our model can extract linguistic concepts, which has not been shown for SR and DR. Furthermore, we demonstrate vector-based compositionality of words in our model, which expands the range of compositionality of EC representations (Piray & Daw 202T to semantic processing.
Our model produces biologically plausible grid-like representations in 2-D space, which supports spatial navigation. Previous studies have revealed that negaaivee and orthogonal constraints are important to obtain realistic grid-like representations (Dordek et al.) 2016; [Sorccher et al.) 2019). Furthermore, recurrent neural networks form grid-like representations through learning path inte- gration, and those representations support efficient spatial navigation (Banino et al.) 2018 Cueva & Wei 2018 Gao et al., 2019). Some of those models have reproduced experimentally observed scaling ratios between grid cell modules (Banino et al.) (2018; [Sorccher et al. (219). However, pre- vious models have not been applied to learning of lingunstic concepts, or other complex conceptual spaces in real-world data. Whittington et al. 2020) proposed a unified model for spatial and non- spatial cognition. However, their model was applied only to simple graph structures and conceptual specificity like our model was not observed.
Analogical inference by our model a same function word embedding methods in NLP (Mikolov ct al. 2013a by Pennington et al. 20144] Levy & Goldberg 2014). However, a unique feature of DST representations is that each dimension of vectors corresponds to a specific concept like concept cells in the human brain (Quiroga 2012 Reber et al.) 2019). Our model provides biological plau- sible interpretation of word embedding: each word is represented by combination of disentangled conceptual units, inference is recombination of those concepts, and such representations emerge through the same constraints with grid cells. was recently shown that transformer- based models Vaswani et al. 2017 Brown et al., 2020), which are currently state-of-the-art models in NLP, gen- crate giid representations when applied to spatial learning (Whittington et al. 2022). Similarly to our model, this finding implies the relationship between spatial and linguistic processing in the brain. However, concept-specific representations has not been found in such model. Furthermore, clear theoretical interpretation in this study depends on the analytical solution for skip-gram (Levy & Goldberg 2014). Such analytical solution is currently unknown for waasformer-basee models.
3 MODEL
3.1
DISENTANGLED SUCCESSOR INFORMATION
Let us assume N₂ discrete states exist in the environment. Successor representation (SR) between two states s and s' is defined as
30
SR(s.8') E
y 88(5 8 (s5
Σ
PP(s s' =s)
1=1
1=0
2

---

Under review as a conference paper at ICLR 2023
where ji,,j) is Kronecker's delta and y is discount factor. We describe how calculate SR in this study in Appendix A.1] SR and its dimension cimension-reduced reduced representations have been viewed as models of hippocampus and entorhinal cortex, respectively (Stachenfeld et al. 2017).
Based on SR, we define successor information (SI) and positive successor information (PSI) as
$I(s, log(S s))) - log(P(s')),
(SI(s, (sx (')
(3)
In this study, we regard this quantity as hippocampal model instead of SR (Figure ITA.
Next, introduce a novel dimension reduction method which we call decorrelative non-negative matrix factorization (decorrelative NMF). Decorrelative NMF can be regarded as variant of NMF (Lee Seung, 1999) with additional constraints of decorrelation. By applying decorrelatve NMF to PSI, we obtain representation vectors called as disentangled successor information (DSI), which regard as model of EC (Figure [A). decorrelative NMF, we obtain D-dimensional vectors ars)) and ((s) (D N₂) by minimization of the following objective function
Σ p(s.ss) - x(s) wsss))² 8,8"
2
+ Bcor ((Corr(i.
1
Sreg E [(1=(())12 + 1|0(())13),
(4)
2
2
iF)
subject non-negative constraints v.)) 2 0, (s(s) 0. p(s, 8') is a weight for the square
1
p(s.s') N.V
PSI(s, pmin M
(5)
where M and V are mean and variance of PSI, respectively, and Pmin is a small value to avoid zeighweight. Corr(i. is correlation between two dimensions in æ(s)
(.ú,(,),,())
Corrli,j) (E,) (i,
where (.(s) - 2, 2s). The first term of the objective function is weighted approx- imation етror minimization, the second term works for decorrelation between dimensions, and the third term regularizes representation vectors. Optimization was performed by Nesterov's acceler- ated gradient descent method (Nesterov 1983) with rectification of (s)s), (s) every iteration. We describe additional details in Appendix
A.2 A.2
3.2 RELATIONSHIPS WITH REINFORCEMENT LEARNING AND WORD EMBEDDING
We show dual interpretation of our model. On the one hand, DSI approximates value estimation of linear reinforcement learning, thus support goal-directed decision making and navigation. On the
Virtual terminal state goal
Place cells
Non-terminal states
Positive successor information (PSI;
Istructure of the envirenmeng
Dimension reduction
Grid calls
Setting
Disentangled successor information
goal
[DSI)
Figure 1: Interpretation of our model. (A) Biological interpretation of our model. PSI model of the hippocampus and DSI as a model of EC. (B) The setting of state transition map in which DSI approximates value functions for spatial navigation.

---

Under review as a conference paper at ICLR 2023
other hand, the same representation approximates word embedding in NLP, thus support semantic computation.
First, our model approximates value functions of linear reinforcement learning (Todorov 2006 2009 Piray & Daw 202T) in the setting of spatial navigation. Linear reinforcement learning as- sumes default policy and imposes additional penalty on deviation from default policy, then we can obtain value functions explicitly by solving linear equations. Let us consider a specific condition in which the environment consists of non-terminal states, and a virtual terminal state is attached to a goal state Sg arbitrarily chosen from non-terminal states (Figure BB.. When the agent gets to the goal, it transits to the terminal state with probability PNT. Furthermore, we assume that reward at non non-terminal states are uniformly negative and reward at the terminal state is positive so that the agent has to take a short path to goal to maximize reward. In this setting, we can obtaín value functions v" (s) in linear reinforcement learning as
A us) =
(c)) - log (*(G)) = (s, N0 x(s) w(sc),
(7)
where (H sc) and (1 sc) are SR and SI under the default policy, respectively. We describe details of derivation in Appendix A.3 Therefore, SI is proportional to value functions for spatial navigation and inner products of DSI vectors approximates value functions. Based on this interpre- represents temporary
tation, basically regard æsss) as a representation of each state, and wss) goal.
Second, DSI is related word embedding methods in NLP (Mikolov al. 2013a| b. Pennington
iet al. 2014: Levy & Goldberg, 2014). In linguistics, pointwise mutual information (PMI) and pos- itive pointwise mutual information (PPMI) are used to measure the degree of coincidence between two words (Levy & Goldberg 2014). They are defined as
P(word,, word,) P(word,) P(wordd,)))))
PMI log
(8)
PPMI
(9)
where P(word, word,) is a coincidence probability of two words (in a certain temporal window). It has been proven that dimension reduction of PMI approximates word embedding method skip- gram (Mikolov et al. al.] [2113 b). and similar performance is obtained using PPMI (Levy & Goldberg,
2014 GloVe (Pennington al. 2014) is also based on this perspective. SI can be written
SI(x, = (SggS((RR,))) 8)) log(P(s')) = log Ei 7 P(s(s = =0
(10)
P(s)
In this formulation, we can see mathematical similarity between PMI and SI by regarding words as states (s = word,, s' = wordy), thus the correspondence between PPMI and PSI. Because of this relationship, has similar properties to word embedding methods. The difference how to count coincidence: the coincidence in SI is evaluated with an asymmetric exponential kernel in SR, in contrast that a symmetric rectangular temporal window is often used in typical word embedding (see Appendix A.4 for further detail).
expect that DSI, which is obtained through dimension reduction of PSI,
3.3 DECORRELATIVE NMF RELATES TO GRID CELLS AND DISENTANGLEMENT
Constraints in decorrelative NMF (non (non-negativity, decorrelation (or orthogonality). and regulariza- tion) are important for generation of grid cells, as shown in previous theoretical studies on grid cells. (Dordek et al.] 2016) Cueva & 20i, Banino et al. 2018 Gao et al. 2019 Sorscher et al.) 2019). They are also biologically plausible because neural activity is basically non-negative and decorre- lation is possible through lateral inhibition. On the other hand, non-negativity (Oja & Plumbley) 2004) and decorrelation (Hyvärinen & Ojal 2000) are also important for extraction of independent components, and it is known that imposing independence in latent space of deep generative models results in the emergence of disentangled representations for visual features (Higgins et al.) 2017). Therefore, in word embedding, we expected that those constraints help emergence of independent and disentangled units for linguistic concepts. Constraints in decorrelative NMF are actually crucial for results obtained in this study (Appendix A.5
4

---

Under review as a conference paper at ICLR 2023
As disentangled visual representations explain single-cell activities in higher-order visual cortex (Higgins et al. 2021), we may similarly interpret conceptual representations in our model as con- cept cells in the human medial temporal lobe (Quiroga 2012). Previous studies suggest that each concept cell respond to a specific concept, whereas population-level activity patterns represent ab- stract semantic structures (Reber al. (019)). Such property consistent with the factorized and distributed nature of disentangled representation vectors.
4 LEARNING REPRESENTATIONS OF PHYSICAL SPACES
In this section, we empirically show that DSI model forms biologically plausible grid-like repre- sentations in 2-D physical space, and they support spatial navigation. These results also apply to conceptual spaces with the 2-D structure, depending on the definition of states.
4.1 LEARNING PROCEDURE
As an environment, we assumed a square room tiled with 30 x 30 discrete states (Figure In each simulation trial, an agent starts at one of those 900 states and transits to one of eight surrounding states each time except that transitions are limited at states along the boundary (the structure was not a torus). Transitions all directions occur with an equal probability. We performed 500 simulation trials and obtained a sequence of 100,000 time steps in each trial. We calculated occurrence proba- bilities (P(s)) and successor representation matrix (SR(s, s')) of 900 states from those sequences, and calculated PSI and DSI (100-dimensional) as described in the Model section. The discount structure with separated and
factor y was set to 0.99. We additionally tested spatial navigation interconnected rooms (sce Figure 3C). In that case, we used the discount factor y 0.999.
4.2 EMERGENCE OF GRID-LIKE REPRESENTATIONS
Here we call each dimension of DSI representation vectors x(s) as neural "unit", and we regard a value in each dimension at each state as neural activity (or neural representation). As shown in Figure many units exhibited grid-like activity patterns in the space. We performed a gridness analysis that has been used in animal experiments (Sargolini et al.) 2006) and found that 51 % of units were classified as grid cells. Similarly, 53% of units w were classified as grid cells.
Furthermore, we checked whether DSI representations in the physical space reproduce property of biological grid cells. Actual grid cells in the rat brain exhibit multiple discrete spatial scales and the ratio between grid scales of adjacent modules is v2 (Stensola et al. 2012). We constructed a distribution of grid scales of DSI units by kernel density estimation, which revealed that multiple peaks of grid scales existed and the ratio between grid scales of adjacent peaks was v² (Figure 2 These results show that DSI model constructs biologically plausible grid-like representations the 2-D physical space. We describe details of analysis methods in Appendix A.6
4.3 NEAR-OPTIMAL SPATIAL NAVIGATION BY DSI VECTORS
As discussed Section 3.2 the inner product of DSI representations approximate value functions for spatial navigation. Therefore, we tested whether DSI representations actually enable near- optimal navigation in the space.
We assume that a start location (state sinit) and a goal location (state sc) are randomly given in each trial such that the shortest path length is minimally 10, and an agent has navigate between them. To solve the task, we define a vector-based state transition rule. Suppose that the agent exists at state 8, and a set of neighboring states of is (s)). Given the goal representation vector ur (SG), value function of neighboring state Snext A(s) is estimated by #(Snext) w(SG), and the agents transits to the state that has a maximum value. This state transition rule can be geometrically interpreted as the choice of movements that has the closest angle with the goal vector in the repre- sentation space (Figure A). Otherwise, we can interpret that the agent estimates value functions by linear readout from like DSI representations. Because of the approximation error, this rule did not always give optimal navigation (the shortest path from the start to the goal). However, the agent could take the shortest path to the goal in 93.9% of 1,000 trials tested (an example is shown in
5

---

Under review as a conference paper at ICLR 2023
Unit 6
Unit 42
Unit 97
offercaa vaite 47)
1.00
NEW
Figure 2: DSI representations for the physical space. (A) The square room tiled with 30x30 dis- crete states, (B) Grid-like DSI representations in the space (upper) and their spatial autocorrelation (lower). (C) The distribution of grid scales of DSI representations. Red and blue broken lines indicate positions of peaks and 30 respectively. (v2)=
A
Str
Goal state
Next state
G Other neigh gebbrlggingg states
Figure 3: Spatial navigation using DSI representation vectors. (A) Geometric interpretation of the state transition rule in the DSI representation space. (B) An example spatial path obtained by DSI- based navigation. (C) Example spatial paths by DSI-based navigation in the four-room environment.
Figure 3B). Furthermore, 97.2% were near-optimal navigation which the actual path length was shorter than 1.1 times the shortest path length. The same framework also worked in relatively complex environment with separated rooms (Figure CC. In this environment, the ratio of optimal and near-optimal navigation was 68% and 82.6%, respectively. We also confirmed that can per- form path integration based on DSI representations using movement-condition recurrent weights (McNaughton al. 2006 Burak Ficte 2009: Oh al. (2015] Gao et 2019) (Appendix A.7]. These results show that DSI representations can support spatial navigation, which corresponds to the contribution of biological grid cells for spatial navigation.
4.4 VECTOR- -ACTER-BASED INFERENCE OF SPATIAL CONTEXTS
We additionally found that we can perform vector-based inference for spatial navigation in novel context. First, we constructed DSI representation vectors in spatial contexts A and B, each of which has barrier (Figure (4A). Then, we created representation vectors for a novel context A+B with two barriers by simply adding representation vectors for familiar contexts A and B (Figure AA). We tested vector- based navigation (described in the section 4.3) in three spatial contexts A, B, and A+B, using of three representations for A, B, and A+B. Naturally, representation vectors for A and B gave the best performance contexts A and B, respectively (Figure BB). Notably, composite representation vectors for A+B achieved the best performance in the context A+B (Figure This

---

Under review as a conference paper at ICLR 2023
Context A
Context B
Context A+B
Context
A
B
A
Infer
Vector
99.1%
46.7%
61.3%
43.3%
98.4%
61.5%
A+
81.5%
82.8%
84.4%
Familiar
Familiar
Novel
Figure 4: Vector-based inference of a novel spatial context. (A) We constructed representation
vectors for novel context A+B square room with two barriers) by the sum of DSI representation vectors for two familiar contexts A and B (with either of two barriers). (B) Rates of near-optimal navigation in various settings of representation vectors and contexts.
result suggests that context. We describe details of the simulation in Appendix A.8
utilize vector-based composition of representations for novel spatial
Additional analysis by multidimensional scaling (MDS) suggest that summing DSI vectors leads to composition an appropriate metric space for the novel context Appendix(A.9), This is potentially useful for composing multiple constraints that change reachability between states in various tasks (such as control of robotic arms and playing computer games), like composition of tasks in soft-Q learning (Haarnoja al. 2018 Makino
5 LEARNING REPRESENTATIONS OF CONCEPTUAL SPACES
In this section, we show that the same DSI model can learn representations for complex conceptual space from linguistic inputs, and those representations support vector- vector-based conceptual inference.
5.1 LEARNING PROCEDURE
We used text data taken from English Wikipedia, which contains 124M tokens and 9376 words (see Appendix(A.100 10 the detail preprocessing). To construct DSI representations, we regarded cach word as a "state", and considered the text data as sequence of 9376 states (N, 9376). Then, applied the exactly same learning procedure as in the experiment 300-dimensional DSI representation vectors for each word. The discount factor y was set to 0.9. The setting of other parameters was the same as the experiment of physical spaces.
physical spaces. We obtained
5.2 EMERGENCE CONCEPT-SPECITE REPRESENTATIONS
As in the previous section, regard each dimension of representation vectors neural unit, and checked how various words activate those units. Specifically, we listed ten words that elicited the highest activities in each unit (TOP-10 words). Consequently, by words related to specific concepts (Figure other examples in Appendix [A. 12), which could be named "game cell" or "president cell",, for example. We quantified this conceptual specificity through WordNet-based semantic similarity between words (Princeton University 2010). We com- pared mean similarity among TOP-10 words and null distribution of similarity between random word pairs, by which we determined statistically significant concept-specific units and quantified the degree of conceptual specificity of each unit (see Appendix [A.I1 details). DSI exhibited the larger number of significantly concept-specific units and higher average conceptual specificity than other eellbestablished word embedding methods such as skip-gram and GloVe (Table (Mikolov al., 2013abb b] Pennington al. 2014 Levy & Goldberg, 2014). We also analyzed conceptual the embedding layer of pretrained BERT model (bert-base-uncee unceeed in Hugging Face transformers) (Devlin et al. 2018: Wolf al. 2020), which was lower than DSI (Table This result shows that our DSI model forms more concept-specific representations than other models.
found that many units activated
specificity of representations
Additional analyses revealed that word representation vectors are non-sparse and distributed (Ap- pendix A.15). Therefore, each word is represented by the combination of concept-specific units
7

---

Under review as a conference paper at ICLR 2023
TOP 10 words for each unit
Unit
Unit 31
Unit
Unit 244
Unit 269
eleventh 2.13
orbit 2.16
playstation 2.38
covid-19 1.84
reagan 1.84
tenth 2.09
spacecraft 2.12
nintendo 2.25
virus 1.76
obama 1,81
ninth 2.00
saturn 1.95
xbox 2.13
infected 1.71
barack 1.78
seventh 1.96
apollo 1.84
console 1.93
pandemic 1.65
bush 1.61
eighth 1.95
lunar 1.80
arcade 1.78
viral 1.64
Trump 1.55
twelfth 1.95
planets 1.78
360 1.62
epidemic 1.62
clinton 1.53
7th 1.93
nasa 1.73
gameplay 1.56
hiv 1,50
presidents 1.35
8th 1.85
mars 1,73
pc 1.46
plague 1.42
roosevelt 1.31
10th 1.84
moon 1.69
mario 1.40
infections 1.39
nixon 1,30
11th 1.84
orbital 1.66
metacritic 1.33
outbreak 1.37
presidential 1.29
Order cell
Space cel/
Game cell
Virus cell
President cell
Figure 5: Concept-specific representations formed by DSI. Ten words that gave the highest activation (TOP-10 words) are shown. We also marked each unit with descriptive label.
Table 1: Evaluation of conceptual specificity. Numbers of evaluated and significantly concept- specific units, the ratio concept-specific units, and average conceptual specificity are summarized.
METHOD EVALUATED SIGNIFICANT
RATIO SPECIFICITY
DSI
285
100
35%
0.32
Skip-gram
297
14
5%
-0.11
CBOW
298
83
28%
0.26
GloVe
297
6
2%
-0.07
PPMI-SVD
297
36
12%
0,06
BERT
755
51
7%
0.03
shared by several related words. For example, "France" can be represented by the combination of units which we could name French cell and country cell (Appendix A.15)
5.3 VECTOR- BACTOR-BASED COMPUTATION IN THE CONCEPTUAL SPACE
Given that DSI and word embedding methods are mathematically similar (Section A.4), we expect that DSI vectors have similar properties representation vectors learned by those word embedding methods. We evaluated the performance of DSI vectors in two tasks that have been used to eval- uate word embedding methods: word similarity and analogical inference (Mikolov et al. 20 b Pennington et al. 2014; Levy & Goldberg 2014). In the word similarity task, we calculated cosine similarity between representation vectors of word pairs, and evaluated the rank correlation between those cosine similarities and human word similarities (WS353 dataset (Agirre et al., 2009): 248/345 word pairs were used). In the analogical inference task, we performed calculation of vectors such as x e(king)) x (man) + x (woman) and checked whether the resultant vector has the maximum co- sine similarity with æ(gueen) (Mikolov's dataset (Mikolov al. 2013a b); 3157/19544 questions were used; examples in Appendix A. 13). The result shows that DSI vectors achieved comparable performance with other well-established word embedding methods (Table(2).
This result indicates that similarity of DSI representation vectors corresponds to semantic similarity. This property is consistent with the experimental observation that population-level pattern similar- ity of concept cell activities represents semantic categories (Reber et al. 2019). By visualizing the structure of DSI representations by MDS, we can actually see clustering of words corresponding to 10 semantic categories used in Reber et al. (2019) (Appendix A.14). Furthermore, conceptual inference is possible through arithmetic composition of DSI vectors. We additionally found that this For example, transfor- mation from "Paris" to "France" corresponds to activation of country cell and deactivation of capital cell, which is possible by summing the difference of Germany and Berlin vectors. (Appendix A.15
inference is intuitive recombination of concept concept-specife specific units in some
8

---

Under review as a conference paper at ICLR 2023
Table 2: Performances of vector-based computations in the conceptual space. The correlation of word similarity evaluated by vectors and humans, and the rate of correct analogical inference.
METHOD SIMILARITY
ANALOGY
DSI
0.70
0.63
Skip-gram
0.69
0.71
CBOW
0.69
0.74
GloVe
0.69
0.71
PPMI-SVD
0.52
0.58
6 DISCUSSION
In this paper, we proposed theoretically interpretable and biologically plausible neural representa- tion model for physical and conceptual spaces. We demonstrated that our DSI model forms grid-like representations in the physical space and concept-specific representations in the linguistic space, which are assumed to correspond to neural representations in EC. Furthermore, we showed that SI mathematically related to linear reinforcement learning and word embedding methods, thus DSI representations support spatial navigation and conceptual inference. These results suggest that can extend the spatial representation model of EC learn and compute linguistic concepts, which apparently seems different computational domain from physical space.
In the section 5.2 we demonstrated concept-specifer representations created from text data. To the best of our knowledge, such property has not been reported in any word embedding methods. However, we unexpectedly found that continuous-bag- bfg-oo-wordsss (CBOW) showed relatively high conceptual specifieity. Although DSI is related PMI, skip-gram and GloVe, we have not found relationship to CBOW. Further clarification of necessary conditions for conceptual specificity is still open problem.
Although DSI has clear mathematical interpretation, how biological neural networks can learn DSI is still unclear. possible solution is an extension of skip-gram neural network with SR, non- negativity, and decorrelation. Because SI corresponds to PMI which is the optimum of skip-gres sip-gram neural network, we can expect that extended sippggram network learns DSI. Building such model and relating it to the circuit mechanism in hippocampus and EC are left for future research.
Our model relates word embedding to conceptual representations the brain. Previously study showed that skip-gram representations support tigh-peeformnce decoding of semantic information from fMRI data (Nishida Nishimoto [2088). Another study revealed that hippocampal theta os- cillation codes semantic distances between words measured in word2vec subspace (Solomon et al.) 2019). These experimental results support our hypothesis. However, recent studies have shown that representations in transformer-based models (Vaswani et al. 2017) such GPT (Brown al. 2020) achieve remarkable performance in linear fitting to neural recording during linguistic pro- cessing (Goldstein al., 2022, Schrimpt al. 2021). A major difference between DSI model and transformer-based models is that DSI representations are basically fixed (static embedding) whereas transformer-basee models flexibly create contexl-dependent representations (dynamic em- bedding). Conceptual interpretation obviously depends on the context, thus activities of concept cells context-dependent (Bausch et al. 2021). Therefore, our DSI model should be extended to process context- -ontext-dependence,, hopetully by combination with other models for learning context- dependent latent cognitive states (Uria et al. 2020 George et al. 2021 Whittington al. 2020).
Another direction of future research is application to general conceptual spaces by learning DSI representations from low-level sensory inputs, like spatial learning from visual and auditory inputs in previous models (Banino et al. [0088] Taniguchi et al., 22088: Uria et (020)) It may be possible by learning discrete states by unsupervised clustering for deep networks (Caron al. 2018). As for the human brain, infants probably form primitive spatial and conceptual representations from sensory signals, and later linguistic inputs enrich those representations. We speculate that real-world sensory data also contain the information of the conceptual space, for which DSI can be extended to learn those structures. Such model would clarify the role of the hippocampal system in computation of general conceptual spaces.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius capaca ca, and Aitor Soroa. A study on similarity and relatedness using distributional and wordnet-based approaches, pp. 19- 27, 2009. URL
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap. Piotr Mirowski, Alexander Pritzel, Martin J. Chadwick, Thomas Degris, Joseph Modayil, Greg Wayne, Hubert Soyer, Fabio Viola, Brian Zhang, Ross Goroshin, Neil Rabinowitz, Razvan Pascanu, Char- lic Beattie, Stig Petersen, Amir Sadík, Stephen Gaffney, Helen King, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector-based navigation using grid- like representations in artificial agents. Nature, 557:429 433, 2018. ISSN 14764687. doi: 10118//415860018--01000
Xiaojun Bao, Eva Gjorgieva, Laura K. Shanahan, James D. Howard, Thorsten Kahnt, and Jay A. Gottfried. Grid-like neural representations support olfactory navigation of a two-dimensional odor space. Neuron, 0e2:1066-175.ee 6 2019. ISSN 10974199. doi: 10.1016 www.eee.............
Marcel Bausch, Johannes Niediek, Thomas P. Reber, Sina Mackay, Jan Boström, Christian E. Elger, and Florian Mormann. Concept neurons in the human medial temporal lobe flexibly represent abstract relations between concepts. Nature Communications, 12, 12 2021. ISSN 20411723. doi: 10. 467-4441---222227777
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voos Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv, 5 2020. URL
http://arxivv org/ abs 2005 14165
Yoram Burak and lla R. Fiete. Accurate path integration continuous attractor network models of grid cells. PLoS Computational Biology, 5, 2009. ISSN 15537358. doi: 10. 11377//jourll.pche 1000291.
Daniel Bush, Caswell Barry, Daniel Manson, and Neil Burgess. Using grid cells for navigation. Neuron, 87:507-520, 8 2015. ISSN 10974199, doi: 10. 1..10666..euu0022255...0000666
Marc-André Carbonneau, Julian Zaidi, Jonathan Boilard, and Ghyslain Gagnon. Measuring disen- tanglement: review of metrics. arxiv, 12 2020. URL http:://arx.v..org//bb////
09276
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clus- tering for unsupervised learning of visual features. pp. 132-149, 2018.
URL
https::/ thecvf..cc.ccc www http////ooteettcccooooo cecccc___1///aeeee 2018/htmllll.mc.
/ /attmmmmaathidde.... for Coo__ECCV_ SBcccv_2018___0 Baapi__ppeer_aae html
Alexandra O Constantinescu, Jill X O'reilly. and Timothy E J Behrens. Organizing conceptual
knowledge in humans with gridlike code. Science, 454:1464-1468, 2016. doi: 10. 10.112666 126/science
aaf0941. URL httt::ttt////t ence. sc www.encemegg.oo.............o..e org/
Christopher J Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. 2018. URL http:://ppeereiiew...
Forumm2.d=B17JT0e00
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5:613-624, 1993. ISSN 0899-7667. doi: 10. 199.16eco..993.55....000 1993.5.4.613,
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arxiv, 10 2018. URL httt:://p//xii.
org/abs/ /1800. 04805
Christian F. Doeller, Caswell Barry, and Neil Burgess. Evidence for grid cells a human memory network. Nature, 463:657-661, 2 2010. ISSN 00280836. doi: 10. 18.
10

---

Under review as a conference paper at ICLR 2023
Yedidyah Dordek, Daniel Soudry, Ron Meir, and Dori Derdikman. Extracting grid cell charac- teristics from place cell inputs using non non-negative principal component analysis. 2016. doi: 00054///Lifc.1004..0000
lla R. Fiete, Yoram Burak, and Ted Brookings. What grid cells convey about rat location. Journal of Neuroxcience, 28:6858-6871, 7 2008. ISSN 02706474. doi: 10. 101523//NEUOSSCI.66.11 2008.
Ruiqi Gao, Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Learning grid cells as vector rep- resentation of self-position coupled with matrix representation of self-motion. 2019. URL http:://.pppprrreeeaaeeee/rrrrrp Fatee/oooummmiiss000h0553
htttps://openreviiee net
Dileep George, Rajeev V. Rikhye, Nishad Gothoskar, Swaroop Guntupalli, Antoine Dedieu, and Clone-structured graph representations enable flexible learning and vi- carious evaluation of cognitive maps. Nature Communications, 12, 12 2021. ISSN 20411723.
Miguel
doi: 10. 10.10///44447-021 -225595..
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim. Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Roi Reichart, Sasha Devore, Adeen Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth A. Norman, Orrin Devinsky, and Uri Hasson. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25:369-380, 2022. ISSN 15461726. doi: 10.1038/ $41593-022-01025
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforemeet learning for robotic manipulation. pp. 6244 6251, 3 2018. URL
bttp://arxiv..rgg/ org/ hwapg///abb//aa 06773
Torkel Hafting, Marianne Fyhn, Sturla Molden, May Britt Moser, and Edvard Moser. Microstruc- ture of spatial map the entorhinal cortex. Nature, 436:801-806, 2005. ISSN 00280836. doi: 10. 10.11388/atuue00321..
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2017. URL dittps://opereete net feet/foorm?ic
Sy2fzU9g1
Irina Higgins, Le Chang, Victoria Langston, Demis Hassabis, Christopher Summerfield, Doris Tsao, and Matthew Botvinick. Unsupervised deep learning identifies semantic disentanglement in single inferotemporal face patch neurons. Nature Communications, 12, 2021. ISSN 20411723, doi: 10. 10.10088641447--22-
A. Hyvärinen and E. Oja. Independent component analysis: algorithms and applications. Neural Nenworks, 13:411-430, 2000. ISSN 08936080. doi: 22.0016600822-600000000022223 URL
https: htt::://1iniiighubb elsevier wtteerr.ocorreeeeeeeeeeeeeeeeeee wttp://wwww.acc.eeeeeeeeee/a///r
Joshua Jacobs, Christoph T. Weidemann, Jonathan F. Miller, Alec Solway, John Burke, Xue Xin Wei, Nanthia Suthana, Michael R. Sperling, Ashwini D. Sharan, Itzhak Fried, and Michael J. Kahana. Direct recordings of grid-like neuronal activity in human spatial navigation. Nature Neuroscience, 16:1 16:1188-1111 188-1190, 2013. ISSN 10976256. doi: 10.1038/m..466.
George Lakoff and Mark Johnson. Conceptual metaphor in everyday language. The Journal of Philosophy, 77:453-486, 1980.
Daniel D. Lee and H. Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401:788-791, 10 1999. ISSN 0028-0836. doi: 10.1038/44565. URL
http::/www.nn nawure.................a.....e.. com/ htmmm/artcleessss...
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. pp. 2177- 2185, 2014.
11

---

Under review as a conference paper at ICLR 2023
Hiroshi Makino. Arithmetic value representation for hierarchical behavior composition. Accepted.
Bruce L. McNaughton, Francesco P. Battaglia, Ole Jensen, Edvard I. Moser, and May Britt Moser. Path integration and the neural basis of the "cognitive map'. Nature Reviews Neuroscience, 7: 663-678, 8 2006. ISSN 1471003X. doi: 10.1038/nm1932.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimaatio of word represen- tations in vector space. arXiv, 1 2013a.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed reprrstttion of words and phrases and their compositionality. arXiv, 2013b. URL htt.p:::/erxiiv.rr///b...
1310. 4546
I. Momennejad, E. M. Russek, H. Cheong, M. M. Botvinick, N. D. Daw, and Gershman. The successor representation in human reinforcement learning. Nature Human Behaviour, 1:680-692, 2017. ISSN 2397-3374. doi: 10. 100033331155---0---1888-8-- URL http: www. nature.
com/ com/ http/aattccesssss1/ 017-0180-
Ida Momennejad. Learning structures: Predictive representations, replay, and generalization. Cur- rent Opinion in Behavioral Sciences, 32:155-166, 2020. ISSN 23521546. doi: 10.1016/j. cobeha.2020.02.017
Yurii Evgen' evich Nesterov. method of solving a convex programming problem with convergence rate 2)k2). In Doklady Akademii Nauk, volume 269, pp. 543-547. Russian Academy of Sciences, 1983.
Satoshi Nishida and Shinji Nishimoto. Decoding naturalistic experiences from human brain activity via distributed representations of words. Neurolmage, 180:232-242, 10 2018. ISSN 10959572. doi: 1..016fj.euroommae2201177
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action- conditional video prediction using deep networks in atari games. pp. 2863-2877 2015.
URL https::/proceeings.. wwwriiss...g.................... cct.cc//pee httppssccc/aapr/21///11ee Pwppaaapperrrr
Erkki Oja and Mark Plumbley. Blind separation of positive sources by globally convergent gra- dient search. Neural Computation, 16:1811-1825, 9 2004. ISSN 0899-7667. doi: 10.
1162008997600413334113 URL lnttps://di htt.://teeee/eeeeeeaatteee//////
1811-1825/6877
Seongmin Park, Douglas S. Miller, and Erie D. Boorman. Inferences on multidimensional social hierarchy use grid-like code. Nature Neuroscience, 14:1222-1301, 9 2021. ISSN 15461726. doi: 10.10088444559--01-0000999
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. pp. 1532-1543, Association for Computational Linguistics, 2014. doi: 10.3115/ v1/D14-1162. URL
1162
Payam Piray and Nathaniel D. Daw. Linear reinforcement learning in planning, grid fields, and cognitive control. Nature Communications, 12, 12 2021. ISSN 2041 doi: 10.1038/ s41467-021-25122-3.
Princeton University. About wordnet. httpss//woorchsttrriintttooo
edu/ 2010.
Rodrigo Quian Quiroga. Concept cells: the building blocks of declarative memory functions. Nature Reviews Neuroscience, 13:587-597. 9 8 2012. ISSN 1471003X. doi: 10. 10.10038/m3251.
Thomas Reber, Marcel Bausch, Sina Mackay, Jan Boström, Christian E. Elger, and Florian Mormann. Representation of abstract semantic knowledge in populations of human single neurons the medial temporal lobe. PLoS Biology, 17, 6 2019. ISSN 15457885. doi:
10.1371/jommallppoo0000000 30000000
12

---

Under review as a conference paper at ICLR 2023
Francesca Sargolini, Marianne Fyhn, Torkel Hafting, Bruce L. McNaughton, Menno P. Witter, May- Britt Moser, and Edvard Moser. Conjunctive representation of position, direction, and velocity in entorhinal cortex. Science, 312:758-7622 5 2006. ISSN 0036 0036-8075, doi: 10.1/26/sc.ence.
1125572. URL https::/p//.. www. ence. www.ooggg/////// /10.///111.. /26/ sci ence. 1125572
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture of lan- guage: Integrative modeling converges on predictive processing. Proceedings of the National URL
Academy of Sciences, 118. 11 2021. ISSN 0027-8424. doi:
htt33//maa.210546611 https://pnas org/ doi ful 100 1073 pttpp/////pppp/......///......./ pras 2 2105646118
Ethan A. Solomon, Bradley C. Lega, Michael R. Sperling, and Michael J. Kahana. Hippocampal theta codes for distances in semantic and temporal spaces. Proceedings of the National Academy of Sciences the United States of America, (16:24343-24355 11 2019. ISSN 10916490. doi: 1t1.77//ppaa.19067999191
Ben Sorscher, Gabriel Mel, Surya Ganguli, and Samuel A Ocko. A unified the- ory for the origin of grid cells through the lens of pattern formation. pp. 10003- neerri http://pww..aaao......oorr./ep/p pdd
10013, 2019. URL.
Kimberly L. Stachenfeld, Matthew M. Botvinick, and Samuel J. Gershman. The hippocampus predictive map. Nature Neuroscience, 20:1643-1655, 2017. ISSN 15461726. doi: 10.1038/mn. 4650.
Hanne Stensola, Tor Stensola, Trygve Solstad, Kristian Fröland, May Britt Moser, and Edvard Moser. The entorhinal grid map is discretized. Nature, 492:72-78, 12 2012. ISSN 00280836.
doi: 10.1////aat 1649.
Akira Taniguchi, Tadahiro Taniguchi, and Tetsunari Inamura. Unsupervised spatial lexical acqui- sition by updating language model with place clues. Robotics and Autonomous Systems, 99:
166-180, 2018. ISSN 09218890. doi: 10.1016)..bbo.2017.0000..
Emanuel
Todorov.
Linearly-solvable markov decision problems.
pp. 1369
1376. 2006.
URL http::://aappss....b.t .iis.. CC/ wtwp//appe///0/////
@806ca13 1rr.6cal3ca3... 3efcaa3449at /2 abtp:aaaaeeebeeeaaaaaaaaaaaaaeea Abstract. html
Emanuel Todorov. Efficient computation of optimal actions. Proceedings of the National Academy of Sciences of the United States of America, 106: 11111147-14488 2009. ISSN 00278424. doi:
10.10773ppas.077100 0710743106.
Benigno Uria, Borja Ibarz, Andrea Banino, Vinicius Zambaldi, Dharshan Kumaran, Demis Has- sabis, Caswell Barry, and Charles Blundell. A model of egocentric to allocentric understand- ing in mammalian brains. biorxiv, 2020. doi: 10.1101/2020.1..11.350 //doi.oor//i1 1101120200.0.. 11. 11. 378141
URL https:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. pp- 5998-6008 6008, 2017.
URL htttos://pppers.nt 3f5ee2435476en 3F5e22155776e1111100ccccccceeee wtt.rraattaattatt......o.. html
cc/ ccc.paaper/211//aaaii
Abstract
James C. R. Whittington, Joseph Warren, and Timothy J. Behrens. Relating transformers to models and neural representations of the hippocampal formation. 12 2022. URL http://
a arxiv. wwt. abs 221222122200 02012.0403
James C.R. Whittington, Timothy Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy E.J. Behrens. The tolman-eichenbaun machine: Unifying space and rela- tional memory through generalization in the hippocampal formation. Cell, 129:1222-2,,3.,2,, 11 2020. ISSN 10974172. doi:
Thomas Wolf, Lysandre Debut. Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger,
13

---

Under review as a conference paper at ICLR 2023
Mariama Drame, Quentin Lhoest, and Alexander M Rush. Transformers: Sfate-of-the-ar nat- ural language processing. pp. 38 45, 2020. doi: 10. 10.16655w.12222.emll--eeeem. URL
htpp:://sclaathhoooo...oo///
emnip-demos
A APPENDIX
A.1 CALCULATION OF SR
SR is a variant value functions in reinforcement learning, thus we can use various methods such as temporal-differenc (TD) learning for the construction. Throughout this study, used a direct count method because we performed only offline processing finite data. In a se- quence of states {51,.... ................................ St....,,87 sr), recursively calculated exponential traces of past states 2s, t) = Σi= 7" (( as
z(s -2(s, t 1) + 5(st.
(11)
and calculated SR from state counts and coincidence counts
SR(s, ΣT= 2(3, t))(s 5))
(12)
11=1 8(st.)) 8)
A.2 DETAILS OF DECORRELATIVE NMF
In decorrelative NMF, we iteratively updated vectors æ(s) and w(s) ) by Nesterov's accelerated gradi- ent descent method to minimize the objective function (Eq. 4). rectifying all elements every iteration. Gradients are
OJ (x)
((PS(() x0() wwss))wu
( + (x))
+ Bcor
(13)
E (+) (5,(ss)) (2,
0J
Σ PS x(s) w(5')) + (reg
(14)
(w)(s')
We note that we regarded mean and variance (s) the correlation ( Σ. (T.
Eq. (6) as constants in the calculation these gradients. Practically, this heuristic did not affect the performance of decorrelation.
Throughout this paper, the learning rate was 0.05 and the number of iteration was 10000. Parameters were Boor Lreg 0.001, and Pmin 0.001.
A.3 MATHEMATICAL RELATIONSHIP OF DSI AND REINFORCEMENT LEARNING
In this section, we show that our model approximates value functions of linear reinforcement learn-
ing [Todorov] 2006 2009 Piray & Daw! 2021) in the setting of spatial navigation.
In linear reinforcement learning, an agent aims to maximize "gain" instead of reward. Assuming default policy ( (any policy available; typically random walk in the case of exploration task), gain function is defined as
(15)
where r(s) is expected reward at the state s and AK (KL) ence between the current policy T(s) and the default policy T (s) (A is relative weight of the cost). Then, previous works have shown that the optimal policy and corresponding value functions can be
the cost imposed on the differ-
determined explicitly by solving linear equations [ooooroy] 2006 2009 Piray & Daw 2021). Here we consider an environment that consists of Nv non-terminal states and Ny terminal states. We define transition probability matrices under the default policy: Pxr is Nx x Ny matrix for
14

---

Under review as a conference paper at ICLR 2023
transitions from non-terminal states to terminal states, and PNN is Ny x Ny matrix for transitions across non non-terminal states. Furthermore, TN and rr are vectors of rewards at non-terminal states and terminal states, respectively. In this condition, a vector of value functions under optimal policy o" = (81),, (8N is obtained as
exp(A (xp(A>''') = MMPTT exp(A (rr)
(16)
where M =
- P is DR (Pira & Daw 2021
To relate v" to SI, consider specific condition in which the environment consists of non-terminal states, and a virtual terminal state is attached to goal state sg arbitrarily chosen from non-terming states (Figure B)) When the agent gets to the goal, it transits to the terminal state with probability PNT- Furthermore, we assume that reward at non-terminal states are uniformly negative and reward the terminal state positive so that the agent has take short path goal to maximize reward. Specifically, we assume all elements of r'y are À log 7, and rTy = - -X(log por +og Pd (sa)) where y is an arbitrary value in the range (0, 1), and (sg) is a probability of visiting the state sg under the default policy. Then, we obtain
exp(A v²) (d(c) (I (I- (I-2PN)) yPNNNNNN--ee ((e))
(17)
where e(to) (0, (O. : 0, 1, 0, ...., 0) (ig is the index of the goal state). Because (I equivalent to a successor representation matrix with a discount factor y (Dayan 1993 Stachenfeld al. 2017). we finally obtain
N is
x 1=) log (S 8c)) log Pd ST (a) x(s) w(sc),
(18)
where (H sG) and (1 sa) are SR and SI under the default policy, respectively. Thus, SI is proportional to value functions for spatial navigation and inner products of DSI vectors approximates value functions. Based on this interpretation, we basically regard a(s) as a representation of each state, and w/ represents temporary goal.
A.4 MATHEMATICAL RELATIONSHIIP OF DSI AND WORD EMBEDDING
In this section, we discuss the relationship of SI and PMI (Levy & Goldberg 2014) in detail. PMI is
P(word;; wordj)
PMI = log P(word,) P(word,)
(19)
where P(word;, word,) is a coincidence probability of two words (in certain temporal window).
To relate PMI SI, we regard words as states: s word,, word,. Furthermore, we consider specific way to count coincidence probability. In typical word embedding, a finite symmetric rectangular window is often used:
W P(s,s' Σ P(s
(20)
where W is window size. Here, we implicitly assumed that same state (word) is not repeated in the temporal window to guarantee that P(s, s') is probability.
However, may arbitrarily calculate coincidence for P(s,s'). Here we evaluate coincidence with an infinite asymmetric exponential kernel as SR:
P(s, s') = (1- Y P(ss = 5'550 = s).
(21)
1=
We introduced a normalization factor (1 - 7) to guarantee that P(s, is less than one ((1 -
7) Ei= 7' =)) Then, PMI becomes
(1 7) 11=0 P(s = S =
PMI log
(22)
P(s)
(og(SR(s, s')) log ((s')) log(1 - 7)
(23)
ST(s, s') log(1 7)-
(24)
If we perform dimension reduction, log(1 (1 - 7) can be ignored because it is a constant. Therefore, we can interpret SI as special case of PMT in our model.
15

---

Under review as a conference paper at ICLR 2023
A.5 RELATIONSHIP BETWEEN MODEL COMPONENTS AND REPRESENTATIONS
To clarify the contribution of each model component to the results in this study, we performed a "lesion study" in which we removed some components in DSI and repeated the same evaluation procedure the main text. We summarize results in Table 3 First, we tested representations ob- tained by singular value decomposition of successor representation (SR-SVD), which was regarded as model of grid cells in a previous study (Stachenfeld et al. 2017). DSI model exceeded SR-SVD in all aspects shown this study. Next, we tested DSI model without decorrelation (Beor = 0) and DSI model without non-negativity (no rectification of representation vectors). Neither modifi- cation impaired the performance navigation and inference, showing the importance of using SI for vector-based computations theoretically expected. In contrast, removing decorrelation and non-negativity significantly impaired the emergence of grid-like units and concept-specific units, respectively. Thus, decorrelative NMF is crucial obtain biologically plausible representations.
Table 3: Contribution of each model component to the performance. Rates of grid-like units, near- optimal navigation, significantly concept-specific units, correct analogical inference.
METHOD
GRID CELL
NAVIGATION CONCEPT CELL ANALOGY
Full DSI
51%
97.2%
35%
63%
Decorrelatio OFF
5%
99.9%
38%
64%
Non-negativity OFF
29%
99.8%
2%
63%
SR-SVD
21%
57.2%
13%
28%
A.6 DETAILS OF EVALUATION OF GRID REPRESENTATIONS
In the section 4.2 we performed the gridness analysis following a previous experimental study (Sargolini et al.) (006). For each unit, we rotated the spatial autocorrelation map (Figure BB lower) and calculated correlations between the original and rotated maps. Gridness was defined as the difference between the lowest correlation at 60 and 120 and the highest correlation at 30", 90 and 150", A unit was classified as a grid cell when gridness exceeds zero
In Figure 2 we constructed a distribution of grid scales. Grid scales were determined as the median of distances between the central peak and the six closest peaks (vertices of inner hexagon) in the spatial autocorrelation map. The kernel function for kernel density estimation was Gaussian with a standard deviation 1.
A.7 PATH INTEGRATION BY DSI
We performed path integration based DSI representations using movement-condittooal recurrent weights. This strategy has been used in previous studies such as grid cell modeling (Gao et al.) 2019) and action-conditional video prediction (Oh et al., 2015). This mechanism also consistent with a conventional biological model for path integration in which head direction signals activate one of attractor networks specialized for different directional shifts of grid patterns (MeNaughton et al. 2006, Burak & Fiete 2009).
We made an estimate of the next representation vector ă.41 by linear transformation of the current representation vector a0(s;) as
L (at)aa(ss)
(25)
where represents a movement (one of eight directional movements in this study) and M M(a)) is movement-conditional recurrent weight matrix. Here, ax(s)) was a DSI representation vector, and we optimized the matrix ((a)) by minimizing prediction error [|æ(S++1) - ((a )x{)) by stochastic gradient descent during random walk on the state transition graph (20 simulation trials of 100,000 time steps). After optimization, we set an initial state so and a sequence of movements {ag, Q1, dr-1), and performed path integration by recursive estimation ø141 = ((a,) ) We determined position each time step by searching state representation vector that has minimum Euclidian distance with the estimated vector (s, = arg min, [0(()) dx112). As shown in Figure 6 this strategy gave accurate estimation of the spatial path from movement signals.
16

---

Under review as a conference paper at ICLR 2023
Correct path
Estimation by path integration
Figure 6: Path integration using DSI model. (Left) Actual path. (Right) Path estimated from DSI vectors updated by movement information.
A.8 DETAILS OF VECTOR-BASER INFERENCE OF THE SPATIAL CONTEXT
In the section we performed rector-based inference for spatial navigation in novel context.
Specifically, we define separated states in contexts A, B, A+B as s s and s A+B where i is positional index which indicates same position all contexts (i 1,2, 900). We constructed representation vectors a(s4), ) w(s()) (s!)), and un()) through direct experiences, then we cre- ated w0s BB+ and w(s A+B)
o0s x(5;¹) + xxs)) w w(s()) w(
(26)
(27)
We performed spatial navigation in given context using of three representations {æ(s²"),
(o(ss")(((s""))) (s;)), {=="s)), )( and {z(sA+BB w( for corresponding positions, fol- lowing the rule described in the section|4 4.3 In figure we show structures of state transition graphs for three contexts A, B, and A+B.
To learn representations in contexts A and B, we sampled sequences of {s( )==,,.,.- .900 and {s/} (s.")_=...... by random walk in context A and B. The procedure was basically same with the section[4exxcee exceppt that increased the number of simulation trials from 500 to 1,000, and state tran- sition to the same position in the other context occurred every 5,000 time steps (transition between x² and )F²). We added this transition to associate the same position in different contexts. It means that we assumed that the setting of barriers can change during the experience but this temporal asso- ciation may be substituted by similarity of sensory inputs across contexts. From sampled sequences, we calculated PSI for all combinations of )=,"). ,900 and {s Ji=1,.. 900,, and calculated 100- dimensional DSI vectors for 1,800 states by simultaneous compression of all states. The discount factor was set to 0.999.
Context A
Context B
Context
Figure 7: Structures of state transition graphs used in the section 4.4
17

---

Under review as a conference paper at ICLR 2023
A.9 VISUALIZATION SPATIAL STRUCTURES REPRESENTED BY DSI VECTORS
In Figure A.9 we visualized metric spaces defined by representation vectors for contexts A and B, and composite vectors for the context A+B, by using multidimensional scaling (MDS). This visualization clearly shows that DSI vectors for A and B capture structures of spatial contexts A and B, and adding those vectors yields appropriate metric space for novel context A+B.
Representation vector A
Representation
Composite vector A+B
vector B
Figure 8: Visualization of metric spaces defined by representation vectors for contexts A and B, and composite vectors for the context A+B using MDS. Red, blue, and green dots correspond to states (positions) in the left, center, and right parts of the original 2-D space (Figure 7. respectively.
A. DETAILS OF PREPROCESSING OF TEXT DATA
we used text data taken from English Wikipedia dump (enwiki-latest-pags- articles, 22-May-2020). We first generated text files from raw data using wikiextrator (https: //github. com/ homm/attardii/ wwikiextracte kenizer, and randomly sampled 100,000 articles containing 1.000 tokens minimum. We lower- cased all characters and removed punctuation characters in the data. After that, we selected words that appeared more than 1,000 times in the data, and substituted all other rare words by <unk> symbol. Finally, we obtained data that contains 124M tokens and 9376 words.
In the section 5
We tokenized texts by nltk Punkt sentense to-
A.11 DETAILS OF EVALUATION OF CONCEPTUAL SPECIFICITY
In the section 5.2 conceptual specificity of each unit was evaluated using WordNet database (Prince- ton University] 2010). In WordNet, word belongs several synsets (sets of cognitive synonyms), and semantic similarity of two synsets can be evaluated from the shortest path length hetween them in the WordNet structure (we used path_ similariy function in nltk library). We defined similarity of two words as the highest similarity among all combinations of synsets of those words. We calcu- lated mean similarity of all combinations of TOP-10 words (ten words that highly activated the unit; Figure 5)) that are available in WordNet. We evaluated only units which had at least five TOP-10 words available in WordNet. Furthermore, we randomly generated 1,000 pairs of words available in WordNet, and generated a null distribution of similarity between words. We defined a signif- icance threshold of similarity as 95 percentile of the null distribution, and unit was classified significantly concept-specific unit if mean similarity of TOP-10 words exceeded the threshold. Furthermore, we quantitatively defined a conceptual specificity of each unit as
Sunit
(28)
Snull
where Sanit is mean similarity of TOP-10 words and snall is the mean of the null distribution. This quantity becomes zero if similarity between TOP-10 words is not different from random pairs, and becomes positive if TOP-10 words are semantically similar. This conceptual specificity was averaged over all evaluated units.
18

---

Under review as a conference paper at ICLR 2023
A.12 EXAMPLE DSI REPRESENTATION FOR WORDS
In the figure P we show TOP-10 words of DSI units without manual selection. We found several non-significant units exhibit conceptual specificity according manual inspection (for example, unit4 may be named as university cell). This is probably because of the limitation of knowledge covered by WordNet. Therefore, we suppose that the current evaluation method tends underesti- mate the number of concept-specific units. However, the comparison across models was fair because we used the same procedure and criteria for all models.
Unit
Unit 2
Unit 3
Unit universitics colleges uc
Unit sheers olivia
faced
transit commuter trains
humid winters summers
overcome facing
compact
due
bus
climate
usc
senalors
owing
buses
nild
consor tium
antony
citing
subway
precipitation
institutes carnegie affiliated
proposes
experiencing
rail
warm
atlas
cape
passenger
temperatures
spaces
amid
metro
cool
alumni
geometry
su ffrre
passengers
cold
selective
rough
Unit 6
Unit 7
Unit
Unit eleventh
Unit
designers scientists
rebounds assists
nbc
nhl
aired
tenth
confirmation
experts
averaged
abc
ninth
hockey
professionals
steals
airing cbs
seventh
maple
consumers
nba
eighth
roller
researchers
average averaging
syndicated
twel
judiciary
filmmakers
partition
programming
7th
flames
lawyers
points
broadcast
8th
devils
composers
prussia
channel
10th
batch
scholars
mvp
espn
11th
senators
Unit
Unit 12
Unit 13
Unit 14
Unit 15
lowest
valve
archive
hatch
rotten
ranked
wheel
archives
deaf
tomatoes
highest
steering
manuscripts
robot
metacritic
ranks
rotating
collections
suffrage
aggregate
literacy
wheels
collection
egg
approval
demographic
cylinder
manuscript
seymour
reviews
rising
trigger
library
salem
grossed
prevalence
shaft
catalogue
wwe
rating
ratings
pump
valuable
monte
byron
rank
gear
documents
resist
apparatus
Figure 9: A part word representations formed by DSI. Ten words that gave the highest activation (TOP-10 10 words) are shown for units 1 15. Those classified as concept-specific units in our analysis are boxed with bold lines.
19

---

Under review as a conference paper at ICLR 2023
A.13 EXAMPLES OF THE ANALOGICAL INFERENCE TASK
In table we show some examples of the analogical inference in Mikolov's dataset. There is relationship "WORD1 is WORD2 as WORD3 WORD4". Then, expected relationship in the vector space is WORD4 by WORD3+WORDD WORDD.. We regarded an inference was correct the actual vector of WORD4 had the largest cosine similarity to the inferred vector among all word representation vectors (except those for WORD1, WORD2, and WORD3). the number of words is 10,000, chance level of the correct answer rate is 0.01% %. Therefore, the performance shown in this study (more than 50%) is far above the chance level.
WORD3. In this study, we performed inference of
Table 4: Examples of analogical relationships in Mikolov's dataset.
WORD1
WORD2
WORD3
WORD4
boy
girl
king
queen
husband
wife
he
she
Paris France
Tokyo
Japan
China Chinese
Italy
Italian
big
bigger
small
smaller
free
freely
quick
quickly
convenient
inconvenient
possible
impossible
A.14 CLUSTERING OF SEMANTIC CATEGORIES IN DSI SPACE
Figure 10 shows the structure DSI word representations visualized by MDS. We arbitrarily chose words based 10 semantic categories used in Reber et al. (2019) We used same dissimilarity metric with Reber et al. (2019) (1 Pearson's 's correlation coefficient).
chair
piano
cheese instrumeRlitarr
bed furniture
bread
internet
software
bee fruit apple
food
computer
insects flower
cloth orange
butterfly animal bird hat
shirt
dog
eagle
lily
rose
Figure 10: Visualization of the representational structure of DSI using MDS. Words are plotted in coordinates embedded in 2-D DSI space. Each color corresponds to a semantic category used in Reber et al. (019).
20

---

Under review as a conference paper at ICLR 2023
A.15 INTUITIVE MECHANISM OF WORD REPRESENTATIONS BY DSI
In this section, we discuss how DSI vectors represent and compute words.
First, analyzed the ratio of each element to the of all elements in DSI vectors. We found that even the largest element accounted for 5% of the sum of all elements on average. (Figure(T]] This result shows that DSI vectors for words are non-sparse and distributed, thus each word is represented by the combination of multiple conceptual units.
0.14
Ratio of each element
to of all elements
0.12
0.10
0.08
0.06
0.04
0.02
0.00
100
200
300
Sorted dimension
Figure 11: Non-sparsity of word representations. Gray lines show ratio of each element to sum of all elements in each word representation vector, and the black line is the average of them. Elements were sorted descending order.
The most active units in each vector (the largest absolute values)
France
Germany
Paris
Berlin
Unit 285
1. Unit 285
Jnit
Unit
2. Unit 142
2. Unit 134
2.
Jnit 281
Jnit 281
France-Part Paris
Germany Berlin Unit 285
France Pance-Germany Jnit 142 Jnil
Paris Rerlin
1. Unit 281
Jnit 142
Unit 285
2. Unit 281
Jnit
TOP 10 words for each unit
Unit 134
Unit 142
U81 stockholm budapest
Unit 285
friedrich
ct
belgium
heinrich
des
finland
ludwwig le
oslo
bulgaria
kari johann du
frankfurt
dermark
countries
von
françois
copenhagen
greece
und
jacques michel
istanbul
colombia
ernst
paulo
swi switzerland
wilhelmm
pays
amsterdam
austria
der
jean
seoul
croatia
German cell
French cell
Capital
Country cell
Figure 12: Words as combination of concepts and inference as recombination. (Upper) The most active units (the largest absolute values) in representation vectors for France, Germany, Paris, Berlin, and difference vectors between them. (Lower) TOP-10 words of active units and their interpretation.
Next, for further clarification, we inspected representations of an example set of words: France, Paris, Germany and Berlin. We can see there are two analogical relationships (country-capital and French-German relationships). We identified the most active units (TOP-2) in DSI vectors for those words, and listed TOP-10 10 words for identified units. As a result, we could see that "France" is represented by the combination of units that could name French cell and country cell, whereas
21

---

Under review as a conference paper at ICLR 2023
"Berlin" is represented by the combination of German cell and capital cell, and (Figure 12 This example also gives a simple interpretation of word similarity in DSI vector space. If words are similar, they share large number of active units, like the country cell shared by representations of France and Germany. Thus, semantic similarity between words increases cosine similarity between word vectors.
Furthermore, we also identified the largest elements (the largest absolute values) in the difference vectors between words, and found that they correspond semantic difference between words (Fig- ure 12). Thus, we can regard analogical inference by DSI vectors as recombination of conceptual units, For example, adding Germany-Been Berlin vector to Paris vector deactivate capital cell and activate country cell, which leads to the transformation of Paris into France.
Such property of the vector space is same conventional word embedding methods, but unique feature of our model is that those analogical relationships are factorized into separated units. We speculate that constraints of decorrelative NMF are sufficient conditions align each semantic factors to each axis of the word vector space, and the mechanism is probably related to how disen- tangled representations emerge in visual feature learning model (Higgins al. 2017 Carbonneau et al. 2020
22


Reviews:
[{"review_id": "vKCha22N1KV", "reviewer": "Reviewer_33UN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a technique called disentangled successor information that extends successor representations (SR) in RL and shows the representations learned by the proposed method resembles grid cells in the hippocampus. The authors apply their technique to spatial navigation and language modeling and show that 1) representations for spatial navigation look like grid cells 2) representations for words resemble concept cells in the brain with each feature more or less corresponding to a single concept. They take these results to argue that perceptual and conceptual procesing in the brain could be using a similar mechanism.\n\nIn more detail, given a set of discrete states, the proposed method first forms the successor matrix S (where Sij is the discounted probability of observing state sj given state si). Then it calculates an information metric called positive successor information from this matrix. Finally, a dimensionality reduction technique based on non-negative matrix factorization is used to extract lower dimensional state representations. This dimensionality reduction technique has an additional objective term to minimize the correlation between different state vectors. This helps to create a more disentangled state representation.\n\nThe authors show that this proposed technique is related to value functions in linear RL and also to word vectors obtained by skip-gram models.\n", "summary_of_the_review": "I think this is overall an interesting paper but it seems more suited for a venue with more neuroscience/cogsci focus. I don't think the results will be very interesting to the ICLR community (for that it needs much stronger empirical results on some relevant task/problem) and as it stands, the empirical results are quite limited.", "strength_and_weaknesses": "Overall I think this is an interesting paper but I'm not sure if ICLR is the right venue for this work. This work mainly tries to establish a connection between their approach (disentangled successor information) and perceptual and conceptual processing in the brain. This is an interesting and valuable endeavor but it's unclear how relevant this would be to the ICLR community. To me it seems like a cogsci/neuro venue or even NeurIPS would be a better choice for this work.\n\nIn general, I think the paper is well written and easy to follow.\n\nIn terms of contributions, the additional step of using decorrelative NMF to get state representations seems novel. However, the connection of successor representations to value functions is well-known and the connection between DSI and value functions seem to follow from this, so the connection is not quite novel. Perhaps the connection to word representations in skip-gram models is novel but given that successor representations are essentially bi-gram models, this is not very surprising.\n\nIn terms of results, there is a nice evaluation of the technique on spatial navigation and on word processing but these seem rather limited. Finding that the learned representations resemble grid cells is interesting but not a very strong evidence since many other techniques also learn representations with grid-like receptive fields.\n\nSimilary for word embedding results, I didn't really find them very strong. The authors show that features in their learned representations are on average more concept specific than some alternative techniqes. Since the brain also seems to have concept specific cells, they take this as evidence for brain using a similar mechanism. However, as far as I know it is still not very clear to what extent the concept representations in the brain are like concept cells. There seems to be many cells that do not cleanly respond to a specific concept for example. Even if this was not an issue (and we knew brain used concept cells), it is still not very strong evidence because other techniques also learn representations with similar properties. In fact, one could also directly use one-hot vectors as word representations and these would be 100% concept specific.\n\nOverall, I think it'd be great to have more evidence supporting the use of representations like DSI in the brain, by perhaps testing out other properties of these representations (like word similarity etc.).\n\nOne major issue with the paper for me is that it conflates conceptual processing with language processing. Language (and words) are only a part of conceptual processing. In fact, one can argue that concepts are more primary than words and language. Then I don't think you can take results with words and use these to make statements about conceptual processing in general. In a couple of places, the authors argue that because their technique can be applied to both spatial navigation and words, it suggests that perceptual processing and conceptual processing use the same mechanisms. However, for the reason I mentioned, this does not follow. Humans certainly have a conceptual understanding of object shape as well. Can the proposed technique also capture key properties of this conceptual space?\n\nRelated to this point, not all conceptual spaces have a 2D (grid-like structure), so it is unclear why this method should work for such conceptual spaces as well.\n\nAlso, the hypothesis that perceptual and conceptual processing might be using the same mechanisms is not novel. There is a long line of research in CogSci that makes this argument, and it'd be good to mention these (for example Lakoff's work).\n\nOther points:\n- One limitation of the method is that is assumes discrete states. Can the method be extended to continuous states? It might be good to mention some ideas along these lines.\n- In 4.4., in what cases would we expect addition to work? If it is only the case where the room is an exact superposition of two previous rooms, then this seems too limited.\n- How about comparing against embedding from next word prediction models like GPT-3? Would these show low concept specificity?\n- In 5.3, many methods can do the same vector based computation in conceptual space (even ones not based on successor representations).\n- In related work, the authors say \"spatial and conceptual processing can be theoretically unified into a single vector-based computational principle\". This is a very strong statement not supported by the results in the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "piubJs2wSpx", "reviewer": "Reviewer_pkWT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a model of hippocampal / entorhinal representation that handles both physical and conceptual space with a common mathematical formulation, drawing on successor representations from RL, and building on dimensionality reduction accounts of grid cells such as the work of Dordek et al. The authors propose a novel dimensionality reduction technique that generates a disentangled pair of vectors. In the case of physical space, the authors show that grid cell-like representations emerge with their method. The authors show that their representations support spatial navigation, where one of the two vectors represents the starting position and the other represents the goal position. The most striking finding of the paper is that the very same method can be used in a language setting to generate word embeddings. The resulting word embeddings exhibit desirable properties familiar from the literature such as conceptual clustering and the ability to support analogical inference through vector algebra.", "summary_of_the_review": "The paper is excellent, in my opinion. The proposed method is novel and makes some quite deep connections that could have important implications. The paper easily merits acceptance at ICLR, unless one of the other reviewers uncovers a serious shortcoming.", "strength_and_weaknesses": "Strengths and weaknesses\n\nStrengths: The paper suggests a profound connection between two superficially different domains of representation - physical and conceptual space. As well as shedding light on the neuroscientific question of how the hippocampal formation functions, the authors’ method potentially provides a powerful method for use in ML. The method is well motivated, both scientifically and mathematically.\n\nWeaknesses: I don’t see any important weaknesses. But I have some questions.\n\nQ1. Section 4.2: Emergence of grid-like representations. You present results for x(s), but I was wondering whether grid-like representations emerge in w(s) too? In the objective function (Eq.6), x(s) and w(s) are treated almost symmetrically, but the correlation term only applies to x(s) not to w(s) (Eq.8). Is this constraint responsible for the grid-like representations?\n\nQ2. Is there an intuitive meaning to w(s) in the word embedding case? It seems that all the results presented for conceptual space use x(s) only. (Is that correct?)\n\nQ3. I’m a little unclear on how the optimisation process works. If x and w are functions that map state vectors to vectors with reduced dimensionality then how are they represented. My reading is they are represented in tabular fashion, i.e. x(s) and w(s’) are represented independently for all s and s’. Is that right? I assume x and s could be approximated by a neural net, but that isn’t the method used here. Is that correct?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "UzuK0h_eLD", "reviewer": "Reviewer_Eei5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors develop a “disentangled successor information” framework for understanding the neural representation for physical and conceptual spaces. The paper then applied this framework to study two problems: grid cells and word embedding.\nOverall, this paper contains some interesting and intriguing ideas. But the technical content is not entirely rigorous.\n\n\n", "summary_of_the_review": "Overall, I enjoyed reading this paper, but there are many question marks upon reading it. I’d be more comfortable endorsing this paper if the work is done in a more rigorous way, and the model assumptions are better justified. Right now, there is uncertainty about the novelty of this paper makes and whether the results are indeed solid.\n\n\n%%%%%%\nI would like to thank the authors for their detailed responses to my comments. Although the responses and the revision clarified some issues, unfortunately, they did not lead to major improvements in the paper in my opinion. The paper has nice potential to be further improved, but at the moment the contribution still seems rather incremental, and the work would benefit by making the arguments more rigorous.", "strength_and_weaknesses": "Strength:\n* The paper studies both grid cells and word embedding problems.\n* The attempt to link grid cells and word embedding is ambitious.\n* Using successor information to study the grid cells is an interesting and novel idea.\n\n\n\nMain concerns: \n* Why decorrelative NMF is a good objective function for the grid cell system is unclear. Eq. (6) needs better motivations. \n\n* It is unclear what determines the structure of the emerging representation. Does the word embed similarly lead to a grid representation? If not, why?\n\n* The improvement of this model (in the case of word embedding ) compared to prior work is in fact very subtle (e.g., Table 2).\n\n* How is the present work different from prior work, e.g., Stachenfeld et al 2017, and Dordek et al 2016 is unclear. This paper performs matrix factorization, while previous work did PCA. Are they really different? This needs to be better explained.\n\n* The similarity between Eq 13 and Eq 15 seems to be superficial. If they are indeed equivalent, the implication seems to be that it should be possible to use an objective function based on Eq. 13 to derive the grid cells.  But this is not discussed or shown.\n\n* Interpreting an information-theoretical quantity as the neural activity seems to be risky. (I am actually not sure if it makes any sense.) This crucial assumption needs to be better justified. \n\n\nOther comments:\nIn Dordek et al., 2016; Sorscher et al., 2019, in addition to the non-negativity, the inhibition surround is also critical for the grid firing fields to emerge. This point should be discussed in more detail.\nSeveral relevant papers on understanding grid cells with machine learning approaches are missing, e.g.,  Cueva & Wei, ICLR, 2018;  Gan, Xie, Zhu, Wu, ICLR, 2019.\nIn Section 4.2, this explanation is not the first that attempts to theoretically explain the grid responses and the scale ratio. Prior work on this should be discussed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2oOeNytI6S", "reviewer": "Reviewer_K3nY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper defines positive successor information in the framework of reinforcement learning, and proposes positive decorrelative nonnegative matrix factorization for dimension reduction of successor information. The paper applies the proposed method to spatial navigation and word embedding, and obtained meaningful results. The paper also analyzes theoretical relationship to linear reinforcement learning. ", "summary_of_the_review": "The paper makes a solid and useful contribution, but it is not extremely novel. ", "strength_and_weaknesses": "Strengths:\n\n(1) The theoretical foundation of the paper is solid, especially relationship with linear reinforcement learning. \n\n(2) The empirical results on grid cells are strong. \n\n(3) The experiments on word embedding are solid. \n\nWeaknesses: \n\n(1) Non-negative matrix factorization is widely used, such as in the recommender system, to factorize the value function or the affinity matrix. The successor representation has also be used to model place cells, which are connected to grid cells via matrix factorization or eigen decomposition. Thus this paper is not entirely novel, although the specific form of decorrelative NMF is new. \n\n(2) Given the large number of grid cells in EC, dimension reduction may not be the most convincing point of view for embedding. It is more about population coding, where generalization does not need to rely on dimension reduction. Other forms of regularizations may also work. \n\n(3) Can your method on grid cells explain path integration? \n\n(4) The sequence of words in NLP is not a trajectory. The contexualized embedding such as BERT seems more reasonable. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject