Language Models Represent Space and Time
Authors: Wes Gurnee
Max Tegmark
Venue: ICLR 2024 poster (2024)
Keywords: Interpretability
world models
probing
Primary area: visualization or interpretation of learned representations
TL;DR: Probing for spatial and temporal features in Llama-2 models.

Abstract: The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.

Published as a conference paper at ICLR 2024
LANGUAGE MODELS REPRESENT SPACE AND TIME
Wes Gurnee & Max Tegmark Massachusetts Institute of Technology {wesg, tegmark edu
ABSTRACT
The capabilities of large language models (LLMM) have sparked debate over whether such systems just learn an enormous collection of superficial statistics set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across dif- ferent entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal co- ordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic in- gredients of a world model.
INTRODUCTION
Despite being trained to just predict the next token, modern large language models (LLMs) have demonstrated an impressive set of capabilities (Bubeck et al. 2023 Wei et al (022). raising ques- tions and concerns about what such models have actually learned. One hypothesis is that LLMs leam a massive collection of correlations but lack any coherent model or "understanding" of the underlying data generating process given text-only training (Bender & Koller 20200 Bisk et al.] 2020). An alternative hypothesis is that LLMs, in the course of compressing the data, learn more compact, coherent, and interpretable models of the generative process underlying the training data, i..., a world model. For instance, Li et al. (2022) have shown that transformers trained with next token prediction to play the board game Othello learn explicit representations of the game state, with Nanda et al. (2023) subsequently showing these representations are linear. Others have shown that [LMS track boolean states of subjects within the context (Li et al.) 2021) and have representations that reflect perceptual and conceptual structure in spatial and color domains (Patel & Pavlick 1 Abdou et al. 2021). Better understanding of if and how LLMs model the world is critical for rea- soning about the robustness, fairness, and safety of current and future Al systems (Bender al.)
2021 Weidinger et al. 2022, Bommasani et al. 2021 Hendrycks et al. 2023 Ngo et al. 2023).
In this work, we take the question of whether LLMs form world (and temporal) models as literally as possible-we attempt to extract an actual map of the world! While such spatiotemporal representa- tions do not constitute a dynamic causal world model in their own right, having coherent multi-scale representations of space and time are basic ingredients required in a more comprehensive model.
Specifically, we construct six datasets containing the names of places or events with corresponding space or time coordinates that span multiple spatiotemporal scales: locations within the whole world, the United States, and New York City addition to the death year of historical figures from the past 3000 years, the release date of art and entertainment from 1950s onward, and the publication date of news headlines from 2010 to 2020. Using the Llama-2 (Touvron et al.) (023) and Pythia BBieerma et al. (023) family of models, we train linear regression probes (Alain & Bengio 2016 Belinkovi 2022 on the internal activations of the names of these places and events at each layer to predict their
real-world location (i.e., latitude/longitude)) time (numeric timestamp).
These probing experiments reveal evidence that models build spatial and temporal representations throughout the early layers before plateauing around the model halfway point with larger models

---

Published as a conference paper at ICLR 2024
Centrert
Predicted Madoid
Figures
Publication Headlines
2020
Lincar probe prodiction (test)
1800
2020
1000
2000
1990
1200
1980
1000
2014
1970
2012
600
Spearman 921
1950
Spearman r=0.925
2010
Spearman r=0.863
2000
Figure 1: Spatial and temporal world models of Llama-2-70b. Each point corresponds to the layer 50 activations of the last token of place (top) or event (bottom) projected on to a learned linear probe direction. All points depicted are from the test set.
consistently outperforming smaller ones ($ 3.1 We then show these representations are (1) linear,
given that nonlinear probes do not perform better ($ 3.2). (2) fairly robust to changes in prompting ($ 3.3), and (3) unified across different kinds of entities (e.g. cities and natural landmarks). We then conduct series robustness checks to understand how our probes generalize across different data distributions ($ 4.1) and how probes trained the PCA components perform ($ 4.2p. Finally, we use our probes to find individual neurons which activate as a function of space or time and use basic causal interventions to verify their importance spatiotemporal modeling, providing strong evidence that the model is truly using these features ($
EMPIRICAL OVERVIEW
2.1 SPACE AND TIME RAW DATASETS
To enable our investigation, we construct six datasets of names of entities (people, places, events, etc.) with their respective location occurrence in time, each different order of magnitude of scale. For each dataset, we included multiple types of entities, c.g., both populated places like cities and natural landmarks like lakes, study how unified representations are across different object types. Furthermore, we maintain enrich relevant metadata to enable analyzing the data with more detailed breakdowns, identify sources of train-test leakage, and support future work on factual recall within LLMs. We also attempt to deduplicate and filter out obscure otherwise noisy data.
Space We constructed three datasets of place names within the world, the United States, and New York City. Our world dataset is built from raw data queried from DBpedia Lehmann et al. (2015). In particular, we query for populated places, natural places, and structures (e.g. buildings or infras- tructure). We then match these against Wikipedia articles, and filter out entities which do not have at least 5,000 page views over a three year period. Our United States dataset is constructed from DB- Pedia and a census data aggregator, and includes the names of cities, counties, zipcodes, colleges, natural places, and structures where sparsely populated or viewed locations were similarly filtered out. Finally, our New York City dataset is adapted from the NYC OpenData points of interest dataset OpenDaaa (203) containing locations such as schools, churches, transportation facilities, and public housing within the city.
2

---

Published as a conference paper at ICLR 2024
Time Our three temporal datasets consist of (1) the names and occupations of historical fig- ures who died between 1000BC and 2000AD adapted from (Annamoradnejaa & Annamoradnejad 2022); (2) the titles and creators of songs, movies, and books from 1950 to 2020 constructed from
DBpedia with the Wikipedia page views filtering technique; and (3) New York Times news headlines from 2010-2020 from news desks that write about current events, adapted from (Bandy) 202T).
Table 1: Entity count and representative examples for each of our datasets.
Dataset
Count
Examples
World
39585
"Los Angeles", "St. Peter's Basilica", "Caspian Sea", "Canary Islands"
USA
29997
"Fenway Park", "Columbia University", "Riverside County"
NYC
19838
"Borden Avenue Bridge", "Trump International Hote!"
Figures
37539
"Cleopatra", "Dante Alighieri", "Carl Sagan", "Blanche of Castile"
Art 31321
"Stephen King's s It", "Queen's Bohemian Rhapsody"
Headlines
28389
"Pilgrims, Fewer and Socially Distanced, Arrive in Mecca for Annual Hajj"
2.2 MODELS AND METHOSS
Data Preparation All of our experiments are run with the base Llama-2 (Touvron 2023) series of auto-regressive transformer language models, spanning billion to 70 billion parameters. For each dataset, we run every entity name through the model, potentially prepended with short prompt, and save the activations of the hidden state (residual stream) on the last entity token for each layer. For set of n entities, this yields an mmode! activation dataset for each layer.
the standard technique of probing Alain & Bengio (2016); Belinkov (2022), which fits simple model on the network activations to predict some target label associated with labeled input data. In particular, given an activation dataset E TR TRnxxmadel, and a target Y containing either the time or two- dimensional latitude and longitude coordinates, we fit linear ridge regression probes
Probing To find evidence of spatial and temporal representations in LLMs,
W = arg min Y AW + AI W 1 = A ATY W
yielding linear predictor Y = AW. High predictive performance on oft-of-sample data indicates that the base model has temporal and spatial information linearly decodable in its representations. although this does not imply that the model actually uses these representations (Ravichander et al, 2020). In all experiments, 2009) on the probe training set.
tune À using efficient leave-out-out cross validation (Hastie et al.
2.3 EVALUATION
To evaluate the performance of our probes we report standard regression metries such as I22 and Spearman rank correlation on our test data (correlations averaged over latitude and longitude for spatial fcatures). An additional metric we compute is the proximity error for each prediction, defined as the fraction of entities predicted to be closer to the target point than the prediction of the target entity. The intuition is that for spatial data, absolute error metrics can be misleading (a 500km error for a city on the East Coast of the United States is far significant than 500km error in Siberia), so when analyzing errors per prediction, we often report this metric to account for the local differences in desired precision.
LINEAR MODELS OF SPACE AND TIME
3.1 EXISTENCE
We first investigate the following empirical questions: do models represent time and space all? If so, where internally in the model? Does the representation quality change substantially with model scale? In first experiment, train probes for every layer of Llama-2-(7B: 13B, 70B) and
3

---

Published as a conference paper at ICLR 2024
World Map
USA Map
NYC Map
0.8
0.6
test R²
0.6
0.4
0.2
0.1
0.2
Historical Figures
Entertainment
Headlines
0.8
0.7
text R2
0.
0.6
0.6
0.4
0.3
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
depth
depth
depth
Uamm-2-70b
Pythie-6.9 9b
Pythie-1b
Model
Pythia
Pythia
Pythia-1.6
Pythia-16000
Figure 2: Out-of-sample R. for linear probes trained every model, dataset, and layer.
(ithia-(160M) 410M, IB, 1.4B, 2.8B, 6.9B} for each of our space and time datasets. Our main results, depicted Figure 2 show fairly consistent patterns across datasets. In particular, both spatial and temporal features can be recovered with linear probe, these representations smoothly increase in quality throughout the first half the layers of the model before reaching plateau, and the representations
accurate with increasing model scale. The gap between the Llama and Pythia models is especially striking, and we suspect due to the large difference in pre-training corpus size (2T and 300B tokens respectively). For this reason, report the rest of our results just the Llama models.
The dataset with the worst performance is the New York City dataset. This was expected given the relative obscurity of most of the entities compared with other datasets. However, this is also the dataset where the largest model has the best relative performance, suggesting that sufficiently large LLMs could eventually form detailed spatial models of individual cities.
3.2 LINEAR REPRESENTATIONS
Within the interpretability literature, there is growing body of evidence supporting the linear rep- resentation hypothesis that features within neural networks are represented linearly, that is, the pres- ence strength of a feature can be read out by projecting the relevant activation on to some feature vector (Mikolov et al. 2013b; Olah et al. 2020 Elhage et al.] (2026)). However, these results are almost always for binary categorical features, unlike the continuous features of space time.
To test whether spatial and temporal features are represented linearly, we compare the performance of linear ridge regression probes with that of substantially more expressive nonlinear MI.P probes of the form (2ReLL((W W + by) + b2 with 256 neurons. Table 2 reports our results and shows that using nonlinear probes results in minimal improvement to R² for any dataset or model. We take this as strong evidence that space and time are also represented linearly (or at the very least are linearly decodable), despite being continuous.
3.3 SENSITIVITY TO PROMPTING
Another natural question is if these spatial or temporal features are sensitive to prompting, that is, the context induce suppress the recall of these facts? Intuitively, for any entity token, an autoregressive model is incentivized to produce a representation suitable for addressing any future possible context question.
4

---

Published as a conference paper at ICLR 2024
World Map
USA Map
NYC Map
0.8
0.8
0.3
Prompt
Prompt
Prompt
test
0.6
empty
empty
empty
random
04
random
02
random
0.4
coords
coords
where_ _ii
empty all caps
where us
0.1
where_ny
Historical Figures
Entertainment
Headlines
0.8
0.8
0.7
0.7
Prompt
0.6
Prompt
0.6
Prompt
0.6
empty
empty
empty
random
0.
random
empty wo periodd
when
release
0.4
when w pepeeoddd
aivee_all_coo caps
empty caps
when wo period
0.4
0.3
layer
layer
layer
Figure 3: Out-ol-sample R² when entity names are included in different prompts for Llama-2-777 2-70b.
To study this, we create new activation datasets where we prepend different prompts to each of the entity tokens, following few basic themes. In all cases, include an "empty" prompt contain- ing nothing other than the entity tokens (and a beginning of sequence token). We then include a prompt which asks the model to recall the relevant fact, e.g., "What is the latitude and longitude of <place>" or "What was the release date <author>'s <book>." For the United States and NYC datasets also include versions of these prompts asking where in the US NYC this loca- tion is, in an attempt to disambiguate common names of places (e.g. City Hall). As baseline we include prompt of 10 random tokens (sampled for each entity). To determine if we can obfuscate the subject, for some datasets we fully capitalize the names of all entities. Lastly, for the headlines dataset, we try probing on both the last token and on period token appended the headline.
We report results for the 70B model in Figure 3 and all models in Figure g We find that explicitly prompting the model for the information, or giving disambiguation hints like that place is in the US or NYC, makes little to no difference in performance. However, we were surprised by the degree to which random distracting tokens degrades performance. Capitalizing the entities also degrades performance, though less severely and less surprisingly, as this likely interferes with "detokenizing" the entity (Elhage et al.) 2022a) Gurnee et al. 2023) Geva et al., 2023). The one modification that did notably improve performance is probing on the period token following headline, suggesting that periods are used to contain some summary information of the sentences they end.
Table 2: Out-of-sample R² of linear and nonlinear (one layer MLP) probes for all models and features at 60% layer depth.
Dataset
Model
Probe
World
USA
NYC
Historical
Entertainment
Headlines
Llama-2-75
Linear
0.881
0.799
0.219
0.785
0.788
0.564
MLP
0.897
0.819
0.204
0.775
0.746
0.467
Llama-2-136
Linear
0.896
0.825
0.237
0.804
0.806
0.645
MLP
0.916
0.824
0.230
0.818
0.808
0.656
Llama-2-70b
Linear
0.911
0.864
0.359
0.835
0.885
0.746
MLP
0.926
0.869
0.312
0.839
0.884
0.739
5

---

Published as a conference paper at ICLR 2024
4 ROBUSTNESS CHECKS
The previous section has shown that the true point in time or space of diverse types of events or locations can be linearly recovered from the internal activations of the mid-to-late layers of LLMs. However, this does not imply (or how) a model actually uses the feature direction learned by probe, as the probe itself could be learning some linear combination of simpler features which are actually used by the model.
4.1 VERIFICATION VIA GENERALIZATION
Block holdout generalization To illustrate potential issue with results, consider the task of representing the full world map. the model has, as we expect it does, an almost orthogonal binary feature for wwwnnnuuntrryy.r then one could construct high quality latitude (longitude) probe by summing these orthogonal feature vectors for each country with coefficient equal to the latitude (longitude) of that country. Assuming place is only one country, such probe would place each entity at its country centroid. However, in this case, the model does not actually represent space, only country membership, and it is only the probe which learns the geometry of the different countries from the explicit supervision.
To better distinguish these cases, we analyze how the probes generalize when holding out specific blocks of data. In particular, we train series of probes, where for each one, we hold out one country, state, borough, century, decade, or year for the world, USA, NYC, historical figure, entertainment, and headlines dataset respectively. We then evaluate the probes the held out block of data. In Table 3 we report the average proximity error for the block of data when completely held out, compared to the error of the test points from that block in the default train-test split, averaged over all held out blocks.
We find that while generalization performance suffers, especially for the spatial datasets, it is clearly better than random. By plotting the predictions of the held out states or countries in Figures [1] and [2] qualitatively clearer picture emerges. That is, the prohe correctly generalizes by placing the points in the correct relative position (as measured by the angle between the true and predicted centroid) but not in their absolute position. We take this as weak evidence that the probes extracting explicitly learned features by the model, but are memorizing the transformation from model coordinates to human coordinates. However, this does not fully rule out the underlying binary features hypothesis, as there could be hierarchy of such features that do not follow country or decade boundaries.
Table 3: Average proximity across blocks of data (e.g., countries, states, decades) when in- cluded in the training data compared to completely held out, Random performance is 0.5.
Dataset
Model
Block
World
USA
NYC
Historical
Entertainment
Headlines
Llama-2-77
nominal
0.071
0.144
0.331
0.129
0. 1.14
0.258
held out
0.170
0.192
0.473
0.133
0.158
0.264
1lama-2-133
nominal
0.068
0.144
0.319
0.121
0.141
0.223
held out
0.156
0.189
0.470
0.126
0.152
0.235
Llama-2-70b
nominal
0.071
0.121
0.262
0.115
0.105
0.182
held out 0.164
0.188
0.433
0.119
0.122
0.200
Cross entity generalization Implicit in our discussion so far is the claim that the model represents the space or time coordinates of different types of entities (like cities or natural landmarks) a unified manner. However, similar the concern that latitude probe could be weighted sum of membership features, a latitude probe could also the sum of different (orthogonal) directions for the latitudes of cities and for the latitudes of natural landmarks.

---

Published as a conference paper at ICLR 2024
Similar to the above, distinguish these hypotheses by training series of probes where the train- test split is performed to hold out all points of a particular entity class rreoors the proximity error for the entities in the default test split compared to when heldout, averaged over all such splits as before. The results suggest that the probes largely generalize across entity types, with the main exception of the entertainment dataset
Table 4: Average proximity error across entity subtypes (e.g. books and movies) when included in the training data compared to being fully held out. Random performance is 0.5.
Dataset
Model
Entity
World
USA
NYC
Historical
Entertainment
Headlines
Llama-2-7b
nominal
0.120
0.206
0.313
0.164
0.224
0.199
held out
0.151
0.262
0.367
0.168
0.305
0.289
Llama 2-13b
nominal
0.117
0.197
0.310
0.153
0.207
0.171
held out
0.147
0.259
0.377
0.159
0.283
0,266
Llama-2-70b
nominal 0.113
0.173
0.266
0.149
0.159
0.144
held out
0.147
0.203
0.322
0.149
0.271
0.219
4.2 DIMENSIONALITY REDUCTION
Despite being linear, our probes still have dmodel learnable parameters (ranging from 4096 to 8192 for the 7B to 70B models), enabling it to engage in substantial memorization. As complementary form of evidence to the generalization experiments, we train probes with 2 to 3 orders of magnitude fewer parameters by projecting the activation datasets onto their largest principal components.
Figure illustrates the test R² for probes trained on each model and dataset over a range of values, compared to the performance of the full d'mode- -iimensioal probe. We also report the test Spearman correlation in Figure [3] which increases much more rapidly with increasing & than the R². Notably, the Spearman correlation only depends on the rank order of the predictions while R² also depends on their actual value. We view this gap as further evidence that the model explicitly represents space and time these features must account for enough variance to be in the top dozen principal components, but that the probe requires more parameters to convert from the model's coordinate system to literal spatial coordinates or timestamps. We also observed that the first several principal components clustered the different entity types within the dataset, explaining why more than few are needed.
Llama-2-7b
Uama-2-13b 2-13b
Uamm--2-7b
0.6
Dataset
World Map
0.4
Map
Historical
0.2
Entertainment
Headlines
full dimension
101
PCA
dimensions
PCA dimensions
Figure 4: Test R. for probes trained activations projected onto i largest principal components for each dataset and model compared to training on the full activations.
We only do this entities which do not make up the majority of the training data (e.g., as is the case with populated places for the world dataset and songs for the entertainment dataset) which is partially responsible for the discrepancies in the nominal cases for JablesBaaddP We note in this case the Spearman correlation is still high, suggesting this is issue with bias generaliza- tion, the different entity types
uniformly distributed in time.

---

Published as a conference paper at ICLR 2024
Absolute Latitude Neuron (7b)
US East Coast Neuron (70b)
Post 2000s Art Neuron (70b)
Place Type (Spearman)
Place Type (Spearman)
Type (Spearman)
(r-0.
2.5
structure (r=0.77)
(r-0. 71)
projection on L20 h7573
(rw0
county (r-0 82)
projection LAD NS177
movie (rwO 70)
structure (r~0 city trwa 77)
-2.5
-2
Negative Latitude Neuron (136)
Histoncal Figure Neuron (7b)
Trump Era Neuron (70b)
News desk (spearman)
projection on L20 M134 Wn
10.0
N3254
progectionn on N017
7.5
Occupation (Spearman)
25
artist 77
Type (Spearman)
populated (r-.0 0...
military 72)
-25
stracture (ra.0 0.71)
researcher (r:00.1 711
natural (r--0 68)
other (rw-0.70)
-5.0
1500
Figure 5: Space and time neurons in Llama-2 models. Depicts the result of projecting activation datasets onto neuron weights compared to true space or time coordinates with Spearman correlation by entity type.
SPACE AND TIME NEURONS
While the previous results are suggestive, none of our evidence directly shows that the model uses the features learned by the probe. To address this, we search for individual neurons with input or output weights that have high cosine similarity with the learned probe direction. That is, we search for neurons which read from or write to a direction similar to the one learned by the probe.
We find that when we project the activation datasets on to the weights of the most similar neurons, these neurons are indeed highly sensitive to the true location of entities in space or time (see Fig- ure 5). In other words, there exist individual neurons within the model that are themselves fairly predictive feature probes. Moreover, these neurons are sensitive to of the entity types within our datasets, providing stronger evidence for the claim these representations are unified.
probes trained with explicit supervision are an approximate upper bound on the extent to which a model represents these spatial and temporal features, then the performance of individual neurons is a lower bound. In particular, we generally expect features to be distributed in superposition (Elhage eet al. 2022b), making individual neurons the wrong level of analysis. Nevertheless, the existence of these individual neurons, which received no supervision other than from next-token prediction, is
very strong evidence that the model has learned and makes use of spatial and temporal features.
We also perform a series of neuron ablation and intervention experiments in Appendix B to verify the importance of these neurons in spatial and temporal modeling.
RELATED WORK
Linguistic Spatial Models Prior work has shown that natural language encodes geographic in- formation (Louwerse & Zwaan 2009 Louwerse & Benesh 2012) and that relative coordinates can be approximately recovered with simple techniques like multidimensional scaling, co-occurrence statistics, or probing word embeddings (Louwerse & Zwaan 2009 Mikolov et al. 2013a Gupta et al. 2015 Konkol al.) 2017). However, these studies only consider few hundred well known cities and obtain fairly weak correlations. Most similar to our work is (Liétard et al. 2021) who probe word embeddings and small language models for the coordinates of global cities and whether
8

---

Published as a conference paper at ICLR 2024
countries share border, but conclude the amount of geographic information learned is "limited," likely because the largest model they study was 345M parameters (500x smaller than Llama 70B).
Neural World Models We consider a spatiotemporal model to be a necessary ingredient within a larger world model. The clearest evidence that such models are learnable from next-token prediction comes from GPT-style models trained on chess (Toshniwal et al. 2022) and Othello games (Li et al, 2022) which were shown to have explicit representations of the board and game state, with further work showing these representations are linear (Nanda et al. 2023). In true L.Ms. Li et al. (2021) show that entity's dynamic properties relations can be linearly read out from representations
at different points in the context. [Abdo et al. (2021) Patel & Pavlick (202T) show LLMs have representations that reflect perceptual and conceptual structure in color and spatial domains.
The point in time or space of event place is particular kind of fact. Our investigation is informed by prior work on the mechanisms factual recall in LLMs (Meng et al.] 2022a b Geva al. 2023) indicating that early-to-mid MLP layers are responsible for outputting information about factual subjects, typically on the last token of the subject. Many of these works also show linear structure, for example in the factuality of a statement (Burns al., 2022) or in the structure of subject-object relations (Hernandez et al. 2023). To our knowledge, our work is unique in considering continuous facts.
Factual Recall
Interpretability More broadly, our work draws upon many results and ideas from the inter- pretability literature (Räuker et al. 2023). especially in topics related to probing (Belinkov) 2022). BERTology (Rogers et 2021)) the linearity hypothesis and superposition (Elhage al.: 2022b) and mechanistic interpretailiity (Olah et al. 2020). More specific results related to our work include Hanna et al. (2023) who find a circuit implementing greater-than the context of years, and Goh et al.. (021) who find "region" neurons in multimodal models that resemble our space neurons.
DISCUSSION
We have demonstrated that LLMs learn linear representations of space and time that are unified across entity types and fairly robust to prompting, and that there exists individual neurons that are highly sensitive to these features. We conjecture, but do not show, these basic primitives underlie a more comprehensive causal world model used for inference and prediction.
Our analysis raises many interesting questions for future work. While we showed that it is possible to linearly reconstruct sample's absolute position in space or time, and that some neurons use these probe directions, the true extent and structure of spatial and temporal representations remain unclear. We conjecture that the most canonical form of this structure is discretized hierarchical mesh where any sample is represented as a linear combination of its nearest basis points at each level of granularity. Moreover, the model can and does use this coordinate system to represent absolute position using the correct linear combination of basis directions in the same way a linear probe would. We expect that models scale, this mesh is enhanced with more basis points, more scales of granularity (c.g. neighborhoods in cities), and more accurate mapping of entities to model coordinates (Michaud et al. 2023). This suggests future work on extracting representations in the model's 's coordinate system rather than trying to reconstruct human interpretable coordinates, perhaps with sparse autoencoders (Cunningham et al. 2023).
We also barely scratched the surface of understanding how these spatial and temporal models are learned, recalled, and used internally, or to what extent these representations exist within a more comprehensive world model. By looking across training checkpoints, it may be possible to localize a point in training when a model organizes constituent is_in i_.inpplll p ace_x x features into coherent geometry else conclude this process is gradual (Liu et al.] 2021). We expect that the model components which construct these representations are similar or identical to those for factual recall (Meng et al. 2022a Geva et al. 2023).
Finally, we note that the representation of space and time has received much more attention in biological neural networks than artificial ones (Buzsáki & Llinás 2017; [Schonbaut et al., (023). Place and grid cells Keefe Dostrovsky 1971) et 2005) in particular are among the most well-studied in the brain and may be a fruitful source of inspiration for future work on L.MMs.

---

Published as a conference paper at ICLR 2024
ACKNOWLED
The authors would like to thank Sam Marks, Eric Michaud, Ziming Liu, Janice Yang. and especially Neel Nanda for their helpful discussions and feedback. W.G. was supported by Dimitris Bertsimas and Open Philanthropy early career grant through the course of this work.
REFERENCES
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? case study in color. arXiv preprint arXiv:2109006 2021.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
Issa Annamoradnejad and Rahimberdi Annamoradnejad. Age dataset: A structured general-pus purepse dataset on life, work, and death of 1.22 million distinguished people. International AAAI Confer- ence on Web and Social Media (ICWSM), 16, 2022.
Jack Bandy. Three decades new york times headlines, 2021. URL wtt.ttttts/t.... //w:::/wwwwwoo http:::/wwwwkkggttt kaggle.
com/ htt//datassee.. hacpettdjjoonbnddy.. /eaennddyyyywwwweeee heapiyynnwwwyrrrr -ow.---ee----eeeealllll ines. Kaggle dataset.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational
Linguistics, 48(1):207-219, 2022.
Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and under- standing in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 5185-5198, 2020.
Emily M Bender, Timnit Gebru, Angelina McMillan- Majoo,, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 202 ACM conference on fairness, accountability, and transparency, pp. 610-623, 2021.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430. PMLR, 2023.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. Experience grounds lan- guage. arXiv preprint 2004 10151, 2020.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.07258 2021.
Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lec, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 23.. 2023.
Collin Burs, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. arXiv preprint arXiv:2212.03827, 2022.
György Buzsáki and Rodolfo Llinás. Space and time the brain. Science, 358(6362):482 485, 2017.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600 2023.
10

---

Published as a conference paper at ICLR 2024
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer EIShowk, Nicholas Joseph, Nova DasSarma, Ben Mann. Danny Hermandez, Amanda Askell, Ka- mal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Ka- davath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCan- dlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022a. http::/teansformee-crruuuss www.eriii...pb///22/////dddd....
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatiield-Dodds Robert Lasenby, Dawn Drain, Carol Chen, al. Toy models of superposi- tion. arXiv preprint arXiv:2209. 10652, 2022b.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. 147677 2023.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3):e30, 2021.
Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Padó. Distributional vectors encode referential attributes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 12-21, 2015.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bert- simas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint 1xXiv:2305.016// 2023.
Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard I Moser. Microstruc- ture of spatial map in the entorhinal cortex. Nature, 436(7052):801-806 2005.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than? Inter- preting mathematical abilities in pre-trained language model. arXiv preprint arXiv:2305.000 00586. 2023.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning; data mining, inference, and prediction, volume 2. Springer, 2009.
Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. arXiv preprini arXiv:2306. /2001, 2023.
Evan Hernandez, Arnb Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. arXiv preprini arXiv:2308.099 19444, 2023.
Michal Konkol, Tomáš Brycheín, Michal Nykl, and Tomáš Hercig. Geographical evaluation of word embeddings. In Proceedings of the Eighth International Joint Conference Natural Language Processing ( (Volume I: Long Pupers), pp. 224-232, 2017.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. Dbpedia- large-scale, multilingual knowledge base extracted from wikipedia, 2015. URL Version 2023.
http:: // //tpp/dibddadc///c a. .rrg
Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint 21Xiv:2106.00737, 2021.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv preprint 2rXiv:2210.13382 2022.
Bastien Liétard, Mostafa Abdou, and Anders Søgaard. Do language models know the way to rome?
arXiv preprint 2rXiv:2 2021.
11

---

Published as a conference paper at ICLR 2024
Leo Z Liu, Yizhong Wang, Jungo Kasai, Hannanch Hajishirzi, and Noah A Smith. Probing across time: What does roberta know and when? arXiv preprint arXiv:2104.07885, 2021.
Max M Louwerse and Nick Benesh. Representing spatial structure through maps and language: Lord of the rings encodes the spatial structure of middle earth. Cognitive science, 36(8):1556-
1569, 2012.
Max M Louwerse and Rolf Zwaan. Language encodes geographical information. Cognitive Science, 33(1):51-73, 2009.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual asso- ciations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022a.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory a transformer. arXiv preprint arXiv:2210.07229, 2022b.
Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling, arXiv preprint arXiv:2303. 13506, 2023.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. Advances neural information processing systems, 26, 2013a.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 2013b.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of sulf-super sequence models. arXiv preprint arXiv:2309.00941
2023. 2023.
Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective, 2023.
NYC OpenData. Points of interest, 2023. URL http:://dttaa..cyoooeeooo.. us/
Accessed: 2023-07-01.
John O' Keefe and Jonathan Dostrovsky. The hippocampus as spatial map: preliminary evidence from unit activity in the freely-moving rat. Brain research, 1971.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3)::00024000,, 2020.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In Inter- national Conference on Learning Representations, 2021.
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Maddfeld-Mennll Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)) pp. 464 IEEE, 2023.
Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? arXiv preprint urXiv:2005.007799 2020.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866, 2021.
Daniel R Schonhaut, Zahra Aghajan, Michael J Kahana, and Itzhak Fried. neural code for time and space in the human brain. Cell Reports, 42(1 1), 2023.
Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Chess testbed for language model state tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11385-1 11333, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint wrXiv:2307.09288. 2023.
12

---

Published as a conference paper at ICLR 2024
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint (tt.i::2206.076666 2022.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 214-229, 2022.
A DATASETS
We describe the construction and post-processing of our data in more detail in addition to known limitations. All datasets and code are available at patpss://itthune com com/wesg52/ wor I
We ran three separate queries to obtain the names, location, country, and associated Wikipedia article of all physical places, natural places, and structures within the DBPedia database Lehmann et al. al. (2015). Using the Wikipedia article link, we joined this information with data from the Wikipedia pageview statistics database g to query how many times this page was accessed over 2018-2020. We use this proxy for whether we should expect an LLM to know of this place or not, and filter those with less than 5000 views over this time period.
World Places
Several limitation worth highlighting. First, data only comes from English Wikipedia, and hence is skewed towards the Anglosphere. Additionally, the distribution of entity types is not uni- form, c.g. we noticed the United Kingdom has many more railway stations than any other country, which could introduce unwanted correlations in the data that may affect the probes. Finally, about 25% of the samples had some sort of state province modifier at the end Iikla County, lowa". Because many of these locations were more obscure or would be ambiguous without the modifier. so chose rearrange the string to be of the from "lowa's Dallas County" such the entity is disambiguated but that we are not probing on token that is a common country or state name.
USA Places The United States places dataset uses structures and natural places within the US from the world places dataset starting point, in addition to another DBPedia for US colleges. We then collect the name, population total, and state for every county"l census data aggregator. We then remove all duplicate county or city names (there are 31 Washington counties in the US!), though we keep any duplicates that have 2x the population has the next largest place of the same name. We also filter out cities with fewer than 500 people, zipcodes with fewer than 10000 (or with population density greater than 50 and population greater than 2000), and any place not in the lower 48 contiguous states (or Washington D.C.).
[4 zipcode F and city from a
NYC Places Our New York City dataset is adapted from the NYC Open Date points of interest dataset (NYC OpenData 2023) containing the names of locations tracked by the city government. This includes the names of schools, places of worship, transit locations, important roads or bridges, government buildings, public housing, and more. Each of these places comes with a complex ID for locations comprised of multiple such buildings (e.g. New York University or LaGuardia airport). We construct our test train splits to make sure that all locations within the same complex are put in the same split to avoid test-train leakage. We filtered out a large number of locations describing the position of bouys in the multiple waterways surrounding NYC.
Historical Figures Our historical figures dataset contains the names and occupation of historical figures who died between 1000BC-2000AD adapted from (Annamoradnejad & Annamoradnejad] 2022). We filtered the dataset to only contain the 350 most famous people who died from each decade, imperfectly measured by the index of their Wikidata entity identifier.
httesss//een.ikippdd...ooss///// http://iiiiiiieeaaaaaaeaeeeee waweriewssttttssiii.i. statistics htttps:/sinpleeass com
httpss//simplengssscom..//
13

---

Published as a conference paper at ICLR 2024
80-
60-
60.5
40-
40.7-
35-
-00
150
lingtlude
longtlude
lingttude
Figures
Enterteinmentt
Headlines
2500
1000
700
1400
1200
1500
COUPE
400-
2000
300
600
200
-500
Figure 6: Distribution of samples in space or time for all datasets.
Art and Entertainment Our art and entertainment dataset consists of the names of songs, movies, and books with their corresponding artist, director, and author release date. We constructed this dataset from DBpedia and similarly filtered out entities which had received less than 5000 page views over 2018-2020. Because many songs or books have fairly generic titles, we include the creator's name in the prompt to disambiguate (e.g. "Stephen Kings' It" for the empty prompt). However, because some artists or authors release many songs or books, we sample our test-train split by creator avoid leakage.
Headlines Our headlines dataset is adapted from scrape of all New York Times headlines of the past 30 years (Bandy 202T). In an attempt to filter out headlines which do not describe an event that could be localized in time, we employ number of strategies. First we filter anything which is not within first 10 pages of the print edition. Second we filter out articles that don't come from the Foreign. National, Politics, Washington, Obits news desks. Third we removed any titles that contained question mark.
14

---

Published as a conference paper at ICLR 2024
B NEURON ABLATIONS AND INTERVENTIONS
To better understand the role of space and time neurons in LLMs, we conduct several neuron ablation and intervention experiments.
Time Intervention We study the effect of intervening on single time neuron (L19.3610; correla- tion with and entertainment release date of 0.77) within Llama-2-7h. Given prompt of the form <media> by <creator> was wr written in 19, pin the activation of the time neuron on all tokens and sweep over a range of pinned values, and track the predicted probability the top five tokens. Results depicted in Figure 7 and show that just adjusting the time neuron activation can change the next token prediction all cases.
Blackbird the Beatles was written in 19
groabiity
0.6
0.4
0.
0.0
Godfather Coppola 19
prediction
probabiity
0.4
0.2
-4
-2
-2
0
-2
(39
activation
119 activation
119 activation
L19 3610 activation
The Stand by Stephen King was written in 19
0.8
proabiity
0.6
0.4
0.
0.
activation
3610
Figure 7: Prediction of decade of publication for famous song, movie, and book when time neuron (L19.3610) is pinned to a particular value, compared to 9 random neurons within the same layer (L19.[0-8] of Llama-2-7b.
Neuron Ablations We also study the effect of zero ablating neurons, and the contexts for which the loss increases the most. For subset of Wikipedia which includes articles corresponding to world places and contemporary art and entertainment, we first run Llama-2-7b as normal and record the loss. Then, for two space neurons and time neurons, we run the model with the neuron activation pinned to 0 (we always pin exactly one neuron to 0). For each neuron, we report the top 10 contexts in which the loss most increased for the next token prediction in Tables
15

---

Published as a conference paper at ICLR 2024
context
true token
loss increase
Bom Jesus has rather dry tropical savanna climate (Köppen
Aw
2.107
line tropical monsoon/humid subtropical climate (Köppen
Am
2.035
of The highest temperatures are reached at the end of the dry season 8.9 "C. In January, the average temperature is a Tropical wet-and-dry climate (Köppen climate classification
March
1.960
8
1.930
Aw
1.876
Goroka has relatively cool tropical monsoon climate (Köppen
Am
1.854
range from 26.4 in January to 7
1.835
wet summers and warm, very winters (Köppen climate classification tropical wet and dry climate/semi-aric climate (Köppen
Am
1.807
Aw
1.783
ably mild, tropical-maritee
climate, (Köppen climate classification
Aw
1.762
Table 5: Contexts with the highest loss when ablating space neuron L20.7573 from Llama-2-7b.
context
true token
loss increase
mark is Mount Etna, one of the tallest active volcanoes in
Europe Europe
1.971
2,800 years, making it one of the oldest cities in
1.676
keeper with 63 caps for Portugal including participation in the 198 15. Tenerife also has the largest number of endemic species in Georgia to the south-west. Mount Elbrus, the highest mountain At one point, the village boasted the longest aluminium rolling mill in centre and the leading economic hub of the Iberian Peninsula and of atican City, sovereign state-and possibly the second largest in name. This is because the British Isles were likely repopulated from the Category 4 stadium by UEFA, hosted matches at the 199
1.631
Europe
1.512
Europe
1.246
Western
1.219
Southern
1.181
Europe
1.103
1.082
1.072
Table 6: Contexts with the highest loss when ablating space neuron L20.7423 from Llama-2-7b.
context
true token
loss increase
was released in June 1993 as the fourth single from the album was released in November 1992 as the second single from her album 93. Yearwood's version was the third single from her album Hot Country Singles & Tracks chart, behind Shania Twain's" was released in February 1992 as the third single from the album and provided additional production on her singles "Like A Prayer" and was released in May 1992 as the fourth single from the album filmmaker Ramesh Aravind in Telugu cinema. P.
He
2.254
He
1.973
He
1.749
Any
1.574
What
1,559
Express
1.481
What
1.367
L
1.328
993 by record label Columbia as the second single from their second studio album Gessle for the duo's 1991 album,
Gold
1.272
Joy
1.253
Table 7: Contexts with the highest loss when ablating time neuron L18.9387 from Llama-2-7b.
context
true token
loss increase
016 the third single from Reyes debut studio album, Louder as the second radio single in support of the band's 's third studio album,
1.082
Life
0.996
Song, but ultimately lost the award Barbra Streisand's Released as the first single from the group's seventh studio album,
The
0.965
Super
0.961
original 30 Carry On films (1958-19
0.912
After listening to No Doubt's s 2002 single"
Under
0.867
ix9ine for his debut mixtape Day69 (201
0.866
BA. It originally featured on the group's fifth studio album, The ck that appeared the Porky Pig cartoons It's
Album
0.864
an
0.805
was written by Andrew Lloyd Webber and Tim Rice and produced by
Fel
0.805
Table 8: Contexts with the highest loss when ablating time neuron L19.3610 from Llama-2-77
16

---

Published as a conference paper at ICLR 2024
C
ADDITIONAL RESULTS
Uama-2-7b
Lama-2-13b
1am0-2-70b
World Map test.e
0.8
empty
prompt
random
0.6
0.6
empty. all caps
0.4
0.4
0.4
0.8
USA
USA Map test
0.6
random
empty
courds
0.4
04
0.4
where,
02
0.25
NYC Map prompt
MYC Map test
empty
random
where
0.15
02
where nyc
Hizorcal Fiquies cest R?
0.8
0.8
0.8
Historical Figures prompt
07
0.6
0.6
when
0.5
055
0.4
0.8
Entertainment prompt
0.6
empty empty all, caps
Entertaimment
04
0.4
0.0
prompt empty
0.6
empty, w_ period when wor_wwooodddd
05
0.4
0.4
0.00
0.25
0.75
100
0.25
0.50
0.75
1.00
0.00
0.25
1.00
Figure 8: Out-of-sample 122 when entity names are included different prompts for all models.
17

---

Published as a conference paper at ICLR 2024
OR
WY
True State (color)
Pred cted Medold
Figure 9: Llama-2-70b layer 50 model of the United states. Points are projections of activations of heldout US places onto learned latitude and longitude directions colored by true state, with median state prediction enlarged. All points depicted are from the test set.
True Continent
North Americe
Europe
Asia
Oceania
Figure 10: Llama-2-70b layer 50 model of the world. Points are projections of activations of heldout world places onto learned latitude and longitude directions colored by true continent. All points depicted are from the test set.
18

---

Published as a conference paper at ICLR 2024
United States
United Kingdom
India
Australia
Tue
-50
Predicted (heldout)
Trin
Italy
South Africa
Syria
50
X
latitude
-50
Japan
Poland
China
Philippines
labtude
-50
Brazil
Mexico
Sweden
Indonesia
Switzerland
Pakistan
Argentina
Netherlands
50
latrude
-50
-100
100
200
-100
100
200
-100
100
200
-100
100
200
longitude
longitude
longitude
longitude
Figure 11: Out-of-sample predictions for each country when the probe training data contains samples from the country as compared to true locations and the mean of the training data. The results imply that the learned feature direction correctly generalizes to the relative position of country but that the probes memorizes the absolute positions.
19

---

Published as a conference paper at ICLR 2024
CA
TX
PA
NY
FL
OH
40
Predicted (holdout)
Tain mean
NJ
NC
MI
GA
VA
50
40
WI
MN
MA
latitude
AZ
OK
lattude
CO
SC
KY
OR
CT
A
40
KS
AR
MS
NM
UT
WV
50
SD
ME
ND
WY
DE
lattude
40
30
-100
-100
Figure 12: Out-of-sample predictions for each state when the probe training data contains samples from the state as compared to true locations and the mean of the training data. The results imply that the learned feature direction correctly generalizes to the relative position of country but that the probes memorizes the absolute positions.
20

---

Published as a conference paper at ICLR 2024
Llama-2-7b
Llam--2-13b
Llmaa-2770b
0.6
Dataset
Map
text
0.4
Historical
USA Map
Entertainment
Headlines
0.0
10
10a
10
10a
10a
10a
PCA dimensions
PCA dimensions
PCA dimensions
Figure 13: Test Spearman rank correlation for probes trained on activations projected onto largest principal components.
21


Reviews:
[{"review_id": "RU5v0jybGs", "reviewer": "Reviewer_xxLT", "summary": "Do LLMs encode an enomous collection of superficial statistics, or do LLMs encode a coherent model of the data generation process (”a world model”)? This paper presents some evidence towards the latter. \n\nThis paper finds that Llama-2 family of models encode linear representations of space and time across multiple scales. Further, this paper finds “space neurons” and “time neurons” that reliably encode spatial and temporal coordinates.", "questions": "- Do the findings generalize to other LLMs, for example bidirectional models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper presents important evidence towards a critical debate (what do LLMs encode).\n- This paper organizes the probing studies in a way that is more comprehensive and rigorous than most probing papers that I have seen. To illustrate that the features are encoded linearly, this paper compares linear (ridge) regression probes and nonlinear MLP probes, and found that the nonlinear probes show minimal improvement in performance in any dataset or model. To illustrate the sensitivity to prompts, this paper tries many different types of prompts and discuss the effects. To test the robustness of the encoding, this paper sets up several block holdout and entity-holdout settings. These combinations of settings make the findings rigorous and compelling.\n- The dimensionality reduction and space & time neuron experiments make this paper even more appealing.", "weaknesses": "The experiments only involve Llama-2 series models, whereas various locations in the paper stretches the claim to be about all LLMs and modern LLMs. I recommend rephrasing some texts into e.g., “LLMs, with Llama as examples” to make the claims better supported by the scope of the experiments.\n\nTypo and comments:\n\n- There are some minor typos. In page 6, a punctuation is needed before the footnote. In the end of the next paragraph, the period should be before the footnote.\n- A related work should be referred: [Distributional vectors encode referential attributes](https://aclanthology.org/D15-1002) (Gupta et al., EMNLP 2015) This is one of the earliest probing papers, and it probed for the attributes involving geographic locations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "czIr7YCGu8", "reviewer": "Reviewer_eYoU", "summary": "This work examines whether representations of entities from the Llama-2 family language models represent important properties such as spatial coordinates for famous landmarks and years for famous people and events. This is done via training linear ridge regression probes on top of the representations extracted from activations at individual layers of the models. The results are that across several datasets, the LLM representations encode space and time information, and that these representations improve through the first half of the model layers and are then stable through the remainder of the model.", "questions": "Q1. Are the neurons that the authors point out as aligning with the space/time directions sufficient, necessary, or not even sufficient for representing these properties? In other words, what happens to the space/time representations if these neurons are ablated and the ridge regression probes are relearned?\n\nOther questions: please respond to weaknesses 2 and 3 above.\n\nMinor point: \npage 3 is the first time when it's clear that what is meant by a spatial representation is the two-dimensional latitude and longitude coordinates. This should be made clear earlier in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- examine an interesting and timely question that will be relevant to the ICLR community\n- well-designed experiments to rule out confounding factors\n- investigate several different open source models", "weaknesses": "W1. This work uses established methods for probing and there is no methodological innovation, though this is on its own not a deal breaker\n\nW2. The research question is motivated as trying to study whether LLMs build a world model, but it's not clear to me why learning important properties of famous landmarks, such as location, and of famous events, such as years, are a sign of a \"world model\". My guess is that these properties co-occur with the names of the landmarks, events, and people in the training datasets and this is how they are learned by the LLM. It would be helpful if the authors can explain more about why they think that the LLM learning these properties is a sign of learning a world model.\n\nW3. Only results from probing experiments are provided as evidence, and only when either the entity is given directly to the model as input or when a prompt that agrees with the task is appended to the context (\"What is the location of..\"). It would be informative to present the accuracy of the model under those prompts (e.g. is the probing needed to answer the question of whether the model has this information?), and also to show whether the representation of time and space is still decodable even if the task the model is asked to perform is not related to the time/location or is even adversarially related.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YMiyNR5plu", "reviewer": "Reviewer_XR65", "summary": "This paper studies the spatial and temporal data representations for LLMs. It utilizes three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) to analyze the internal representations in the Llama-2 model family. The study finds that LLMs develop linear, robust representations of space and time, unified across various entity types. It also identifies specific \"space neurons\" and \"time neurons\" within these models, indicating that LLMs form structured knowledge about space and time. The research, which employs linear regression probes on model activations to predict real-world locations or times, underscores the potential of LLMs in developing complex world models, impacting their robustness and application in AI systems.", "questions": "- What if the space or time neurons are removed/masked? How would it influence the models’ understanding of space or time?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper presents an exploration into the internal workings of large language models (LLMs), particularly focusing on how these models internalize and represent continuous variables including spatial and temporal dimensions. \n2. The investigation is well-conducted, employing rigorous experiments and analysis methods to support that the probing results are not merely superficial statistics. The identification of specific \"space neurons\" and \"time neurons\" within these models is an innovative aspect to understand LLMs.\n3. This study provides evidence that large language models possess an intrinsic comprehension of world models. This insight can enhance researchers' understanding of how information is represented within large language models.", "weaknesses": "1. One of the paper's limitations is its potential overlap with findings already established in the word embedding literature. Earlier works in this area, such as those by Mikolov et al. (2013), have demonstrated that simpler word embedding models can capture relational information and regularities in a linear fashion. This raises the question of whether the findings in this paper are genuinely novel or simply an extension of what is already known about linear representations in language models. The paper could strengthen its contribution by more directly addressing how its findings with large language models (LLMs) differ significantly from those with simpler models, and by delving deeper into the implications of these differences for the field.\n2. The assertion that LLMs learn literal 'world models' may be overstated. The concept of a 'world model' in the context of AI and cognitive science typically refers to a comprehensive and dynamic representation of the external world, including an understanding of causal relationships and the ability to predict future states. The paper's findings, while impressive, primarily demonstrate that LLMs can encode spatial and temporal information linearly. This is a far cry from the richer and more dynamic conception of a world model. A more accurate claim might be that LLMs are capable of forming structured and useful representations of spatial and temporal information, but these do not necessarily constitute comprehensive world models. Further exploration into how these representations are utilized by LLMs in real-world tasks, and how they compare to human cognitive processes, could provide a more nuanced understanding of their nature and limitations.\n3. Though the analysis experiments are useful and abundant within the given datasets and LLMs, the scope of data and experiments is limited. In particular, the datasets used, while diverse, primarily represent well-known entities and locations. This could limit the generalizability of the findings to less common or more ambiguous entities. The study largely focuses on data in English and from a Western perspective. Exploring how these models represent space and time in other languages and cultural contexts could provide valuable insights into their versatility and limitations. Also, the study is conducted on the Llama-2 model family. Testing the findings across different models and architectures would strengthen the argument that these capabilities are inherent to LLMs in general, rather than specific to a particular model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HFw8ZIRtW4", "reviewer": "Reviewer_MD6k", "summary": "This paper probes language models (specifically the Llama-2 type) for representations of spatial location of place names at different levels of granularity, as well as placement in time of people, products and events.\nActivations in language models are mapped to explicit geographical and temporal coordinates via linear probes. The paper finds that the tested language models in general have robust representations of both spatial and temporal location; locations within New York City however are mapped with relatively lower accuracy. Additionally the paper localizes specific neurons especially sensitive to location in either space or time.", "questions": "Proximity error is defined as \"fraction of predictions closer to the target point than the actual prediction\". I don't understand what this means; should this read \"fraction of datapoints closer to the target point than the actual prediction\"?\n\nI don't understand how the mere presence of specific neurons correlated to spatial/temporal probes counts as evidence that these representations are used by the model. A more convincing evidence would involve some intervention on these neurons.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The works uses straightforward methods and well thought out experimental setup. Experiments are exhaustive. The results are presented in a clear and easy to follow fashion.", "weaknesses": "The main problem with this work is the serious lack of awareness and engagement with very similar work carried out not so long ago [1,2]. As a results, the paper doesn't acknowledge that qualitatively similar results obtain even in extremely simplistic models applied to text such as SVD and LSA, and overinterprets the results as evidence of the language models analyzed possessing \"a coherent model of the data generating process—a world model\". \nIf the spatial and temporal representations found here count as a world  model, it's a a very partial and almost trivial one. It certainly is very far from a complete model of the process generating textual data in general. The paper would be much improved if it seriously toned down these overly dramatic claims.\n\nA minor methodological quibble: longitude wraps around, so linear correlation is not an ideal measure here. \n\n[1] Louwerse, M.M., & Zwaan, R.A. (2009). Language Encodes Geographical Information. Cognitive science, 33 1, 51-73 .\n\n[2] Louwerse, M.M., & Benesh, N. (2012). Representing Spatial Structure Through Maps and Language: Lord of the Rings Encodes the Spatial Structure of Middle Earth. Cognitive science, 36 8, 1556-69 .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)