Monotonic Representation of Numeric Properties in Language Models (2024)

Abstract: Language models (LMs) can express factual knowledge involving numeric properties such as Karl Popper was born in 1902. However, how this information is encoded in the model’s internal representations is not understood well. Here, we introduce a method for finding and editing representations of numeric properties such as an entity’s birth year. We find directions that encode numeric properties monotonically, in an interpretable fashion. When editing representations along these directions, LM output changes accordingly. For example, by patching activations along a "birthyear" direction we can make the LM express an increasingly late birthyear. Property-encoding directions exist across several numeric properties in all models under consideration, suggesting the possibility that monotonic representation of numeric properties consistently emerges during LM pretraining. Code: https://github.com/ bheinzerling/numeric-property-repr A long version of this short paper is available at: https://arxiv.org/abs/2403.10381

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 175–195 August 11-16, 2024 ©2024 Association for Computational Linguistics
Language models (LMs) can express factual knowledge involving numeric properties such as Karl Popper was born in 1902. However, how this information is encoded in the model’s internal representations is not understood well. Here, we introduce a method for finding and editing representations of numeric properties such as an entity’s birth year. We find directions that encode numeric properties monotonically, in an interpretable fashion. When editing representations along these directions, LM output changes accordingly. For example, by patching activations along a "birthyear" direction we can make the LM express an increasingly late birthyear. Property-encoding directions exist across several numeric properties in all models under consideration, suggesting the possibility that monotonic representation of numeric properties consistently emerges during LM pretraining. Code: https://github.com/ bheinzerling/numeric-property-repr
A long version of this short paper is available at: https://arxiv.org/abs/2403.10381

heading: 1 Introduction
text: Language models (LMs) can express factual knowledge (Petroni et al., 2019; Jiang et al., 2020; Roberts et al., 2020; Heinzerling and Inui, 2021; Kassner et al., 2021). For example, when queried In which year was Karl Popper born? Llama 2 (Touvron et al., 2023) gives the correct answer 1902. While the question if LMs “know” anything at all is subject of debate (Bender and Koller, 2020; Hase et al., 2023b; Mollo and Millière, 2023; Lederman and Mahowald, 2024), empirical work has progressed from behavioral analysis focused on the accuracy and robustness of knowledge expression (Shin et al., 2020; Jiang et al., 2021; Zhong et al., 2021; Youssef et al., 2023) to representational analysis aimed at understanding how fac-
tual knowledge is encoded1 in model parameters (De Cao et al., 2021; Mitchell et al., 2021; Meng et al., 2022) and activations (Hernandez et al., 2023; Merullo et al., 2023; Geva et al., 2023; Gurnee and Tegmark, 2023).
However, representational analysis has mainly targeted entity-entity relations such as Warsaw is the capital of Poland. How LMs encode factual knowledge involving numeric properties, such as an entity’s birthyear, is less understood. Unlike entity-entity relations, numeric properties have natural ordering and monotonic structure. While this structure is intuitive for humans, LMs encounter numeric properties mostly in form of unstructured textual mentions. This raises the question if LMs learn to represent numeric properties appropriately, according to their structure.
Here, we devise a simple method for identifying and manipulating representations of numeric properties in LMs. We find low-dimensional subspaces that strongly correlate with numeric properties across models and numeric properties, thereby confirming and extending prior observations of representations of numeric properties in LMs (Liétard et al., 2021; Faisal and Anastasopoulos, 2023; Gurnee and Tegmark, 2023; Godey et al., 2024). Going beyond prior work (see §A), we show that by causally intervening along directions in these subspaces, LM output changes correspondingly. That is, we find a monotonic relationship between the intervention and the quantity expressed by the LM. For example, an entity’s year of birth shifts according to the strength and sign of the intervention along a “birthyear” direction (Fig. 1). Taken together, our findings suggest that LMs represent numeric properties in a way that reflects their natural structure and that such monotonic representations consistently emerge during LM pretraining.
1We say “X is encoded in Y” as shorthand for “X can be easily extracted from Y”. See caveats in §5.
175
Terminology. We briefly clarify important terms. A quantity consists of a scalar numeric value paired with a unit of measurement. A numeric property is a property that can be described by a quantity, e.g., birthyear, population size, geographic latitude. A numeric attribute is an instance of a numeric property, associated with a particular entity. For example, Karl Popper has the numeric attribute birthyear:1902. By linear representation we denote the idea that a numeric attribute is encoded in a linear subspace of a LM’s activation space. A monotonic representation is a linear representation characterized by a monotonic relationship between directions in activation space and the value of the encoded numeric attribute. That is, as activations shift along a particular direction the value of the corresponding numeric attribute increases or decreases monotonically.

heading: 2 Finding Property-Encoding Directions
text: Motivation. While numeric properties generally map naturally onto simple canonical structures, such as number lines or coordinate systems, it is not obvious that pretraining on largely unstructured data enables LMs to appropriately represent such structures. Our goal is to find out if and how numeric properties are encoded in the geometry of LM representations. How could such an encoding look like? Based on two arguments, we hypothesize that numeric properties are encoded in lowdimensional linear subspaces of activation space.
The first argument rests on a key principle in representation learning: a model generalizes if and only if its representations reflect the structure of the data (Conant and Ashby, 1970; Liu et al.,
2022). To the degree that current LMs generalize, in the sense of achieving non-trivial performance on benchmarks involving knowledge of numeric properties (Petroni et al., 2019), we can expect that their representations reflect the structure of numeric properties. Since the natural structure of many numeric properties is low-dimensional, we expect to find low-dimensional structure in the representations of a well-performing model.
As second argument we adduce the linear representation hypothesis, which posits a correspondence between concepts and linear subspaces (Elhage et al., 2022; Park et al., 2023; Nanda et al., 2023). If the linear representation hypothesis is true,2 this would imply that numeric properties are encoded in linear subspaces. For brevity, we will call a low-dimensional linear subspace of a LM’s activation space a direction, regardless of whether it is one- or multi-dimensional.
Method. Motivated by the hypothesis that numeric properties are encoded as directions in activation space, we now devise an experimental setup for finding out if such directions exist. A common choice for identifying linear structure is principal component analysis (PCA; Pearson, 1901). However, PCA looks for directions of maximum variance, while we want to find directions that maximally covary with model outputs. This kind of problem can be solved with partial least squares regression (PLS; Wold et al., 2001).
Concretely, for a given numeric property we collect n entities that have this property. For each
2For positive evidence, see Marks and Tegmark (2023); Merullo et al. (2023); Tigges et al. (2023); Jiang et al. (2024)
entity e we encode a prompt with a LM to obtain entity representation xe of dimension d. That is, X = [x1 · · · xn]T ∈ Rn×d. We also collect the quantity ye expressed by the LM, i.e., Y = [y1 · · · yn]T ∈ Rn. Having collected entity representations X and associated LM outputs Y , we fit a k-component PLS model to predict Y from a k-dimensional subspace of X . We vary the number of components k and record goodness of fit via the coefficient of determination R2.
Results. After selecting six frequent numeric properties in Wikidata (Vrandečić and Krötzsch, 2014), for each property we sample n = 1000 popular3 entities and prompt the LM (in English) for the corresponding attribute (See samples of entities and prompts in §B). As entity representation we take the hidden state of the entity mention’s last token at layer l, choosing l as described in §F.
PLS regression results for Llama 2 13B representations are shown in Fig. 2 and results for additional models in §C. All properties can be predicted well (R2 ≥ 0.79), with the exception of elevation (R2 = 0.43). Across all six properties, PLS identifies small sets of predictive components. For example, a PLS model with k = 7 components achieves a goodness of fit of R2 = 0.91 when predicting birthyear attributes from entity repre-
3We define popular entities as those in the top decile of the rank mean of Wikidata degree and Wikipedia article length.
20 0 20 PLS component 1
20
10
0
10
PL S
co m
po ne
nt 2
-3760
1832
1905
1937
1962
2468

heading: 3 Effect of Property-Encoding Directions
text: of attribute instances i, j with quantities qi ̸= qj and their representations x⃗i, x⃗j , there exists a steering vector u⃗ so that x⃗i − x⃗j ∈ Cone(u⃗), where Cone(v⃗) = {αv⃗ : α > 0} is the cone of vector v⃗.
Linearity of representations requires that representations lie in a cone, but says nothing about their ordering. To model the structure of numeric properties, we introduce the constraint that the ordering of quantities is preserved in representation space.
Definition 2 (Monotonic representation of numeric properties). A numeric property is represented monotonically if it is represented linearly in Cone(u⃗) and for all triples of attribute instances h, i, j with quantities qh > qi > qj and representations x⃗h, x⃗i, x⃗j the following holds: x⃗h − x⃗j = αhj u⃗ and x⃗i − x⃗j = αij u⃗ if and only if αhj > αij .
There are many ways to operationalize this definition. One is to prepare a series of monotonic representations in Cone(u⃗) by varying α and then testing if these representations yield monotonic output changes, which is what we will do now.
Method. Viewing the LM as a causal graph (Meng et al., 2022; McGrath et al., 2023), we intervene via activation patching (Vig et al., 2020; Wang et al., 2022; Zhang and Nanda, 2024) and observe the effect on model output. Unlike the common setup in which one replaces activations from one input with activations from a different input, we
patch activations along directions, similar to the manipulation method of Matsumoto et al. (2022).
Specifically, for each of the top K directions u⃗k ∈ Rd, k ∈ [1 . .K] found by PLS, we prepare patches p⃗s,k = αsu⃗k with edit weights αs and step index s ∈ [1 . . 80]. Lacking a principled method for choosing edit weights αs, we set their range to the minimum and maximum PLS loadings on each property’s training split. This choice yields patches covering the empirical range of activation projections onto direction u⃗k. After sampling ntrain = 1000 popular entities for each of the six numeric properties we first fit PLS models for each property, then apply activation patches p⃗s,k to the representations of ntest = 100 held-out entities and for each entity record the LM’s expressed quantity ys,k. To evaluate monotonicity, i.e., the notion that small (large) edit weights αs should have a small (large) effects and that negative (positive) weights should decrease (increase) the expressed quantity ys,k, we quantify the intervention effect via the ranked Spearman correlation ρ(αs,k; ys,k).
Results. We are interested in the effects and side effects on model output when patching activations along property-specific directions. Looking at effects first, we plot mean effects of directed activation patching across six numeric properties in Fig. 4. We see that there are properties
for which directed activation patching has highly monotonic effects, e.g., birthyear (ρ = 0.84), elevation (ρ = 0.88), or work period start (ρ = 0.90), suggesting that these properties have highly monotonic representations. Other properties exhibit a smaller degree of monotonic editability, e.g., longitude (ρ = 0.55) and population (0.65), suggesting that LM representations do not encode these properties as well. Figures for other models (see §G) lead to similar conclusions.
Having observed the effects of our interventions we now turn to their side effects on the expression of properties that were not the target of the intervention. For example, if we fitted a PLS regression to find “birthyear” directions, birthyear is our targeted property and all other properties, such as death year or longitude are non-targeted properties. Using the directions found in §2, we prompt LMs for non-targeted attributes, perform activation patching with weight αs along a direction found for the targeted property and record expressed quantities y′s,k. To see if non-targeted properties are affected in a similar monotonic fashion as targeted ones, we quantify the side-effect of directed activation patching as the mean Spearman correlation ρ(αs, y′s,k), taken over 100 entities per property. We perform this procedure for all combinations of targeted and non-targeted properties, including three additional properties, and show results in Fig. 5. In this figure, diagonal entries show the mean effect on targeted properties and off-diagonal entries the size of side-effects. For Llama 2 7B, the mean effect size ρ̄ = 0.65 ± 0.12 (mean of diagonal entries),
is not much larger than the mean side-effect size ρ̄ = 0.53 ± 0.11 (mean of off-diagonal entries). In contrast, for Llama 2 13B the effect size of ρ̄ = 0.85 ± 0.07 is much larger than the size of side effects (ρ̄ = 0.58 ± 0.18). A plausible explanation is that in Llama 2 7B properties share a subspace which encodes generic numeric or smalllarge ranges that are mapped to specific quantities depending on context, while the representation space of Llama 2 13B is more akin to a mixture of generic-numeric and property-specific subspaces. More work is needed to test this hypothesis.
The analysis of side-effects is complicated by real correlations between properties: Birthyear and death year distances are bounded by the human life span, latitude and population are correlated since the Earth’s northern hemisphere is more populous, etc. Consequently, one might argue that, say, editing an entity’s birthyear should also affect LM output when querying the entity’s death year.

heading: 4 Conclusions
text: We used partial least-squares regression to identify low-dimensional subspaces of activation space that are predictive of the quantity an LM expresses when queried for numeric attributes such as an entity’s birthyear. We then performed activation patching along directions in these subspaces and observed corresponding changes in model output. Our results suggest that LMs learn monotonic representations of numeric properties and that these representations exist in all of the examined LMs.

heading: 5 Limitations

heading: 5.1 General limitations of representational analysis
text: None of the language models studied in this work are embodied agents or otherwise capable of embodied cognition. Lacking direct sensorimotor grounding (Harnad, 1990; Mollo and Millière, 2023; Harnad, 2024), LMs cannot directly perceive, let alone precisely measure, the numerical attributes of which we claim to have found monotonic representations. It follows that any such representations are an artifact of distributional patterns in their training data, and that the best one can hope for is isomorphy between model representations and the properties of the real-world entities to which we tie those representations.
Leaving the groundedness of representations aside, the idea that concepts, knowledge, or behavior are “encoded” in neural representations might seem intuitively appealing, but has been strongly criticized, on theoretical grounds in the context of biological and artificial neural networks in general (Brette, 2019), and on empirical grounds in the context of pretrained language models in particular (Hase et al., 2023a; Niu et al., 2024).
Analysis of LM representations also has wellknown limitations. Under the mild assumption that there exists a bijection between inputs and their representations, all information extractable from the input, i.e., the natural language prompt, can also be extracted from the LM’s representation of that sequence (Pimentel et al., 2020b). Hence the question to be answered by representational analysis is not whether a feature of interest can be extracted or not, but how easy it is to extract. How to best quantify “ease of extraction” (Pimentel et al., 2020b) is an open question, although methods have been proposed (Pimentel et al., 2020a; Voita and Titov, 2020).

heading: 5.2 Specific limitations of the representational analysis conducted in this work
text: The low-dimensional linear subspaces found in this work allow relatively “easy” extraction when compared to the nominally high dimensionalities of activation space, but are still higher-dimensional than necessary, since the represented structures (e.g., years, geographic coordinates) are canonically oneto two-dimensional. Furthermore, activation space is nominally high-dimensional but its intrinsic dimension is believed to be much lower (Li et al.,
2018; Aghajanyan et al., 2021; Razzhigaev et al., 2024). For example Razzhigaev et al. (2024) provide estimates for the intrinsic dimension of various LMs, ranging from about 10 to 70 dimensions (the models used in our experiments are not covered). If we view a non-linear, non-monotonic representation of full intrinsic dimensionality as the most complex encoding with worst-case ease of extraction, and one- to two-dimensional linear monotonic encodings as the simplest representation with optimal ease of extraction, then the low-dimensional subspaces we found fall somewhere between these bounds. Whether they are low-dimensional relative to the models’ intrinsic dimension is currently unknown. Put differently, if the intrinsic dimension of Llama 2 7B turns out to be, say, 10, then finding, a 10-dimensional subspace that encodes all latitude information (see §D) is not surprising, but necessary.
While we found evidence for monotonic representation of numeric properties, it is likely that our causal interventions via activation patching along one-dimensional directions are too simplistic, considering the fact that according to our PLS regression results, numeric properties are encoded in lowbut not one-dimensional subspaces. Hence it is possible that a more refined editing method operating on higher-dimensional directions will allow more precise control over LM output. Furthermore, our analysis is limited to popular entities, frequent numeric properties, and English queries, i.e., the combination most likely to be well-represented in the LM training data.
Acknowledgements. This work was supported by JST CREST Grant Number JPMJCR20D2 and JSPS KAKENHI Grant Number 21K17814.

heading: A Additional related work
text: Shaped by the locality of physical reality, the locality of human experience (Prystawski et al., 2023) gives rise to distributional patterns of language use. Such patterns include patterns of geographic and temporal coherence (Heinzerling et al., 2017), which reflect spatiotemporal proximity of real-world entities. These patterns can be picked up by statistical models and allow, e.g., to predict geographic information from co-occurrence statistics of cities mentioned in news articles (Louwerse and Zwaan, 2009). Probing static word vector representations for numeric attributes of geopolitical entities, Gupta et al. (2015) obtain good relative rankings, but do not evaluate absolute values nor analyze the geometry of representations. Continuing this line of research, Liétard et al. (2021) probe LM representations for GPS coordinates. Perhaps due to the—by current standards—small scale of the studied LMs, they find only limited success but report that larger models appeared to encode more geographic information. Faisal and Anastasopoulos (2023) measure how well the geographic proximity of countries can be recovered from LM representations but differ from our work in their focus on the impact of politico-cultural factors.
Closest to our work is the analysis of geo-temporal information encoded in Llama 2 representations by Gurnee and Tegmark (2023). Our work corroborates their finding of linear subspaces of activation space which are predictive of numeric attributes, but is distinct in three important aspects. First, as we show in §2, the subspaces found PCA, as used by Gurnee and Tegmark, are of considerably higher dimensionality (50− 100) than the subspaces found by partial least-square regression (2− 17). Our finding thus tightens the upper bound on the complexity of numeric property representation in recent LMs. Second, we make explicit and formalize the notion of monotonic representation. Third, our interventions via directed activation patching (§3) found one-dimensional directions with fine-grained effects on the expression of numeric attributes, across all numeric properties and models we analyzed, thereby establishing a causal relationship between monotonic representations and LM behavior.

heading: B Data sample

heading: C Regression on entity representations: Additional figures

heading: D Regression on entity representations: Additional analysis
text: E PLS projections of entity representations: Additional figures

heading: F Choice of probing and edit locus
text: Varying token position and layer, we edit the hidden state at this locus as described in §3 and record the Spearman correlation between edit strength and the change in the quantity (here: birthyear) expressed by the model. Correlation is highest (0.99) in the region between layers 0.2 and 0.4 and the last subword token of the entity mention and the following token. Based on this, we choose the last mention token and the middle point at layer l = 0.3 as locus for the regression experiments in §2 and activation patching experiments in §3, across all numeric properties and LMs, but acknowledge that a more exhaustive search would likely find better probing and edit loci.
A question left open so far is where activation patching should be performed. While automatic methods for localizing model components and subnetworks of interest have been proposed (Conmy et al., 2023;
Kramár et al., 2024), for simplicity we perform a coarse search across layers and token positions for one numeric property and use the found setting for all experiments (see §F). In addition to this edit locus, we also search for an edit window, whose purpose is to counteract iterative inference effects (McGrath et al., 2023; Rushing and Nanda, 2024). Layer-wise we find that a window of ±2 layers around the edit locus is most effective, which is smaller than the ±5 layers used in prior work (Meng et al., 2022; Hase et al., 2023a). We also implement a token-wise window (Monea et al., 2024), finding that in addition to the last entity mention token, patching up to two token representations to the left and one token representation to the right works best for the prompts in our experiments. Typically, this token window size covers the entity mention and the main verb or last token of the prompt, depending on the numeric property (see prompts in §B). In summary, we patch activations in a 5-layer window centered on layer l = 0.3 and an up-to 4-token window surrounding the last entity mention token. To improve output format adherence, we append the instruction One word answer only to all prompts.

heading: G Edit curves for additional language models

heading: H Effect of property-encoding directions: Model output examples
text: Table 3 gives examples of how numeric attribute expression changes as a result of directed activation patching. Patching along “birthyear” directions results in the expression of different years, although the degree of monotonicity, as quantified by Spearman correlation ρ, varies. Patching along the top “population” direction causes the model to generate a range of outputs that can be interpreted as population sizes, although the largest values are more suited to a planetary than a municipal scale. The sequence of outputs has rather sudden jumps, e.g., from 40,000 (unedited model, αs = 0.00) to 10 million after taking the first step in the “larger population” direction (αs = 0.10). The pattern of jumps and plateaus is plausibly connected to several factors such as tokenization effects and the likely high frequency of certain numerals (1.3 billion: population of China at some point in time; 7.5 billion: population of Earth, etc.) in the training data, but we leave a detailed investigation to future work. The pattern also indicates that activation space, while apparently monotonic, is not linear in this direction. The intervention also induces a switch from positional notation (40,000) to named numbers (million, billion), which showcases effects beyond single tokens.

heading: I Software
text: The following is a list of the main libraries used in this work:
• Numpy (Harris et al., 2020)
• Scikit-learn (Pedregosa et al., 2011)
• Pytorch (Paszke et al., 2019)
• Transformers (Wolf et al., 2020)
• seaborn (Waskom, 2021)
• Matplotlib (Hunter, 2007)
• SciPy (Virtanen et al., 2020)
• Pandas (Pandas development team, 2020)
We thank all authors and the open source community in general for creating and maintaining publicly and freely available software.

heading: Future Work
text: ) in the training data, but we leave a detailed investigation to future work. The pattern also indicates that activation space, while apparently monotonic, is not linear in this direction. The intervention also induces a switch from positional notation (40,000) to named numbers (million, billion), which showcases effects beyond single tokens