Venue: NAACL 2025

# The Geometry of Numerical Reasoning:

Language Models Compare Numeric Properties in Linear Subspaces

 Ahmed Oumar El-Shangiti\({}^{1}\) Tatsuya Hiraoka\({}^{1}\) Hilal AlQuabeh\({}^{1}\)

Benjamin Heinzerling\({}^{3,2}\) Kentaro Inui\({}^{1,2,3}\)

\({}^{1}\) Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

\({}^{2}\)Tohoku University

\({}^{3}\)RIKEN

ahmed.oumar@mbzuai.ac.ae

###### Abstract

This paper investigates whether large language models (LLMs) utilize numerical attributes encoded in a low-dimensional subspace of the embedding space when answering questions involving numeric comparisons, e.g., _Was Cristiano born before Messi?_. We first identified, using partial least squares regression, these subspaces, which effectively encode the numerical attributes associated with the entities in comparison prompts. Further, we demonstrate causality, by intervening in these subspaces to manipulate hidden states, thereby altering the LLM's comparison outcomes. Experiments conducted on three different LLMs showed that our results hold across different numerical attributes, indicating that LLMs utilize the linearly encoded information for numerical reasoning.

## 1 Introduction

Language models (LMs) store large amounts of world knowledge in their parameters Petroni et al. (2019); Jiang et al. (2020); Roberts et al. (2020); Heinzering and Inui (2021); Kassner et al. (2021). While prior work has evaluated parametric knowledge mainly via behavioral benchmarks, more recent work has analyzed how knowledge is represented in activation space, for example, localizing relational knowledge to specific layers and token representations Meng et al. (2022); Geva et al. (2023); Merullo et al. (2024) or identifying subspaces that encode numeric properties such as an entity's birth year Heinzerling and Inui (2024). However, analysis of LM-internal knowledge representation has been limited to simple factual recall, e.g., for queries like "When was Cristiano born?" Answer: 1985 or "When was Messi born?" Answer: 1987. If and how the mechanisms responsible for simple factual recall also participate in more complex queries, e.g., "Is Cristiano older than Messi?", is not understood so far. A possible mechanism by which an LLM answers this query is a multi-step process consisting of first recalling the respective birth years of the two entities, comparing the two years, and then selecting a corresponding answer.

Herein, we focus on LLM's ability of arithmetic operations Dehaene (2011). The LLM's ability to handle numbers has been discussed after the advent of pre-trained language models Spithourakis and Riedel (2018); Wallace et al. (2019). With modern LLMs such as the LLaMA family Touvron et al. (2023), Heinzerling and Inui (2024) shows that LLMs map numerical attributes such as _Cristiano, born-in, 1985_) and _Messi, born-in, 1987_) to low-dimensional (Linear) subspaces and prove that those subspaces are used during knowledge extraction. However, it is not clear whether the LLMs use those subspaces to solve logical reasoning such as the relation _Cristiano, born-before, Messi_).

In this study, we tackle the research question: **do LLMs leverage the linear subspace of entity-numerical attributes when solving numerical reasoning tasks?** We investigate whether the linear subspace is indeed used in the logical reasoning tasks. We first show the LLMs' capability to solve

Figure 1: Summary of our approach. We extract contextualized numeric attribute activations and then train \(k\)-components PLS model on the activations to predict their values and then use the first component of the PLS model to do an intervention at the last token of the second entity in the logical comparison.



the numerical reasoning tasks from the viewpoint of behavioral observation: testing the performance of the reasoning task with in-context learning (SS3). We then examine the representations of LLMs (SS4). We identify the linear subspace corresponding to the numerical attributes with partial least-squares (PLS [20]) and intervene in the representation to test whether the model utilizes the linearly represented information (see Figure 1).

The experimental results on the three numerical properties (the birth/death year of a person and the latitude of location) and on three LLMs (LLama3 8B [5], Mistral 7B [6], and Qwen2.5 7B [9] all instruction based models) demonstrate that LLMs leverage the numerical information represented in the linear subspace for the reasoning tasks.

## 2 Outline of Experiments

This section outlines our methodology to investigate the process of LLMs to solve the numerical reasoning.

### Model and Dataset

In this work, we focus on the three numerical properties: the birth years of person entities, the death years of person entities, and the latitudes of location entities. Table 1 exemplifies the questions and expected responses for both tasks. For the knowledge extraction task, we create the question-answer pairs by extracting 5,000 entities alongside their numerical attributes from Wikidata [10]. After filtering out entities that the LLM does not know (SS3.1), we created the 5,000 questions about numerical reasoning that include two entities each. For all experiments, we used Llama3-8B-instruction following model [5] as the LLM and later validate our finding on two additional models (see SS 4.3).

### Design of Experiments

We conducted the experiments in two phases to investigate the LLM's ability to utilize the linear subspace for numerical reasoning.

Data Pre-processing (SS3):We began by evaluating the LLM's ability to handle both knowledge extraction and numerical reasoning tasks by inputting questions and evaluating its response. To focus the subsequent experiments on entities for which the LLM has reliable numerical knowledge, we filtered out any entities that the LLM could not answer correctly during this initial behavioral experiment.

Internal Representation Experiments (SS4):In the second phase, we examined the inner workings of the LLM when solving the knowledge extraction (SS4.1) and the numerical reasoning (SB.1). Here, we focus on analyzing the hidden state of each entity representation at a particular layer for knowledge extraction. For the case of numerical reasoning, we investigated the activations of the last token's representation. We denote the hidden state of the \(i\)-th input at the \(l\)-th layer as \(h_{i}^{(l)}\). To investigate whether knowledge of numerical attributes is stored in low-dimensional subspaces, we applied PLS [20] for each representation [11]. Partial Least Squares (PLS) offers an alternative to Principal Component Analysis (PCA) for dimensionality reduction, especially when predicting one set of variables from another. PLS seeks to maximize the covariance between the input matrix **X** and the response matrix **Y** by projecting both onto a latent space. Through PLS, we identified components that represent the linear structure of each numerical attribute, allowing us to analyze how the LLM might utilize these subspaces for reasoning. To further test this, we intervened in the hidden state \(h_{i}^{(l)}\) by incorporating the 1st PLS component \(v\), as follows:

\[h_{i}^{(l)}\gets h_{i}^{(l)}+\alpha\frac{v}{\|v\|}, \tag{1}\]

where \(\alpha\) is a hyperparameter derived from the first PLS component, and \(||v||\) is the Euclidian norm (L2-norm) of the vector \(v\). Intuitively, this intervention edits the numerical attribute captured by the LLM. For instance, if the numerical information _(Cristiano, born-in, 1985)_ is shifted to _(Cristiano, born-in, 2020)_, an LLM that genuinely relies on a linear subspace for reasoning would adjust its interpretation accordingly, reflecting the change in its responses (Figure 1).

\begin{table}
\begin{tabular}{p{56.9pt} p{113.8pt} p{113.8pt}} \hline
**Experiment** & **Question** & **Response** \\ \hline \multirow{3}{*}{**Extraction**} & Birth year of Albert Einstein? What is Isaac Newton’s year of death? Latitude of Cairo? & 1879 \\  & Einstein born before Newton? Enstein died before Newton? Is Cairo’s latitude higher than Jerusalem’s? & No 

Data Pre-processing

The purpose of this experiment is to assess whether the LLM possesses knowledge of the numerical attributes of the entities prepared for this study, and to evaluate its capability to perform numerical reasoning tasks. Additionally, by conducting behavioral experiments focused on information extractions, we aim to filter out entities for which LLM lacks sufficient knowledge, therefore creating a refined dataset to be used in the subsequent numerical reasoning tasks. For both tasks, extraction and reasoning, we prepared ten distinct prompts. The prompts that demonstrated the best performance in preliminary tests were selected for further investigation of the internal representations (SS4). Appendix 4 lists the complete list of prompts in the experiments.

### Knowledge Extraction

To assess the LLM's knowledge extraction of entity numerical attributes, we conducted a zero-shot question-answering task, in which we asked direct questions about numerical attributes for various entities. The results summarized in the top half of Table 2, demonstrate that the LLM correctly answered at least 67% of the prepared questions with the best-performing prompt for each task.

### Numerical Reasoning

For the numerical reasoning task, we created 5,000 question samples using a pair of unique entities, selected after filtering out those that the LLM could not answer correctly in SS3.1. Each question was designed to prompt the model to perform numerical reasoning, with binary (Yes/No) answers indicating correctness. The results, shown in the bottom half of Table 2, reveal varying levels of accuracy across different prompts. The LLM achieved around 75% for birth/death year prediction, but only 56% for latitude-related questions, suggesting differences in task difficulty.

## 4 Internal Representation Experiments

This experiment aims to train a PLS model to identify low-dimensional linear subspaces within the activation space, which could potentially be efficient in predicting numerical attributes for various entities. We then demonstrated the causal relationship within these subspaces by implementing targeted interventions which shows that indeed there is a causal effect between the identified linear subspaces and the logical comparison answers by the model. We validate our hypothesis by running three models on three numerical attributes.

We also fitted another PLS model to evaluate Yes/No comparison reasoning related to these numerical attributes (see appendix B.1).

### Prediction of numerical attributes with PLS

The training procedure consists of the following steps: (1) we first filter out the entities that the model predicted their comparison incorrectly (Section 3.2). (2) We feed a context vector that contains the comparison prompt (e.g., _Was Cristiano born prior to Messi?_) (3) We extract the hidden states of the last token of each entity from the LLM's hidden states at a particular layer. (4) These hidden states are then used to train a PLS model with a \(5\) component to predict the corresponding numerical attribute of each entity based on their corresponding model representation (activations). Figure 2 depicts the results achieved by \(five\) components PLS model, measured by the coefficient of determination \(R^{2}\). The goodness of fit exceeds \(0.8\) for all measured properties, indicating that the information encoded in these attributes can be extracted with low-dimensional (linear) subspaces.

### Intervention using PLS Components Vector

While the previous experiments with the PLS model establish correlation, they do not demonstrate causality. For this purpose, we perform interventions at a particular token within a designated model layer, chosen based on the correlation strength identified in predicting numerical attributes from each task (Section 4.1). We fix the first entity and intervene at the last token of the

\begin{table}
\begin{tabular}{l c r r r r r r r r r r}  & \multicolumn{9}{c}{Prompts} \\ Task & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\ \hline BP & 66.0 & 70.0 & 67.4 & 66.2 & 72.3 & 67.6 & 66.9 & 66.6 & 68.2 & 71.3 \\ DP & 63.4 & 65.5 & 61.5 & 61.5 & 67.0 & 65.0 & 63.3 & 60.1 & 61.7 & 66.1 \\ LP & 47.6 & 72.0 & 69.0 & 70.0 & 69.0 & 68.5 & 61.5 & 61.9 & 69.0 & 69.0 & 66.6 \\ \end{tabular} \begin{tabular}{l l l l l l l l l l l l}  & \multicolumn{9}{c}{Prompts} \\ Task & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\ \hline BP & 66.0 & 70.0 & 67.4 & 66.2 & 72.3 & 67.6 & 66.9 & 66.6 & 68.2 & 71.3 \\ DP & 63.4 & 65.5 & 61.5 & 61.5 & 67.0 & 65.0 & 63.3 & 60.1 & 61.7 & 66.1 \\ LP & 47.6 & 72.0 & 69.0 & 70.0 & 69.0 & 68.5 & 61.5 & 61.9 & 69.0 & 69.0 & 66.6 \\ \end{tabular} 
\begin{tabular}{l l l l l l l l l l l l}  & \multicolumn{9}{c}{Prompt} \\ DC & 53.5 & 50.3 & 74.8 & 58.7 & 50.5 & 50.2 & 50.3 & 61.8 & 50.1 & 56.6 \\ LC & 53.0 & 56.0 & 50.0 & 37.8 & 55.0 & 51.2 & 55.0 & 50.0 & 50.0 & 50.2 \\ \end{tabular}
\end{table}
Table 2: Experiments 1 and 2’s Results for three tasks, and 10 different prompts for each. The accuracy of exact matching is reported, except for the Latitude task, where we relaxed the predicted and ground truth to be rounded to the integer part. **BP**: Birth Prediction, **DP**: Death Prediction, **LP**: Latitude Prediction, **BC**: Birth Comparison, **DC**: Death Comparison, **LC**: Latitude Comparison 

second entity. This token's hidden state is then updated by a scaled version of the first component direction from the PLS model to the original hidden state \(h_{i}^{(l)}\) as illustrated in equation (1).

In Figure 3 we compare the effect of our intervention per layer against a random vector from the normal distribution. It is measured by the Effect of Intervention metric (EI) (equation 2), \(f\) and \(f^{\prime}\) are the clean and patched models.

\[\text{EI}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\left[f(x_{i})\neq f^{\prime}(x_{ i})\right] \tag{2}\]

The results clearly demonstrate the superiority of our intervention method, particularly evident in Subfigures \(a\) and \(b\). In subfigure \(c\), related to the Latitude numeric attribute, the gap between our method and the baseline narrows, suggesting that the direction may not be significant for this attribute. This could reflect the mode's nearly random response in the behavior experiment (Section 3.2). Additionally, the intervention's effect is notable only in the first \(\approx 50\%\) of the model layers, after which it diminishes to zero, aligned with prior research on inference time theory. We also tested the generalization of our approach on unseen samples, as shown in appendix, Figure 9 and additional models (see SS 4.3).

Figure 4: \(R^{2}\) score of predicting entity’s birth years attributes, using a 5-Component PLS model trained on Mistral 7B Instruct activations.

Figure 5: \(R^{2}\) score of predicting entity’s birth years attributes, using a 5-Component PLS model trained on Qwen2.5 7B Instruct activations.

Figure 3: The effect of the intervention—specifically, the ratio of flipped answers after performing intervention—was analyzed within the identified model subspace of each layer and compared to the effects observed in a randomly selected direction sampled from a normal distribution.

 



### Experiments on Additional Models

To further validate our hypothesis generalization, we run the same experiments on two additional language models for the _birth_ property. Those additional models are Mistral-7B-intruct Jiang et al. (2023) and Qwen2.5-7B-Instruct Team (2024).

PLS models trained on models' activation have crossed an \(R^{2}\) score of \(0.8\) suggesting that the information encoded in those models' activations can be extracted using low-dimensional (linear) subspaces (see Figure 4 and Figure 5).

The Effect of Intervention (EI) results shown in Figures 6 and 7 of the Mistral 7B Instruct and Qwen-2.5 7B Instruct models, respectively, demonstrate the same behavior seen in the previous experiments. For the EI of Mistral, we can see that the peak was around the \(11\)th layer and then continued to decrease until it finally disappeared around the \(16\) layer (Figure 6). When compared to other models, Qwen2.5 has shown two clear differences. First, we can observe two peaks for the EI with almost the same value of the EI, early around the \(third\) layer and later one around layer \(12\), while other models have shown only one peak. Second, unlike other models, Qwen2.5 7B kept bouncing around almost the same EI values and suddenly become None at around layer \(16\) (Figure 7). One reason that might explain the difference between Qwen2.5 7B and other models, is that Qwen2.5 7B uses only \(28\) layers, while other models in the experiments are formed of \(32\) layers.

## 5 Conclusion

In this research, we empirically demonstrate that the model answers numerical reasoning questions, such as "Was Cristiano born before Messi?" using a two-step process. First, it extracts numerical attributes for each entity from a linear subspace. The second step involves utilizing these linear directions to answer the logical question. Specifically, subspaces are identified through PLS regression, where directions in low-dimensional subspaces of the activation space encode numerical property information. We illustrate this approach using three numerical attributes: Birth, Death, and Latitude across three LLMs. The reasoning step is validated using causal interventions along the direction of the first component of the PLS model, where these interventions successfully alter the model's answers.

## 6 Ethical Statement

Our work adheres to the ACL Code of Ethics and maintains a high standard of ethical research practice. We ensure that our methodology, data usage, and model development follow responsible AI principles, and that there are no ethical violations in our study. Our research does not involve the use of sensitive or private data, nor does it contribute to any potential harm or bias propagation. We remain committed to transparency, fairness, and the responsible application of large language models in line with ACL's ethical guidelines.

Figure 6: The effect of the intervention(i.e. the ratio of the flipped answers) in the identified subspace in each layer of the Mistral 7B Instruct model, compared to a random direction from a normal distribution.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \multirow{2}{*}{Task Model} & \multicolumn{9}{c|}{**Properties**} \\ \cline{2-11}  & & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\ \hline \multirow{2}{*}{BP} & Mostai 7B & 72.65 & 72.63 & 74.68 & 75.36 & 73.64 & 75.44 & 73.36 & 74.81 & 72.90 & 73.56 \\  & Quen2.57 & 70.82 & 40.36 & 33.95 & 34.95 & 35.97 & 35.97 & 35.92 & 36.36 & 72.81 & 36.97 & 34.32 \\ \hline \multirow{2}{*}{BC} & Mostai 7B & 71.90 & 64.81 & 65.10 & 65.11 & 65.18 & 57.65 & 53.00 & 67.66 & 66.68 & 60.00 \\  & Quen2.57 & 79.20 & 58.10 & 38.88 & 58.22 & 49.76 & 40.54 & 62.00 & 73.84 & 9.16 & 6.68 \\ \hline \end{tabular}
\end{table}
Table 3: Exact Matching Accuracy of Natural 7B and Qwen2.5 7B Models on Birth Date Numerical property extraction and Comparison Tasks Across Prompt Variations. All models are instruction-based models. **BP:** Birth Prediction and **BC:** Birth Comparison tasks are evaluated.

Figure 7: The effect of the intervention(i.e., the ratio of the flipped answer) in the identified subspace in each layer of the Qwen2.5 7B instruct model, compared to a random direction from a normal distribution.





## 7 Limitations

This work has several limitations we plan to address in future work:

* Error Analysis: While the experimental results demonstrate the model's ability to map numerical properties to low-dimensional subspaces and use them for reasoning tasks, we have not conducted a thorough error analysis to understand the model's types of mistakes. Identifying patterns in erroneous outputs could guide improvements in both model design and training.
* Limited Scope of Numerical Attributes: Our experiments are restricted to three types of numerical attributes: birth year, death year, and geographic latitude. It remains unclear whether our findings extend to a broader range of numerical properties, such as financial data, time intervals, or other continuous variables. We plan to investigate this in future work.
* Intervention Hyperparameter Sensitivity: The success of the intervention experiments relies heavily on the choice of the scaling factor \(\alpha\) applied during the intervention. We have not explored the full sensitivity of the model's performance to this hyperparameter, which could introduce biases or instability in real-world applications.

## References

* B Brown (2020) longage models are few-shot learners. arXiv preprint arXiv:2005.14165. Cited by: SS3.
* D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei (2022) Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493-8502. External Links: Document, Link Cited by: SS1.
* Stanislas Dehaene (2011)The number sense: how the mind creates mathematics. Oxford University Press. External Links: ISBN 978-0-38

ney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharic Delpiere Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafori, Abha Jain, Adam Kelsey, Adam Shajinfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alex Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yuuns, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdandan, Beau James, Ben Maurer, Benjamin Leonndri, Bernise Huang, Beth Loyd, Beto De Paola, Bhargavi Parjanjee, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasi, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damoin Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Estbean Aracute, Evan Dunbar, Evan Smothers, Feri Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzman, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanzeri, Han Zou, Hannah Wang, Hanwen Zhao, Harroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Dallaji, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenmy Zhen, Jereny Rezienstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingji Yang, Joe Cummings, Jon Cavvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyple Huang, Lailing Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Markus, Matan Hasson, Matthew Lennite, Matthias Resco, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Kenenically, Michael L. Seltzer, Michal Valko, Michelelle Restrepo, Mihir Patel, Mik Vytsakov, Mikyed Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhnini Santhanam, Natascha Parks, Natasha White, Navyata Bauxa, Naya Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavvolt Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Cheronguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittrner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafii Ayub, Raghotham Murthy, Raghy Nagami, Rahul Mitra, Raymond Lal, Rebekkah Hogan, Robin Battety, Rocky Wang, Rohan Mahsewari, Russ Howes, Ruuty Rinot, Said Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Shengaho Lin, Shengxin Cindy Zh, Shiva Shankar, Shuquiang Zhang, Shinong Wang, Sncha Agarwal, Soji Sayjugige, Soumirth Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzok Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vittor Albiero, Vlad Inonescu, Vlad Poenaru, Vlad Tabieri Muhailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojiang Wu, Xiaolan Wang, Xide Xia, Xiilun Wu, Xinibo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The llama 3 herd of models. _Preprint_, arXiv:2407.21783.
* [65] Mor Geva, Jasmin Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12216-12235, Singapore. Association for Computational Linguistics.
* [66] Benjamin Heinzerling and Kentaro Inui. 2021. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1772-1791, Online. Association for Computational Linguistics.
* [67] Benjamin Heinzerling and Kentaro Inui. 2024. Monotonic representation of numeric properties in language models. _Preprint_, arXiv:2403.10381.
* [68] Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2023. Linearity of relation decoding in transformer language models. _arXiv preprint arXiv:2308.09124_ 

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil-laume Lample, Lucile Saulnier, Leibo Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed (2023). External 7b. PreprintarXiv:2310.06825. Cited by: SS4.
* Zhengbao Jiang et al. (2020)How can we know what language models know?. Transactions of the Association for Computational Linguistics8, pp. 423-438. Cited by: SS4.
* Nora Kassner et al. (2021)Multilingual learning knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, pp. 3250-3258. External Links: Link, Document Cited by: SS4.
* K. Meng, D. Bau, A. Andonian, and Y. Belinkov (2022)Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems35, pp. 17359-17372. Cited by: SS4.
* J. Merullo, C. Eickhoff, and E. Pavlick (2024)A mechanism for solving relational tasks in transformer language models. External Links: Link Cited by: SS4.
* J. Niu, A. Liu, Z. Zhu, and G. Penn (2024)What does the knowledge neuron thesis have to do with knowledge?. arXiv preprint arXiv:2405.02421. Cited by: SS4.
* K. Park, Y. Joong Choe, and V. Veitch (2023)The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658. Cited by: SS4.
* M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer (2018)Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 2227-2237. Cited by: SS4.
* F. Petroni, T. Rocktaschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, and A. Miller (2019)Language models as knowledge bases?. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 2463-2473. Cited by: SS4.
* A. Radford and K. Narasimhan (2018)Improving language understanding by generative pretraining. In Pre-print, Cited by: SS4.
* A. Roberts, C. Raffel, and N. Shazeer (2020)How much knowledge can you pack into the parameters of a language model?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 5418-5426. External Links: Link, Document Cited by: SS4.
* Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, E. Wallace, and S. Singh (2020)AutoPrompt: eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 4222-4235. External Links: Link, Document Cited by: SS4.
* Georgios Spithourakis and Sebastian Riedel (2018)Numracy for language models: evaluating and improving their ability to predict numbers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 2104-2115. External Links: Link, Document Cited by: SS4.
* Owen Team (2024)Qwen2.5: A party of foundation models. External Links: Link Cited by: SS4.
* H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023)LIMA 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: SS4.
* Denny Vrandecic and Markus Krotzsch (2014)Wiki-data: a free collaborative knowledgebase. Communications of the ACM57, pp. 78-85. Cited by: SS4.
* I. Vulic, E. Maria Ponti, R. Litschko, G. Glavas, and A. Karbonen (2020)Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7222-7240. Cited by: SS4.
* E. Wallace, Y. Wang, S. Li, S. Singh, and M. Gardner (2019)Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 5307-5315. External Links: Link, Document Cited by: SS4.
* S. Wold, M. Sjostrom, and L. Eriksson (2001)Pls-regression: a basic tool of chemometrics. Chemometrics and Intelligent Laboratory Systems58, pp. 109-130. Cited by: SS4.
* W. Zhang, C. Wan, Y. Zhang, Y. Cheung, X. Tian, X. Shen, and J. Ye (2024)Interpreting and improving large language models in arithmetic calculation. arXiv preprint arXiv:2409.01659.

 

Background

Generative-Transformer Language Models.Transformer models, particularly in generative contexts, have revolutionized natural language processing tasks due to their self-attention mechanisms. These models map an input sequence \(x_{1},x_{2},\ldots,x_{n}\) to a corresponding sequence \(y_{1},y_{2},\ldots,y_{m}\) using multi-layer perceptron, and multi-head self-attention layers, which compute attention scores based on the query-key-value system. Mathematically, for a given layer \(l\), the attention output \(A_{l}\) is computed as:

\[A_{l}=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V \tag{3}\]

where \(Q\), \(K\), and \(V\) are the query, key, and value matrices, and \(d_{k}\) is the dimension of the keys. By stacking multiple layers of these attention mechanisms and multi layer percptron, transformers efficiently capture long-range dependencies in text. The autoregressive nature of generative transformers allows them to generate coherent text sequences by predicting the next token based on previous tokens.

Representation Analysis of Transformer Language Models.Representation analysis of transformers has revealed important insights into how these models store and manipulate information across layers. Research has shown that transformer language models develop complex, hierarchical representations that can be understood by analyzing the attention patterns and hidden states at different layers Niu et al. (2024). For example, studies have found that early layers capture syntactic structures, while deeper layers capture more semantic information Hernandez et al. (2023). Recent work also uses probing techniques to analyze how specific linguistic features are represented, contributing to a growing understanding of model interpretability Vulic et al. (2020).

Intervention and Activation Patching.One technique that has gained attention in the analysis of neural models, including transformers, is **activation patching**. This involves replacing activations in a specific layer with those from another input in order to study the effect of those activations on the final output. By intervening at different points within the model, researchers can better understand how information is processed and transformed throughout the network. This method has been useful in dissecting how specific neurons or attention heads contribute to a model's behavior, allowing for targeted interventions that shed light on model interpretability.

Linear Hypothesis in Representation.The **linear hypothesis** posits that the representations formed by transformer models are linearly separable. This means that complex patterns, such as syntactic and semantic categories, can be distinguished by applying a linear transformation to the learned embeddings Park et al. (2023). The key idea here is that the hidden representations of different tasks or features align in such a way that linear classifiers can achieve good performance with minimal processing, a phenomenon observed across a range of neural architectures. Connecting this with the previous analysis, it appears that transformers structure their internal space in a way that is amenable to linear separation of features, thus facilitating tasks such as classification and regression.

Partial Least Squares (PLS).Partial Least Squares (PLS) offers an alternative to Principal Component Analysis (PCA) for dimensionality reduction, especially when predicting one set of variables from another. PLS seeks to maximize the covariance between the input matrix \(\mathbf{X}\) and the response matrix \(\mathbf{Y}\) by projecting both onto a latent space. The key idea is to find latent variables \(\mathbf{T}=\mathbf{XW}\) and \(\mathbf{U}=\mathbf{YC}\) that best capture this covariance.

The predictive relationship between \(\mathbf{X}\) and \(\mathbf{Y}\) is then modeled as:

\[\hat{\mathbf{Y}}=\mathbf{XWP}^{T}, \tag{4}\]

where \(\hat{\mathbf{Y}}\) is the predicted output matrix, \(\mathbf{P}\) are the loadings, and the quality of this prediction can be assessed using the coefficient of determination \(R^{2}\). The \(R^{2}\) value measures how well the model explains the variance in \(\mathbf{Y}\), where higher values indicate a better fit between predicted and actual outputs.

PLS is preferred over regression when predictors (or columns of \(\mathbf{X}\)) are not independent or when the number of predictors exceeds the number of observations, making it suitable for high-dimensional data. For transformers, applying PLS helps uncover how input embeddings influence predictions by focusing on the shared variance between input features and outputs Heinzerling and Inui (2024).

 

Related Work

After the appearing of pre-trained language models such as ELMo Peters et al. (2018), BERT Devlin et al. (2019), and GPT Radford and Narasimhan (2018), researchers have had interests in the numerical capability of language models. Spithourakis and Riedel (2018) evaluates the pre-trained language models from viewpoints of the output capability of numerical tokens, the behavioural side of the numeracy. Wallace et al. (2019) focused on the numerical knowledge stored in the embeddings, which is the internal side of the numeracy. Zhang et al. (2024) investigated the internal working of the recent large language models when processing arithmetic calculation.

Knowledge of entities such as named entity has also been payed attention to by many researchers. Considering the pre-trained language models as a knowledge base Petroni et al. (2019); Jiang et al. (2020), behavioral Shin et al. (2020) and internal Meng et al. (2022); Dai et al. (2022) analysis have been studied.

With much larger scale of language models such as GPT3 Brown (2020) and LLaMA Touvron et al. (2023) and the technique of in-context learning, the capability of reasoning acquired by the language models has started to be discussed. Merullo et al. (2024) examined the internal working of language models when solving the reasoning task of the entity-entity relation such as _(Paris, capital-of, France)_. Heinzerling and Inui (2024) provides a deeper observation of the reasoning of the entity-numeric relation such as _(Dijkstra, born-in, 1930)_. They reveal that the entity-numeric relations are stored in the language models' representation as keeping their monotonic structure. Following this work, we further dive into the numerical reasoning that requires the extraction of the entity-numeric knowledge and the comparison of the two numerical information such as _(Bellman, born-before, Dijkstra)_.

### Logical Comparison with PLS

In this experiment, we feed the entire context vector containing a comparison into the model and extract the last hidden state of the last token for each comparison sample. We train a PLS model on these activations to predict the comparison results (i.e. Yes or No). We aim to make sure that the Yes/No task is predictable from model activations using a low-dimensional (linear) subspace. Figure 8 illustrates the accuracy of the \(5\)-components PLS model in predicting the comparison results giving the model activations. The model shows near-perfect performance of the Birth and Death tasks, while less robust on the Latitude task. This outcome is consistent with findings from the Behavioral experiments in Section 3.2.

 

* [25] M. C. Bader, J. S. Hubert, and A. K. Jain, "A new approach to the detection of a large number of missing data samples in multi-view images," _IEEE Transactions on Image Processing_**10**(4):639-678, 2000.
* [26] D. R. Bishop, L. W. Tukey, and P. G. Rubin, "The lasso method for multivariate regression problems," _Annals of Statistics_**10**(1):1-10, 1975.
* [27] T. E. Friedman, "On the lasso method for multivariate regression problems," _Journal of Machine Learning Research_**10**(1):1-1, 1975.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{|p{142.3pt}|p{142.3pt}|p{142.3pt}|} \hline
**Birth** & **Death** & **Latitude** \\ \hline Did \{entity\_x\} come into the world earlier than \{entity\_y\}? Answer with Yes or No. & Did \{entity\_x\} die before \{entity\_y\}? Answer with Yes or No. & Is \{entity\_x\} located at a higher latitude than \{entity\_y\}? Answer Yes or No. \\ \hline Is \{entity\_x\}’s birthdate before \{entity\_y\}’s? Respond with Yes or No. & Did \{entity\_x\} pass away earlier than \{entity\_y\}? Respond with Yes or No. & Is \{entity\_x\} farther north than \{entity\_y\}? Answer Yes or No. \\ \hline Was \{entity\_x\} born prior to \{entity\_y\}? Output only Yes or No. & Was \{entity\_x\}'s death prior to \{entity\_y\}? Provide only Yes or No. & Does \{entity\_x\} have a higher latitude value than \{entity\_y\}? Answer Yes or No. \\ \hline Did \{entity\_x\} enter life before \{entity\_y\}? Answer with Yes or No. & Did \{entity\_x\} pass on before \{entity\_y\}? Answer Yes or No. & Comparing latitudes, is \{entity\_x\} north of \{entity\_y\}? Answer Yes or No. \\ \hline Was \{entity\_x\}’s birth earlier than \{entity\_y\}’s? Output only Yes or No. & Did \{entity\_x\} die first compared to \{entity\_y\}? Respond only with Yes or No. & In terms of latitude, is \{entity\_x\} above \{entity\_y\}? Answer Yes or No. \\ \hline Was \{entity\_x\} born first compared to \{entity\_y\}? Respond with Yes or No. & Was \{entity\_x\}’s death earlier than \{entity\_y\}’s? Answer with Yes or No. & Is the latitude of \{entity\_x\} greater than the latitude of \{entity\_y\}? Answer Yes or No. \\ \hline Is \{entity\_x\} older than \{entity\_y\}? Reply only with True or False. & Did \{entity\_x\} precede \{entity\_y\} in death? Reply only with True or False. & Geographically, is \{entity\_x\} at a more northern latitude than \{entity\_y\}? Answer Yes or No. \\ \hline Did \{entity\_x\} precede \{entity\_y\} in birth? Respond only with True or False. & Did \{entity\_x\} pass before \{entity\_y\}? Respond only with True or False. & Does \{entity\_x\} have a more northerly latitude compared to \{entity\_y\}? Answer Yes or No. \\ \hline Did \{entity\_x\} arrive before \{entity\_y\}? Answer only with True or False. & Did \{entity\_x\} die earlier than \{entity\_y\}? Answer only with Yes or No. & Is \{entity\_x\} positioned at a latitude north of \{entity\_y\}? Answer Yes or No. \\ \hline Is \{entity\_x\} senior to \{entity\_y\}? Reply only with Correct or Incorrect. & Did \{entity\_x\} pass away first compared to \{entity\_y\}? Reply with Correct or Incorrect. & Considering only latitude, is \{entity\_x\} more northward than \{entity\_y\}? Answer Yes or No. \\ \hline \end{tabular}
\end{table}
Table 4: Comprehensive list of prompts for our three tasks: for Birth, Death, and Latitude