Interpreting Arithmetic Reasoning in Large Language Models using Game-Theoretic Interactions
Authors: Leilei Wen
Liwei Zheng
Hongda Li
Lijun Sun
Zhihua Wei
Wen Shen
Venue: NeurIPS 2025 poster ()
Keywords: Explainable Artificial Intelligence
Large Language Models
Arithmetic Reasoning
Interactions
Interpretability
Deep Learning
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: In recent years, large language models (LLMs) have made significant advancements in arithmetic reasoning. 
However, the internal mechanism of how LLMs solve arithmetic problems remains unclear.
In this paper, we propose explaining arithmetic reasoning in LLMs using game-theoretic interactions.
Specifically, we disentangle the output score of the LLM into numerous interactions between the input words.
We quantify different types of interactions encoded by LLMs during forward propagation to explore the internal mechanism of LLMs for solving arithmetic problems.
We find that (1) the internal mechanism of LLMs for solving simple one-operator arithmetic problems is their capability to encode operand-operator interactions and high-order interactions from input samples.
Additionally, we find that LLMs with weak one-operator arithmetic capabilities focus more on background interactions.
(2) The internal mechanism of LLMs for solving relatively complex two-operator arithmetic problems is their capability to encode operator interactions and operand interactions from input samples.
(3) We explain the task-specific nature of the LoRA method from the perspective of interactions.

Interpreting Arithmetic Reasoning in Large Language Models using Game-Theoretic Interactions
Leilei Wen' Liwei Zheng'
Hongda Li!
Lijun Sun'
Zhihua Wei!
Wen Shen
'Tongii University, Shanghai, China (2331934, 2432010, 2410907, sunlijun, zhihua_wei, wenshen]@tongja edu. cn
Abstract
In recent years, large language models (LLMs) have made significant advancements in arithmetic reasoning. However, the internal mechanism of how LLMs solve arith- metic problems remains unclear. In this paper, we propose explaining arithmetic reasoning in LLMs using game- theoretic interactions. Specifically, the output score of the LLM into numerous interactions between the input words. We quantify different types of interactions encoded by LLMs during forward propa- gation to explore the internal mechanism of LLMs for solving arithmetic problems. We find that (1) the internal mechanism of LLMss for solving simple one-operator arithmetic problems is their capability to encode operand-operator interactions and high-order interactions from input samples. Additionally, weak one-operator arithmetic capabilities focus more on background interactions. (2) The internal mechanism of LLMs for solving relatively complex two-operator arithmetic problems is their capability to encode operator interactions and operand interactions from input samples.
disentangle
find that LLMs with
(3) We explain the task-specific nature of the LoRA method from the perspective of interactions.
Introduction
In recent years, the arithmetic reasoning capabilities of large language models (LLMs) have improved significantly, but the internal mechanism still unclear. Some studies identified neurons that have great effects on arithmetic reasoning Yu and Ananiadou [022 Rai and Yao 2024). Other studies evaluated the impact of modifying words in input arithmetic questions on neuron activations and network outputs [Stolfo et al. 2023 Zhang et al. 2024]. However, previous studies have not mathematically guaranteed that the explanations faithfully reflect the arithmetic reasoning logic of LLMs. Recently, series of studies have used game- theorete interactions between input variables to explain the representation power of traditional DNNs [Ren et al. 2024 Deng et al., 2024]. e.g., the interaction between "green" and "hand" forms the concept of "beginner." The interaction has been proven to be faithful explanations by a series of theoretical guarantees [Rnn et al. 2023a
Inspired by these studies, we use interactions to explain the arithmetic reasoning capability of LLMs. "Whut is 2 plus 7? Answer is," the interaction S - - {2, plus, 7}) represents AND relationship between the words in S, which is equivalently encoded by the LLM. Each interaction S contributes a numerical effect Is to push the output score towards generating the answer "9." Thus, we construct logic model ø (æ) based on interactions. The faithfulness of using the logical model to explain an LLM is reflect as follows.
As Figure shows, given an input arithmetic question æ, e.g., "Whut
As Figure aa) shows, given the same input sentence the output score of the LLM is equal to the output of the logical model, that is, v(x) = & (ax) EsgN Is, where the set containing all the
"Corresponding author.
Note that each interaction is equivalently encoded by the entire DNN, rather than by specific neuron.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

P( p('9"11)
(xx)=
MathCoderr LB-BB
Equivalently
SEW
modeling operator Interactions
background Interactions
Nomallzzn
Namaaliz
operand interactions
operand-operitore itt.rtttttttctiionss
(pinn -129
12.082.08
12phag=1.77 11.7
4yhha2=-0.11 0.10
LLM
[Whicolaj=100 1.09
22.7==227
1.118 1.18
(whhucio) 0.72
[uaiaa) 12.50 0.50
7--070
[whukAsswwrre) -.29
Layer
operator interactions
operand
operand intoractions
Inpor Sentonce x
What
plus
Antwwe
1s
operator
ground
opernnd-operator internctions
background interactions
(a)
(b)
Figure I: Illustration of using interactions to explain the internal mechanism of LLMs for arithmetic reasoning. (a) Given arithmetic question æ consisting of words, let N = (211 æ2...., T. each æ; represents a single word in the input question. We use a logic model based on interactions, i.e., @ (z) Eegw Is to explain the detailed inference patterns encoded by LLMs. Note that superscripts a and b are not input to the LLM, but are used to distinguish two words of "is" in two positions. (b) Based on interactions, we investigate how an LLM encodes different types of interactions during forward propagation when solving arithmetic problems.
where
words in i, i.e., N= {a), (2,, #2......
). This means that the output score of the LLM on any input sentence can be represented as the sum of the effects of all interactions.
mask input words in æ, the output score of LLM the masked sentence masked is equal to the output the logical model, that is, v(z:"asked) = of ((x: masked)
More crucially, no matter how
EscN "asod triggers S) 3)-22 -s. This means that the output score of an LLM on any masked sentence can be represented as the sum of the effects of all triggered interactions. For example, if we mask the input word "7" in Figure aa) and obtain the masked sentence "What is 2 plus [akd_] Answer is." then all interactions that contain "7" (e.g.. S {2, plus, 7}) will not be triggered, meaning that 1(x maseed triggers S) Is = O. As result, these non non-triggenn triggered interactions have no effects on the network output. Please see Theorem for details.
In this way, interactions can be considered as inference patterns encoded by LLM. Therefore, we quantify how LLMs encode different interactions during forward propagation. We find that different LLMs have preferences for encoding certain interactions. For example, as Figure 2 shows, when
computing features closer to the output layer, the Llemma-7B [Azerbayey et al. 2023 model tends to encode interactions that contain both operands and operators, while the OPT-1.3B [Zhang et al.)
2022b model tends to encode interactions that only contain background words.
Inspired by the above observations, we further define four types of interactions encoded by the LLM for arithmetic reasoning. including (1) operand interactions, (2) operator interactions, (3) operand-operator interactions, and (4) background interactions, shown in Figure [ Furthermore, we propose a new metric to quantify the focus of LLMs on different types of interactions and discover the following insights.
Insight I: The internal mechanism LLMs for solving simple one-operator arithmetic prob- lems is their capability to encod operand-operator interactions and high-order interactions from input samples. For simple one-operator arithmetic problems, we observe that LLMs with strong performance tend to increase their focus on operand-operator interactions and high-order interactions, while reducing their focus on operator interactions and background interactions. A high-order interaction contains large number of input words, while low-order interaction contains small number of input words. In contrast, LLMs with weak performance tend to focus on background interactions and extremely low-order interactions.
Insight 2: The internal mechanism of L.LMs for solving relatively complex two-operator arith- metic problems is their capability to encod operator interactions and operand interactions from input samples. As an LLM learns to solve two-operator arithmetic problems, we observe that it consistently focuses more operator interactions and gradually increases its focus operand interactions during the final stages of training.
'It is common to use specific token embedding to mask input variables of a DNN, but there are still no unified masking strategies. Please Appendix A for introduction masking strategies.

---

Insight 3: We explain the task-specific nature of the LoRA method from the perspective of interactions. As an LLM with strong one-operator arithmetic capabilities is fine-tuned using LoRA to solve relatively complex two-operator arithmetic problems, its capability to solve simpler one-operator problems declines. This phenomena is consistent with the task-specific nature of LoRA (Hu et al. 2022]. From the perspective of interactions, we observe that the LLM reduces its focus on operand-operatee interactions and high-order interactions, which helps explain the underlying mechanism behind the task-specific nature of LoRA in arithmetic reasoning tasks.
2 Related Work
Explaining arithmetic reasoning in LLMs. Some studies identified and analyzed key neurons to explain arithmetic reasoning in LLMs. Yu and Ananiadou (2024) identified an internal logic chain.
Rai and Yao [2024] investigated neuron activations and identified reasoning- eeeooneg-rellted neurons. Some
studies identified key components by changing the inputs of the LLM to perturb activations and
then measuring the changes in the output logits. [Hanna et al. [0233 explained the capability of the
GPT2-small model on the "greater than" task by identifying specific circuit. Zhang et al. [2024]
further selectively fine-tuned the key components to boost the LLM's arithmetic performance. Stolfo
et al. [023]] used causal mediation analysis framework to provide mechanistic explanation of
how LLMS solve arithmetic problems. |Wu et al. 2023) proposed a method to scale causal analysis of language models to billions of parameters. However, previous studies failed to provide strong mathematical support for their explanations. In contrast, in this paper, we use interactions to explain how LLMs solve arithmetic tasks, which has been proven be faithful explanation.
Using game-theoretic interactions to explain DNNs Ren et al. 2023a] proposed using interac- tions to explain DNNs and provided series of theoretical guarantees to ensure the faithfulness of this explanation. series of studies further explored using interactions to explain the representational
power of DNNs, including adversarial robustness [Wang et al. [022]] adversarial transferability [Wang et al. 2020), and the overfitting problem [Ren et al. 2023b] Zhou et al. 2023). Additionally, Deng et al. I demonstrated that DNNs struggle to encode mid-order interactions. Ren et al.
[224] discovered that DNNs encode interactions of different orders in two distinct phases. Some other studies used interactions to analyze common mechanisms across various deep-learning methods. Deng et al. 2024] showed that the core mechanism of fourteen attribution methods can be explained as redistribution of interactions. Zhang al. [022aa] found that the twelve previous methods for improving transferability all reduce interactions between local adversarial perturbations. However, due terabytes of data and billions of parameters in LLMs,, well the inherent complexity of arithmetic reasoning tasks, whether interactions can be used to explain the arithmetic reasoning of LLMs while ensuring the faithfulness of the explanation remains to be verified.
3 Using interactions to explain inference patterns encoded by LLMs
3.1 Preliminaries: disentangling the network output using interactions
Given a DNN and an input sample æ with variables indexed by .N == (w), 22........ J. we can obtain 2n masked sentences (ær | T' C NN by masking each of n variables. Specifically, ær denotes the input sentence when we keep variables in T' unchanged and mask variables in N'T 2a denotes the input sentence when all variables in æ are masked Ren et al. (2021) have proven that there exists surrogate logical model d6.) that can well predict/fit the scalar output of the DNN for all 2n different masked sentences, as Theorem [ shows.
VT C (xT) = o (xx) +
1(S@xr) Is.
(1)
SCN,SPO
where the trigger function (S]ær) represents an AND relationship among the input variables in set C N. That is, if all variables are present in T, then 1 (5] = 1. Otherwise, if any variable in S is m then (S]@T) = O. Each interaction S has a scalar weight Is.
Theorem (Universal matching property, as proven by Ren et al. (023a)). ). When the scalor weight Is in the logical model ((-) is defined as follows,
VS S N. + 1.
Σ (-1)18-151 (æg)),
(2)
S'CS
3

---

then we have VT S N. the outpur score of the DNN v (ær) = & (æT).
The universal matching property guarantees the theoretical faithfulness of using interactions to explain the DNN. That is, the interaction truly reflects the inference logic of the DNN. Besides, the interaction has been proven to serve as the basis for many game- theoretic metrics, further ensuring its theoretical faithfulness. Please see Appendix for more details about game-theorete metrics.
3.2 Explaining the L.M using interactions
Although interactions have been widely used to explain traditional DNNs, the following two chal- lenges remain in using interactions explain the forward propagation process of the LLM.
(1) How to quantify interactions encoded by an LLM in intermediate layers. Given an LLM with L layers we set the output score (xr) ) at layer as follows.
(ET)) )) ( (xx) AT N o = cos( (xT)), (xx)) ( la (() (x )l2
(((()
(3)
where f(l) E IRd denotes the embedding of the last input token The score " (xT) measures the cosine similarity between (x)x) and (xx). If the value of e (xr) is large, then it indicates that the feature (xr) of the masked sentence is similar with the feature j (xxx) of the original sentence. This means that input variables in T have great impact on the feature of the I-th layer. Thus, in Equation ( the interaction computed based on the score can be considered as an inference pattern encoded by the features at the 1-th layer. Note that we use (xT) p (æx) to replace (xT), where y() (xx) denotes the embedding when we mask all input variables in æ. This shifting operation is used to ensure that (O) (ma) O.
(2) Why we use words instead of tokens input variables. is because the tokenizers of different LLMs adopt different strategies for encoding numbers. For example, the Llama-2-7B (Touvron et al.
2023 model divides the word "15" into two tokens "I" and "5," while the OPT-1.3B [Zhang et al.
2022b model obtains single token "15." To ensure a fair comparison of interactions encoded by different LLMs, we treat all tokens within a word as a single input variable. When generating the masked sentence ær, we use a specific padding token (see Appendix B for details) to mask all tokens that belong to the words in N \
Based on the above two settings, we quantify interactions in LLMS when solving arithmetic problems. Figure 2 shows the top 10 interactions (i.e., those interactions with largest absolute values) encoded by the Llemma-7B model [Azerbayev et al., (023) and the OPT-1.33 model (Zhang et al.. 2022b when solving a one-operator problem "How much 4 times 2? Answer is." The Llemma-7B model predicts the correct answer "8," while the OPT-1.3B model predicts the incorrect answer "4." We observe that the Llemma-7B model mainly focuses on interactions that contain only background words in an early (5th) layer and mainly focuses on interactions containing both operands and operators in late (30th) layer. comparison, the OPT-1.3B model mainly focuses on interactions that contain only background words in a late (24th) layer, which might be the reason why the O.T-1.3B model answers incorrectly.
To better observe the changing trends of different interactions encoded by an LLM during forward propagation, Figure reports the curves of interactions encoded by LLMs when computing features. To ensure fair comparison of interactions at different layers, we compute the normalized strength
|r
of interactions at different layers, that is, 20 where z = Esga (T As Figure 2 shows, the Llemma-78 7B model enhances its focus on interactions containing both operands and operators (e.g.. S S= {is". 4, fimes, 2} during forward propagation. In comparison, the OPT-1.3B model does not show a preference for any interactions in the middle layers and shifts its focus to interactions that contain only background words (e.g., 5 {is*}) in the very few layers close to the output layer. Therefore, interactions provide us with a new perspective to explain the internal mechanism of LLLMs for arithmetic reasoning.
PPlesse see Appendix D for details about network architectures and chosen layers. "We follow Wendler et al. 2024) to focus the embedding of the last input tnken at each layer 1, the last input token captures the information from the entire sentence. We also conduct experiments to verify that the mid-layer features of other tokens remain nearly unchanged when input is masked. Please see Appendix for details. To this end, only use the embedding of the last input token compute ( (xT).

---

Given a simple onе-operator arithmetic problem as input: "Hlow much times 2? Answer is
The I. Llemma-7B model predicts: 8 (Correct)
The P model predicts: 4 (Incorrect) The 3th layer
The 5th layer
The 30th layer
The 24th layer
(muchar Attehurr"AAmmme Almes
1.34
11001 00.80
(nnwwrr -051
Howwss A 7
Answer]=
[Howomuchis" A.tma:aa.....meaeaaaaaaa Artmm))
(LAnser 1.10
[e=".Answer 0.32
Ho Sltowmuchhis
00.47
Answer) 0.05
[BlowAnswer) -0031
Iis Allimen
|-
(AAnwarr 1..1.7
(muddh,syy=-033
How 2
(Bo Ans
(poowie)m-0030
out Altimen Ammwer 2
(10 (1ttmee )
[p="AAAnswer] =0.30
[muchisP AriiieiiiPiiiia....... {
[[st,a)=0.30
(mmcchis* A.tmmm 2,16) m0.40
Amarhse/AAAnemmm 1-11
(1imm,JsP) ( -0.005
ftimes=0.300
!/muchiss A.tmee
(muchute Anoweras))=-0000
1e-3
2.5
(1ss times,
Normodiant tian Iutaryttinn dhinghh
S=(is")
S=fis Limes, Answer)
[Annwerr)
Inserration Normol Land
(dmes)
$= (tiees Answer)
mm(uh times)
1.0
Climes, Answer,
[4,2]
0.5
(How, much, is 1c))
-How 2, 2,Anwerr, (42)
[much, Is")
5- (much, Is', times,
- How, is. 4, Answer)
5- [How,HH isf, 4,times A,teess,Answer,.k
10
20
100
20
Layer
Laper
(a) Interactions encoded by the Llemma-7B model
(b)
Interactions encoded by the 1.T-1.3B model
Figure Visualization of top 10 interactions encoded by LLMs when computing features different layers. Results show that the Llemma-7B model mainly focuses on interactions containing both operands (in blue) and operators (in orange) late layer (i.e., the 30th layer). comparison, the OPT-1.3B model mainly focuses on interactions that contain only background words in a late layer (i.e., the 24th layer), which may be the reason why the OPT-1.3B model answers incorrectly, Note that superscripts a and b are not input to the LLM, but are used to distinguish two words of "is."
3.3 Defining and quantifying different types of interactions
classify the variables in input arithmetic query æ into the following three categories: operand variables æ''4 operator variables " and background variables xh Thus, all 2n interactions can be classified the following four types to describe the inference patterns encoded by LLMs for arithmetic reasoning.
Inspired by the above observations obtained in Section 3.2
Operand interactions. If an interaction S contains at least one operand variable but no operator variables, we regard S as an operand interaction. Let (2ud denote the set of all operand interactions.
12994
{S 3 upd SAA E S).
(4)
Operator interactions. If an interaction S contains at least one operator variable but no operand variables, regard S as operator interaction. Let 12 denote the set of all operator interactions.
~0 {S 1 x S ad S}.
(5)
Operand-operator interactions. If an interaction S contains at least one operand variable and at least one operator variable, regard S an operand-operator interaction. Let opepé-opr denote the set of all operand-openee operraor interactions.
pood-opr = (S 3x SSA E
(6)
Background interactions. If an interaction 5 contains neither operand nor operator variables, we regard S background interaction. Let 8 denote the set of all background interactions.
0 = {S A opd
A E S}.
(7)
Note that the union of the four sets above is the complete set of 2" interactions. According to Theorem [, the sum of numerical effects of these four types of interactions can accurately fit the output scores of the LLM, thereby ensuring the f'attfulnesss of the analysis. We design the following metric quantify how an LLM focuses on specific type of interaction Da (S2 SP9 opp 52
Definition (Focality on a specific type of interaction). Let R (ITT)) denote the focality on a specific type of interaction Dowe at layer I. It computed follows,
Esegos 11
11 (12
(8)
5

---

where z = EscN 1 is a normalization term used to ensure a fair comparison of the interaction effects across different layers.
In Equation higher ( (2**0) value suggests that the LLM focuses more on this type of interaction 52 when computing features at layer I. If Rd (S2 = 1, that means the strength of this type of interaction 2 encoded by the LLM equal to the average strength of all interactions encoded by the LLM, which reveals that the LLM does not show preference for this type of interaction. If R (S > 1, that means the strength of this type of interaction 2 exceeds the average, indicating that the LLM exhibits a preference for this type of interaction.
We also quantify interactions of different orders to measure the representation complexity of an LLM. The order m of an interaction S is defined as the number of variables in S, that is, = |S]. We design the following metric to quantify how an LLM focuses on m-order interactions.
Definition 2. (Focality on interactions of a specific order). Let (!) denote the focality on m-order interactions at layer I. It is computed as follows.
Eist=m| (
Vm {1,
! Km
2
If Km has larger value when m is higher, it indicates that the LLM encodes interactions of greater complexity when computing features at layer I. If x has a larger value when m is lower, indicates that the LLM encodes interactions of lower complexity when computing features at layer I.
4 Comparative studies
In this section, we conduct comparative studies to analyze the internal mechanism of LLMs for arithmetic reasoning (see Section [41]. We also fine-tune an LLM to improve its capability to solve arithmetic problems and explore how the LLM encodes different types of interactions during the training process (see Section 4.2)
LLMs. We use interactions to analyze seven LLMs for arithmetic reasoning, including the OPT-1.3B
Zhang et al., [202bb] model, the GPT-J-6B (Wang and Komatsuzaki] [022] model, the Llama-2-7B
Touvron et al. 20233 model, the Llemma-7B [Azerbayev et al.] [023] model, the MathCoder- L-7B [Wang et al., 2023] model, the C1 [Wang et al. 2023 model, and the
CodeLlama- 13B Roziere et al. 2023 model. Appendix B shows how to mask words for these LLMs.
Table 1: Overall accuracy (%) of different
Data. We follow Karpas et al. 2022 Razeghi et al.
[2022]. Stolfo et [2023] to conduct experiments on a set of arithmetic problems hand-crafted by hu-
LLMs arithmetic queries.
Model
1-opr
2-opr
mans, including 6 templates for one-operator two-
OPT-1.3B
3.2
operand queries and 29 templates for two-operator three-operand queries. For example. "The sun of nn and 122 is" and "What is the ratio between ny minus n22 and nz? The answer is." Each template for one-operator queries includes all four arithmetic operators, i.e., {+.- -, x, +) and each template for two-operator queries corresponds to unique com- bination of two operators. Please see Appendix [F for details of templates. For each template of one-operator queries, we generate 20 prompts by randomly sampling operands (#1, nt2) [ and for each template of two-operator queries, we generate prompts following the same procedure. Table the overall accuracy of different LLMs one-operator queries and two-operator queries. Please see Appendix for details of the accuracy tests. We observe that the Llama-2-7B model, the Llemma-77 7B model, the MathCoder-L--BB model, the MathCoder-CLL perform well on one-operator queries, while the 0.T-1.3B model and the GPT-J-6B model perform relatively poorly. However, for two-operator queries, all seven LLMs perform poorly.
GPT-J-6B
14.7
5.8
Llama-2-77
65.1
10.1
Llemma-7B
75.1
15.3
MathCoder-LL-B
74.0
8.2
MathCoder-CCL-TBB
62.6
9.3
CodeLlama- 13B
71.1
15.0
OPT-1.3B Fine-tuned
83.6
69.7
model, and the CodeLlama- 13B model
'We sample operands from [1, ..... 9) since some LLMs tokenize each digit as an independent token, such the Llama-2-7B 2-BB model.

---

sparazer interacionns operand intar acdorr www.-pndd--ppaaaaeo..o..o insaraciions baciground interaciona
OP1-1.58
GPT-448
Lamma--77
Liemma-78
MethCodee-178
MerhCodde--C-788
Codeal Lema- 138
1.1
8*0
3
LD
13
6.7
LayoT1
Layor
Layert
Layert
Layor!
Loyar!
Layer!
Figure 3: Focality across different types of interactions R encoded by LLMs during forward propagation. Each curve in the figure is averaged over various one-operator arithmetic queries.
fot-order ze-2nd-onder
3thordrr
4th-rrder
6th-ondor
Bthvrrdr
7th
OPT:1.38
GB
Liama
Liemma 7B
MathCoder 78
MathCader CL-7B
Codeliaro 12B
x!0
1.
1.4
L0
LayoTt
Layer
Layort
Figure 4: Focality across interactions of different orders (0)) encoded by LLMs during forward propagation. Each curve in the figure is averaged over various one-operator arithmetic queries.
4.1 Exploring the internal mechanism of LLMs for solving arithmetic problems
In this subsection, we analyze how LLMs focus different types of interactions during forward propagation and obtain the following insights.
Insight 1: The internal mechanism of LLMs for solving simple one-operator arithmetic prob- lems is their capability to encode operand-operateee Figure 3 propagation. The results are averaged over all one-operator queries. We observe that in LLMS with strong one-operator arithmetic capabilities (the Llama-2-7B model, the Llemma-7B model, the MathCoder-L-7B model, the MineCoddr-L-77 model, and the CodeLlama- 13B model), the focality on operand-operator interactions R (S29dd-pr) tends to increase and exceeds 1.0 in the later layers, while the focality on operator interactions (I2PP))))) and the focality on background interactions (12 tend to fall below 1.0. We notice that in the MathCoder-L-7B model, the focality operator interactions and the focality on background interactions increase and exceed 1.0 in the very few layers close to the output layer. However, the encoding of operand-operator interactions in the later layers remains sufficient to support its arithmetic reasoning. This suggests that solving one-operator arithmetic problems needs L.LMs increase their focus operand-operator interactions, and reduce their focus on operator interactions and background interactions in the later layers. Note that in the Llama-2-7B model, the Llemma-7B model, and the MathCoder-L-77 model, the focality on operand interactions Rd (I2%²) also tends to increase and exceeds 1.0 in the later layers, which may contribute their strong performance on one-operator arithmetic tasks.
interactions and high-order interactions. reports the focality (()) on different types of interactions encoded by LLMs during forward
In comparison, in LLMs with weak ope-operator arithmetic capabilities (the OPT-1.3B model and the GPT-J-6B model), the focality on different types of interactions is similar and remains around 1.0 in the middle layers, while the focality on background interactions 17 suddenly increases in the very few layers close to the output layer. This suggests that these L.MMs do not exhibit clear preference for any specific type of interaction in the middle layers, but subsequently shift their focus to background interactions in the very few layers close to the output layer, which weakens their performance on one-operator arithmetic tasks.
Figure A reports the focality x(() interactions of different orders encoded by LLMs during forward propagation. The results are averaged over queries from a single one-operator template 7 We observe that in LLMs with strong ope-operator arithmetic capabilities (the Llama-2-7B model, the Llemma- 7B model, the MathCoder-L-7B model, the MathCoder-CLL- model), the focality high-order interactions tends increase and exceeds 1.0 in the later layers. This suggests that solving one-operator arithmetic problems needs L.MMs to increase their focus on high-order interactions in the later layers. In comparison, in LLMs with weak one-operator arithmetic capabilities (the DPT-113B model and the GPT-J-6B model), the focality on high-order interactions in the middle layers is around 1.0, while in the very few layers close the output layer, the focality on
model, and the CodeLlama-
"As different templates correspond to different maximum orders, i.c., the number of input words. Please see Appendix for more results from other templates, which lead to the same conclusion.
7

---

Fine-tuned on ona-operator queries
OPT-1.38
poim!
3olnt
points $9.6
Aoc Aocureey)))) on one operatr quartes
3.2
27.4
$1.4
69.4
23
15
speeatrr interadions
oporand Interadi ons
nop opurand-cjjeatzz interactions -on www.gamnnniieeeeeeeeoooooo
13
13 Layer!
Layer
Layer!
Layer!
Layer
Layer
2rccanbe
Abbrrder
20
4.0
Sharder Blberdar
Utvonder -aw
Stwander
19
Lager
Layerz
Laygrr
Layyr
Lager
Laywr
Figure 5: Visualizing the dynamic changes in different types and orders of interactions encoded by the OPT-1.3B model during the training process on one-operator queries. Each curve is averaged over various one-operator queries. Results show that as accuracy improves, the model gradually increases its focus sperand-operaton interactions (top) and high-order interactions (bottom).
the extremely low-oder interactions suddenly increases. This suggests these LLMs exhibit insufficient encoding of higher-order interactions in the middle layers and tend to focus on extremely low-order interactions, which weakens their performance on one-operator arithmetic tasks.
4.2 Dynamic encoding of different types of interactions during the training process
We further explore how an LLM learns to solve arithmetic problems. That is, we investigate how an LLM encodes different types of interactions when trained on arithmetic problem data. To this end, we use the LoRA method [Hu et al.
2022 to fine-tune the OPT-1.3B model on arithmetic queries in
the following three different ways (1) We fine-tune the OPT-1.3B model on ope-operator queries, improving its accuracy on one-opertoor queries from 3.2% to 83.6%. This version is termed the OPT-1.3B-OM Bne model. (2) We fine-tune the OPT-1.3B model on two-operator queries, improving its accuracy on two-operator queries from 1.7% to 69.7%. (3) Building upon the OPT-1.3B-O0 model, we further train on two-operator queries. We analyze the dynamic encoding of different types of interactions during the above three training processes and obtain the following insights.
Figure [5 reports the dynamic changes in different types of interactions encoded by the OPT-1.3B model during training on one-operator queries. The results of R( are averaged over all one-operator queries, and the results of (1))) are averaged over queries from a single one-operator template y We observe that as the accuracy of the OPT-1.33 model on one-operator queries improves, the focality on operand-operator interactions ( (1p7 and the focality on high-order interactions gradu- ally increase. while the focality on operator interactions ((() (SP+) and the focality on background interactions R² (22*) gradually decrease. The results validate our Insight
We also observe that the OPT-1.3B model consistently enhances its focus on background interactions in the very few layers close to the output layer during the training process. This may be due to the inherent preference of the 1.T-1.3B model, which tends to enhance its focus on background interactions the very few layers close the output layer, Figure shows.
The internal mechanism of LLMs for solving relatively complex two-operator arithmetic problems is their capability to encode operator interactions and operand interactions. Figure rreports the dynamic changes in different interactions encoded by the OPT-1.3B model during training two-operator queries. The results are averaged over all two-operator queries. We observe that the focality on operator interactions R (S remains consistently high throughout the training process. As the accuracy of the OPT-1.3B model two-operator queries improves, the focality on operand interactions R()) (12md) gradually increases in the later layers during the final stage of training. This suggests that solving two-operator arithmetic problems requires an LLM to focus more on operator interactions and operand interactions. We notice that the OPT-1.3B model consistently focuses on background interactions, which may be due to its inherent preference. Notably, the focality on operand-operator interactions gradually increases in the very few layers close to the output layer, which may contribute the model's strong performance on two-operator arithmetic tasks.
Insight 2:
PPlease see Appendix H for details about model training.
8

---

Fine-tuned on two-operator queries
OPT-1138
pain!
por12
part
poiil
point
Accusscy"% on twovipenttor quaeries
1.7
15.3
31.9
53.4
60
08.7
14
openar inaractoon.
opera interaatoon
-on operand operator Interactions background imtractioss
09
Layorz
Layar
Layar
Layar
Layer
Figure 6: Visualizing the dynamic changes in different types of interactions encoded by the OPT-1.3B model during the training process on two-operator queries. Each curve is averaged over various two-operator queries. Results show that the OPT-1.3 model consistently focuses more on operator interactions and gradually increase its focus on operand interactions during the final stage of training.
Fine-tuned on two-operator quaries
OPT-1
comm
point
O
0
Acourang (h) on ono-operalzr queries
32.1
25.4
1.
opc
810
0.
badig
13
15
25
19
4.0
64
6.
tsl-onder
6thoreer
Prisonder Thbarder
2.0
4iyarder Sthorddr
00-arder
13
25
25
13 13
25 19
Figure 7: Visualizing the dynamic changes in different types and orders of interactions encoded by the OPT-1.3B- One model during the training process on two-operator queries. Each curve is averaged various one-operator queries. Results show that accuracy declines, the model reduces its focus on operand-operator interactions (top) and high-order interactions (bottom).
Insight We explain the task-specific nature of the LoRA method from the perspective of interactions. Figure 7 reports the dynamic changes in different types of interactions encoded by the OPT-1.3B-OLL model during training on two-operator queries. The results of R(0) are averaged over all one-operator queries, and the results of a() are averaged over queries from single one- operator template B We observe that the accuracy of the OPT-1.3B-One model on one-operator queries declines, which means that the OPT-1.3B-Oht model gradually forgets how to solve simpler arithmetic problems while learning more complex knowledge. Meanwhile, the accuracy on two-operator queries steadily improves during training (please Appendix M for details). This phenomena is consistent with the task-specific nature of LoRA Hu et al. 2022]. From the perspective of interactions, we observe that the focality operand-operator interactions (1 and the focality high-order interactions gradually decrease during the training process, with most values tending to stay around 1.0. The results help explain the underlying mechanism behind the task-specific nature of LoRA in arithmetic reasoning tasks and validate our Insight 1.
Conclusion and discussions
In this paper, we use interactions to provide a deep understanding of the internal mechanism LLMs for arithmetic reasoning. Through comparison studies of different types and orders of interactions encoded by LLMs during forward propagation, we find that the internal mechanism of LLMs for solving simple one-operator arithmetic problems is their capability to encode operand-operaton interactions and high-order interactions. We further fine-tune an LLM to explore how an LLM encodes different types and orders of interactions when learning to solve arithmetic problems. We find that the internal mechanism of LLMs for solving relatively complex two-operator arithmetic problems is their capability to encode operator interactions and operand interactions. We also explain the task specific nature of the LoRA method from the perspective of interactions.
Limitations. As the number of input variables increases significantly, the computational cost of interaction analysis indeed rises. address this issue, we can analyze informative variables while treating uninformative variables as background or adopt some other methods (see Appendix D. On the other hand, we have only studied simple arithmetic problems and have not yet extended our research to more complex math word problems. In the future, will work on this.

---

Acknowledgment
This work is partially supported by the National Nature Science Foundation of China NN.66376619.66206177)
References
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with polynomial time algorithm for shapley value approximation. In International Conference on Machine Learning, pages 272-281. PMLR, 2019.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos. Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint urXiv:2310.1066 2023.
Lu Chen, Siyu Lou, Benhao Huang, and Quanshi Zhang. Defining and extracting generalizable interaction primitives from dnns. arXiv preprint arXiv:2401.16316 2024.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in neural information processing systems, 30, 2017.
Huiqi Deng, Qihan Ren, Hao Zhang, and Quanshi Zhang. Discovering and explaining the representa- tion bottleneck of dnns. arXiv preprint arXiv:2111.062. 2021.
Huiqi Deng, Na Zou, Mengnan Du, Weifu Chen, Guocan Feng. Ziwei Yang, Zheyang Li, and Quanshi Zhang. Unifying fourteen post-hoc attribution methods with taylor interactions. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF international conference computer vision, pages 2950-2958, 2019.
Michel Grabisch and Marc Roubens. An axiomatic approach to the concept of interaction among
players in cooperative games. International Journal of game theory, 28:547-565, 1999.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36:76033-76060, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, et Mrkl systems: A modular, neuro-symbolic ar- chitecture that combines large language models, external knowledge sources and discrete reasoning. arXiv preprint urXiv:2205.00444 2022.
Scott Lundberg. A unified approach to interpreting model predictions. arXiv:1705.07874, 2017.
arXiv preprint
Daking Rai and Ziyu Yao. An investigation of neuron activation unified lens explain chain-of- thought eliciting arithmetic reasoning of llms. arXiv preprint arXiv:2406. /2288, 2024.
Yasaman Razeghi, Robert L Logan IV. Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning, arXiv preprint arXiv:2202.0721 2022.
Jie Ren, Mingjie Li. Qirui Chen, Huiqi Deng, and Quanshi Zhang. Towards axiomatic, hierarchical, and symbolic explanation for deep models. 2021a.
Jie Ren, Zhanpeng Zhou, Qirui Chen, and Quanshi Zhang. Can we faithfully represent masked states to compute shapley values on dnn? arXiv preprint 200. 2021b.
10

---

Jie Ren, Mingjie Li, Qirui Chen, Huiqi Deng, and Quanshi Zhang. Defining and quantifying the emergence of sparse concepts in dnns. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 20280-20289 2023a.
Qihan Ren, Huiqi Deng, Yunuo Chen, Siyu Lou, and Quanshi Zhang. Bayesian neural networks avoid encoding complex and perturbation-sensitive concepts. In International Conference on Machine Learning, pages 28819. PMLR, 2023b.
Qihan Ren, Yang Xu, Jumpeng Zhang, Yue Xin, Dongrui Liu, and Quanshi Zhang. Towards the dynamics of a dnn learning symbolic interactions. arXiv preprint arXiv:2407.19198 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Lloyd S Shapley. A value for n-person games. Contribution to the Theory of Games, 2, 1953.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. arXiv preprins arXiv:2305.15005, 2023.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pages 3319-3328, PMLR, 2017.
Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The shapley taylor interaction index. PMLR, 2020.
In International conference on machine learning, pages
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397), 2023.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A billion parameter autoregressive language model, 2021.
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in lims for enhanced mathematical reasoning. arXiv preprint 10.0v:2310003331, 2023.
Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach interpreting and boosting adversarial transferability. arXiv preprint arXiv:2010.04055. 2020.
Xin Wang, Shuyun Lin, Hao Zhang, Yufei Zhu, and Quanshi Zhang. Interpreting attributions and interactions of adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1095-1104, 2021.
Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do Ilamas work in english?
on the latent language of multilingual transformers. arXiv preprint arXiv:2442.11058 2024.
Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. Advances in neural information processing systems, 36:78205--88222 2023.
Zeping Yu and Sophia Ananiadou. Interpreting arithmetic mechanism in large language models through comparative neuron analysis. arXiv preprint arXiv:2409.14144, 2024.
Quanshi Zhang, Xin Wang, Jie Ren, Xu Cheng, Shuyun Lin, Yisen Wang, and Xiangming Zhu. Proving common mechanisms shared by twelve methods of boosting adversarial transfcrability. arXiv preprint arXiv:2207.1 1694, 2022a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.000 00068, 2022b.

---

Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. Interpreting and improving large language models in arithmetic calculation. arXiv preprint arXiv:2409.01659, 2024.
Huilin Zhou, Hao Zhang, Huiqi Deng, Dongrui Liu, Wen Shen, Shih-Han Chan, and Quanshi Zhang. Concept-level explanation for the generalization of a dnn. arXiv preprint arXiw:2302.13391, 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims from the introduction and abstract are reflected in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize other setrings.
It is fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Conclusion and Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not diseussed in the paper.
The authors are encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low or images taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
For each theoretical result, the full set of assumptions and a complete (and correct) proof are included the main paper and Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details are included in the main paper and in Appendix B Appendix H Appendix
Appendix P
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-souree dataset or instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While the code and dataset generated are not yet ready for anonymous open sourcing, we plan to open-source the code and the data with appropriate licensing.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cco
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are included in the main paper and in Appendix B Appendix ( Appendix H Appendix K
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: For all experiments, the do sample parameter of the large language model was set to False to ensure deterministic outputs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK to report 1 1-iigma error bars, but one should state The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have added computational resources needed in the Appendix K
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't ' make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our work is to study arithmetic reasoning in large language models, and doesn't t violate any code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss scalability and limitations of our work in conclusion. Appendix ?? and Appendix[[ Since our work about the internal mechanism of arithmetic reasoning in LLMs, we do not believe that there are any direct negative implications.
Guidelines:
16

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considcrations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is not focused towards releasing novel pretrained language models, image generators, or scraped datasets. For the data generated using cxisting arithmetic reasoning templates with randomly sampled one- and two-operand numbers, and the large language model fine-tuned on the data based on existing pretrained models, we believe there risks of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We make sure to cite all relevant and related work. The templates we use are publicly available.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
17

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paasssaaperrree thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not release any new assets in this work. For the data generated in our work, we include templates mentioned in Appendix
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not do any crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector:
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not do any experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We only use LLMs for translation and grammar checking.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy atttpe///eerrrpppccc///oooeeeer htcc/Conferencees22000 what should or should not described.
/LLM) for
19

---

A Strategies of masking input variables
In the research of attribution methods, it is common use specific token embedding mask input variables a DNN [Lundherg, 2017 AAncona et al. 2019 Fong et al. 2019]] and use changes in network outputs on the masked samples to estimate attributions of input variables. However, each method has certain limitations. The mean baseline value [Dabkowski and Gal 2017]. i.e., setting the baseline value for each input variable its mean across all samples, introduces additional signals. e.g., grey dots in images. Similarly. the zero baseline value [Ancona et al.) [019 [Sundararajan et al. 2017], i.e., setting baseline values for all input variables to zero, would also introduce additional signals to the input, such as black dots.
B Details about how to mask input words for different LLMs
In this paper, we analyze seven LLMs for arithmetic reasoning. including the OPT-1.3B model, the GPT-J-6B model, the Llama-2-7B model, the Codel model, the MathCoder-L-7BB model, the MathCoder-CUI CL-BB model, and the Llemma-77 7B model. For the OPT-1.33 model. we use the "
" token with the token id = 2 to mask the words in N'.. For the GPT-J-6B model, we use the "< endoftext 1>" with the token id 50256 to mask the words in N.. For the other five models, including the Llama-2-7B model, the CodeLlama-2-1 the MathCoder-CL- model, and the Llemma-7B model, with Llama as their base model, we use the "<unk>" with the token id = 0 to mask the words in N.T.
model, the MathCoder-L-7B model,
C Properties of interactions
The Harsanyi interaction 1s. i.c., the interaction in this paper, can explain the elementary mechanism of existing game-theoretic metrics [Ren et al. [02 including the Shapley value (Shapley [953]. the Shapley interaction index [Grabisch and Roubens 1999]. and the Shapley-Taylon interaction index (Sundararajan et al. 2020].
(1) Connection to the Shapley value (Shapley 1953]. Let Φ(t) denote the Shapley value of input variable i, given the input sample x. Then, the Shapley value Φ(1) can be explained as the result of uniformly assigning attributions of each Harsanyi interaction to each involving variable i, i.e., (()) = Σ SscVi / /u(gs- This also proves that the Shapley value fair assignment of attributions from the perspective of the Harsanyi interaction.
(2) Connection to the Shapley interaction index [Grabisch and Roubens 1999]. Given subset of variables T C N in an input sample æ, the Shapley interaction index 1 1hShpple can be represented as |T Shapley = EscNAr TS ISUT. In other words, the index y Shhppey can be explained as uniformly allocating Is such that S" = SUT to the compositional variables of S", if we treat the coalition of variables in T as a single variable.
(3) Connection to the Shapley Taylor interaction index [Sundararajan et al. 2020). Given subset of
variables T C N in an input sample æ, the k-th order Shapley Taylor interaction index 1 (hhapley Taylor can be represented as a weighted sum of interaction effects, i.e., 1 "Shapeey-Taylor [ TTylor
JT if [T] < k;
EsgNAT (IS)+t)) -1 "ISUT [T] k; and 1r7h yShhaple-
if [T) k.
Given an input sample æ. the Harsanyi interaction 1s satisfies seven desirable axioms in game theory [Ren et al. 2021a], including the efficiency, linearity, dummny, symmetry, anonymity, recursive and interaction distribution axioms.
(1) Efficiency axiom. The output score of a model can be decomposed into interaction effects of different patterns, i.e., v(x) = EsCN Is.
(2) Linearity axiom. If we merge output scores of two models w and as the output of model u, i.c.,
SA N, u((s)) v(xs)) ) + w(ass), then their interaction effects p( and f(() can also be merged as AS C N. I = 1 + (wr)
(3) Dummy axiom. If variable i N is dummy variable, i.e., AS C NV o( v(xs) v()),)), then it has no interaction with other variables, VD + T C (V(i),,ruu)=
20

---

(4) Symmetry axiom. If input variables i.j E N cooperate with other variables in the same way, SS S N fi (() = o(zsu())). then they have the same interaction effects with other variables, VS S N (i.j), ss [su())-
(5) Anonymity axiom. For any permutations on N, we have AS _ N, S a {i(t) i 8), and the new model TE is defined by (TT) v(zg). This indicates that interaction effects are not changed by permutation.
T( where
(6) Recursive axiom. The interaction effects can be computed recursively. For i E N and S S (1), the interaction effect of the pattern S {i} is equal to the interaction effect of with the presence of is always present)
minus the interaction effect of S with the absence of i, i..., VS S NM),), /utfi rs
always present) denotes the interaction effect when the variable i is always present as a constant context, i.e. 1 is always present)
Is
ELcs(-1) ( ( v()))()))
(7) Interaction distribution axiom. This axiom characterizes how interactions are distributed for "interaction functions" [Sundararajan et al. 2020]. An interaction function VT parameterized by a subset of variables T' is defined as follows: VS C N, ur (æs)
if C S; otherwise, (r(x)) O. The function vy models pure interaction among the variables in T because only if all variables in T are present the output value will be increased by c. The interactions encoded in the function ur satisfy Ir = c, and AA + T, Ig = O.
D Details about network architectures and chosen layers for experiments in section
OPT-1.3B model. The OPT-1.3B model is composed of the following parts: one word embedding layer (namely Embedding Layer), one position embedding layer (namely OPTLearnedPasitionall bedding), 24 OPT decoder modules (namely OPTDecolerLayyr), and one linear output layer (namely Linear Layer). The architecture can be summarized Embedding Layer OPTLearnedPP tionalEmbedding + (OPTDecoderLaver)>> X24 + Linear Layer. Each module of OPTDecoderLayer contains a self-attention mechanism layer (namely OPTAttention). an activation function (namely ReLU), a layer normalization operation (namely LayerNorm), two fully connected layers (namely Linear), and final layer normalization operation (namely LayerNorm). In our experiments, we selected all 24 OPDDecoderLayer modules and conducted experiments based on the output features of the LayerNorm operation.
GPT-J-6B model. The GPT-J-6B model includes a single word embedding layer (namely Embedding Layer), followed sequentially by 28 Transformer blocks (namely GPTJBlock), culminating in an output layer normalization (namely LayerNorm) and a linear output layer (namely Linear Layer). Layer-
This architecture can be suceinetly described as: Embedding Layer - [GPTJBllck])228 Norm > Linear Layer. Delving into the details of each module of GPTJBlock, it comprises three integral components, including a layer normalization (namely LayerNorm). a self-attention mecha- nism (namely GPTJAttention), and a feed-forwart network (namely GPTMLP). In our experiments, we selected all 28 GPT.Bllock modules and conducted experiments based on the output features of the LayerNorm operation.
Other Llama-based models. The other five models based on Llama include the Llama-2-7B model, the CodeL model, the MathCoder-L--B model, the MathCoder-CLL7B model, and the Llemma-7B model. These models share the same architecture, which is composed of various elements: a single word embedding layer (namely Embedding Layer), followed by a sequence of 32 LlamaDecoderLayerr an output layer normalization layer (namely LlamaRMSNorm), and culminating in linear output layer (namely Linear Layer). This architecture can be summarized as Embedding > LlamaRMSNorm - Linear Layer. Delving into each module it integrates multiple components, including
Layer [LlamaDecoderLLnn of Laama layer (namely .lamaAttention), a feed-forward network layer (namely LlamaMLP) encompassing several linear layers and activation functions, an input layer normalization (namely Input LayerNormi) and a post-attention layer normalization (namely Post Attention LayerNorm). the latter serving to normalize the output from the self-attention mechanism layer. In our experiments, we selected all 32 modules of Laama and conducted experiments based on the output features of the LayerNorm operation.
atf-ntttnnion mechanism
21

---

Xy (maac,, 5)
Xy [Row, much.
2, [How, 511
X+ (How, Hoow,L.Aeee L.nwwer,ds)
1.0
1.0
1.0
Klon -o- Ioon 10
0.5
0.5
0.5
Stan -g- Tooen 11
10m Taeen
Reaz Thoen
Layer
25
33
25 33
33
23
33
Figure 8: The visualization of ( at different token positions across various layers of CodeLlama for the prompt "How much is 5 plus 1? Answer is" under different masked inputs. Results show that except for the final tokens of the complete sentence (i.e., token 10 and token 13), the v(1) values of other tokens in the middle layers remain almost unchanged.
Table 2: Question templates for one-operator arithmetic queries.
Type
Addition
Subtraction
How ii plus mg? Answer
How much my minus viz? is What is ni minus na? Answer is
2 What is ni plus ri?? Answer is
3 How much is the sum of ni and ng? Answer is ny and vı? Answer
How much is the difference between and niz Answer is What difference between nı and nz? Answer The difference between ii and Ry is
What
5 The sum of 2y and my is
6 Given two numbers and nz. the sum of them is
Given numbers and the difference between them is
Multiplication
Division
How much ny ny? Answer
How much my ny? Answer
2 What is by times nl? Answer is
What is ni over ng? Answer
3 What is the result of times ri?? Answer
What is the result of
nz? Answer is
4 How much is the product
and n2? Answer
How much the ratio between n, and n.? Answer
5 The product in and ny
The ratio of ny and ng is
6 Given two numbers by and ng, the product of them
Given two numbers and n2, the ratio between them is
E The features of other tokens in the middle layers remain unchanged
Through experiments, we found that, except for the final token of the complete sentence, the ( values at other token positions the middle layers remain almost unchanged. Figure & shows the ()()) values at different token positions across various layers of CodeLlama for the prompt "How much is plus Answer is" under different masked inputs ry- The prompt consists of tokens (
, How. much, is, space, 5, plus, space, 1, ?, answer, is, space). Except for token 10 (?) and token 13 (space), the y(!) values of other tokens in the middle layers remain almost unchanged.
F Prompt Templates
report the question templates used prompts for the model for and two-operator queries, respectively. For two-operator queries, we use one query template for each of the 29 possible two-operation combinations. To enable the model output the answer directly, appended "The answer is" at the end of each template.
In Table 2 and
G Performance of the LLMs
To accurately and fairly evaluate the LLMs' capabilities solve one-operator and two-operator arithmetic problems, we evaluated the accuracy all LLMs on the one-operator and two-operator test sets used for fine-tuning in Appendix
For the OPT-1.3B and GPT-J-6B models, we treat each operand as a single token, while for other LLMs, each number is split into multiple tokens ("0", "I", "2".... "9") by the tokenizer.
22

---

Table 3: Question templates for two-operator arithmetic queries.
Type
Formula
Format
f ((A B)
Sum A and B and multiply by C
f (A+ B+
What is the sum of and the product of and C?
f ((A - B) C)
What is product of minus and
f = (A/(B/C)
How much A divided by the ratio between B and C?
f = (A -BCC)
What is the difference between A and the product of and C?
6
f = (A (B-C))
How much A times the difference between B and C?
7
f == ((A + B)/C)
What is the ratio between plus B and C?
8
f (A (B-C)
How much A minus the diffrence between B and C?
f ((A B)/C)
What is the ratio between minus B and C?
f (A B/C)
What is difference between A and the ratio between and How much divided bu the sum of B and C?
f = (A/(++O)
f = (A/(B-C)
How much A divided by the difference between B and C? what is the sum of A and the ratio between and C?
13
f == B/C)
14
f == (A* (B/O))
How much is A times the ratio between B and C?
15
f (A*
How much is the sum of A times B and C? How much A times the of B and C?
f (A (B+)))
f (A/B+O)
How much the sum of divided by B and C?
f == (A/B/C)
How much divided by B divided by C?
19
f == (A/B-O)
How much the difference between divided by B and C? How much A divided by B times C?
20
f f= CA)
21
[= (4 (=(A(B++O)
How much is A minus the sum of B and C?
22
f= B-C)
How much is the difference between A times B and C?
23
f f= (A)(B+
How much A divided by the product of B and C? How much A minus B plus C?
24
f f= (A -++C)
f f= B++
How much A plus plus How much A minus B minus C??
26
f == (A-B- -B)
27
f = B/C)
How much A times B divided by C?
28
f B-+B-O)
How much is A plus minus C?
29
f B+C)
How much is A times B times C?
H
Fine-tuning Details
For the one-operator data, we sample 500 examples for each of the 6 natural language templates in Table[aand their corresponding original mathematical expressions, resulting total of 3,500 samples. For the two-operator data, we sample 1,000 examples for each of the 29 natural language templates in T resulting in total of 29,000 samples. We set the maximum operand value to 100 and exclude problems with final results exceeding 1000. For single-operator data, we use an 8/2 train-test split, while for two-operator data, we use a 9/1 train-test split.
We fine-tune the OPT-1.3B model using the LoRA method (Hu et al. 2022 one-operator and two-operator samples. For the one-operator templates, we train the model for 10 epochs with a batch size of 16. For the two-operator templates, we train the model for 20 epochs with a batch size of 32. The training uses learning rate of 8e-4 with linear decay scheduler. The LoRA configuration includes rank of 8, LoRA alpha of 32, and dropout of 0.05.
I More experimental results
Figure shows that LLMSs with good arithmetic capabilities gradually focus more on high-order interaction patterns. Figure [0 illluttraees that the OPT-1.3B model gradually increases its focus
23

---

Inverder -inonneerder --- -tvorde
4thvarde
Sihvrddr
-thvrrde -is 7lbvoree -- -i-Ibhoree litvord
OPT-138 GB
Liama-2-8
Liemme 7B
MahCCddrLL7B
MathCoder
CodeLiame 138
4.2
4.8
2.0
27
Figure 9: Comparing the normalized average strength (0)) interaction patterns different orders encoded by LLMs during forward propagation. Each curve in the figure averaged over various one-operator arithmetic queries, corresponding to template Table
one-operator querles
OPT-1
Ascusscy()) un one operator queries
67.6
sperand
basiground Internationnoooionn
13
14
13
14
25
7.0
20 order -i- -itthh ondor
4th-odder - Bthooddor
1.4
Sth-rder
13 Layer/
19
Laywr
Figure 10: Visualizing the dynamic process different interaction patterns encoded by the OPT- 1.3B model during the training process. Each curve is averaged over various one-operator queries, corresponding to (a) template 0 and (b) template in Table
high-order interaction patterns during the learning process of simple arithmetic problems. Figure [] demonstrates that the OPT-1.3B model gradually decreases focus on high-order interactions while leamingg relatively complex arithmetic problems.
Information about the use of AI assistants.
In this paper, Al tools such as DeepSeek were used for translation and grammar checking.
K Computational budget
We conducted our experiments on an NVIDIA GeForce RTX 4090 24GB GPU. For the Llama-2- 7B model, the computation time per one-operator sample is around 30 seconds, while that for two-operator sample is around 60 seconds.
Fine-tuned on two-operator queries
OPT One
Time coin: 40.1
Time poin1
Time soint
Time J0int4
Time point
Accuracy
83.8
34.0
$2.1
28.7
25.4
ope inter ora speraed intraction..
8/0
opo and crotor interaatins sedigrounnt indernations
Layar?
Laper?
Layer
Layar
Layer
Layw?
4.0
4.0
falcander -.s Silvrreer 2nd Ethoorder
6.0
Thvrrde Tihverdrr
Layor2
Layerz
Layor
Layorr
Layor
Layor
Figure 11: Visualizing the dynamic process of different interaction patterns encoded by the OPT-1.3B- One model during the training process. Each curve averaged over various one-operator queries, corresponding to (a) template 0 and (b) template 3 in Table

---

L
Limitations
When the number of input variables increases significantly. the computational cost of interaction analysis indeed rises. To deal with this issue, we can adopt the following approaches.
Analyzing informative variables while treating uninformative variables background to reduce computational complexity while preserving core semantic information. Previous research (Chen] al. 2024 has demonstrated that selecting informative input variables does not fundamentally compromise the faithfulness of interactions.
Phrase-level aggregation, where related tokens merged into meaningful phrases without compro- mising semantic integrity, can reduce the computational burden.
Exploring approximation methods, such as variable selection based on attention weights and prioritizing tokens that have a greater impact on model predictions for interaction analysis, can help reduce computational overhead.
M Accuracy curve of the OPT-13B-One model on two-operator queries
We visualize the accuracy of the OPT-1.3B-One model on two-operator arithmetic queries across different checkpoints. As shown in Figure [2] the model demonstrates gradual improvement in performance during the training process. We also highlight five selected checkpoints (referred to as TimePoints through 5) that are used in Figure
Accuracy on two-operator queries
5
Accuracy
0.60
Selected Checkpoints
0.55
0.50
Time-Point 4
Accuracy
Time Poin-Ponnt
0.45
0.40
Time-Point
0.35
0.3Ame--Poin
200
400
1000
1200
1400
1600
Checkpoint
Figure 12: Accuracy of the OPT-1.3B-OL model two-operator queries evaluated at different checkpoints. Red markers indicate selected TimePoints 1-5 used in Figure 7
25


Reviews:
[{"review_id": "TwDUTXDFBy", "reviewer": "Reviewer_pSEa", "summary": "This paper proposes explaining arithmetic reasoning in LLMs through game-theoretic interactions, quantifying operand-operator, operator-operand, and background interactions across layers. It identifies distinct interaction encoding patterns in LLMs for single-operator vs. multi-operator problems and links LoRA’s task-specificity to interaction shifts. This work provides a novel perspective for explaining arithmetic reasoning in LLMs, which may help users and developers understand, use, and improve LLMs.", "questions": "See the weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Faithfulness guarantees via Harsanyi interactions provide a mathematically sound foundation, distinguishing it from heuristic attribution methods.\n- The layer-wise analysis of interaction types effectively reveals distinct reasoning mechanisms for different problem types.\n- Comprehensive experiments have been conducted to demonstrate the different types of interactions and validate the proposed insights.\nWeaknesses\n- Source code is not opened.\n- The interaction effect can fit the output (Theorem 1), but since this is an interpretability study, whether the causal necessity can be verified?\n- Experiments use templated arithmetic questions. It’s unclear how it performs in free‐form math problems.", "was_revised": true, "final_justification": "In authors' rebuttal, my concerns have been well addressed.", "rebuttal": "Thank you for your great efforts on the review of this paper. We will try our best to answer all your questions.\n\n$\\color{blue}Q1:$ **\"Source code is not opened.\"**\n\n**A:** Thank you. Due to the conference policy, we cannot provide any links during the rebuttal phase. The code will be released publicly upon acceptance.\n\n$\\color{blue}Q2:$ **Ask for verify the causal necessity. \"The interaction effect can fit the output, but since this is an interpretability study, whether the causal necessity can be verified?\"**\n\n**A:** Thank you. This is a very good question. **Causal necessity** means that a certain factor is necessary for the model's prediction, i.e., without this factor, the model would not be able to produce the same output. **In our paper, the causal necessity is reflected in the influence of input variables on both the interaction effects and the LLM outputs.**\n\nIn our paper, given an input $\\boldsymbol{x}$ and an LLM, we can construct a **logical model** based on interactions to accurately fit the output of the LLM. The output of the logical model can be decomposed into the sum of the effects of all interactions in $\\boldsymbol{x}$. \n\nGiven an LLM, changing the input $\\boldsymbol{x}$ leads to changes in the LLM's output. This process reflects the **causal necessity** of input variables, while interactions are the outcome derived from the above causal analysis. However, we cannot directly change interaction effects to influence the LLM's output.\n\nWe **conducted new experiments** to demonstrate the causal necessity of the input variable (Table 1). Specifically, we input two prompts into the Mathcoder-L-7B model that differ **only in the operator** (\"+\" vs. \"×\"). **We observe that both the model’s output and the focality on different types of interactions change accordingly.** \n\nTable 1: Changes in **model output** and **focality across different types of interactions at Layer $l$** when flipping the operator in arithmetic prompts. (Mathcoder-L-7B)\n\n**Prompt**: \"*How much is 2 **plus** 4? Answer is* \"\n**Model output**: \"*6*\"  \n\n|Interaction Type|Layer 10|Layer 26|Layer 31|\n|-|-|-|-|\n|Operand|0.97|0.85|0.78|\n|Operator|1.02|0.66|0.76|\n|Operand-Operator|1.04|1.30|1.33|\n|Background|0.96|0.88|0.90|\n\n**Prompt**: \"*How much is 2 **times** 4? Answer is* \"\n**Model output**: \"*8*\" \n\n|Interaction Type|Layer 10|Layer 26|Layer 31|\n|-|--|--|--|\n|Operand|0.99|0.92|0.92|\n|Operator|0.98|0.89|0.99|\n|Operand-Operator|1.01|1.11|1.08|\n|Background|1.00|1.03|1.02|\n\n\n$\\color{blue}Q3:$ **Ask for new experiments on free-form math problems. \"Experiments use templated arithmetic questions. It’s unclear how it performs in free‐form math problems.\"**\n\n**A:** Thank you. A good question. We have **conducted new experiments** on a free-form math problem : \"*Henry has 1 notebook, then Henry receives 8 notebooks. What's the total number of notebooks that Henry has?*\" \nWe categorize input variables into 4 types: **numbers** (i,e., \"*8*\", \"*1*\"), **operators** (i.e., \"*receives*\", *\"the total number of\"*), **entity words** (including names and objects, i.e., \"*Henry*\", \"*notebook*\", \"*notebooks*\"), and background words (all words other than the above three categories).\n\nFollowing the same setting as in our paper, we also define four types of interactions:\n\n1. **Operand interactions**: contain only numbers (no operators)\n\n2. **Operator interactions**: contain only operators (no numbers)\n\n3. **Operand-operator interactions**: contain both numbers and operators\n\n4. **Background interactions**: contain neither numbers nor operators\n\nWe  conducted experiments on the Mathcoder-L-7B model. We observe that during the forward propagation, the LLM’s focus on operand-operator interactions (see Table 2) gradually increases, along with its focus on higher-order interactions (7th-order and 8th-order in Table 3). **This is consistent with the Insight 1 in our paper. This experiment demonstrates that our method can be extended to free-form math problems.**\n\nTable 2: Focality across **different types of interactions** at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)\n\n|Interaction Type|Layer 5|Layer 9|Layer 18|Layer 20|Layer 30|\n|-|-|-|-|-|-|\n|Operand|0.97|0.98|0.86|0.88|1.00|\n|Operator|1.10|**1.11**|1.09|1.04|0.97|\n|Operand-Operator|0.96|0.95|**1.13**|**1.16**|**1.02**|\n|Background|**1.11**|1.10|0.15|0.14|0.14|\n\nTable 3: Focality  across **interactions of different orders**  at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)\n\n|Interaction Order|Layer 5|Layer 9|Layer 18|Layer 20|Layer 30|\n|-|-|-|-|-|-|\n|1|**2.02**|**2.05**|2.10|1.86|1.31|\n|2|1.26|1.22|0.98|0.93|0.83|\n|3|1.06|1.03|0.85|0.77|0.79|\n|4|0.96|0.95|0.90|0.84|0.88|\n|5|0.88|0.89|0.99|1.03|1.07|\n|6|0.79|0.85|1.08|1.29|1.36|\n|7|0.67|0.83|1.52|1.96|1.91|\n|8|0.59|0.83|**3.10**|**3.85**|**3.13**|"}, {"review_id": "aLQcz3yegK", "reviewer": "Reviewer_XdQB", "summary": "This paper introduces a game-theoretic method for studying inside transformer models when they do arithmetic. It shows that any LLM’s numeric output can be exactly written as the sum of Harsanyi interaction terms over all subsets of input tokens. By grouping them into “operand-only”, “operator-only”, “operand-operator”, and “background” categories, they find that strong models gradually amplify high-order, “operand-operator” interactions in late layers, thereby binding numbers to their operators. Using LoRA for fine-tuning, model gain on complex two-operator tasks at the cost of forgetting simpler one-operator reasoning, illustrating how tuning can reshape the collaboration patterns.", "questions": "1. Can you use other distance metrics (e.g. Euclidean) alongside cosine similarity to capture both angular and magnitude changes?\n2. Can you provide a simple intervention study to produce the predicted jumps in interaction scores and output, thereby demonstrating causality? For example, flipping the operator token from ‘×’ to ‘+’.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper is well-structured, with a solid theoretical foundation backed by rigorous game-theoretic proofs and clear mathematical derivations.\n2. This paper provides layer-wise focality scores that clearly show how and when a model learns to bind numbers with operators. Besides, the methodology is model-agnostic and can be used on any transformer without changing its code.\n\nWeaknesses:\n1. The paper's empirical analysis is based on several assumptions; in my opinion, they are too strong to be fully convincing. The authors use cosine similarity (Eq.3) to quantify hidden-state changes across layers, which missed large magnitude shifts in a single dimension. Without metrics, like Euclidean distance, the analysis risks overlooking substantial activation shifts.\n2. Harsanyi interactions require summing over 2^n subsets, which is infeasible when input length grows beyond a handful of tokens.\n3. This paper does not compare its interaction decomposition with other interpretability or attribution techniques.", "was_revised": true, "final_justification": "The rebuttal has addressed my main concerns. First, the supplemented  Euclidean distance experiments confirmed the robustness of their findings beyond cosine similarity. Second, they proposed practical strategies with additional experiments to mitigate the scalability issue of Harsanyi interactions, which supports and verifies the feasibility. They also included the comparisons with Shapley value and an operator-flip intervention study, which strengthened the empirical support for their claims. Although the scalability remains a challenge, the additional results improve the paper's credibility. Therefore, I tend to borderline accept (rating 4) this paper.", "rebuttal": "Thank you for your great efforts on the review of this paper. We will try our best to answer all your questions.\n\n$\\color{blue}Q1:$ **\"The paper's empirical analysis is based on several assumptions which are too strong to be fully convincing. The authors use cosine similarity (Eq.3) to quantify hidden-state changes across layers, which missed large magnitude shifts in a single dimension. Without metrics, like Euclidean distance, the analysis risks overlooking substantial activation shifts.\"**\n\n**A:** Thank you. We would like to clarify whether the reviewer’s concern — *\"they are several assumptions which are too strong to be fully convincing\"* — refers specifically to our use of cosine similarity in Eq.3. If so, we **conducted new experiments** using **Euclidean distance**.\n\nWe replaced the cosine similarity in the original Eq.3 with Euclidean distance, and normalized it to the range $[0,1]$: \n\n$v^{(l)}(x_T) = 1 - (EuclideanDistance(f^{(l)}(x_T), f^{(l)}(x_N)) - d_{min}) / (d_{max} - d_{min} + ε)$\n\n**The experimental results confirm our main conclusion:** For simple one-operator arithmetic problems, LLMs that perform well (i.e., the LLaMA-2-7B model) tend to focus more on operand-operator interactions (Table 1) and high-order interactions (Table 2) during forward propagation. While LLMs with weak performance (i.e., the GPT-J-6B model) tend to focus more on background interactions (Table 3) and extremely low-order interactions (Table 4).\n\nTable 1: Focality across **different types of interactions** at Layer $l$  using Euclidean distance (**LLaMA-2-7B**).\n\n|Interaction Type|Layer 1|Layer 8|Layer 16|Layer 23|Layer 31|\n|-|-|-|-|-|-|\n|Operand|1.01|0.91|0.78|0.84|0.96|\n|Operator|0.94|0.96|0.83|0.64|0.63|\n|Operand-Operator|**1.07**|**1.15**|**1.40**|**1.44**|**1.34**|\n|Background|0.82|0.88|0.63|0.52|0.48|\n\nTable 2:  Focality across **interactions of different orders** at Layer $l$  using Euclidean distance (**LLaMA-2-7B** template 1). \n\n|Interaction Order|Layer 1|Layer 8|Layer 16|Layer 23|Layer 31|\n|-|-|-|-|-|-|\n|1|0.76|0.72|0.51|0.62|0.82|\n|2|0.62|0.42|0.23|0.27|0.46|\n|3|0.83|0.62|0.42|0.35|0.60|\n|4|0.96|0.94|0.74|0.57|0.75|\n|5|1.12|1.28|1.16|1.09|1.04|\n|6|1.41|1.67|2.12|2.37|1.82|\n|7|1.75|2.03|4.38|5.41|3.81|\n|8|**2.14**|**3.56**|**10.88**|**13.67**|**10.14**|\n\nTable 3: Focality across **different types of interactions** at Layer $l$  using Euclidean distance (**GPT-J-6B**).\n\n|Interaction Type|Layer 3|Layer 9|Layer 15|Layer 21|Layer 27|\n|-|-|-|-|-|-|\n|Operand|0.62|0.66|0.69|0.72|0.68|\n|Operator|0.97|1.02|1.02|0.94|0.53|\n|Operand-Operator|0.63|0.67|0.72|0.77|0.74|\n|Background|**3.28**|**2.98**|**2.75**|**2.58**|**3.18**|\n\nTable 4:  Focality across **interactions of different orders** at Layer $l$ using Euclidean distance (**GPT-J-6B** template 1). \n\n|Interaction Order|Layer 3|Layer 9|Layer 15|Layer 21|Layer 27|\n|-|-|-|-|-|-|\n|1|**5.58**|**3.69**|**3.15**|**2.85**|**10.15**|\n|2|0.75|0.76|0.77|0.78|0.65|\n|3|0.82|0.87|0.89|0.90|0.72|\n|4|0.86|0.93|0.95|0.96|0.73|\n|5|0.89|0.97|0.99|1.00|0.72|\n|6|0.92|1.00|1.01|1.02|0.69|\n|7|0.95|1.03|1.00|1.01|0.61|\n|8|0.91|0.94|0.80|0.78|0.81|\n\n$\\color{blue}Q2:$ **\"Harsanyi interactions require summing over 2^n subsets, which is infeasible when input length grows beyond a handful of tokens.\"**\n\n**A:** Thank you. A good question! When the input length grows beyond a handful of tokens, the computational cost of interaction analysis indeed rises.\n\nTo deal with this issue, we can adopt the following approaches.\n\n1. **Analyzing informative variables while treating uninformative variables as background** to reduce computational cost while preserving core semantic information. Previous research [Chen et al., 2024] [Ren et al., 2024] [cite 1] has demonstrated that selecting informative input variables does not fundamentally compromise the faithfulness of interactions.\n2. **Phrase-level aggregation**, where related tokens are merged into meaningful phrases without compromising semantic integrity, can reduce the computational cost.\n\nWe have **conducted new experiments using the two approaches** described above. Specifically, we extended a simple one-operator arithmetic problem \"*1+8*=\" into a longer prompt:  \"*Henry has 1 notebook, then Henry receives 8 notebooks. What's the total number of notebooks that Henry has?*\" We categorize input variables into 3 types: **numbers** (i,e., \"*8*\", \"*1*\"), **operators** (i.e., \"*receives*\", *\"the total number of\"*), and **entity words** (including names and objects, i.e., \"*Henry*\", \"*notebook*\", \"*notebooks*\"). By using **Approach 1**, all other words not belonging to the above three categories are treated as background. By using **Approach 2**, the phrase *\"the total number of\"* is grouped as a single variable.\n\nFollowing the same setting as in our paper, we also define four types of interactions:\n\n1. **Operand interactions**: contain only numbers.\n\n2. **Operator interactions**: contain only operators.\n\n3. **Operand-operator interactions**: contain both numbers and operators.\n\n4. **Background interactions**: contain neither numbers nor operators.\n\nWe conducted experiments on the Mathcoder-L-7B model. **The results confirm our Insight 1**: The internal mechanism of LLMs for solving simple one-operator arithmetic problems is their capability to encode operand-operator interactions  (Table 5) and high-order interactions (Table 6) during the forward propagation.\n\nNevertheless, **we acknowledge that when $n$ is too large, the computation becomes infeasible. In fact, we have discussed this issue in the *Limitation* section of our paper.**\n\nTable 5: Focality across **different types of interactions** at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)\n\n|Interaction Type|Layer 5|Layer 9|Layer 18|Layer 20|Layer 30|\n|-|-|-|-|-|-|\n|Operand|0.97|0.98|0.86|0.88|1.00|\n|Operator|1.10|**1.11**|1.09|1.04|0.97|\n|Operand-Operator|0.96|0.95|**1.13**|**1.16**|**1.02**|\n|Background|**1.11**|1.10|0.15|0.14|0.14|\n\nTable 6: Focality across **interactions of different orders**  at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)\n\n|Interaction Order|Layer 5|Layer 9|Layer 18|Layer 20|Layer 30|\n|-|-|-|-|-|-|\n|1|**2.02**|**2.05**|2.10|1.86|1.31|\n|2|1.26|1.22|0.98|0.93|0.83|\n|3|1.06|1.03|0.85|0.77|0.79|\n|4|0.96|0.95|0.90|0.84|0.88|\n|5|0.88|0.89|0.99|1.03|1.07|\n|6|0.79|0.85|1.08|1.29|1.36|\n|7|0.67|0.83|1.52|1.96|1.91|\n|8|0.59|0.83|**3.10**|**3.85**|**3.13**|\n\n[cite 1] Li M. et al. Does a neural network really encode symbolic concepts? ICML, 2023.\n\n$\\color{blue}Q3:$ **\"This paper does not compare its interaction decomposition with other interpretability or attribution techniques.\"**\n\n**A:** Thank you. A good question! **Traditional interpretability and attribution methods**, such as Grad-CAM and Shapley values, often lack rigorous theoretical guarantees. In contrast, the interactions has been proven to be faithful explanations by a series of theoretical guarantees [Ren et al., 2023a]. The faithfulness of using interactions to explain an LLM is reflect as follows.\n\n1. The output score of the LLM on any input sentence can be represented as the sum of the effects of all interactions.\n2. The output score of the LLM on any masked sentence can be represented as the sum of the effects of all triggered interactions.\n\nMoreover, most attribution methods focus on the influence of individual input variable. However, a DNN does not independently use each variable for inference. Instead, a DNN relies on interactions among input variables for inference. For example, the interaction between \"*green*\" and \"*hand*\" forms the concept of \"*beginner*,\" a meaning that cannot be derived from either word alone.\n\nNevertheless, we **conducted new experiments using the Shapley value-based method** on the same single-operator templates. We computed Shapley values for different types of input variables during the forward propagation of the LLM. The results show that LLMs that perform well (i.e., the LLaMA-2-7B model) tend to focus most on operators, followed by operands, and lastly on background words (Table 7). However, **the Shapley value-based method fail to capture the joint effect of operands and operators on the model’s reasoning process.**\n\nTable 7: Average Shapley values for different types of input variable across layers. (LLaMA-2-7B)\n\n|Variable Type|Layer 1|Layer 8|Layer 16|Layer 23|Layer 31|\n|-|-|-|-|-|-|\n|Operand|**0.16**|0.09|0.14|0.17|0.16|\n|Operator|0.13|**0.15**|**0.17**|**0.18**|**0.18**|\n|Background|0.12|0.13|0.11|0.10|0.10|\n\n\n$\\color{blue}Q4:$ **\"Can you provide a simple intervention study to produce the predicted jumps in interaction scores and output, thereby demonstrating causality? For example, flipping the operator token from ‘×’ to ‘+’.\"**\n\n**A:** Yes, we **followed your advice to conduct experiments** to demonstrate the causality of the input operator variable (Table 8). Specifically, we input two prompts into the Mathcoder-L-7B model that differ **only in the operator** (\"+\" vs. \"×\"). **We observe that both the model’s output and the focality on different types of interactions change accordingly.** \n\nHowever, we cannot directly change the interaction effect to observe output changes, as it is an outcome of causal analysis based on how input changes lead to output differences, not a controllable variable in the process.\n\nTable 8: Changes in **model output** and **focality across different types of interactions at Layer $l$** when flipping the operator in arithmetic prompts. (Mathcoder-L-7B)\n\n**Prompt**: \"*How much is 2 plus 4? Answer is* \"\n**Model output**: \"*6*\"  \n\n|Interaction Type|Layer 10|Layer 26|Layer 31|\n|-|-|-|-|\n|Operand|0.97|0.85|0.78|\n|Operator|1.02|0.66|0.76|\n|Operand-Operator|1.04|1.30|1.33|\n|Background|0.96|0.88|0.90|\n\n**Prompt**: \"*How much is 2 times 4? Answer is* \"\n**Model output**: \"*8*\" \n\n|Interaction Type|Layer 10|Layer 26|Layer 31|\n|-|--|--|--|\n|Operand|0.99|0.92|0.92|\n|Operator|0.98|0.89|0.99|\n|Operand-Operator|1.01|1.11|1.08|\n|Background|1.00|1.03|1.02|"}, {"review_id": "xiTITCFqnZ", "reviewer": "Reviewer_5hoz", "summary": "This paper introduces a game-theoretic framework to interpret the arithmetic reasoning capabilities of Large Language Models (LLMs). By disentangling the LLM's output score into contributions from various interactions between input words, the authors categorize these interactions into operand, operator, operand-operator, and background types, and also analyze their order (complexity). Through extensive experiments on one- and two-operator arithmetic problems across a range of LLMs, the study reveals that strong LLMs for simple problems primarily rely on operand-operator and high-order interactions, while for complex problems, they focus on operator and operand interactions. The paper also provides an interaction-based explanation for the task-specific nature of LoRA fine-tuning, demonstrating how it can lead to a decline in performance on simpler tasks by shifting the model's focus away from relevant interaction types.", "questions": "Please refer to the weakenesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\nThe paper applies game-theoretic interactions (specifically Harsanyi interactions and derived focality metrics) to explain the internal mechanisms of LLMs for arithmetic reasoning. This provides an approach to interpret complex model behavior, moving beyond less rigorous neuron-level analysis or perturbation studies.\nThe classification of interactions into operand, operator, operand-operator, and background types, along with the analysis of interaction orders, offers a framework for understanding how LLMs process arithmetic queries. \nThe study visualizes how interaction patterns change during the fine-tuning process. This dynamic analysis reveals the learning trajectory of LLMs, showing how they adapt their focus on different interaction types and orders as they acquire new arithmetic skills, and also explains phenomena like LoRA's task-specific nature.\n\n[Weaknesses]\nThe analysis of interactions in intermediate layers relies solely on the embedding of the last input token and its cosine similarity to the unmasked input. This is a simplification for complex transformer architectures, which distribute information across all tokens and layers via attention mechanisms. Might this choice lead to an incomplete or potentially biased understanding of how interactions are truly encoded and evolve within the model's internal states?\n\nThe paper acknowledges the absence of unified masking strategies and employs specific tokens (
, 
, <unk>) for masking. The choice of masking token may influence the model's internal representations and the resulting interaction values. \n\nThe study focuses exclusively on \"simple arithmetic problems\" (one- and two-operator queries) and explicitly states that it has not extended research to \"more complex math word problems\". This limits the generalizability of the findings.\n\nThe \"focality\" metric (Eq 8, 9) quantifies the strength of interaction types or orders by summing absolute interaction values. While useful for measuring overall \"focus,\" this approach may disregards the direction (positive or negative contribution) of interactions, which is crucial for understanding the precise nature of the model's reasoning.", "was_revised": true, "final_justification": "Thanks for the authors rebuttal, which addresses my most concerns. I keep my positive score.", "rebuttal": "Thank you for your great efforts on the review of this paper. We will try our best to answer all your questions.\n\n$\\color{blue}Q1:$ **\"The analysis of interactions in intermediate layers relies solely on the embedding of the last input token is a simplification for complex transformer architectures. Might this choice lead to an incomplete or potentially biased understanding?\"**\n\n**A:** Thank you. **We have conducted experiments and found that the variation in middle-layer representations is primarily concentrated at the last token.** Please see Appendix E for detailed results and analysis. Therefore, to simplify computation, we only focus on the change in the embedding of the last token, while ignoring the negligible changes in the embeddings of other tokens.\n\nNevertheless, **we show an example to support the above claim.** We tested a sample using the LLaMA-2-7B model: \"*How much is 5 plus 1? Answer is*\". This example contains 13 tokens: `['
', '▁How', '▁much', '▁is', '▁', '3', '▁times', '▁', '1', '?', '▁Answer', '▁is', '▁']`.\n\nWe focused on a masked sentence: \"*How much is 5 [mask] [mask]? Answer is ,*\" Table 1 reports the cosine similarity between embeddings of the original input and that of the masked input ($v^{(l)}$). Results in Table 1 show that, except for the final token of the complete sentence, that is, token 10 ('*?*') and token 13 (the last token '*▁*'), the cosine similarity between the embeddings of the original input and the masked input is nearly identical (close to 1). Only the embedding of the last token shows a significant change.\n\nMoreover, **several prior studies [Wendler et al. 2024] [Zhang et al. 2024] claim that in the Transformer architecture, the representation of last token includes information from all previous tokens.**\n\nTable 1: $v^{(l)}$ at different token positions across layers for the masked input \"*How much is 5 [mask] [mask]? Answer is ,*\".\n\n|Token|Layer 5|Layer 20|Layer 30|\n|-|-|-|-|\n|token_1|0.00|0.00|0.00|\n|token_2|1.00|1.00|1.00|\n|token_3|1.00|1.00|0.99|\n|token_4|1.00|1.00|0.99|\n|token_5|1.00|1.00|0.99|\n|token_6|1.00|1.00|0.99|\n|token_7|1.00|1.00|1.00|\n|token_8|1.00|1.00|1.00|\n|token_9|1.00|1.00|1.00|\n|token_10|0.95|0.95|0.91|\n|token_11|1.00|1.00|0.99|\n|token_12|1.00|1.00|0.99|\n|token_13|**0.57**|**0.71**|**0.59**|\n\n$\\color{blue}Q2:$ **\"The paper acknowledges the absence of unified masking strategies and employs specific tokens may influence the model's internal representations and the resulting interaction values.\"**\n\n**A:** Thank you. A good question. We apply masking to remove the semantic meaning of the corresponding token, thereby effectively blocking its influence on the LLM's output. \n\nSince not all LLMs provide a *[pad_token]*, we use the model's *[unk_token]* as a placeholder, which represents an out-of-vocabulary token. At present, different LLMs offer different *[unk_token]* without specific semantic meaning, so we adopt different placeholders for different models accordingly.\n\nIn recent years, some studies have proposed learning an optimal masking strategy for each model through training. However, such approaches are time-consuming, and thus we have not explored them in this paper. **Currently, there is no unified standard for masking, and we also highlight this point in Appendix A.**\n\n$\\color{blue}Q3:$ **\"The study focuses exclusively on \"simple arithmetic problems\", has not extended research to \"more complex math word problems\". This limits the generalizability of the findings.\"**\n\n**A:** Thank you. We **conducted new experiments** on a more complex math word problem: \"*Henry has 1 notebook, then Henry receives 8 notebooks. What's the total number of notebooks that Henry has?*\" This prompt is essentially an extension of a simple one-operator arithmetic problem \"*1+8*=,\" but contains richer contextual information. \n\nWe categorize input variables into 4 types: **numbers** (i,e., \"*8*\", \"*1*\"), **operators** (i.e., \"*receives*\", *\"the total number of\"*), entity words (including names and objects, i.e., \"*Henry*\", \"*notebook*\", \"*notebooks*\"), and background words (all words other than the above three categories).\n\nFollowing the same setting as in our paper, we also define four types of interactions:\n\n1. **Operand interactions**: contain only numbers (no operators)\n\n2. **Operator interactions**: contain only operators (no numbers)\n\n3. **Operand-operator interactions**: contain both numbers and operators\n\n4. **Background interactions**: contain neither numbers nor operators\n\nWe  conducted experiments on the Mathcoder-L-7B model. **The results confirm our main conclusion (Insight 1)**: The internal mechanism of LLMs for solving simple one-operator arithmetic problems is their capability to encode operand-operator interactions (Table 2) and high-order interactions (Table 3) during the forward propagation.\n\nTable 2: Focality across **different types of interactions** at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)\n\n|Interaction Type|Layer 5|Layer 9|Layer 18|Layer 20|Layer 30|\n|-|-|-|-|-|-|\n|Operand|0.97|0.98|0.86|0.88|1.00|\n|Operator|1.10|**1.11**|1.09|1.04|0.97|\n|Operand-Operator|0.96|0.95|**1.13**|**1.16**|**1.02**|\n|Background|**1.11**|1.10|0.15|0.14|0.14|\n\nTable 3: Focality  across **interactions of different orders**  at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)\n\n|Interaction Order|Layer 5|Layer 9|Layer 18|Layer 20|Layer 30|\n|-|-|-|-|-|-|\n|1|**2.02**|**2.05**|2.10|1.86|1.31|\n|2|1.26|1.22|0.98|0.93|0.83|\n|3|1.06|1.03|0.85|0.77|0.79|\n|4|0.96|0.95|0.90|0.84|0.88|\n|5|0.88|0.89|0.99|1.03|1.07|\n|6|0.79|0.85|1.08|1.29|1.36|\n|7|0.67|0.83|1.52|1.96|1.91|\n|8|0.59|0.83|**3.10**|**3.85**|**3.13**|\n\n$\\color{blue}Q4:$ **\"The \"focality\" metric (Eq 8, 9) sums absolute interaction values. This approach may disregards the direction (positive or negative contribution) of interactions, which is crucial for understanding the precise nature of the model's reasoning.\"**\n\n**A:** Thank you. A good question! We **conducted new experiments** by dividing interactions into **positive** and **negative** categories. \n\nFor different types of interactions:\n1. Positive value: $R_{+}^{(l)}(\\Omega^{\\text{type}}) = \\frac{\\mathbb{E}_{S \\in \\Omega^{\\text{type}},\\ I_S^{(l)} > 0} I_S^{(l)}}{Z^{(l)}}$ (Table 4)\n2. Negative value: $R_{-}^{(l)}(\\Omega^{\\text{type}}) = \\frac{\\mathbb{E}_{S \\in \\Omega^{\\text{type}},\\ I_S^{(l)} < 0} I_S^{(l)}}{Z^{(l)}}$ (Table 5)\n\nFor interactions of different orders:\n1. Positive value: $\\kappa_{m,+}^{(l)} = \\frac{\\mathbb{E}_{|S|=m,\\ I_S^{(l)} > 0} I_S^{(l)}}{Z^{(l)}}$ (Table 6)\n2. Negative value: $\\kappa_{m,-}^{(l)} = \\frac{\\mathbb{E}_{|S|=m,\\ I_S^{(l)} < 0} I_S^{(l)}}{Z^{(l)}}$ (Table 7)\n\n**Results show that, in the later stage of forward propagation, the LLM shows the highest focus on operand-operator interactions (Table 4 and Table 5)  and high-order interactions (8th-order and 9th-order in Table 6 and Table 7). The results confirm our main conclusion (Insight 1)**: The internal mechanism of LLMs for solving simple one-operator arithmetic problems is their capability to encode operand-operator interactions and high-order interactions during the forward propagation. \n\nIn summary, regardless of being positive or negative, interactions with large absolute values are considered to have a significant impact on the model’s output (i.e., whether pushing the prediction toward or away from the target answer, they still indicate strong influence).\n\nTable 4:   Focality across **different types of interactions** at Layer $l$ – **Positive values**.  (Mathcoder-CL-7b)\n\n|Interaction Type|Layer 1|Layer 8|Layer 16|Layer 23|Layer 31|\n|-|-|-|-|-|-|\n|Operand|1.00|1.03|1.01|0.88|0.84|\n|Operator|1.02|0.95|0.92|0.97|1.11|\n|Operand-Operator|1.00|0.96|**1.07**|**1.23**|**1.23**|\n|Background|**1.03**|**1.07**|0.94|0.77|0.87|\n\n\nTable 5: Focality across **different types of interactions** at Layer $l$ – **Negative values**.  (Mathcoder-CL-7b)\n\n|Interaction Type|Layer 1|Layer 8|Layer 16|Layer 23|Layer 31|\n|-|-|-|-|-|-|\n|Operand|-1.00|**-1.04**|-1.00|-0.88|-0.81|\n|Operator|**-1.02**|-1.00|-0.90|-0.96|-1.00|\n|Operand-Operator|-1.00|-0.98|**-1.06**|**-1.23**|**-1.23**|\n|Background|-0.99|-0.98|-0.88|-0.68|-0.74|\n\n\nTable 6: Focality across **interactions of different orders**  at Layer $l$ – **Positive values**.  (Mathcoder-CL-7b template 4).\n\n|Interaction Order|Layer 1|Layer 8|Layer 16|Layer 23|Layer 30|\n|-|-|-|-|-|-|\n|1|**1.56**|**2.27**|**1.43**|1.18|1.45|\n|2|0.00|0.46|0.00|0.64|0.30|\n|3|1.11|1.41|0.94|0.72|0.79|\n|4|0.00|0.61|0.00|0.87|0.38|\n|5|0.94|0.97|1.01|1.04|1.00|\n|6|0.00|0.70|0.24|1.07|0.65|\n|7|0.85|0.57|1.11|1.71|2.01|\n|8|0.00|0.74|0.33|1.39|1.97|\n|9|0.78|0.25|0.82|**2.40**|**2.34**|\n\nTable 7: Focality across **interactions of different orders**  at Layer $l$ – **Negative values**.  (Mathcoder-CL-7b template 4).\n\n|Interaction Order|Layer 1|Layer 8|Layer 16|Layer 23|Layer 30|\n|-|-|-|-|-|-|\n|1|0.00|0.00|0.00|0.00|0.00|\n|2|**-1.26**|**-1.66**|-1.00|-0.52|-0.79|\n|3|0.00|-0.54|0.00|-0.77|-0.27|\n|4|-1.01|-1.22|-0.95|-0.98|-0.83|\n|5|0.00|-0.70|-0.13|-1.00|-0.47|\n|6|-0.89|-0.77|**-1.07**|-1.21|-1.35|\n|7|0.00|-0.69|-0.17|-1.13|-0.88|\n|8|-0.81|-0.37|-1.03|**-2.37**|**-2.84**|\n|9|0.00|-0.81|-0.17|-1.89|-2.50|"}, {"review_id": "uag6r97RVL", "reviewer": "Reviewer_8cBX", "summary": "Using game-theoretic measures, the paper explains arithmetic reasoning capabilities for LLMs through strengths of operand-operator interactions.", "questions": "- What is the motivation behind the cosine similarity as the choice for $v^{(l)}$, as opposed to say the mean squared difference? \n\n- In Sec 4.1 the operand interactions do not seem to show  any consistent trends. What is the consequence of this behavior?\n\n- Were embeddings considered at all? They could serve as a basic sanity check, since the tokens are not supposed to interact at the level of embeddings.\n\n- It was not clear to me what the message of Appendix E was.", "limitations": "There were no particularly important limitations that stood out to me that was not addressed by the authors. Given that this is a preliminary investigation of game-theoretic measures on LLMs, there are lots of scope for future directions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper extends game-theoretic measures to LLMs, opening a different avenue for interpretability for such models.\n- The experiments performed were quite extensive, and various evidence is provided to validate each of the claims made in the paper. \n- The paper provides clear information regarding the template prompts, data sources, models chosen, etc.\n- The metric chosen is easily generalizable to other settings. \n\n\nWeaknesses:\n\n- The paper studies the effect of each layer in one go, in other words, effects of attention heads are not studied separately. Given that the attention is primarily responsible for the token-token interaction, studying the cosine similarity of the head outputs could be illuminating.\n\n- The score is averaged in a manner such that anti-parallel and parallel vectors are given equal importance. While I understand that without absolute values, the average would be \"lower\", it would be interesting to look at histogram or higher moments to understand if such weightage is appropriate.", "was_revised": true, "final_justification": "The paper answers my questions to adequate degree, and clarifies some subtle points not explicitly mentioned originally. However, it does not prompt me to increase my rating.", "rebuttal": "Thank you for your great efforts on the review of this paper. We will try our best to answer all your questions.\n\n$\\color{blue}Q1:$ **\"The paper did not study effects of attention heads separately. Given that the attention is primarily responsible for the token-token interaction, studying the cosine similarity of the head outputs could be illuminating.\"**\n\n**A:** Thank you. This is a valuable question. Following your advice, we **conducted new experiments to study effects of attention heads separately.** Specifically, we used **attention scores of the last token** to compute $v^{(l)}$ and then calculated the cosine similarity using this attention vector.\n\nWe conducted experiments on all **32 (0-31) attention heads** of the MathCoder-L-7B model and observed consistent results across all of them: the model gradually focus more on **operand-operator interactions** (Table 1) as well as **higher-order interactions** (7th-order and 8th-order in Table 2), which aligns with our **Insight 1**. An example result for **head 15** is shown as follows.\n\nTable 1: Focality across **different types of interactions** at Layer $l$ using **head 15** outputs. (Mathcoder-L-7B)\n\n|Interaction Type|Layer 2|Layer 9|Layer 12|Layer 23|Layer 31|\n|-|-|-|-|-|-|\n|Operand|1.08|1.08|1.00|0.98|0.97|\n|Operator|0.99|0.72|0.94|0.94|0.89|\n|Operand-Operator|0.89|**1.16**|**1.07**|**1.08**|**1.14**|\n|Background|**1.10**|0.56|0.85|0.89|0.78|\n\nTable 2:  Focality across **interactions of different orders** at Layer $l$ using **head 15** outputs. (Mathcoder-L-7B template 1)\n\n|Interaction Order|Layer 2|Layer 9|Layer 12|Layer 23|Layer 31|\n|-|-|-|-|-|-|\n|1|**1.80**|0.48|0.96|0.84|0.89|\n|2|1.35|0.60|0.97|0.71|0.72|\n|3|1.16|0.79|0.97|0.74|0.73|\n|4|1.00|0.97|0.98|0.86|0.85|\n|5|0.83|1.17|1.02|1.08|1.11|\n|6|0.63|1.44|1.08|1.60|1.59|\n|7|0.49|1.81|1.18|2.31|2.38|\n|8|0.47|**2.30**|**1.31**|**3.19**|**3.06**|\n\n$\\color{blue}Q2:$ **\"The score is averaged in a manner such that anti-parallel and parallel vectors are given equal importance. It would be interesting to look at histogram or higher moments to understand if such weightage is appropriate.\"**\n\n**A:** Thank you. A interesting question! **To further investigate whether using the absolute value is appropriate, we analyzed the distribution of interaction effects.** Specifically, we computed the skewness (with zero as the axis of symmetry) of interaction effects at each layer. The results (Table 3) show that the maximum skewness is 0.033 (< 0.05), indicating that the distribution does not exhibit a strong bias toward either the positive or negative side. Therefore, treating positive and negative values with equal importance (i.e., using the absolute values) is reasonable.\n\nTable 3: Skewness (centered at 0) of interaction effects (MathCoder-CL-7B template 3).\n\n|Statistic|Layer 1|Layer 8|Layer 16|Layer 23|Layer 31|\n|-|-|-|-|-|-|\n|Skewness|0.008|0.006|**0.033**|0.014|-0.003|\n\n$\\color{blue}Q3:$ **\"What is the motivation behind the cosine similarity as the choice for $ v^{(l)}$ , as opposed to say the mean squared difference?\"**\n\n**A:** Thank you for the thoughtful question. We chose **cosine similarity** for computing $v^{(l)}$ because it effectively captures the directional alignment between hidden representations. In Transformer-based architectures, semantic similarity is often encoded in the direction of vector representations rather than their magnitude. This makes cosine similarity **a natural and widely adopted choice**, not only in language modeling, but also in other domains such as recommender systems, where it has proven effective in measuring similarity between user and item embeddings.\n\nOn the other hand, **mean squared difference (MSE)** measures absolute deviation in Euclidean space. While useful in some contexts, it conflates directional changes with scale differences, which may not be meaningful in the high-dimensional embedding spaces of LLMs. \n\nNevertheless, we **conducted new experiments using the mean squared difference**. We replaced the cosine similarity with the mean squared difference, and normalized it to the range $[0,1]$: \n\n$v^{(l)}(x_T) = 1 - (MSE(f^{(l)}(x_T), f^{(l)}(x_N)) - d_{min}) / (d_{max} - d_{min} + ε)$\n\n**The experimental results confirm our main conclusion:** For simple one-operator arithmetic problems, LLMs that perform well (i.e., LLaMA-2-7B) tend to focus more on operand-operator interactions (Table 4) and high-order interactions (Table 5). While LLMs with weak performance (i.e., GPT-J-6B) tend to focus more on background interactions (Table 6) and extremely low-order interactions (Table 7).\n\n**While MSE and cosine similarity yield similar conclusions, we tend to adopt cosine similarity in this paper to avoid the impact of feature scale differences.**\n\nTable 4: Focality across **different types of interactions** at Layer $l$ using **MSE**. (LLaMA-2-7B)\n\n|Interaction Type|Layer 1|Layer 8|Layer 16|Layer 23|Layer 30|\n|-|-|-|-|-|-|\n|Operand|0.99|0.94|0.82|0.90|0.93|\n|Operator|0.94|0.98|0.90|0.71|0.67|\n|Operand-Operator|**1.10**|**1.10**|**1.31**|**1.34**|**1.33**|\n|Background|0.80|0.91|0.71|0.60|0.53|\n\nTable 5:  Focality  across **interactions of different orders** at Layer $l$ using **MSE**. (LLaMA-2-7B template 1)\n\n|Interaction Order|Layer 1|Layer 8|Layer 16|Layer 23|Layer 30|\n|-|-|-|-|-|-|\n|1|0.81|0.74|0.60|0.77|0.75|\n|2|0.49|0.38|0.26|0.34|0.39|\n|3|0.80|0.61|0.46|0.40|0.51|\n|4|1.02|0.93|0.79|0.60|0.72|\n|5|1.18|1.31|1.20|1.10|1.10|\n|6|1.38|1.70|2.01|2.25|2.02|\n|7|1.64|2.10|3.94|5.05|4.15|\n|8|**1.91**|**2.56**|**8.23**|**11.23**|**8.88**|\n\nTable 6: Focality across **different types of interactions** at Layer $l$ using **MSE**. (GPT-J-6B)\n\n|Interaction Type|Layer 1|Layer 6|Layer 11|Layer 17|Layer 23|\n|-|-|-|-|-|-|\n|Operand|0.79|0.71|0.71|0.74|0.77|\n|Operator|1.33|1.10|1.10|1.04|0.97|\n|Operand-Operator|0.81|0.69|0.72|0.77|0.82|\n|Background|**1.85**|**2.71**|**2.61**|**2.43**|**2.25**|\n\nTable 7:  Focality  across **interactions of different orders** at Layer $l$​ using **MSE**. (GPT-J-6B template 1)\n\n|Interaction Order|Layer 1|Layer 6|Layer 11|Layer 17|Layer 23|\n|-|-|-|-|-|-|\n|1|**4.94**|**2.71**|**2.27**|**2.00**|**1.95**|\n|2|1.18|0.82|0.81|0.82|0.81|\n|3|0.86|0.91|0.92|0.93|0.92|\n|4|0.77|0.96|0.98|0.99|0.99|\n|5|0.80|0.99|1.01|1.03|1.03|\n|6|0.93|1.02|1.04|1.04|1.06|\n|7|1.07|1.03|1.04|1.03|1.05|\n|8|1.92|0.97|0.98|0.92|0.95|\n\n$\\color{blue}Q4:$ **\"In Sec 4.1 the operand interactions do not seem to show any consistent trends. What is the consequence of this behavior?\"**\n\n**A:** Thank you. That's a great question. Among the five LLMs that perform well on one-operator arithmetic problems (as shown in Table 8), the LLaMA-2-7B, Llemma-7B, and MathCoder-L-7B models show increasing focus on **operand interactions** in deep layers, while the CodeLlama-13B and MathCoder-CL-7B models do not show this trend. On the other hand, the CodeLlama-13B and MathCoder-CL-7B models show a more significant encoding of **operand-operator interactions** compared to the other three models. Among five LLMs, the LLaMA-2-7B model shows the weakest encoding of operand-operator interactions.\n\nThe Llemma-7B and MathCoder-L-7B models achieve the highest accuracy on one-operator arithmetic tasks (Table 8), suggesting that their strong focus on operand interactions is beneficial for arithmetic reasoning. For the **LLaMA-2-7B** model, we believe that its limitation in encoding operand-operator interactions is mitigated by its stronger focus on operand interactions.\n\nAs for the **CodeLlama-13B** and **MathCoder-CL-7B** model, although they do not tend to focus more on operand interactions, their strong modeling of **operand-operator interactions** is sufficient to support accurate arithmetic reasoning.\n\nTable 8: Accuracy (%) of LLMs on one-operator arithmetic tasks.\n\n|Model|1-opr Accuracy(%)|\n|-|-|\n|Llemma-7B|75.1|\n|MathCoder-L-7B|74.0|\n|CodeLlama-13B|71.1|\n|Llama-2-7B|65.1|\n|MathCoder-CL-7B|62.6|\n\n$\\color{blue}Q5:$ **\"Were embeddings considered at all? They could serve as a basic sanity check, since the tokens are not supposed to interact at the level of embeddings.\"**\n\n**A:** Thank you. This is an excellent question. As you correctly pointed out, tokens are not expected to interact at the embedding layer. Indeed, we verify this by computing interaction effects at the embedding layer, and **all values turn out to be zero**. \n\n$\\color{blue}Q6:$ **\"It was not clear to me what the message of Appendix E was.\"**\n\n**A:**  Thank you for pointing this out. Appendix E aims to justify our choice of using the last token's embedding to compute $v^{(l)}$, that is, the cosine similarity between embeddings of the original input and that of the masked input. \n\nTo further illustrate this, we **conducted the same experiment** on the LLaMA-2-7B model with the prompt \"*How much is 5 plus 1? Answer is .*\" Results in Table 9 show that, except for the final token of the complete sentence, that is, token 10 ('*?*') and token 13 (the last token '*▁*'), the $v^{(l)}$ is nearly identical (close to 1). Only the embedding of the last token shows a significant change. Therefore, to simplify computation, we only focus on the change in the embedding of the last token, while ignoring the negligible changes in the embeddings of other tokens.\n\nMoreover, **several prior studies [Wendler et al. 2024] [Zhang et al. 2024] claim that in the Transformer architecture, the representation of last token includes information from all previous tokens.**\n\nTable 9: $v^{(l)}$ at different token positions across layers for the masked input \"*How much is 5 [mask] [mask]? Answer is .*\"\n\n|Token|Layer 5|Layer 20|Layer 30|\n|-|-|-|-|\n|token_1|0.00|0.00|0.00|\n|token_2|1.00|1.00|1.00|\n|token_3|1.00|1.00|0.99|\n|token_4|1.00|1.00|0.99|\n|token_5|1.00|1.00|0.99|\n|token_6|1.00|1.00|0.99|\n|token_7|1.00|1.00|1.00|\n|token_8|1.00|1.00|1.00|\n|token_9|1.00|1.00|1.00|\n|token_10|0.95|0.95|0.91|\n|token_11|1.00|1.00|0.99|\n|token_12|1.00|1.00|0.99|\n|token_13|**0.57**|**0.71**|**0.59**|"}]

Author rebuttal:

Meta-review: This paper presents a novel framework for interpreting arithmetic reasoning in LLMs through the lens of game-theoretic Harsanyi interactions. The authors provide strong theoretical grounding and extensive empirical analysis, including intervention studies, comparison with Shapley values, and experiments on both templated and free-form math problems. The work offers new insights into how LLMs encode operand-operator interactions and high-order interactions across layers, and explains task-specific behavior in LoRA fine-tuning. Therefore I recommend acceptance.
Decision: Accept (poster)