Title: Interpreting Arithmetic Reasoning in Large Language Models using Game-Theoretic Interactions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Explainable Artificial Intelligence

Large Language Models

Arithmetic Reasoning

Interactions

Interpretability

Deep Learning
Abstract: In recent years, large language models (LLMs) have made significant advancements in arithmetic reasoning. 
However, the internal mechanism of how LLMs solve arithmetic problems remains unclear.
In this paper, we propose explaining arithmetic reasoning in LLMs using game-theoretic interactions.
Specifically, we disentangle the output score of the LLM into numerous interactions between the input words.
We quantify different types of interactions encoded by LLMs during forward propagation to explore the internal mechanism of LLMs for solving arithmetic problems.
We find that (1) the internal mechanism of LLMs for solving simple one-operator arithmetic problems is their capability to encode operand-operator interactions and high-order interactions from input samples.
Additionally, we find that LLMs with weak one-operator arithmetic capabilities focus more on background interactions.
(2) The internal mechanism of LLMs for solving relatively complex two-operator arithmetic problems is their capability to encode operator interactions and operand interactions from input samples.
(3) We explain the task-specific nature of the LoRA method from the perspective of interactions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUsing game-theoretic measures, the paper explains arithmetic reasoning capabilities for LLMs through strengths of operand-operator interactions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper extends game-theoretic measures to LLMs, opening a different avenue for interpretability for such models.\n- The experiments performed were quite extensive, and various evidence is provided to validate each of the claims made in the paper. \n- The paper provides clear information regarding the template prompts, data sources, models chosen, etc.\n- The metric chosen is easily generalizable to other settings. \n\n\nWeaknesses:\n\n- The paper studies the effect of each layer in one go, in other words, effects of attention heads are not studied separately. Given that the attention is primarily responsible for the token-token interaction, studying the cosine similarity of the head outputs could be illuminating.\n\n- The score is averaged in a manner such that anti-parallel and parallel vectors are given equal importance. While I understand that without absolute values, the average would be \"lower\", it would be interesting to look at histogram or higher moments to understand if such weightage is appropriate.\n\nQUESTIONS:\n- What is the motivation behind the cosine similarity as the choice for $v^{(l)}$, as opposed to say the mean squared difference? \n\n- In Sec 4.1 the operand interactions do not seem to show  any consistent trends. What is the consequence of this behavior?\n\n- Were embeddings considered at all? They could serve as a basic sanity check, since the tokens are not supposed to interact at the level of embeddings.\n\n- It was not clear to me what the message of Appendix E was.\n\nLIMITATIONS:\nThere were no particularly important limitations that stood out to me that was not addressed by the authors. Given that this is a preliminary investigation of game-theoretic measures on LLMs, there are lots of scope for future directions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a game-theoretic framework to interpret the arithmetic reasoning capabilities of Large Language Models (LLMs). By disentangling the LLM's output score into contributions from various interactions between input words, the authors categorize these interactions into operand, operator, operand-operator, and background types, and also analyze their order (complexity). Through extensive experiments on one- and two-operator arithmetic problems across a range of LLMs, the study reveals that strong LLMs for simple problems primarily rely on operand-operator and high-order interactions, while for complex problems, they focus on operator and operand interactions. The paper also provides an interaction-based explanation for the task-specific nature of LoRA fine-tuning, demonstrating how it can lead to a decline in performance on simpler tasks by shifting the model's focus away from relevant interaction types.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\nThe paper applies game-theoretic interactions (specifically Harsanyi interactions and derived focality metrics) to explain the internal mechanisms of LLMs for arithmetic reasoning. This provides an approach to interpret complex model behavior, moving beyond less rigorous neuron-level analysis or perturbation studies.\nThe classification of interactions into operand, operator, operand-operator, and background types, along with the analysis of interaction orders, offers a framework for understanding how LLMs process arithmetic queries. \nThe study visualizes how interaction patterns change during the fine-tuning process. This dynamic analysis reveals the learning trajectory of LLMs, showing how they adapt their focus on different interaction types and orders as they acquire new arithmetic skills, and also explains phenomena like LoRA's task-specific nature.\n\n[Weaknesses]\nThe analysis of interactions in intermediate layers relies solely on the embedding of the last input token and its cosine similarity to the unmasked input. This is a simplification for complex transformer architectures, which distribute information across all tokens and layers via attention mechanisms. Might this choice lead to an incomplete or potentially biased understanding of how interactions are truly encoded and evolve within the model's internal states?\n\nThe paper acknowledges the absence of unified masking strategies and employs specific tokens (
, 
, <unk>) for masking. The choice of masking token may influence the model's internal representations and the resulting interaction values. \n\nThe study focuses exclusively on \"simple arithmetic problems\" (one- and two-operator queries) and explicitly states that it has not extended research to \"more complex math word problems\". This limits the generalizability of the findings.\n\nThe \"focality\" metric (Eq 8, 9) quantifies the strength of interaction types or orders by summing absolute interaction values. While useful for measuring overall \"focus,\" this approach may disregards the direction (positive or negative contribution) of interactions, which is crucial for understanding the precise nature of the model's reasoning.\n\nQUESTIONS:\nPlease refer to the weakenesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a game-theoretic method for studying inside transformer models when they do arithmetic. It shows that any LLM’s numeric output can be exactly written as the sum of Harsanyi interaction terms over all subsets of input tokens. By grouping them into “operand-only”, “operator-only”, “operand-operator”, and “background” categories, they find that strong models gradually amplify high-order, “operand-operator” interactions in late layers, thereby binding numbers to their operators. Using LoRA for fine-tuning, model gain on complex two-operator tasks at the cost of forgetting simpler one-operator reasoning, illustrating how tuning can reshape the collaboration patterns.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper is well-structured, with a solid theoretical foundation backed by rigorous game-theoretic proofs and clear mathematical derivations.\n2. This paper provides layer-wise focality scores that clearly show how and when a model learns to bind numbers with operators. Besides, the methodology is model-agnostic and can be used on any transformer without changing its code.\n\nWeaknesses:\n1. The paper's empirical analysis is based on several assumptions; in my opinion, they are too strong to be fully convincing. The authors use cosine similarity (Eq.3) to quantify hidden-state changes across layers, which missed large magnitude shifts in a single dimension. Without metrics, like Euclidean distance, the analysis risks overlooking substantial activation shifts.\n2. Harsanyi interactions require summing over 2^n subsets, which is infeasible when input length grows beyond a handful of tokens.\n3. This paper does not compare its interaction decomposition with other interpretability or attribution techniques.\n\nQUESTIONS:\n1. Can you use other distance metrics (e.g. Euclidean) alongside cosine similarity to capture both angular and magnitude changes?\n2. Can you provide a simple intervention study to produce the predicted jumps in interaction scores and output, thereby demonstrating causality? For example, flipping the operator token from ‘×’ to ‘+’.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes explaining arithmetic reasoning in LLMs through game-theoretic interactions, quantifying operand-operator, operator-operand, and background interactions across layers. It identifies distinct interaction encoding patterns in LLMs for single-operator vs. multi-operator problems and links LoRA’s task-specificity to interaction shifts. This work provides a novel perspective for explaining arithmetic reasoning in LLMs, which may help users and developers understand, use, and improve LLMs.

STRENGTHS AND WEAKNESSES:
Strengths:
- Faithfulness guarantees via Harsanyi interactions provide a mathematically sound foundation, distinguishing it from heuristic attribution methods.
- The layer-wise analysis of interaction types effectively reveals distinct reasoning mechanisms for different problem types.
- Comprehensive experiments have been conducted to demonstrate the different types of interactions and validate the proposed insights.
Weaknesses
- Source code is not opened.
- The interaction effect can fit the output (Theorem 1), but since this is an interpretability study, whether the causal necessity can be verified?
- Experiments use templated arithmetic questions. It’s unclear how it performs in free‐form math problems.

QUESTIONS:
See the weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01