Activation Patching Analysis of Arithmetic In-Context Learning Prompts in Large Language Models
Authors: Christian Hepting, Carsten Eickhoff, Seyed Ali Bahrainian


Reviewer paper summaries: The paper focuses on studying the use of in-context demonstration for arithmetic tasks. Typically, it designs 7 sets of experiments to assess the roles of specific tokens with in-context demonstration for arithmetic task solving. With 100% accuracy as the baseline, it modifies the examples in the prompting by changing it into english language, changing its correctness, and rearranging the order of ICD structure. Based on such counterfactual manipulation of ICD's tokens, it observes significant drops on accuracy. It finds that early MLP layers interacting with attention modules and representation consistency have significant impact in the in-context learning process for arithmetic answer.

The paper investigates the capability of large language models (LLMs) to solve arithmetic tasks via in-context learning (ICL). Specifically, it examines the crucial model components for ICL of arithmetic tasks using causal mediation analysis. The findings reveal that the early multi-layer perceptron (MLP) layers are essential for ICL. Additionally, the paper explores how manipulating different components of in-context demonstration (ICD), such as the ICD result and ICD result position, affects these important model components. The study shows that the representation (number or text) of the ICD is more significant than the arithmetic correctness of the ICD.
Summary of strengths: (1) It provides sufficient experiments and tests performance change under every potential counterfactual settings.

(2) It provides detailed information about experimental settings and dataset usage.

(3) It provides comprehensive examples of the patching effects of the MLP and Attention modules.

The paper is well-motivated and addresses the important problem of interpreting arithmetic tasks in large language models (LLMs) through in-context learning. The experimental setup is well-designed and clearly presented.
Summary of weaknesses: (1) The discussion and experiments writing part is a little bit verbose, a large part of the discussion and conclusion is basically a recap of the previous content. The writing can be shortened a lot.

(2) The experiments are limited to single token arithmetic testing. Therefore, the conclusion that draws from the paper is highly restricted to single situations. More complicated situations with large numbers might lead different conclusions.

(3) In many works, it is commonly understood that the initial layers of neural networks are responsible for processing low-level information like syntax, while the higher layers handle high-level information such as semantics. The patching effect mentioned in the context does not seem very convincing to me. This is because altering the lower layers is generally expected to produce a more significant change in the final output compared to modifying the upper layers. I wonder if this is because the paper focuses on a simple task, causing the large language model to overfit and thus not require a deep understanding of semantic information.

(4) It lacks in-depth analysis, it points out the importance of representation consistancy, but there is no in-depth analysis on how heavy this situation is.

Although the paper investigates the well-motivated problem of interpreting LLMs' ability to perform arithmetic tasks via in-context learning (ICL), it lacks novel findings. 
1. The importance of the embedding layer for the task seems obvious. 
2. The significance of the representation of the operator and operand for better ICL performance has been previously studied. 
3. The ability of LLMs to perform even with incorrect demonstrations of ICL has also been identified before. 
4. Finally, the paper only investigates ICL with a single demonstration, which may not generalize to scenarios with a higher number of demonstrations.
Comments, suggestions and typos: More case study can be included in the main section to help compare what types of error each experimental setting has.

**Missing related work on MI for ICL**:  
1. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., ... & Olah, C. (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1), 12. 
2. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., ... & Olah, C. (2022). In-context learning and induction heads. Transformer Circuits Thread, 2022. 
3. Bansal, H., Gopalakrishnan, K., Dingliwal, S., Bodapati, S., Kirchhoff, K., & Roth, D. (2022). Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale. arXiv preprint arXiv:2212.09095. 
4. Ren, J., Guo, Q., Yan, H., Liu, D., Qiu, X., & Lin, D. (2024). Identifying semantic induction heads to understand in-context learning. arXiv preprint arXiv:2402.13055.
Overall assessment scores: 3

2.5