Comparative Neuron-Level Analysis of Related Language Models
Authors: Jia Peng Lim, Zhiqiang Hu, Hady W. Lauw


Reviewer paper summaries: The paper proposes a method to extract token distributions from individual neurons. The token distributions are then used to compare the same language model fine-tune on different domains.
Summary of strengths: Interesting method to estimate token distribution from neurons in the residual stream. It is potentially useful to determine whether a language model has been exposed to a certain domain during pre-training.
Summary of weaknesses: Despite being an interesting approach to model analysis, the exposition is very difficult to follow, and the math notation is not introduced consistently to the reader (suggestions are in the next text box). Overall, the paper would need a major rewrite to be at the bar of an EMNLP publication.
Comments, suggestions and typos: Suggestions to address weaknesses:
1. Make the goal of the paper clear and consistent: 
    - “This work proposes to investigate related language model pairs […] by examining the mechanistic difference between weights […]” (001-007)
    - “[…] we propose a comparative analysis, showing a causal link between logit distributions […] and token frequency […] (484-488)
2. There are many grammatically incorrect sentences in the paper (e.g., 046-049) or confusing (e.g., “As such, the interpretability of such approaches (which one?) […]” in 055-058; This low […] in 012-014; “It is also paramount to determine if these (which one?) changes are intuitive […] in 088-091)
3. Define concepts first and then use them. For example, “macro-perspective” is used in the abstract, but it is only (kind of) defined in 067-070. Most importantly, the “projected logit distribution” is not clear what it is. You have used it since the beginning of the paper but never defined
4. Eq 1 seems incorrect: shouldn’t you remove the second term (the attention)
5. The math notation is confusing and not introduced to the reader consistently. For example, in 151 you introduce `B_{M, d}` but do not define M; in 171 you introduce `M_i(x)` but do not explain what it is (is it a list of logits per token?). 
6. You use the term “data points” (which, from the context, I think should be “tokens”)
7. “Foundation” models not “Foundational”
8. Keep the ACL template consistent: do not indent `\paragraphs{}` (e.g., 067, 216, etc)
Overall assessment scores: 2