Title: Hierarchical Frequency Tagging Probe (HFTP): A Unified Approach to Investigate Syntactic Structure Representations in Large Language Models and the Human Brain
Venue: NeurIPS 2025
Accepted: yes
Keywords: Syntactic structure processing

LLM

Syntactic alignment
Abstract: Large Language Models (LLMs) demonstrate human-level or even superior language abilities, effectively modeling syntactic structures, yet the specific computational units responsible remain unclear. A key question is whether LLM behavioral capabilities stem from mechanisms akin to those in the human brain. To address these questions, we introduce the Hierarchical Frequency Tagging Probe (HFTP), a tool that utilizes frequency-domain analysis to identify neuron-wise components of LLMs (e.g., individual Multilayer Perceptron (MLP) neurons) and cortical regions (via intracranial recordings) encoding syntactic structures. Our results show that models such as GPT-2, Gemma, Gemma 2, Llama 2, Llama 3.1, and GLM-4 process syntax in analogous layers, while the human brain relies on distinct cortical regions for different syntactic levels. Representational similarity analysis reveals a stronger alignment between LLM representations and the left hemisphere of the brain (dominant in language processing). Notably, upgraded models exhibit divergent trends: Gemma 2 shows greater brain similarity than Gemma, while Llama 3.1 shows less alignment with the brain compared to Llama 2. These findings offer new insights into the interpretability of LLM behavioral improvements, raising questions about whether these advancements are driven by human-like or non-human-like mechanisms, and establish HFTP as a valuable tool bridging computational linguistics and cognitive neuroscience. This project is available at https://github.com/LilTiger/HFTP.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an extension of the experimental method of Ding et al. (2016) to both a  dataset of human subjects with intracranial sEEG recording and a fleet of text-based Transformer language models. This method revealed, in both Transformer models and brains, individual units (or electrodes) which exhibited structure tracking (=frequency tracking) at the phrase- and sentence-level. They found that different language models exhibited vastly different distributions of phrase- and sentence-selectivity, but that within any model, the distribution of phrase- and sentence-level neurons is strongly correlated across layers. This latter fact does not hold in a similar comparison applied across brain regions, suggesting a distinct gross architecture for syntactic processing between models and the human brain.\n\nSTRENGTHS AND WEAKNESSES:\nThis is an ambitious paper which transfers a popular method from cognitive neuroscience to AI, and draws broad conclusions about an architectural dissociation between models and humans as a result. The experiments are extensive on the modeling side, covering a large class of LLMs.\n\nHowever, there are some major analytic and experimental issues that make this paper difficult to accept at this point. These are given below with asterisks. Other conceptual / presentational issues are also highlighted.\n\n- 4.1 MLP neurons represent sentences and phrases\n\t- You find a strong correlation between the number of sentence and phrase neurons within-model across layers (L254–257). This is interesting. How do you reconcile this with existing claims that LLMs distribute different syntactic functions across distinct layers (e.g. Tenney et al. (2019))?\n\t- \\* L237–246: The interpretation of differences between phrase- and sentence-selectivity layer distributions across models is pretty unsatisfying (\"reflecting a more integrated approach ... suggesting a balanced intermediate strategy ... indicating an emphasis on initial-stage syntactic processing\"). These labels aren't very meaningful without further analysis. Given that this is a core finding of your paper (and part of the critical mismatch with the brain findings) I think you need to do more work to detail what kind of process it is you are locating which is in wildly distant layers across different LLMs. You could use probing methods, for example, to ask whether the populations you're picking out correspond to theoretical syntactic concepts.\n- 4.3 SRDM analysis\n\t- Your SRDM method evaluates the cosine distance between two frequency spectra, and is thus sensitive to differences across the entire spectrum. But we know by the controlled design of your setup that there are only a few bands that actually matter. For simplicity and interpretability, why not just take the amplitude of the two relevant peaks at 1 and 2 Hz? Is there something we gain with the cosine distance metric here?\n\t- How exactly are SRDMs computed at each model layer? Are you averaging SRDMs across all neurons? Or just the ones found to be sentence/phrase-selective?\n\t- I am having a lot of trouble interpreting Table 1.\n\t\t- These are text-based models and yet, for each model, the best- or near-best-predicted region is A1. How can this be? This does not match, for example, the performance distribution of the ECoG study of Goldstein et al. (2022), who seem to find highest predictability in frontal regions.\n\t\t- \\* We need error bars on these values. Consider, for example, bootstrapping the set of stimuli that enter into each SRDM in order to estimate nonparametric confidence intervals, and/or bootstrap-sampling the subjects that enter into the SRDM evaluation.\n\t- \\* Comparisons between ROIs and hemispheres here are likely confounded with major between-subject differences in signal quality and SRDM performance, since the placement of electrodes is highly variable across subjects. As far as I can tell this was not taken into account in any evaluations. I would suggest running per-ROI model–brain alignment within-subject, and then using a mixed-effects regression model to explain per-ROI and per-hemisphere performance controlling for random subject-level intercepts.\n- General questions\n\t- Why do you not analyze at the syllable level (4 Hz) as done in Ding et al. (2016)?\n- Minor questions/comments\n\t- Table 1: What do bold numbers indicate? This is not labeled anywhere as far as I can see.\n\t- Why is there no second constraint (as in Definition 2) on electrode responses as well? Without this, the comparison is not exactly matched: you're finding MLP neurons that are *exclusively* sentence- and/or phrase-tuned, but electrodes which are tuned (not necessarily exclusively). Do you not have the relevant control materials for the sEEG data?\n\nDing, N., Melloni, L., Zhang, H. _et al._ Cortical tracking of hierarchical linguistic structures in connected speech. _Nat Neurosci_ **19**, 158–164 (2016). https://doi.org/10.1038/nn.4186\n\nGoldstein, A., Zada, Z., Buchnik, E. _et al._ Shared computational principles for language processing in humans and deep language models. _Nat Neurosci_ **25**, 369–380 (2022). https://doi.org/10.1038/s41593-022-01026-4\n\nIan Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [BERT Rediscovers the Classical NLP Pipeline](https://aclanthology.org/P19-1452/). In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4593–4601, Florence, Italy. Association for Computational Linguistics.\n\nQUESTIONS:\nPlease minimally address the points given with asterisks above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Hierarchical Frequency Tagging Probe (HFTP), a unified framework for investigating syntactic structure representations in both LLMs and the human brain using frequency-domain analysis. The authors apply FFT to LLM activations and human sEEG recordings to identify neurons/channels that respond to sentence (1 Hz) and phrase (2 Hz) frequencies. They test six LLMs (GPT-2, Gemma, Gemma 2, Llama 2, Llama 3.1, GLM-4) and find that while LLMs process syntax in analogous layers, the human brain uses distinct cortical regions. Representational similarity analysis reveals stronger alignment between LLMs and the left hemisphere, with divergent trends in model upgrades.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe unified framework for comparing LLMs and brain representations is novel and addresses an important gap in understanding computational vs. biological language processing. The frequency-domain approach is creative and well-motivated, building appropriately on established HFT paradigms. The comprehensive evaluation across multiple LLMs with proper controls (randomized versions) is commendable.\n\nWeaknesses:\n1. The artificial temporal constraints imposed on LLMs are problematic. Constraining activations to 4 Hz and artificially defining time scales seems unmotivated and creates an artificial mapping that may not reflect genuine syntactic processing mechanisms in these models.\n2. The high correlations between sentence and phrase neurons across all LLMs raise questions about whether the method is actually detecting distinct syntactic processes or simply identifying general linguistic sensitivity with slight frequency variations.\n3. Generalizability is limited by restriction to Chinese sEEG data and Chinese/English text corpora. The authors acknowledge this but it significantly constrains the broader applicability of findings. The relatively simple syntactic structures tested may not capture the full complexity of syntactic processing.\n4. The theoretical justification for why frequency-domain analysis should reveal syntactic representations in LLMs lacks depth. While the biological motivation is clear, the computational rationale is underdeveloped.\n\nQUESTIONS:\n1. How do you justify the 4 Hz sampling constraint for LLMs? This seems to force a temporal structure that doesn't naturally exist in these models. Have you tested other frequencies or more naturalistic approaches?\n2. The correlations between sentence and phrase neurons are extremely high (0.754-0.994). Doesn't this suggest you're detecting general linguistic processing rather than distinct syntactic levels? How do you address this concern?\n3. Given the restriction to Chinese brain data, how confident are you that these alignment patterns would hold for English speakers or other languages? Can you provide any cross-linguistic validation?\n4. You tested relatively simple syntactic structures. How would HFTP perform on more complex syntactic phenomena like long-distance dependencies, garden-path sentences, or recursive structures?\n5. Have you tested whether similar frequency patterns emerge for non-syntactic linguistic features (e.g., semantic or phonological patterns) to ensure syntactic specificity?\n\nLIMITATIONS:\nThe authors adequately acknowledge the language-specific limitations and restricted model coverage. However, they could better address the fundamental methodological concern about artificial temporal constraints and the implications of extremely high sentence-phrase correlations. The computational overhead and scalability of the approach across different model architectures also deserves discussion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work uses a method first introduced by Ding et al. (2016) to analyze the syntactic structure of very simple sentences. The authors adapt this method to the study of large language models, by looking at the frequency components of the response of a large language model to carefully designed stimuli (here consisting in sentences of four words, carefully put together into groups that reflect word, phrase or sentence level). This makes it possible to have a common method that allows for a direct comparison between the language processing of the language models and the human brain. Such a comparison is provided, making use of a corpus of sEEG recordings.\n\nSTRENGTHS AND WEAKNESSES:\nHaving the same method to analyze both large language models and the brain activity is certainly valuable, and as far as I know, the method of Ding et al. (2016) has never been used to analyze the linguistic processing of large language models.\n\nOne important issue at first sight is the fact that the authors do not specify that the model that they call GPT-2 is actually not the original GPT-2. If this was the case, all the results would be hard to believe, given that the original GPT-2 model was primarily trained on English data, while here it is the model that achieves the best performance at their brain/models comparison in processing Chinese data. Buried in Appendix C, and not announced in the main paper (even the cite to Radford et al., 2019, is misleading, as found in Table 3), there is actually the information that the GPT-2 model that is used is not the original version but another version pretrained on a Chinese corpus. This is far from being a detail and should be stated upfront, right from the presentation of the models, as otherwise the results seem very puzzling.\n\nAn important limitation of the study concerns the generalization to naturalistic text. It is said that the method generalizes \"seamlessly to naturalistic text\" (see Conclusion), but the evidence for such a statement is rather weak. There is only a small appendix (appendix E) addressing this point, and it is actually based on Chinese only, and, as discussed by the authors, due to properties specific to the Chinese language. How all this would work for languages like English is not discussed and is far from obvious -- this goes counter to the statement that the method generalizes, which is misleading. This should be discussed and acknowledged in a more transparent way.\n\nThe origins of the differences between models are not much investigated, notably with respect to their training data. Also, more work on understanding the role of the syntactic neurons would be appreciated, possibly by ablating these syntactic neurons vs. other neurons.\n\nThe Related Work section cites \"methodological inconsistencies\" across previous studies, without providing more details -- it would be nice to state what these inconsistencies are.\n\nMinor points.\n\nThe text contains some typos (eg a few places with no space after a full stop, \"fruquency\").\n\nTable 3, the number of MLP neurons for Gemma 2 seems wrong, as the Gemma 2 paper reports that the feedforward dim for Gemma 2 9B is 28672. Please verify these numbers.\n\nQUESTIONS:\nWhy focus only on the hidden layer of the MLPs? The neurons at the output of a transformer layer (output of the MLP, of size d_model) might be interesting to look at. One could imagine that certain syntactic neurons might be visible at the finer resolution of the hidden layer of the MLP but not when putting them all together down to the dimension of the embedding, due to some superposition; but conversely, having a finer resolution might yield responses that are not visible under the criteria that are chosen here but might be visible when combined together at the output of each layer. \n\nConcerning the methods, what do you do when a word is split into several tokens?\n\nWill you make the sEEG corpus available?\n\nWhat is the rationale behind the use of bold font in Table 1?\n\nLIMITATIONS:\nAn important limitation is the generalization to naturalistic text, which is not acknowledged by the authors (see Weaknesses section above).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Hierarchical Frequency Tagging Probe (HFTP), a novel method designed to investigate how LLMs and the human brain represent syntactic structures. HFTP utilizes frequency-domain analysis to identify specific computational units (MLP neurons in LLMs and cortical regions in the brain) that encode hierarchical linguistic elements like phrases and sentences.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe introduction of the Hierarchical Frequency Tagging Probe (HFTP) is a useful contribution as it extends frequency-domain analysis to LLM internals and brain activity in a unified framework, enabling direct comparison of syntactic processing across artificial and biological systems.\nThe authors do a systematic probing of six diverse LLMs (GPT-2, Gemma, Gemma 2, Llama 2, Llama 3.1, GLM-4) revealing patterns of syntactic specialization and how these evolve across model versions. They report results showing that models encode sentence and phrase structures at different layers, with differences across architectures (e.g., Gemma vs. Llama families).\nHFTP could be a valuable tool that bridges computational linguistics and cognitive neuroscience, offering new insights into the interpretability of LLMs and their improvements\n\n\n\nWeaknesses\nThe approach of \"artificially defining a time scale\" with a 4 Hz sampling rate for LLM activations is a design decision to enable frequency-domain analyses when drawing a comparison with human brain data (as LLMs do not inherently have a time dimension). However, it means that the \"time-course information\" is imposed on LLMs, as it's not inherent to LLM input structures like it is for the brain. While this is a well-explained design choice, one could argue that it doesn't truly show how LLMs “process time” on their own, but rather how they can adapt to an externally imposed rhythm. Additionally, this might prevent applicability to more realistic language corpora.\nThe paper reveals a divergence: a model (Gemma 2) shows increased brain similarity compared to Gemma, while Llama 3.1 exhibits reduced alignment with the human brain compared to Llama 2, despite both being newer models with enhanced task performance. This is contrast with the authors’ explanation on brain alignment (around line 288) where they report that GPT-2, a model from 2019, has the highest correlation with human activity, and Gemma 2 outperforms Gemma due to modelling and training innovations. Overall the results seem mixed, making it impossible to draw any conclusion of the impact of LLM advances on brain alignment or going beyond empirical evaluation of models.\n\nMinor\n“To enhance performance on complex tasks—such as reasoning and higher cognitive functions—Llama 3.1 and Gemma 2 may reduce their specialized processing of syntactic structures (sentences and phrases), reallocating neurons to support these advanced capabilities.”. LLMs do not have any cognitive functions, I suggest the authors tone down such claims of human-like behavior of LLMs.\n\nProposing a unified framework for comparing syntactic structure representations in LLMs and the human brain is a valuable and timely contribution. The Hierarchical Frequency Tagging Probe (HFTP) offers a novel and methodologically consistent way to bridge AI and neuroscience. However, the core design choice of imposing a 4 Hz temporal structure on LLM activations—while necessary for frequency-domain comparison—raises interpretability concerns. Without ablations or alternative temporal mappings, it remains unclear whether the observed frequency patterns reflect intrinsic model behavior or artifacts of imposed structure. Moreover, the mixed trends in model–brain alignment across newer LLM versions highlight the difficulty of drawing principled conclusions about architectural progress. As it stands, HFTP enables compelling empirical evaluations, but additional controls are needed before it can support stronger theoretical claims.\n\nQUESTIONS:\nDid the authors account for tokenization across LLM when processing English data? How do they think their artificial time-resolution method can scale to more natural data?\nHow do the authors envision the generalizability of their brain-LLM alignment findings to English-speaking brains, especially considering potential cross-linguistic differences in syntactic processing?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work. Just a small note: authors state \"NA\" for the question about discussing potential positive and negative societal impacts, justifying this by saying their work is \"foundational and methodological\". While it is true that this research is foundational, authors should reflect on potential indirect or long-term negative applications, even for foundational work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}