How Programming Concepts and Neurons Are Shared in Code Language Models
Authors: name: Amir Hossein Kargaran; name: Yihong Liu; name: François Yvon; name: Hinrich Schütze
Published: 2025-06-01 16:24:13

Abstract: Several studies have explored the mechanisms of large language models (LLMs)
in coding tasks, but most have focused on programming languages (PLs) in a
monolingual setting. In this paper, we investigate the relationship between
multiple PLs and English in the concept space of LLMs. We perform a few-shot
translation task on 21 PL pairs using two Llama-based models. By decoding the
embeddings of intermediate layers during this task, we observe that the concept
space is closer to English (including PL keywords) and assigns high
probabilities to English tokens in the second half of the intermediate layers.
We analyze neuron activations for 11 PLs and English, finding that while
language-specific neurons are primarily concentrated in the bottom layers,
those exclusive to each PL tend to appear in the top layers. For PLs that are
highly aligned with multiple other PLs, identifying language-specific neurons
is not feasible. These PLs also tend to have a larger keyword set than other
PLs and are closer to the model's concept space regardless of the input/output
PL in the translation task. Our findings provide insights into how LLMs
internally represent PLs, revealing structural patterns in the model's concept
space. Code is available at https://github.com/cisnlp/code-specific-neurons.