How does Alignment Enhance LLMs' Multilingual Capabilities? A Language
  Neurons Perspective
Authors: name: Shimao Zhang, name: Zhejian Lai, name: Xiang Liu, name: Shuaijie She, name: Xiao Liu, name: Yeyun Gong, name: Shujian Huang, name: Jiajun Chen

Multilingual Alignment is an effective and representative paradigm to enhance
LLMs' multilingual capabilities, which transfers the capabilities from the
high-resource languages to the low-resource languages. Meanwhile, some
researches on language-specific neurons reveal that there are language-specific
neurons that are selectively activated in LLMs when processing different
languages. This provides a new perspective to analyze and understand LLMs'
mechanisms more specifically in multilingual scenarios. In this work, we
propose a new finer-grained neuron identification algorithm, which detects
language neurons~(including language-specific neurons and language-related
neurons) and language-agnostic neurons. Furthermore, based on the
distributional characteristics of different types of neurons, we divide the
LLMs' internal process for multilingual inference into four parts: (1)
multilingual understanding, (2) shared semantic space reasoning, (3)
multilingual output space transformation, and (4) vocabulary space outputting.
Additionally, we systematically analyze the models before and after alignment
with a focus on different types of neurons. We also analyze the phenomenon of
''Spontaneous Multilingual Alignment''. Overall, our work conducts a
comprehensive investigation based on different types of neurons, providing
empirical results and valuable insights for better understanding multilingual
alignment and multilingual capabilities of LLMs.