On Relation-Specific Neurons in Large Language Models
Authors: name: Yihong Liu, name: Runsheng Chen, name: Lea Hirlimann, name: Ahmad Dawar Hakimi, name: Mingyang Wang, name: Amir Hossein Kargaran, name: Sascha Rothe, name: François Yvon, name: Hinrich Schütze
Organization: CIS, LMU Munich

In large language models (LLMs), certain neurons can store distinct pieces of
knowledge learned during pretraining. While knowledge typically appears as a
combination of relations and entities, it remains unclear whether some neurons
focus on a relation itself -- independent of any entity. We hypothesize such
neurons detect a relation in the input text and guide generation involving such
a relation. To investigate this, we study the Llama-2 family on a chosen set of
relations with a statistics-based method. Our experiments demonstrate the
existence of relation-specific neurons. We measure the effect of selectively
deactivating candidate neurons specific to relation r on the LLM's ability to
handle (1) facts whose relation is r and (2) facts whose relation is a
different relation r' neq r. With respect to their capacity for encoding
relation information, we give evidence for the following three properties of
relation-specific neurons. (i) Neuron cumulativity. The neurons for
r present a cumulative effect so that deactivating a larger portion of them
results in the degradation of more facts in r. (ii) Neuron
versatility. Neurons can be shared across multiple closely related as well as
less related relations. Some relation neurons transfer across languages.
(iii) Neuron interference. Deactivating neurons specific to one
relation can improve LLM generation performance for facts of other relations.
We will make our code publicly available at
https://github.com/cisnlp/relation-specific-neurons.