Attention Mechanisms Perspective: Exploring LLM Processing of
  Graph-Structured Data
Authors: name: Zhong Guan, name: Likang Wu, name: Hongke Zhao, name: Ming He, name: Jianpin Fan

Attention mechanisms are critical to the success of large language models
(LLMs), driving significant advancements in multiple fields. However, for
graph-structured data, which requires emphasis on topological connections, they
fall short compared to message-passing mechanisms on fixed links, such as those
employed by Graph Neural Networks (GNNs). This raises a question: ``Does
attention fail for graphs in natural language settings?'' Motivated by these
observations, we embarked on an empirical study from the perspective of
attention mechanisms to explore how LLMs process graph-structured data. The
goal is to gain deeper insights into the attention behavior of LLMs over graph
structures. We uncovered unique phenomena regarding how LLMs apply attention to
graph-structured data and analyzed these findings to improve the modeling of
such data by LLMs. The primary findings of our research are: 1) While LLMs can
recognize graph data and capture text-node interactions, they struggle to model
inter-node relationships within graph structures due to inherent architectural
constraints. 2) The attention distribution of LLMs across graph nodes does not
align with ideal structural patterns, indicating a failure to adapt to graph
topology nuances. 3) Neither fully connected attention nor fixed connectivity
is optimal; each has specific limitations in its application scenarios.
Instead, intermediate-state attention windows improve LLM training performance
and seamlessly transition to fully connected windows during inference. Source
code: https://github.com/millioniron/LLM_exploration{LLM4Exploration}