Self-supervised Quantized Representation for Seamlessly Integrating
  Knowledge Graphs with Large Language Models
Authors: name: Qika Lin; name: Tianzhe Zhao; name: Kai He; name: Zhen Peng; name: Fangzhi Xu; name: Ling Huang; name: Jingying Ma; name: Mengling Feng
Published: 2025-01-30 03:40:20

Abstract: Due to the presence of the natural gap between Knowledge Graph (KG)
structures and the natural language, the effective integration of holistic
structural information of KGs with Large Language Models (LLMs) has emerged as
a significant question. To this end, we propose a two-stage framework to learn
and apply quantized codes for each entity, aiming for the seamless integration
of KGs with LLMs. Firstly, a self-supervised quantized representation (SSQR)
method is proposed to compress both KG structural and semantic knowledge into
discrete codes (\ie, tokens) that align the format of language sentences. We
further design KG instruction-following data by viewing these learned codes as
features to directly input to LLMs, thereby achieving seamless integration. The
experiment results demonstrate that SSQR outperforms existing unsupervised
quantized methods, producing more distinguishable codes. Further, the
fine-tuned LLaMA2 and LLaMA3.1 also have superior performance on KG link
prediction and triple classification tasks, utilizing only 16 tokens per entity
instead of thousands in conventional prompting methods.