Executable Knowledge Graphs for Replicating AI Research
Authors: name: Yujie Luo; name: Zhuoyun Yu; name: Xuehai Wang; name: Yuqi Zhu; name: Ningyu Zhang; name: Lanning Wei; name: Lun Du; name: Da Zheng; name: Huajun Chen
Published: 2025-10-20 17:53:23

Abstract: Replicating AI research is a crucial yet challenging task for large language
model (LLM) agents. Existing approaches often struggle to generate executable
code, primarily due to insufficient background knowledge and the limitations of
retrieval-augmented generation (RAG) methods, which fail to capture latent
technical details hidden in referenced papers. Furthermore, previous approaches
tend to overlook valuable implementation-level code signals and lack structured
knowledge representations that support multi-granular retrieval and reuse. To
overcome these challenges, we propose Executable Knowledge Graphs (xKG), a
modular and pluggable knowledge base that automatically integrates technical
insights, code snippets, and domain-specific knowledge extracted from
scientific literature. When integrated into three agent frameworks with two
different LLMs, xKG shows substantial performance gains (10.9% with o3-mini) on
PaperBench, demonstrating its effectiveness as a general and extensible
solution for automated AI research replication. Code will released at
https://github.com/zjunlp/xKG.