Hephaestus: Improving Fundamental Agent Capabilities of Large Language
  Models through Continual Pre-Training
Authors: name: Yuchen Zhuang, name: Jingfeng Yang, name: Haoming Jiang, name: Xin Liu, name: Kewei Cheng, name: Sanket Lokegaonkar, name: Yifan Gao, name: Qing Ping, name: Tianyi Liu, name: Binxuan Huang, name: Zheng Li, name: Zhengyang Wang, name: Pei Chen, name: Ruijie Wang, name: Rongzhi Zhang, name: Nasser Zalmout, name: Priyanka Nigam, name: Bing Yin, name: Chao Zhang

Due to the scarcity of agent-oriented pre-training data, LLM-based autonomous
agents typically rely on complex prompting or extensive fine-tuning, which
often fails to introduce new capabilities while preserving strong
generalizability. We introduce Hephaestus-Forge, the first large-scale
pre-training corpus designed to enhance the fundamental capabilities of LLM
agents in API function calling, intrinsic reasoning and planning, and adapting
to environmental feedback. Hephaestus-Forge comprises 103B agent-specific data
encompassing 76,537 APIs, including both tool documentation to introduce
knowledge of API functions and function calling trajectories to strengthen
intrinsic reasoning. To explore effective training protocols, we investigate
scaling laws to identify the optimal recipe in data mixing ratios. By continual
pre-training on Hephaestus-Forge, Hephaestus outperforms small- to medium-scale
open-source LLMs and rivals commercial LLMs on three agent benchmarks,
demonstrating the effectiveness of our pre-training corpus in enhancing
fundamental agentic capabilities and generalization of LLMs to new tasks or
environments.