Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic
  Empirical Study
Authors: name: Yuqi Zhu; name: Yi Zhong; name: Jintian Zhang; name: Ziheng Zhang; name: Shuofei Qiao; name: Yujie Luo; name: Lun Du; name: Da Zheng; name: Huajun Chen; name: Ningyu Zhang
Published: 2025-06-24 17:04:23

Abstract: Large Language Models (LLMs) hold promise in automating data analysis tasks,
yet open-source models face significant limitations in these kinds of
reasoning-intensive scenarios. In this work, we investigate strategies to
enhance the data analysis capabilities of open-source LLMs. By curating a seed
dataset of diverse, realistic scenarios, we evaluate models across three
dimensions: data understanding, code generation, and strategic planning. Our
analysis reveals three key findings: (1) Strategic planning quality serves as
the primary determinant of model performance; (2) Interaction design and task
complexity significantly influence reasoning capabilities; (3) Data quality
demonstrates a greater impact than diversity in achieving optimal performance.
We leverage these insights to develop a data synthesis methodology,
demonstrating significant improvements in open-source LLMs' analytical
reasoning capabilities.