Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic
  Empirical Study
Authors: name: Yuqi Zhu, name: Yi Zhong, name: Jintian Zhang, name: Ziheng Zhang, name: Shuofei Qiao, name: Yujie Luo, name: Lun Du, name: Da Zheng, name: Huajun Chen, name: Ningyu Zhang

Large Language Models (LLMs) hold promise in automating data analysis tasks,
yet open-source models face significant limitations in these kinds of
reasoning-intensive scenarios. In this work, we investigate strategies to
enhance the data analysis capabilities of open-source LLMs. By curating a seed
dataset of diverse, realistic scenarios, we evaluate models across three
dimensions: data understanding, code generation, and strategic planning. Our
analysis reveals three key findings: (1) Strategic planning quality serves as
the primary determinant of model performance; (2) Interaction design and task
complexity significantly influence reasoning capabilities; (3) Data quality
demonstrates a greater impact than diversity in achieving optimal performance.
We leverage these insights to develop a data synthesis methodology,
demonstrating significant improvements in open-source LLMs' analytical
reasoning capabilities.