EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies
Authors: name: Xavier Hu, name: Jinxiang Xia, name: Shengze Xu, name: Kangqi Song, name: Yishuo Yuan, name: Guibin Zhang, name: JinCheng Ren, name: Boyu Feng, name: Li Lu, name: Tieyong Zeng, name: Jiaheng Liu, name: Minghao Liu, name: He Zhu, name: Yuchen Eleanor Jiang, name: Wei Wang, name: Wangchunshu Zhou

Long-horizon planning is widely recognized as a core capability of autonomous LLM-based agents; however, current evaluation frameworks suffer from being largely episodic, domain-specific, or insufficiently grounded in persistent economic dynamics. We introduce EcoGym, a generalizable benchmark for continuous plan-and-execute decision making in interactive economies. EcoGym comprises three diverse environments: Vending, Freelance, and Operation, implemented in a unified decision-making process with standardized interfaces, and budgeted actions over an effectively unbounded horizon (1000+ steps if 365 day-loops for evaluation). The evaluation of EcoGym is based on business-relevant outcomes (e.g., net worth, income, and DAU), targeting long-term strategic coherence and robustness under partial observability and stochasticity. Experiments across eleven leading LLMs expose a systematic tension: no single model dominates across all three scenarios. Critically, we find that models exhibit significant suboptimality in either high-level strategies or efficient actions executions. EcoGym is released as an open, extensible testbed for transparent long-horizon agent evaluation and for studying controllability-utility trade-offs in realistic economic settings.