Title: Agentic Design of Compositional Machines
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Agents, LLM, Machine Design, 3D Generation, Reinforcement Learning, Symbolic Program, Compositionality

Abstract: The design of complex machines stands as both a marker of human intelligence and a foundation of engineering practice. Given recent advances in large language models (LLMs), we ask whether they, too, can learn to create. We approach this question through the lens of compositional machine design, where machines are assembled by selecting and placing mechanical components to satisfy functional demands—such as locomotion or manipulation—within a simulated physical environment. To support this investigation, we introduce BesiegeField, a testbed built on the physics-based game Besiege, which enables part-based construction, physical simulation, and reward-driven evaluation. Using BesiegeField, we benchmark state-of-the-art LLMs with agentic workflows and identify key capabilities required for success, including spatial reasoning, strategic assembly, and instruction-following. As current open-source models fall short, we explore reinforcement learning (RL) as a path to improvement. We curate a compositional design dataset, conduct RL finetuning and ablation studies, and highlight open challenges at the intersection of language, design, and physical reasoning.

TL;DR: We introduce the novel task of compositional machine design with physical functionality using LLMs. We introduce an environment called BesiegeField with the game Besiege, in which we evaluate LLMs with both agentic and RLVR methods.