BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing
Authors: name: Yunqi Gu, name: Ian Huang, name: Jihyeon Je, name: Guandao Yang, name: Leonidas Guibas

3D graphics editing is crucial in applications like movie production and game
design, yet it remains a time-consuming process that demands highly specialized
domain expertise. Automating this process is challenging because graphical
editing requires performing a variety of tasks, each requiring distinct skill
sets. Recently, vision-language models (VLMs) have emerged as a powerful
framework for automating the editing process, but their development and
evaluation are bottlenecked by the lack of a comprehensive benchmark that
requires human-level perception and presents real-world editing complexity. In
this work, we present BlenderGym, the first comprehensive VLM system benchmark
for 3D graphics editing. BlenderGym evaluates VLM systems through code-based 3D
reconstruction tasks. We evaluate closed- and open-source VLM systems and
observe that even the state-of-the-art VLM system struggles with tasks
relatively easy for human Blender users. Enabled by BlenderGym, we study how
inference scaling techniques impact VLM's performance on graphics editing
tasks. Notably, our findings reveal that the verifier used to guide the scaling
of generation can itself be improved through inference scaling, complementing
recent insights on inference scaling of LLM generation in coding and math
tasks. We further show that inference compute is not uniformly effective and
can be optimized by strategically distributing it between generation and
verification.