Generating Physically Stable and Buildable LEGO Designs from Text
Authors: name: Ava Pun, name: Kangle Deng, name: Ruixuan Liu, name: Deva Ramanan, name: Changliu Liu, name: Jun-Yan Zhu
Organization: Generative Intelligence Lab (CMU)

We introduce LegoGPT, the first approach for generating physically stable
LEGO brick models from text prompts. To achieve this, we construct a
large-scale, physically stable dataset of LEGO designs, along with their
associated captions, and train an autoregressive large language model to
predict the next brick to add via next-token prediction. To improve the
stability of the resulting designs, we employ an efficient validity check and
physics-aware rollback during autoregressive inference, which prunes infeasible
token predictions using physics laws and assembly constraints. Our experiments
show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO
designs that align closely with the input text prompts. We also develop a
text-based LEGO texturing method to generate colored and textured designs. We
show that our designs can be assembled manually by humans and automatically by
robotic arms. We also release our new dataset, StableText2Lego, containing over
47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed
captions, along with our code and models at the project website:
https://avalovelace1.github.io/LegoGPT/.