IvanBanny/dirtflow-v0

Dirtflow v0

Research | Model | Engine | Mod | Demo

Dirtflow generates Minecraft builds from text and the blocks already around them. Select a region, describe what belongs there, and the model proposes an edit using full Minecraft block states (including properties such as the direction a stair faces). The surrounding world stays fixed.

The loop shows 36 selected builds, from structure sampling through material sampling to the final result after physics cleanup. Playback timing does not measure inference speed.

For example, a request for "a small stone watchtower with a wooden roof" supplies both the prompt and a snapshot of the selected region and its surroundings. The model first samples the shape, then assigns block states to it. In the Minecraft mod, the result is a proposal you can preview before accepting.

I'm releasing the Stage 1 340k EMA / Stage 2 400k LIVE pair, with the corrected material-stage inference policy. Both stages and all required model assets are included here.

Model details

Detail	Value
Model	Two-stage flow model for Minecraft generation and editing
Geometry	TRELLIS XL-based structure flow, 340,000 updates, EMA weights
Materials	Continuous block-state flow, 400,000 updates, complete LIVE weights
Text encoder	Frozen CLIP ViT-L/14, 77-token context
Input	Text, observed block states, an editable region and read-only context
Output	An edit in the Minecraft 1.21.11 full-block-state vocabulary
Model canvas	64 x 64 x 64 blocks, including context
Output vocabulary	29,669 legal full block states
Weight precision	FP32 storage; BF16 denoiser execution on CUDA

EMA means exponential moving average. LIVE means the trained parameters at the selected update. All four material-stage modules use the 400k LIVE parameters, including the embedding and categorical head. The geometry stage retains its frozen codec and the selected EMA parameters.

Generation and editing

The engine supplies task conditioning alongside the prompt. Each task uses the same rectangular edit boundary.

Task	Behavior
Build / replace	Generate inside the selection. Existing blocks inside it may change or disappear.
Complete	Ask the model to complete a structure. Blocks inside the selection may still change.
Repair	Condition on a damaged structure and retain its occupied positions as material candidates. The material stage can still choose air.
Materials	Change block states at the existing occupied positions while preserving occupancy. Shape generation and physics cleanup are skipped.

The engine restores the original blocks outside the selection. This gives an exact edit boundary; how well the generated build fits its surroundings still depends on the model. The padded snapshot must fit inside the 64-block canvas without resizing. The current engine supports up to 58 blocks per editable axis with three blocks of context on each side. Larger context shells reduce the space available for the edit.

Using the model

Dirtflow uses a custom inference engine. Follow the installation commands in the engine README, activate its environment, and run:

dirtflow-engine

The first run downloads this 9.3 GB (8.65 GiB) bundle at the revision pinned by the engine. Later runs use the cached copy without network access. The engine checks every file's SHA-256 hash before loading and keeps the model loaded between requests. Once it is ready, the Minecraft mod can submit a selection and prompt.

Use dirtflow-engine --download-only to fetch and verify the model without a GPU or running server. The cache lives in $HF_HOME/dirtflow-engine/<revision>; HF_HOME defaults to ~/.cache/huggingface and can be set to another drive. Interrupted downloads resume when you run the command again.

For a separate local copy:

hf download IvanBanny/dirtflow-v0 \
    --revision v0.1.0 \
    --local-dir ./dirtflow-v0
dirtflow-engine --model ./dirtflow-v0

Keep both stages and their assets together. --model uses only local files:

dirtflow-v0/
    bundle.json
    stage1.pt
    stage2.pt
    assets/
        clip/
        class-mapping.npz
        game-schema.json.gz
        input-registry.json
        clip-manifest.json
        clip-source-config.json
        clip-derivation.json

File	Contents	Size
stage1.pt	Selected geometry weights and frozen structure codec	4.47 GB
stage2.pt	Complete LIVE material-stage weights	4.32 GB
assets/clip/model.safetensors	Frozen CLIP text weights	0.49 GB

bundle.json records the source checkpoints and inference policy, with a SHA-256 hash for every required payload. The engine verifies these hashes before loading. The .pt files contain inference tensors loaded with PyTorch's weights_only=True; optimizer state is excluded.

The reference runtime uses Python 3.12, PyTorch 2.12.0 with CUDA 13.2, and Transformers 5.13.0. GPU execution requires an NVIDIA GPU with BF16 support and a compatible driver. The default profile keeps the inactive stage in CPU memory to reduce VRAM use. CPU loading is supported, but interactive generation speed on CPU remains untested. GPU memory use depends on the selection and its occupied context; a region fitting the canvas can still exceed your available VRAM.

Start with 30 Euler steps per stage, guidance scale 5 and one candidate. Player serving uses batched guidance and physics cleanup by default (except for the Materials task). The checkpoint-selection comparisons used sequential guidance with physics disabled. Those settings can produce different block states even with the same seed. Record the runtime, sampler settings and seed when comparing outputs.

Required inference policy

This pair uses the corrected continuous material model. After selecting the candidate coordinates, the engine sets normalized editable confidence to exactly zero. It preserves the validity flag and all other conditioning features. This correction leaves the original checkpoint weights unchanged.

Use FP32 parameter storage and native FP32 position frequencies, with BF16 autocast for CUDA denoising. Rounding the stored weights or position frequencies to BF16 changes the numerical model. The bundle records these requirements and the engine rejects incompatible profiles.

Architecture and training

The geometry stage starts from Microsoft TRELLIS Text XL. A frozen structure codec compresses occupancy into a 16 x 16 x 16 latent grid. A flow Transformer samples that grid, conditioned on text and the observed world. Decoding gives occupancy scores, from which the engine selects candidate block positions.

The material stage samples eight-dimensional continuous features at those positions. Sparse convolutional blocks and a Transformer combine spatial and text conditioning; a categorical head maps the sampled features to legal block states. Text conditioning comes from OpenAI CLIP ViT-L/14.

Training quantity	Geometry stage	Material stage
Selected update	340,000	400,000
Training presentations	2,040,000	2,400,000
Global batch size	6	6
Selected weights	EMA, with frozen codec tensors	Complete LIVE

A presentation is one sampled training example; repeated crops and tasks can come from the same source build. The runs used 8-bit AdamW, FP32 parameters and BF16 autocast. Stage 1 stays frozen during material training.

The source Minecraft corpus contains 469,648 build rows, with generated captions in five styles. Training samples crops and editing tasks from this corpus, so the number of distinct builds seen by a checkpoint depends on the sampling. Stage 1 also uses geometry replay derived from the Sketchfab and GitHub subsets of TRELLIS-500K; the continuous Stage-2 recipe uses Minecraft data without replay. Captions describe source builds and can miss details of a sampled crop. The training corpus is not included here.

Evaluation and limitations

The 400k LIVE material checkpoint was selected on small development panels: 84 attempts from 24 source scenes, plus a separate authored panel of 24 task scenes and eight prompt pairs. Visual review was nonblinded and performed by an AI agent. Some tasks and prompt pairs favor other checkpoints. Human preference and performance across a broader player population remain unmeasured by these comparisons.

The model is intended for building experiments and research on Minecraft generation and editing. Review the proposed edit before applying it to a world you care about.

• Material choices can be patchy or unrelated to the prompt. Terrain can become water-heavy. Confidence correction reduces a documented material failure but leaves these broader quality limits.
• The material stage cannot add geometry outside the candidate positions supplied by Stage 1. A missing roof or support needs a change in the generated shape.
• Completion and repair can alter existing blocks inside the selection. Use the Materials task when occupancy must stay fixed.
• CLIP's context is limited to 77 tokens. Long prompts can be truncated. The development comparisons do not establish reliable following of precise directional instructions.
• Legal block states do not guarantee a functional building, connected redstone or a physically stable result. Physics cleanup handles a subset of placement rules. The block-state output does not generate entities or block-entity data such as chest contents and sign text.
• Samples can change across devices, precision settings and guidance modes. A seed alone is insufficient for exact reproduction.

License and attribution

The Dirtflow weights and model card are licensed under Apache 2.0. TRELLIS and CLIP retain their MIT licenses; their notices are included in LICENSES/TRELLIS.txt and LICENSES/CLIP.txt. The bundle derives its geometry model and structure codec from TRELLIS and includes CLIP's frozen text encoder and tokenizer. Its text-only CLIP export omits the vision model and projection heads.

The model license does not relicense source builds, training datasets or Minecraft assets. This is an independent project, unaffiliated with Mojang or Microsoft.