PandacatAI/lingbot-world-v2-mps

Infinite Worlds with Versatile Interactions

Robbyant Team

---

We present LingBot-World 2.0 (also known as LingBot-World-Infinity), an advanced iteration of LingBot-World featuring four distinct upgrades.

• Unbounded Interaction Horizon: Our model achieves an unbounded interaction horizon while maintaining consistent output quality, benefiting from a carefully crafted causal pretraining paradigm.
• Rapid Response Time: Through distilling a real-time variant from the base model, our system guarantees rapid response time, sufficient to drive 720p video streams at 60 fps.
• Highly Diverse Interactive Elements: Compared to the previous version, this update introduces highly diverse interactive elements, comprising a broader spectrum of actions (e.g., attacking, archery, spell-casting, and shooting) alongside a richer variety of text-driven events.
• Agentic Harness: We pioneer the integration of an agentic harness within the domain of world modeling, wherein a pilot agent is tasked with planning and executing character behaviors, while a director agent is responsible for synthesizing novel environmental elements as the scene progresses.

Fork note. This repository is a fork of robbyant/lingbot-world-v2 maintained by PandacatAI. It adds single-device Apple Silicon (MPS) support with a bounded memory footprint, a disk prompt cache, and an optional 8-bit MLX text encoder; the model weights and architecture are unchanged. See the causal_fast 1.3B bullet under Inference for the Mac instructions. The fork is distributed under the original CC BY-NC-SA 4.0 license.

🚀 Try it now

The real-time version of LingBot-World-Infinity is available on two platforms. We thank Reactor and LingGuang for their support:

• International (Web): Experience it on Reactor.
• Domestic (Mobile): Experience it on LingGuang.

Note: Reactor and LingGuang provide a convenient way to try LingBot-World-Infinity in real time. In our official setup, the model runs at full capability. To experience our official demo, join us at WAIC 2026.

🎬 Demo Gallery

✨ For more high-fidelity and compelling demos, please visit our Project Page.

Apple Silicon (this fork)

1.3B causal-fast on an M4 Pro (48 GB unified memory), 480×832, 109 frames @ 16 fps. Prompt and camera path from examples/03. Generated in 14.4 min at a 16.1 GB process-footprint peak. Also on YouTube.

🔥 News

• Sep. 13, 2026: 🍎 Apple Silicon fork (PandacatAI/lingbot-world-v2-mps) — single-device MPS inference for the 1.3B causal-fast model, 8-bit MLX text encoder, disk prompt cache.
• Sep. 10, 2026: 🎉 We release the remaining full model variants: the 14B model’s causal-pretrained and bidirectional variants, and the 1.3B model’s causal-fast variant.
• Jul. 9, 2026: 🎉 We release the technical report, inference code, and models for LingBot-World-Infinity.

📋 TODO

• Release the causal-fast inference code and model of the 14B model
• Release the causal-pretrained model of the 14B model
• Release the bidirectional model of the 14B model
• Release the causal-fast model of the 1.3B model

⚙️ Quick Start

This codebase is built upon Wan2.2. Please refer to their documentation for installation instructions.

Installation

Clone the upstream repo, or this Apple Silicon fork from Hugging Face:

# upstream (14B / multi-GPU)
git clone https://github.com/robbyant/lingbot-world-v2.git
cd lingbot-world-v2

# this fork (1.3B / MPS) — `huggingface-cli` was replaced by `hf`
pip install -U huggingface_hub
hf download PandacatAI/lingbot-world-v2-mps --local-dir lingbot-world-v2-mps
cd lingbot-world-v2-mps

Install dependencies:

# Ensure torch >= 2.4.0
pip install -r requirements.txt

Install flash_attn (CUDA multi-GPU only; skip on macOS):

pip install flash-attn --no-build-isolation

Model Download

Model	Model Type	Model Size	Download Links
lingbot-world-v2-14b-causal-fast	causal-fast	14B	🤗 HuggingFace 🤖 ModelScope
lingbot-world-v2-14b-causal-pretrain	causal-pretrain	14B	🤗 HuggingFace
lingbot-world-v2-14b-bid	bidirectional	14B	🤗 HuggingFace
lingbot-world-v2-1.3b-causal-fast	causal-fast	1.3B	🤗 HuggingFace

Download models using the Hugging Face CLI (hf; formerly huggingface-cli):

pip install -U huggingface_hub
hf download robbyant/lingbot-world-v2-14b-causal-fast --local-dir ./lingbot-world-v2-14b-causal-fast

Download models using modelscope-cli:

pip install modelscope
modelscope download robbyant/lingbot-world-v2-14b-causal-fast --local_dir ./lingbot-world-v2-14b-causal-fast

Inference

We provide generate.py for causal inference with KV caching, which processes video frames chunk-by-chunk instead of all at once.

• causal_fast — 480P, multi-GPU:

  torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 480*832 --ckpt_dir lingbot-world-v2-14b-causal-fast --image examples/03/image.jpg --action_path examples/03 --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 361 --local_attn_size 18 --sink_size 6 --prompt "A serene lakeside scene with a lone tree standing in calm water, surrounded by distant snow-capped mountains under a bright blue sky with drifting white clouds — gentle ripples reflect the tree and sky, creating a tranquil, meditative atmosphere."
  

You can also use the provided run_fast.sh script:

bash run_fast.sh <weights_dir> <frame_num>
# e.g. bash run_fast.sh lingbot-world-v2-14b-causal-fast 361

• causal_fast — 1.3B model, single device (Apple Silicon / MPS, or one GPU):

  bash run_mps.sh <weights_dir> <frame_num>
  # e.g. bash run_mps.sh ckpt 361
  

  The 1.3B checkpoint needs no flash_attn and no torchrun, so on macOS you only need the base requirements (pip install -r requirements.txt without the flash-attn line; run_mps.sh uses .venv/bin/python when present). On MPS the pipeline keeps the process footprint at roughly 16 GB for 480×832 output at any frame count: the text encoder is released after the prompt is encoded, the DiT is released before VAE decode, and the VAE runs spatially tiled (--vae_tile_size, exact with the default halo). To measure the peak of a run, wrap it with python tools/memwatch.py -- python generate.py ....

  Assemble the weights once (~7 GB; the 1.3B checkpoint is natively in diffusers format and loads as-is):

  pip install -U huggingface_hub mlx
  hf download robbyant/lingbot-world-v2-1.3b-causal-fast \
      --local-dir ckpt/transformers
  hf download Wan-AI/Wan2.1-T2V-14B Wan2.1_VAE.pth --local-dir ckpt
  hf download google/umt5-xxl --local-dir ckpt/google/umt5-xxl
  hf download PandacatAI/umt5-xxl-enc-mlx-q8 \
      --local-dir ckpt/umt5-xxl-enc-mlx-q8
  

  Prompt embeddings are cached on disk (<ckpt_dir>/t5_cache/, a few KB to 2 MB each). A prompt seen in any earlier run skips the text encoder entirely. --t5_cache_dir relocates the cache, --t5_cache_dir '' disables it.

  The I2V conditioning tensor is cached the same way (<ckpt_dir>/cond_cache/, ~10 MB each; --cond_cache_dir, '' to disable). It depends only on the image and the output geometry — not the prompt, seed or frame count — so an interactive session that stays on one image does no VAE encode work at all after the first clip.

  The text encoder is an 8-bit MLX quantization of umt5-xxl (PandacatAI/umt5-xxl-enc-mlx-q8): 5.6 GB on disk instead of 10.6 GB, T5-phase peak 8 GB instead of 14 GB, ~1 s per prompt. --t5_backend auto (the default) uses <ckpt_dir>/umt5-xxl-enc-mlx-q8 whenever it exists, --t5_backend torch forces the original bf16 checkpoint. To (re)build the MLX encoder from ckpt/models_t5_umt5-xxl-enc-bf16.pth: pip install mlx && python tools/convert_t5_mlx.py --ckpt_dir ckpt --bits 8 --check. Measured against the bf16 torch encoder the 8-bit embeddings differ by ~3% relative — the same amount torch-on-MPS differs from torch-on-CPU — and a clip generated from them matches the torch one at 39 dB PSNR. 4-bit was measured too and rejected (17–28% error).

Local benchmark (Apple Silicon)

Measured on a MacBook Pro with M4 Pro and 48 GB unified memory, macOS, PyTorch MPS, 1.3B causal-fast, 480×832, tiled VAE (core=32 latent px, halo=16), local_attn_size=18, sink_size=6, examples/03 camera path. Peak footprint is the process's physical-memory high-water mark (the figure that actually OOMs); RSS is much smaller because MPS keeps most weight pages file-backed.

Frames (requested / written)	Text encoder	Wall time	Peak footprint	Peak RSS
121 / 109	MLX 8-bit (prompt cache hit)	14.4 min	16.1 GB	2.8 GB
361 / 269	torch bf16	33.5 min	16.1 GB	3.3 GB

The 121-frame run (the demo above) spent ~4 min in VAE encode, 5.2 min in 7 DiT chunks (30→49 s/chunk as the KV window fills), and ~5 min in VAE decode. Footprint does not grow with frame count past ~16 GB: the encoder, DiT and VAE are never resident together, and the VAE is spatially tiled. On a 13-frame clip the MLX encoder's process peak is 11.8 GB versus 14.0 GB for torch bf16.

The table above predates the conditioning-encode change. The VAE no longer encodes the zero-padded tail of the conditioning tensor: past latent frame 25 the encoder's output is bitwise constant in bfloat16 (which is the precision the DiT's patch_embedding sees under autocast), so those frames are broadcast instead of computed. Encode cost therefore stops scaling with clip length — 26 temporal chunks at any frame_num — which is worth ~7% at 121 frames but 70% at 361, where a paired same-process measurement gave 1048.6 s → 349.6 s (3.0x). With a warm cond_cache the encode is skipped entirely (measured 190.5 s → 0.1 s). Both are bitwise identical to the uncut encode at bfloat16 — verified element-wise on the conditioning tensor and end-to-end as byte-identical output video — and the cut can be disabled with LINGBOT_ZERO_TAIL_CUT. tools/vae_zero_fixpoint.py reproduces the measurement.

Wrap any command with python tools/memwatch.py -- ... to record the same counters.

Deployment

We do NOT plan to release our deployment code. If you would like to deploy our model yourself, please refer to the LingBot-World deployment in SGLang or flashdreams.

📚 Related Projects

• LingBot-World

📜 License

This project is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). The project is available for non-commercial use only: you may share and adapt it with proper attribution, but derivative works must be distributed under the same license. Please refer to the LICENSE file for the full text, including details on rights and restrictions.

The Apple Silicon fork in this repository is a derivative work and is distributed under the same CC BY-NC-SA 4.0 license. The MLX text encoder published separately at PandacatAI/umt5-xxl-enc-mlx-q8 is derived from the Apache-2.0 licensed umt5-xxl / Wan2.1 text encoder and carries its own license.

✨ Acknowledgement

We would like to express our gratitude to the Wan Team for open-sourcing their code and models. Their contributions have been instrumental to the development of this project.

📖 Citation

If you find this work useful for your research, please cite our paper:

@article{lingbot-world-v2,
      title={Infinite Worlds with Versatile Interactions}, 
      author={Zelin Gao and Qiuyu Wang and Jiapeng Zhu and Jingye Chen and Zichen Liu and Qingyan Bai and Jiahao Wang and Yufeng Yuan and Hanlin Wang and Yichong Lu and Ka Leong Cheng and Haojie Zhang and Jian Gao and Tianrui Feng and Yuzheng Liu and Yao Yao and Yinghao Xu and Xing Zhu and Yujun Shen and Hao Ouyang},
      journal={arXiv preprint arXiv:2607.07534},
      year={2026}
}