anonymousICLR2027/many-views-one-world

SyncWM: Inference Release

What this release is

This is the anonymous ICLR 2027 SyncWM artifact for mixed-expert video inference. It packages the public inference implementation and two step-8500 delta checkpoints. It does not include the LingBot-World base model, the VAE, or dataset-derived condition caches.

The checkpoints are adapters, not standalone generative models. Both require the external base model and inputs listed below.

Included files/checkpoints

Path	Contents
checkpoints/low_interaction_step8500.pt	LOW adapter, LoRA, dense encoder, state projector, and interaction projector
checkpoints/high_adapter_step8500.pt	HIGH adapter, LoRA, dense encoder, and state projector
tools/	Mixed-route sampler and condition modules
vendor/lingbot/	Project-compatible LingBot inference runtime
configs/mixed_route.json	Public route and sampling configuration
checkpoint_manifest.json	Checkpoint hashes and complete tensor inventory
docs/	Detailed inference and checkpoint contracts

Mixed LOW/HIGH route

The sampler uses both experts during denoising:

HIGH non-interaction adapter
        t >= 0.947
             |
      boundary sigma 0.947
             |
         t < 0.947
LOW interaction-enabled adapter

Interaction conditioning is active only on LOW. See docs/INFERENCE.md for the dense-resolution contract, interaction projection, condition fusion, and routing details.

External dependencies

Dependency	Source/version	Expected path or interface
LingBot-World camera base	TODO: publish or identify the exact public revision	Directory passed to --ckpt-dir, containing low_noise_model/ and high_noise_model/
Wan2.1 VAE	TODO: identify the exact public source/revision	<LINGBOT_BASE>/Wan2.1_VAE.pth
UMT5-XXL text encoder	Weight format used by the runtime; exact public revision TODO	<LINGBOT_BASE>/models_t5_umt5-xxl-enc-bf16.pth
UMT5 tokenizer	Runtime identifier google/umt5-xxl; exact pinned revision TODO	Available through the local Hugging Face cache or normal Transformers resolution
Dense cache	User-provided SyncWM dense-condition data	[7,240,416] map manifest passed to --map-manifest
Aligned cache	User-provided clip/sample alignment	JSONL passed to --cache-manifest
State and interaction cache	User-provided temporally aligned sidecar data	JSONL passed to --state-cache-manifest, with referenced NPZ files
Source clip inputs	User-provided initial frame and camera inputs	Clip directory containing image.jpg, poses.npy, intrinsics.npy, and optionally prompt.txt

Install a compatible Python/PyTorch environment using the packaged dependency profiles. The provided CUDA profile targets CUDA 12.4; the installed PyTorch build must also be compatible with the host NVIDIA driver.

Quick start

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements/cuda124.txt
python -m pip install -r requirements/base.txt
python -m pip install flash-attn==2.7.4.post1 --no-build-isolation

Run from the release root after preparing the external dependencies:

PYTHONPATH="$PWD/tools:$PWD/vendor/lingbot" \
python tools/sample_memory_dense_adapter_interaction_eval_v1.py \
  --lingbot-repo vendor/lingbot \
  --ckpt-dir /path/to/lingbot-world-base-cam \
  --adapter-checkpoint-low checkpoints/low_interaction_step8500.pt \
  --adapter-checkpoint-high checkpoints/high_adapter_step8500.pt \
  --map-manifest /path/to/dense_map_manifest.json \
  --cache-manifest /path/to/aligned_cache_manifest.jsonl \
  --state-cache-manifest /path/to/state_interaction_manifest.jsonl \
  --clip-root /path/to/clips \
  --out-dir outputs/mixed_route \
  --split test \
  --clip-id EXAMPLE_CLIP_ID \
  --interaction-variant true \
  --boundary-override 0.947 \
  --latent-frames 21 \
  --chunk-size 3 \
  --base-sampling-steps 70 \
  --base-guide-scale 5.0 \
  --base-shift 3.0 \
  --target-height 480 \
  --target-width 832 \
  --seed 20260801 \
  --no-offload-model

This configuration requests an 81-frame, 832x480 video at 16 FPS. The shown seed is the sampler default; use the evaluation-specific seed when reproducing a particular reported result.

Checkpoints

Both files are sanitized inference-only PyTorch checkpoints. Optimizer state, training paths, run directories, and resume metadata are excluded. All retained tensor keys, shapes, dtypes, and values were checked against their source training checkpoints.

Checkpoint	Step	Interaction projector	SHA256
low_interaction_step8500.pt	8500	Yes, 134 -> 128	380ce32fbc80a33d8bb5b05fba3446f644036c0e294dc9f9f5c7d7b901dd66e4
high_adapter_step8500.pt	8500	No	123c337889ec9a887f40127b74a7e0c23d4bc00702de943bf2c2711878a9a9fd

Reproducibility notes

• The packaged code and checkpoint structures have passed static validation, including CLI parsing, state-group compatibility, tensor inventories, and checkpoint hashes.
• A complete mixed-route generation has not been rerun from this packaged release under the currently available hardware conditions.
• LOW uses a [7,240,416] runtime dense condition. HIGH was trained with [7,176,320], but at runtime it consumes the same shared [7,240,416] manifest as LOW. This mixed-resolution lineage is a known validation caveat.
• Exact public revisions for the external base model, VAE, and text-model assets remain to be pinned before claiming a fully self-contained artifact.

Detailed docs

• docs/INFERENCE.md: routing, input contracts, condition fusion, runtime dense resolution, base-model layout, and outputs.
• docs/CHECKPOINTS.md: checkpoint structure and state groups.
• checkpoint_manifest.json: exact checkpoint sizes, hashes, tensor names, shapes, dtypes, and parameter counts.
• LICENSE and THIRD_PARTY_NOTICES.md: project and third-party licensing information.