EnyaWoooo/ctflowv2-vlm3d2026

CTFlow — VLM3D Challenge 2026 Submission

Text-to-CT volume generation model (radiology report → synthetic chest CT volume), submitted to the VLM3D / CT-RATE challenge, 2026 edition.

Architecture

Autoregressive flow-matching transformer (STDiT / DiffuserSTDiT) operating in a latent space (FLUX-style VAE), generating a CT volume block-by-block conditioned on a BERT (BiomedVLP-CXR-BERT) text embedding of the radiology report and a declared voxel spacing. A learned super-resolution adapter upsamples the native 256-in-plane latent decode to the final 512x512 output.

Production pipeline (this submission)

• Selection: draws 5 candidates per prompt (independently verified seeds, not a naive offset formula), picks the one whose z-axis jerk is closest to the group median (a proxy for motion-plausibility), after a length gate and a motion floor filter.
• CT-CLIP quality safety net: scores the picked candidate against its own report text using the real CT-CLIP model (dual-tower CTViT + BERT). If the score is below 0.5, redraws 2 backup-seed candidates and republishes whichever of the combined pool scores highest on CLIP — no second jerk-median pass.
• Geometry: xy = 0.75mm (SR adapter upsampled to 512x512), z spacing declared adaptively so every volume's block spans a fixed 384mm physical extent; the model is conditioned on a coarser z (3.0mm) than what's declared, which was found empirically to draw more varied content.
• HU: clipped to [-1000, 1000].
• Compile: torch.compile(dynamic=True) (default mode — max-autotune was found to add large, unpredictable recompilation cost across the many distinct sequence lengths this pipeline produces).

Contents

• inference_ft_vae11k_v4_d3_seeded_ext384.py — production inference entrypoint
• Dockerfile, entrypoint.sh — submission container build
• ct_clip_src/ — vendored CT-CLIP source (patched: offline-safe tokenizer load)
• CTFlow/ — shared model/dataset code
• models/ — small configs; large weights are in weights.zip
• weights.zip — denoiser (spacing-fine-tuned, round 2), VAE, FLUX VAE, CT-CLIP, BiomedVLP-CXR-BERT-specialized weights
• sr_adapter.pt — trained super-resolution adapter (256→512 in-plane)
• submission_fast2retry.tar.gz — the exact built+packaged image that was submitted
• val_reports_*.json — small held-out validation prompts used during development

Usage

Input is a JSON file, a list of {"input_image_name": <output filename>, "report": <radiology report text>} objects. Output is one .nii.gz per input, written to /output.

Option A — load the prebuilt image (fastest)

docker load -i submission_fast2retry.tar.gz          # creates ctflow-submission:fast2retry
unzip weights.zip -d weights_extracted/               # or mount the individual files yourself

docker run --rm --gpus all \
  -v $PWD/weights_extracted:/weights:ro \
  -v $PWD/my_prompts_dir:/input:ro \                  # must contain prompts.json
  -v $PWD/my_output_dir:/output \
  ctflow-submission:fast2retry

entrypoint.sh auto-detects the GPU count and runs one prompt-shard per GPU via torchrun.

Option B — rebuild from source

docker build -t ctflow-submission:local -f Dockerfile .
# then run exactly as in Option A, substituting the new image tag

Notes

• /weights must contain (paths exactly as they appear inside weights.zip): CT-CLIP_v2.pt, vae_step11000.pt, ctflow_vae_ft_spacing_ckpt4500_r2/denoiser_ema/{config.json,diffusion_pytorch_model.safetensors}, FLUX_vae_checkpoint/{config.json,diffusion_pytorch_model.safetensors,rgb_imagenet.pt}, BiomedVLP-CXR-BERT-specialized/model.safetensors.
• A single prompt costs roughly 1–3 minutes of A100 time (more if the CT-CLIP safety net triggers a retry); budget accordingly for large prompt lists.
• CTFLOW_CLIP_THRESHOLD (env var, default 0.5) controls the CT-CLIP retry/fallback trigger threshold described above.

Citation

If you use CTFlow, the fine-tuned VAE, or this pipeline, please cite the base paper and, for this year's system specifically, the write-up repo:

@article{wang2025ctflow,
  title   = {CTFlow: Video-Inspired Latent Flow Matching for 3D CT Synthesis},
  author  = {Wang, Jiayi and Reynaud, Hadrien and Erick, Franciskus Xaverius and Kainz, Bernhard},
  journal = {arXiv preprint arXiv:2508.12900},
  year    = {2025}
}

@misc{wang2026ctflowv2,
  title        = {CTFlow v2: VLM3D 2026 Submission},
  author       = {Wang, Jiayi and Reynaud, Hadrien and N\"{u}tzel, Felix and Durug\"{o}l, \"{O}mer Faruk and Spasojevic, Teodora and Hagen, Luca and Kainz, Bernhard},
  year         = {2026},
  howpublished = {\url{https://github.com/WongJiayi/CTFlow-v2}},
  note         = {Model weights and inference code: \url{https://huggingface.co/EnyaWoooo/ctflowv2-vlm3d2026}}
}