aqualaguna/MOSS-Image-SFT-OpenVINO

MOSS-Image-SFT for OpenVINO (Intel Arc GPUs)

OpenMOSS-Team/MOSS-Image-SFT (5B single-stream DiT + Qwen3.5-9B text encoder + FLUX.2 small-decoder VAE) converted to OpenVINO IR and run on an Intel Arc Pro B60 24 GB with OpenVINO 2026.4. No PyTorch GPU, no IPEX: the networks run in the OpenVINO GPU plugin, the sampler is plain numpy / torch-cpu host code.

It ran as a production text-to-image service on a B60 for two weeks (Oct–Nov 2026) and is published as-is. Not affiliated with OpenMOSS. Original model licence: Apache-2.0 (see LICENSE).

What is in here

folder	content	size
te/	Qwen3.5-9B text encoder, 4 IR parts of 8 layers (Gated-DeltaNet as ov::Loop), fp16, + token embeddings	15 GB
dit_fp16/	DiT in 4 segments: text_refine, dit_a (blocks 0–21), dit_b (block 22), dit_c (rest), fp16	9.7 GB
dit_int8/	same graph, NNCF per-channel INT8_ASYM weights, every ffn_down kept fp16 ("keepdown")	6.1 GB
vae/	VAE decoder, f16 IR (+ f32 IR used for the CPU check)	161 MB
tokenizer/	tokenizer + chat template	20 MB
code/	pipeline, HTTP server, exporters, reference + verification scripts	<1 MB

The DiT is split so the negative (CFG) branch can skip block 22, as the original pipeline does (uncond_skip_layers=(22,)). Dynamic shapes: any size in multiples of 16.

Fidelity (vs the original fp32 PyTorch pipeline on CPU)

part	fp16 IR	int8 (keepdown)
text encoder	cosine ≥ 0.999997	(same fp16 TE)
DiT velocity	cosine ≥ 0.999995, rel. L2 0.2–0.3 %	cosine 0.9991–0.9999, rel. L2 1.3 / 3.8 / 4.2 % (t = 1000 / 500 / 100)
VAE	64.6 dB	–
whole sampler, 256², 4 steps	54 dB	40.2 dB

Note: the GPU plugin inserts a DynamicQuantize before each compressed matmul, so "weight-only" int8 also quantizes activations on the fly. DYNAMIC_QUANTIZATION_GROUP_SIZE=0 disables it (40.0 dB, but 14 % slower, no faster than fp16). The OV kernel cache key ignores that property: use a separate CACHE_DIR when testing it.

Speed and memory (Arc Pro B60 24 GB, 1024×1024, CFG 3)

	s/step	30 steps (default)	50 steps	peak GPU memory
fp16	2.26–2.35	~72 s	~2 min	15.6–17.8 GB; 1536×1152 21.3 GB; 1536² does not fit
int8	2.01–2.09	63–65 s	101–105 s	11.8–13.8 GB; 1536² / 2048×1152 ~20.5 GB

Cold start with an empty kernel cache: ~75 s to load, first int8 image +47 s. The cache then holds ~29 GB; warm start ~11 s. Defaults that looked best by eye: 30 steps, guidance 3, shift 5. (8-step distill LoRAs were tried and rejected.)

Benchmarks (Arc Pro B60 24 GB, OpenVINO 2026.4.1, measured 2026-10-20)

All warm, kernel cache filled, five test prompts (10–300 tokens, one JSON layout prompt), CFG 3, shift 5.

Time per image at 1024×1024 (≈ 3 s fixed: text encoder 2–3 s + VAE 0.3 s, plus steps × s/step):

model	s/step	50 steps	30 steps	20 steps	8 steps (computed)
fp16	2.26–2.35	116–121 s	70–74 s	48–50 s	~21 s (CFG) / ~12 s (guidance 1)
int8 keepdown	2.01–2.09	101–105 s	63–65 s	43–46 s	~19–20 s (CFG) / ~11 s (guidance 1)

guidance: 1 skips the negative pass (one forward per step), as the original pipeline does.

DiT time per step by size (fp16, 2 forwards: cond + negative with block 22 skipped):

512²	1024²	1536×1152	1536²
0.5 s	2.26 s	4.4 s	6.2 s

Text encoder (Qwen3.5-9B, one of 4 parts, random embeddings, timing only): 0.08 s at 1024 tokens, 0.18 s at 2048, 0.34 s at 4096. In practice 2–4 s per request including compiling the 4 parts from the cache. VAE decode: 0.2 s at 1024², 0.4–1.9 s at 1536×1152.

GPU memory (peak_gpu_mb from the OV device; "process" = what the host saw for the whole process):

model	1024²	1536×1152	1536² / 2048×1152
fp16	15.6–17.8k (process 19.4k)	21.3k	24.2k: does not fit 24 GB
int8 keepdown	11.8k first image, 13.1–13.8k next (process 15.9k after ten)	16–17k	20.3–20.5k (process 21.5k)

Host RAM: 1.6 GB peak (token embeddings memory-mapped).

Load times: warm start ~11 s; model switch fp16 ↔ int8 0.8–7.6 s (10–17 s with a cold cache). Empty kernel cache: 75 s to ready, then the first int8 call 47 s (DiT 22 s, text encoder 18.5 s, VAE 1 s); the cache then holds ~29 GB.

Fidelity detail (vs the original PyTorch code, fp32, CPU, same weights):

check	result
text encoder, 32 layers fp16 on GPU	cosine ≥ 0.999997, rel. L2 0.1–0.25 % per pass (official bf16 itself: 0.6–1.9 % off fp32)
DiT velocity 512², fp16	cosine ≥ 0.999995, rel. L2 0.2–0.3 %
DiT velocity, int8 keepdown	rel. L2 1.3 / 3.8 / 4.2 % (t = 1000 / 500 / 100)
DiT velocity, plain per-channel int8 (not shipped)	1.3 / 5.0 / 5.5 %, sampler 36.2 dB
DiT velocity, group-128 int8 (not shipped)	1.1 / 3.9 / 4.5 %, sampler 39.7 dB
VAE decode fp16 GPU	64.6 dB (512²) / 64.9 dB (1024²); f32 on CPU 91.5 / 87.2 dB
whole sampler 256², 4 steps	fp16 54.0 dB, int8 keepdown 40.2 dB

Not measured: picture quality beyond a person's eye on a few prompts, text-rendering accuracy, comparison with other models or with the official bf16 pipeline on a GPU.

Use

Requirements (tested): openvino 2026.4.1, numpy 2.4, torch 2.12 (cpu), transformers 5.20, tokenizers 0.23, pillow, aiohttp (server only). Lay the folders out as models/{te,tokenizer,vae,dit_sft,dit_sft_int8_keepdown} (rename dit_fp16 → dit_sft, dit_int8 → dit_sft_int8_keepdown) or edit VARIANTS in code/moss_pipeline.py.

import sys; sys.path.insert(0, "code")
from moss_pipeline import MossImage
from PIL import Image

pipe = MossImage(models_dir="models", device="GPU", cache_dir="ov_cache", variant="sft-int8")
img, stats = pipe.generate("a red fox in fresh snow, morning light", width=1024, height=1024, steps=30, guidance=3.0, seed=1)
Image.fromarray(img).save("fox.png")

code/server.py --port 8109 serves the same thing over HTTP (POST /v1/mossimage). Hard-coded paths (/opt/mossimage/...) in the scripts are from the original box; adjust them.

Rebuild from the original weights

CPU only, from code/: te_export.py --parts 0,1,2,3, dit_export.py --which sft, vae_export.py --dtype f16, then int8_compress.py <dit_sft> <dit_sft_int8_keepdown> --keep-down. References and checks: ref_*.py, verify_*.py, ref_e2e.py. Trap: torch.jit.trace's check_trace re-runs the forward and can hang for 10+ min under CPU contention (patched off).

Known limits

• fp16 DiT residual stream reaches ~55 000 (fp16 max 65 504); the sampler checks for non-finite values.
• The text encoder drifts on very long prompts in fp16 (the official bf16 pipeline is itself 33 % off fp32 at 1532 tokens); MOSSIMAGE_TE_PRECISION=f32 fixes it at +7 GB per part.
• Image quality was judged by eye only, on a handful of prompts.

Image editing build (community Moss-Image-Edit LoRA baked in): see aqualaguna/MOSS-Image-Edit-OpenVINO.