creador/mv-adapter-sd21-onnx

MV-Adapter (image + geometry → 6 views) on Stable Diffusion 2.1-base — ONNX graphs for the browser

ONNX conversion of MV-Adapter's image-and-geometry-guided multi-view generation (mvadapter_ig2mv_sd21.safetensors of huanngzh/mv-adapter, revision 6de4033df6b53366f3c009d22f5ec434bb55e59f) fused into the UNet of sd2-community/stable-diffusion-2-1-base (revision 4e63672c03103b6c636b8fb4119ba982469b2955), split into the graphs WaveSlice runs locally with onnxruntime-web on WebGPU to paint the sides of a photographed object its photo never saw. The VAE and the text embeddings come from Stable Diffusion 2.1-base (sd2-community/stable-diffusion-2-1-base, revision 4e63672c03103b6c636b8fb4119ba982469b2955).

What was changed

• The UNet carries MV-Adapter's decoupled row/column multi-view self-attention (DecoupledMVRowColSelfAttnProcessor2_0, six views) and reference cross-attention as part of the graph; the T2IAdapter residuals, the multi-view and reference scales and the reference hidden states are inputs. Its 16 hs_i outputs are the hidden states in front of each self-attention, which is what the reference pass caches, so the same graph does the reference pass (t = 0, both scales 0).
• Every scaled-dot-product attention is a single com.microsoft.MultiHeadAttention node (onnxruntime's contrib operator), which onnxruntime runs as tiled flash attention instead of materialising the scores.
• The text encoder is not shipped: the embeddings of the fixed prompts are precomputed in text.bin.
• The noise schedule is MV-Adapter's ShiftSNR (interpolated, scale 8) over the base model's betas; schedule.json carries its alphas_cumprod.

The conversion scripts are in tools/mvadapter-onnx of the WaveSlice repository, with the cameras and the rasterisation conventions the control maps need.

Files

file	size	sha256	contents
unet.onnx	0.004 GB	66928f02446ed6a2d28a2360af233d301a49ca9b58e9490da2cc66490719901c	UNet graph (fp16): SD UNet with MV-Adapter's decoupled row/column multi-view self-attention and reference cross-attention fused in; weights in unet.onnx.data
unet.onnx.data	1.930 GB	7a604460731810788cfdcf45c9af7eda00ee79b2187bcc8fab5188795210611c	the UNet's weights (external data of unet.onnx)
cond_encoder.onnx	0.156 GB	7bb1994a5cfd1e38c3a6f8732cce34f3252078f65797bcc26ec9cc7cd3c9e394	T2IAdapter over the six position + normal maps (fp16)
vae_encoder.onnx	0.068 GB	6baff042f69f212d7fc3be3381abd6f42d0b2f204f39da40771cedfe34a9df8b	SD2.1 VAE encoder, image → latents × 0.18215 (fp16)
vae_decoder.onnx	0.099 GB	3e01d206d996938e3580dd49976f101465b18d6a9f6952ba522741616ab12741	SD2.1 VAE decoder, latents → image (fp16)
text.bin	0.000 GB	4224beb0f4666467bbe52d3e3aea66a912f43ab8a572a16de47a823fac9b2b73	OpenCLIP-H embeddings of the negative prompt (row 0) and "high quality" (row 1), float16 [2, 77, 1024]
schedule.json	0.000 GB	d2666658a466e1afecb55aea77813177bd339cae36cc66aaa322bd1b983962e0	the shifted noise schedule (1000 alphas_cumprod), prediction type, prompts
unet_layers.json	0.000 GB	979f9a53ad476b75d63a650a270e53bd32dd14381c20444eecf1bba2b902bdc7	the 16 self-attention layers whose hidden states the reference pass caches, with their shapes

Inputs and outputs

• cond_encoder.onnx: control float16 [6, 6, 512, 512] (positions + 0.5 and normals / 2 + 0.5 of the mesh in the six cameras, grey 0.5 behind) → a0 [6, 320, 64, 64], a1 [6, 640, 32, 32], a2 [6, 1280, 16, 16], a3 [6, 1280, 8, 8].
• vae_encoder.onnx: image float16 [1, 3, 512, 512] in [−1, 1] → latents [1, 4, 64, 64] (posterior mean × 0.18215).
• vae_decoder.onnx: latents float16 [N, 4, 64, 64] (÷ 0.18215 first) → image [N, 3, 512, 512] in [−1, 1].
• unet.onnx: sample [B, 4, 64, 64], timestep [1] float32, text [B, 77, 1024], mv_scale [], ref_scale [], a0..a3 [B, …], ref_0..ref_15 [B, S_i, C_i] (float16 unless noted) → noise [B, 4, 64, 64], hs_0..hs_15 [B, S_i, C_i]. B is 6 without classifier-free guidance or 12 with it (unconditional half first, its ref_i zero). Reference pass: the graph on the reference latents at t = 0 with both scales 0 and zero ref_i; its hs_i (first row) repeated over the views are the ref_i of every step.

The graphs need an onnxruntime with the com.microsoft contrib operators (the CPU, CUDA and WebGPU providers have them; onnxruntime-web ≥ 1.30 on WebGPU is what WaveSlice uses).

Licences

• The UNet is a derivative of sd2-community/stable-diffusion-2-1-base (Stable Diffusion 2.1-base, by Stability AI), licensed under CreativeML Open RAIL++-M; its use restrictions apply. The VAE and the text embeddings are derivatives of Stable Diffusion 2.1-base (CreativeML Open RAIL++-M; its use restrictions apply to them).
• MV-Adapter weights and code: Apache-2.0 (LICENSE-MV-Adapter), © the MV-Adapter authors (Zehuan Huang et al., "MV-Adapter: Multi-view Consistent Image Generation Made Easy").