Longpipe: real-time portrait matting (official ONNX)
Official ONNX exports of the Longpipe models: real-time person matting for video calls and live video (virtual backgrounds, background blur), plus the face-keypoint, face-mesh and optical-flow heads that power touch-up, auto-reframe and temporal stability.
Try it in your browser: longpipe.dev/demo, or the demo Space.
Longpipe is primarily a browser SDK (npm install longpipe,
GitHub, live demo) that runs these
networks as hand-written WebGPU/WebGL2 shaders. These ONNX files let you use the same trained weights
anywhere ONNX Runtime runs: Python, native apps, servers, mobile, or onnxruntime-web.
- License: MIT, code and weights. Training data provenance: longpipe.dev/docs/licensing.
- Weights: the SDK's current production weights (weights release
v0.0.5, served atcdn.longpipe.dev/models/v/0.0.5/and used by SDK 0.0.16), exported directly from the PyTorch training checkpoints that produce those files. Every ONNX file is checked against PyTorch (max abs diff โค 7e-5, most โค 1e-6); seeonnx/manifest.json.
Quality and speed
Average alpha error on the P3M-10K validation set (499 images): every Longpipe tier beats MediaPipe on both MAE and IoU.
In the browser, the SDK's zero-copy GPU pipeline is faster than MediaPipe or BodyPix despite running larger models (browser numbers; ONNX Runtime speed depends on your runtime and hardware):
Tiers
Five tiers trade quality for cost. Each is a small "base" network (EfficientNet-Lite encoder + decoder) wrapped in a high-resolution U-Net wrapper that refines edges at canvas resolution.
| Tier | Canvas (landscape, HรW) | Base res | matting size |
Intended hardware |
|---|---|---|---|---|
xs |
240ร384 | 80ร128 | 3.6 MB | lowest-end devices |
small |
224ร384 | 112ร192 | 3.6 MB | Chromebooks, low-end laptops |
medium |
320ร512 | 160ร256 | 13.0 MB | mid-range |
large |
400ร640 | 160ร256 | 14.2 MB | mid-range laptops |
xl |
768ร1280 | 192ร320 | 29.2 MB | Apple Silicon, discrete GPUs |
Every model has a fixed input shape. Each tier ships a landscape and a portrait (H and W swapped)
version. Resize your frame to the canvas size (bilinear) before inference; the network does not need
the exact aspect ratio.
Files
onnx/<tier>/matting_<orientation>.onnx canvas RGB -> refined alpha (what the SDK renders)
onnx/<tier>/matting_base_<orientation>.onnx base-res RGB -> alpha (faster, softer edges)
onnx/<tier>/face_<orientation>.onnx canvas RGB -> 5 face-keypoint heatmaps
onnx/<tier>/flow_<orientation>.onnx two canvas frames -> backward optical flow
onnx/landmark_mesh.onnx 256x256 face crop -> 478 face-mesh points (shared by all tiers)
onnx/manifest.json shapes, sha256, parity vs PyTorch
All inputs are RGB in [0, 1], NCHW float32, batch 1. No mean/std normalization needed (the landmark model's ImageNet normalization is built into the graph).
| Graph | Input | Output |
|---|---|---|
matting |
rgb [1,3,H,W] at canvas res |
alpha [1,1,H,W], in [0,1] |
matting_base |
rgb [1,3,h,w] at base res |
alpha [1,1,h,w], in [0,1] |
face |
rgb [1,3,H,W] at canvas res |
heatmaps [1,5,h',w'], probabilities (sigmoid applied). Channels: left eye, right eye, nose, left mouth corner, right mouth corner. Grid is base/8 (medium, large, xl) or base/4 (xs, small) |
flow |
frames [1,6,H,W] at canvas res: previous frame in channels 0โ2, current frame in 3โ5 |
flow [1,2,h',w'] backward flow (dx, dy): for each point on the current frame's grid, where it was in the previous frame, in base-resolution pixels. Grid is base/4 (base/2 on xs). Use it to warp the previous alpha onto the current frame |
landmark_mesh |
rgb [1,3,256,256] face crop |
landmarks [1,478,2], (x, y) in crop coordinates, 0โ1 |
Quick start (Python)
import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from PIL import Image
path = hf_hub_download("Longpipe/longpipe", "onnx/large/matting_landscape.onnx")
sess = ort.InferenceSession(path)
H, W = sess.get_inputs()[0].shape[2:] # 400, 640
frame = Image.open("me.jpg").convert("RGB")
x = np.asarray(frame.resize((W, H), Image.BILINEAR), np.float32).transpose(2, 0, 1)[None] / 255
alpha = sess.run(None, {"rgb": x})[0][0, 0] # [H, W] in [0, 1]
# composite onto a background at the original resolution
a = np.asarray(Image.fromarray((alpha * 255).astype(np.uint8)).resize(frame.size, Image.BILINEAR)) / 255
bg = np.zeros_like(np.asarray(frame), dtype=np.float32); bg[..., 1] = 200 # green
out = np.asarray(frame) * a[..., None] + bg * (1 - a[..., None])
Image.fromarray(out.astype(np.uint8)).save("composited.png")
Face crop for the landmark model
The 478-point mesh model expects the crop the SDK makes from the face heatmaps:
- For each of the 5 heatmap channels, find the peak, then refine it with a soft-argmax over a 7ร7 window (ยฑ3 cells) around the peak. Don't use the hard argmax alone; it jitters.
- Take the bounding box of the 5 refined points (in heatmap cells). The crop is a square centered
on that box, with side = 2.4 ร max(box width, box height). Map the center by
(cell + 0.5) / grid size. - Skip the face if the weakest of the 5 peaks is below 0.15.
- Resample that square to 256ร256 (bilinear) and run
landmark_mesh.
Video: temporal stability (how the SDK uses flow)
The matting networks are static (one frame in, one alpha out). The SDK gets temporally stable mattes from the flow head: each frame it predicts backward flow from the current frame into the previous one, warps the previous stabilized alpha along that flow, and blends it with the fresh prediction through a flow-gated stabilizer (gated on flow magnitude and divergence, to avoid ghosting at occlusion edges). On lower tiers the matting network runs only every 2nd or 3rd frame and the held alpha is warped forward in between. The face heatmaps get the same treatment. This blending logic lives in the SDK (TypeScript/WGSL), not in these graphs. For a single-file temporal step, see the community port below.
Notes
- Exported with ONNX opset 17 (
large: opset 18). Tested with ONNX Runtime 1.23 (CPU). - Each
face/flow/mattingfile contains its own copy of the tier's encoder. The SDK shares one encoder pass across all heads; if you need that, run the SDK. - Community port: imbcmdth/longpipe-onnx converted the same weights by reconstructing them from the SDK's packed GPU format (xs/small/large/xl), and includes a single-graph temporal step with the flow stabilizer. Thanks! This repo is the official export, made directly from the training checkpoints, covering all five tiers in both orientations.
Links
- Website and docs: longpipe.dev ยท architecture
- SDK: github.com/sb2702/longpipe ยท npm
- Built by the founder of Vectorly (effects SDK, acquired 2021). Also used for people-mode background removal in katana.video/remove-background.