Longpipe: real-time portrait matting (official ONNX)

Official ONNX exports of the Longpipe models: real-time person matting for video calls and live video (virtual backgrounds, background blur), plus the face-keypoint, face-mesh and optical-flow heads that power touch-up, auto-reframe and temporal stability.

Try it in your browser: longpipe.dev/demo, or the demo Space.

Longpipe is primarily a browser SDK (npm install longpipe, GitHub, live demo) that runs these networks as hand-written WebGPU/WebGL2 shaders. These ONNX files let you use the same trained weights anywhere ONNX Runtime runs: Python, native apps, servers, mobile, or onnxruntime-web.

  • License: MIT, code and weights. Training data provenance: longpipe.dev/docs/licensing.
  • Weights: the SDK's current production weights (weights release v0.0.5, served at cdn.longpipe.dev/models/v/0.0.5/ and used by SDK 0.0.16), exported directly from the PyTorch training checkpoints that produce those files. Every ONNX file is checked against PyTorch (max abs diff โ‰ค 7e-5, most โ‰ค 1e-6); see onnx/manifest.json.

Quality and speed

Average alpha error on the P3M-10K validation set (499 images): every Longpipe tier beats MediaPipe on both MAE and IoU.

Longpipe vs MediaPipe quality on P3M-10K

In the browser, the SDK's zero-copy GPU pipeline is faster than MediaPipe or BodyPix despite running larger models (browser numbers; ONNX Runtime speed depends on your runtime and hardware):

Longpipe vs MediaPipe and BodyPix speed

Tiers

Five tiers trade quality for cost. Each is a small "base" network (EfficientNet-Lite encoder + decoder) wrapped in a high-resolution U-Net wrapper that refines edges at canvas resolution.

Tier Canvas (landscape, Hร—W) Base res matting size Intended hardware
xs 240ร—384 80ร—128 3.6 MB lowest-end devices
small 224ร—384 112ร—192 3.6 MB Chromebooks, low-end laptops
medium 320ร—512 160ร—256 13.0 MB mid-range
large 400ร—640 160ร—256 14.2 MB mid-range laptops
xl 768ร—1280 192ร—320 29.2 MB Apple Silicon, discrete GPUs

Every model has a fixed input shape. Each tier ships a landscape and a portrait (H and W swapped) version. Resize your frame to the canvas size (bilinear) before inference; the network does not need the exact aspect ratio.

Files

onnx/<tier>/matting_<orientation>.onnx       canvas RGB -> refined alpha          (what the SDK renders)
onnx/<tier>/matting_base_<orientation>.onnx  base-res RGB -> alpha                (faster, softer edges)
onnx/<tier>/face_<orientation>.onnx          canvas RGB -> 5 face-keypoint heatmaps
onnx/<tier>/flow_<orientation>.onnx          two canvas frames -> backward optical flow
onnx/landmark_mesh.onnx                      256x256 face crop -> 478 face-mesh points (shared by all tiers)
onnx/manifest.json                           shapes, sha256, parity vs PyTorch

All inputs are RGB in [0, 1], NCHW float32, batch 1. No mean/std normalization needed (the landmark model's ImageNet normalization is built into the graph).

Graph Input Output
matting rgb [1,3,H,W] at canvas res alpha [1,1,H,W], in [0,1]
matting_base rgb [1,3,h,w] at base res alpha [1,1,h,w], in [0,1]
face rgb [1,3,H,W] at canvas res heatmaps [1,5,h',w'], probabilities (sigmoid applied). Channels: left eye, right eye, nose, left mouth corner, right mouth corner. Grid is base/8 (medium, large, xl) or base/4 (xs, small)
flow frames [1,6,H,W] at canvas res: previous frame in channels 0โ€“2, current frame in 3โ€“5 flow [1,2,h',w'] backward flow (dx, dy): for each point on the current frame's grid, where it was in the previous frame, in base-resolution pixels. Grid is base/4 (base/2 on xs). Use it to warp the previous alpha onto the current frame
landmark_mesh rgb [1,3,256,256] face crop landmarks [1,478,2], (x, y) in crop coordinates, 0โ€“1

Quick start (Python)

import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from PIL import Image

path = hf_hub_download("Longpipe/longpipe", "onnx/large/matting_landscape.onnx")
sess = ort.InferenceSession(path)
H, W = sess.get_inputs()[0].shape[2:]                    # 400, 640

frame = Image.open("me.jpg").convert("RGB")
x = np.asarray(frame.resize((W, H), Image.BILINEAR), np.float32).transpose(2, 0, 1)[None] / 255
alpha = sess.run(None, {"rgb": x})[0][0, 0]             # [H, W] in [0, 1]

# composite onto a background at the original resolution
a = np.asarray(Image.fromarray((alpha * 255).astype(np.uint8)).resize(frame.size, Image.BILINEAR)) / 255
bg = np.zeros_like(np.asarray(frame), dtype=np.float32); bg[..., 1] = 200   # green
out = np.asarray(frame) * a[..., None] + bg * (1 - a[..., None])
Image.fromarray(out.astype(np.uint8)).save("composited.png")

Face crop for the landmark model

The 478-point mesh model expects the crop the SDK makes from the face heatmaps:

  1. For each of the 5 heatmap channels, find the peak, then refine it with a soft-argmax over a 7ร—7 window (ยฑ3 cells) around the peak. Don't use the hard argmax alone; it jitters.
  2. Take the bounding box of the 5 refined points (in heatmap cells). The crop is a square centered on that box, with side = 2.4 ร— max(box width, box height). Map the center by (cell + 0.5) / grid size.
  3. Skip the face if the weakest of the 5 peaks is below 0.15.
  4. Resample that square to 256ร—256 (bilinear) and run landmark_mesh.

Video: temporal stability (how the SDK uses flow)

The matting networks are static (one frame in, one alpha out). The SDK gets temporally stable mattes from the flow head: each frame it predicts backward flow from the current frame into the previous one, warps the previous stabilized alpha along that flow, and blends it with the fresh prediction through a flow-gated stabilizer (gated on flow magnitude and divergence, to avoid ghosting at occlusion edges). On lower tiers the matting network runs only every 2nd or 3rd frame and the held alpha is warped forward in between. The face heatmaps get the same treatment. This blending logic lives in the SDK (TypeScript/WGSL), not in these graphs. For a single-file temporal step, see the community port below.

Notes

  • Exported with ONNX opset 17 (large: opset 18). Tested with ONNX Runtime 1.23 (CPU).
  • Each face / flow / matting file contains its own copy of the tier's encoder. The SDK shares one encoder pass across all heads; if you need that, run the SDK.
  • Community port: imbcmdth/longpipe-onnx converted the same weights by reconstructing them from the SDK's packed GPU format (xs/small/large/xl), and includes a single-graph temporal step with the flow stabilizer. Thanks! This repo is the official export, made directly from the training checkpoints, covering all five tiers in both orientations.

Links

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Space using Longpipe/longpipe 1