VIAME/Stereo-Template-Matcher

VIAME Stereo Template Matcher

Epipolar template matching for stereo correspondence, exported from VIAME as a single ONNX graph. Given a set of points in the left image of a calibrated stereo pair, it returns the matching points in the right image, plus match scores.

This is VIAME's stereo measurement method 1 (epipolar_template_matching): for each source point it walks the epipolar curve implied by the calibration, sampling candidate depths, and picks the best NCC (TM_CCOEFF_NORMED) template match. Pair it with two-view triangulation to measure real-world lengths — the usual application is measuring fish in stereo camera rigs.

There are no learned weights. The graph is pure geometry plus normalized cross-correlation, which is why it is 89 KB. Nothing here was trained, and there is no training data or evaluation set.

Files

File	Size	Notes
stereo_match.onnx	89 KB	opset 18, IR 8, float32 throughout

Baked-in constants

Two parameters are frozen into the graph at export time and are not runtime inputs:

Constant	Value
template_size	13 (13×13 NCC patch)
num_samples	5000 (depth samples along the epipolar curve)

These match configs/pipelines/interactive_stereo_template.conf in VIAME, which is what the DIVE desktop interactive stereo service loads. Note they are not the export script's own defaults (25 / 5000) — re-exporting without explicit flags produces a model that loads identically but matches differently. To reproduce this exact file:

python plugins/onnx/export_stereo_mapping.py --model match \
    --out stereo_match.onnx --template-size 13 --num-samples 5000

Inputs

All float32. The world frame is the left camera, so the left camera is normally R_left = I, t_left = 0, and the right camera carries the rig's relative pose. Matrices are row-major.

Name	Shape	Meaning
left_gray	[H, W]	Left image, grayscale, 0–255
right_gray	[Hr, Wr]	Right image, grayscale, 0–255
points_left	[P, 2]	Source points (x, y) in left-image pixels
K_left	[3, 3]	Left intrinsics
dist_left	[8]	Left distortion [k1, k2, p1, p2, k3, k4, k5, k6], zero-padded
R_left	[3, 3]	Left rotation (identity in the usual convention)
t_left	[3]	Left translation (zero in the usual convention)
K_right	[3, 3]	Right intrinsics
dist_right	[8]	Right distortion
R_right	[3, 3]	Rig rotation, left → right
t_right	[3]	Rig translation, in calibration units
min_depth	scalar	Near bound of the depth search
max_depth	scalar	Far bound of the depth search

Grayscale must use BT.601 luma (0.299 R + 0.587 G + 0.114 B) to match the OpenCV BGR2GRAY the reference implementation uses.

Outputs

Name	Shape	Meaning
right_points	[P, 2]	Matched points in right-image pixels
best_score	[P]	Best NCC score
second_score	[P]	Best NCC score outside a template_size neighborhood of the winner

Acceptance thresholds

The graph deliberately does not apply a score threshold — it returns the best match unconditionally, so the host decides what to accept. VIAME's two reference hosts disagree, and both are defensible:

Host	Threshold	Uniqueness ratio
interactive_stereo_template.conf (DIVE desktop)	0.5	none
plugins/onnx/run_epipolar_onnx.py	0.2	0.85

The uniqueness test, where used, rejects a match when second_score / best_score > ratio — i.e. the winner was not clearly better than an unrelated candidate elsewhere on the curve. Useful on repetitive texture.

Search range

The graph takes a depth range, but disparity is usually the more natural way to think about it. Convert with:

min_depth = fx * baseline / max_disparity
max_depth = fx * baseline / min_disparity

where fx = K_left[0][0] and baseline = ||t_right||.

A disparity range of 2–300 px matches the DIVE desktop interactive stereo config, but this is scene-dependent: VIAME's batch measurement pipelines ship 7–724 for other rigs. Too wide invites false matches; too narrow misses the target entirely. Calibrate it to how far the same object actually shifts between your two cameras.

Usage

Python

import numpy as np, onnxruntime as ort

sess = ort.InferenceSession("stereo_match.onnx", providers=["CPUExecutionProvider"])
out = sess.run(None, {
    "left_gray":   left.astype(np.float32),      # [H, W], 0-255
    "right_gray":  right.astype(np.float32),
    "points_left": np.array([[330.4, 234.8]], np.float32),
    "K_left": K1, "dist_left": d1, "R_left": np.eye(3, dtype=np.float32),
    "t_left": np.zeros(3, np.float32),
    "K_right": K2, "dist_right": d2, "R_right": R, "t_right": T,
    "min_depth": np.float32(fx * baseline / 300),
    "max_depth": np.float32(fx * baseline / 2),
})
right_points, best_score, second_score = out
accepted = best_score >= 0.5

Browser / Node (onnxruntime-web)

The graph is small and CPU-only, so it runs comfortably in a browser via the WASM execution provider — this is how DIVE warps a detection from one camera to the other with no backend. A complete client-side implementation lives in client/dive-common/use/stereo/ (see Kitware/dive#1709), covering calibration parsing, the search-range conversion above, and two-view triangulation.

Not a transformers.js model

This is a bespoke geometry graph, not a transformer. It has no config.json, no tokenizer or image processor, and no architecture in the transformers.js registry, so pipeline() / AutoModel will not load it. Use onnxruntime (or onnxruntime-web) directly.

Swapping left and right

The graph always matches left → right. To warp a point annotated on the right camera, invert the rig instead of swapping the inputs: the new world frame is the old right camera, so R' = Rᵀ and T' = -Rᵀ·T, with the intrinsics and distortion swapped between sides.

Accuracy

Validated against the VIAME C++ / Python reference implementation to roughly a quarter pixel. Note the exporter's own verification reports a sub-pixel right_points difference between eager PyTorch and onnxruntime (about 0.04 px at template_size=13) caused by ties between equally-scoring epipolar candidates; scores match to ~1e-8.

License and provenance

CC-BY-4.0. Produced by plugins/onnx/export_stereo_mapping.py in VIAME, whose core infrastructure is BSD-3-Clause.