BigStitcher interest point embedding: rotation and mirror invariant, stacked extents (rotinv-stack64)
A small 3-D convolutional network that turns a detected interest point (a DoG detection in a light-sheet volume) into a 64-D unit vector describing its surroundings at five scales, invariant to the orientation of the view: two detections of the same physical structure in two views that are rotated or mirrored against each other get similar vectors, so BigStitcher can match points of a multi-view acquisition by cosine similarity. Exported from pcmatch; evaluated inside BigStitcher by its own Java ONNX evaluator, no runtime needed.
The invariance is built in two places. The standardisation inside the graph takes its statistics over the in-image voxels inside the sphere of radius 8 around the cube's centre and sets everything outside the sphere to 0, so a rotated cube standardises to the rotated standardised cube; the training then shows the network rotations up to 180 degrees and reflections. For tiled acquisitions and views in the same orientation use the translation-invariant model JaneliaSciComp/bigstitcher-embedding-translation-stack64, which is more descriptive there.
Files
| file | content |
|---|---|
rotinv-stack_sigma5-30_mask_floor_ds1345678_p2000_s0_local_neg3r500_mirror_ft-allmv.onnx |
the graph, opset 18, 22 operator types (Conv, Relu, MaxPool, Gemm, ReduceL2 and the elementwise ops of the standardisation, with the sphere mask as a constant) |
rotinv-stack_sigma5-30_mask_floor_ds1345678_p2000_s0_local_neg3r500_mirror_ft-allmv.onnx.json |
the sidecar BigStitcher reads: tensor contract, sampling configuration (rotation_invariant: true), training record, fingerprint |
Both files must stay next to each other under these paired names.
Fingerprint: 861ed356e946ace5e099b1b9362d6c0c0560e9149a9c697f9cba96fa66622850 (sha256 of the configuration and all weights). Every array of vectors written with this model carries it as model.fingerprint; BigStitcher refuses to match vectors of different fingerprints, so embed all views of a dataset with one model.
Input and output
Identical to the translation-invariant model, so the same sampling code serves both:
- Input
patches, float32[N, 5, 16, 16, 16], axes N, channel, z, y, x. Channel k is a cube of 16ยณ samples centred on the point with an edge of k ร sigma for k = 5, 8, 13, 20, 30, where sigma is the detection scale of the point in full-resolution pixels. The cube is isotropic in physical units (the voxel size corrects the anisotropy), sampled trilinearly from the pyramid level whose resolution matches the sample spacing, never finer than the level the detection ran at. Raw intensities with any gain or offset; the standardisation is part of the graph. Samples outside the image are NaN and are masked inside the graph. The caller does nothing about the orientation of the view: the cubes are sampled along the view's own axes. - Output
descriptor, float32[N, 64], L2-normalised rows.
Matching
Cosine similarity between the vectors of two views. A pair is a candidate when the best partner's similarity is above 0.3 and passes a ratio test (0.85 in similarity space) against the second best; RANSAC on the candidates gives the view transformation. For views of unknown relative orientation run RANSAC with a rigid or affine model and without a search radius (or with a generous one); the candidates are fewer and less pure than on tiles (see the numbers below), and RANSAC does the rest.
Training
Two stages. Stage 1 (2026-10-31) trained the translation-invariant recipe on the seven tile sets under the rotation regime: rotations up to 180 degrees, reflections with probability 0.5, directional blur applied as one physical blur across the five channels; 4,000 self-supervised steps followed by 10,000 cross-view steps (HardNet, margin 0.15). Stage 2, this release, fine-tuned that model for 12,000 steps on the seven tile sets plus the real rotated view pairs of four multi-view light-sheet datasets: a Drosophila embryo acquired from seven angles (nuclei and fluorescent beads), an expansion-microscopy sample, a C. elegans embryo and a nuclei-stained sample. 99,569 positives in 358 pair groups (corresponding detections of registered view pairs, tolerance 2 px, at most 2,000 per pair); the real rotated pairs with light augmentation only (rotations up to 8 degrees, 6 % affine jitter), the tile pairs with the full regime. InfoNCE loss (temperature 0.1) with hard negatives drawn within 500 px of the anchor (3 per anchor, 2 of them the nearest) inside the loss; batches of 1,024 drawn locally (one view pair per batch); per-point sigma; learning rate 1e-4; seed 0; final loss 1.76. pcmatch commit f5ccf8c, 2026-11-04.
Results
Descriptor-level recall on real rotated view pairs (the nearest descriptor of the other view is the true partner): 21 % (nuclei) and 18 % (beads) on the Drosophila embryo, 63 % on the nuclei-stained sample, 68 % on the expansion-microscopy sample, and 72 % on a multi-view dataset that is in no training set. The stage-1 model reached 3 %, 3 %, 21 %, 30 % and 51 % on the same pairs, the translation-invariant model 0.1 to 2 %.
Under exact transformations the vectors stay put: with one view of the stitching dataset rotated by exactly 90 degrees the model recovers 55 % of the registration truth (59 % unrotated); with that view's cubes reflected along x, y or z 56 % (the translation-invariant model: 86 % unrotated, 0 % rotated or reflected).
On tiles of one orientation it finds 66 to 89 % of the translation-invariant model's true correspondences (for example 1,682 vs 1,946 on the stitching dataset at a 20 px radius, 7,460 vs 10,538 on the mouse head at 500 px), so tile stitching stays with that model.
Scope and limits
- Invariant to rotations and reflections of the view; the price is descriptiveness, see the tile numbers above.
- Trained on beads and nuclei-like blobs detected by a DoG; other structures may embed poorly. The rotated-pair recall depends strongly on the sample (18 to 72 % above).
- Points closer than about 15 sigma to the image border lose part of their largest cubes (masked, not zero-filled); far fewer such points in large volumes.
Export and verification
PyTorch 2.11.0 (dynamo exporter), opset 18. Checked against onnxruntime 1.31.1 on 64 points: maximal absolute difference 5.1e-7, minimal cosine 0.9999999. BigStitcher's pure-Java evaluator reproduces pcmatch's vectors of a 256-point fixture to 6.9e-6.
How to use
- BigStitcher:
Interest Points > Tools > Compute Interest Point Embeddings > Rotation invariant (Hugging Face)lists this model; then match with theEmbeddingsmethod and a rigid or affine transformation model. - BigStitcher-Spark (release 0.2.0 or later):
embed-interestpoints -x dataset.xml -l <label> --model hf://JaneliaSciComp/bigstitcher-embedding-rotinv-stack64, thenmatch-interestpoints -x dataset.xml -l <label> -m EMBEDDINGS ...with a rigid or affine model.
In both, the revision is pinned to the repository's commit at download time and recorded with the vectors; the files are downloaded once into ~/.bigstitcher/models.
BigStitcher, BigStitcher-Spark and pcmatch sample the cubes identically; the stored vectors agree to float rounding.