TuesdayCrowd/transkun-onnx

Transkun — transformer-only ONNX export + decode spec

A self-contained ONNX export of Transkun (Yujia Yan's Neural Semi-CRF piano transcriber, 0.984 MAESTRO note F1) that runs the full model in-process with no Python/PyTorch at runtime. This is not a drop-in .onnx transcriber: torch.fft.rfft and the custom semi-CRF backtracking are not ONNX-exportable, so the mel front end and the Viterbi decode are provided as a documented decode spec + a reference decoder.

Attribution. The model, weights and architecture are the work of Yujia Yan, Frank Cwitkowitz and Zhiyao Duan ("Skipping the Frame-Level: Event-Based Piano Transcription with Neural Semi-CRFs", NeurIPS 2021). This is an independent export + decode spec, not affiliated with or endorsed by the authors. Upstream: https://github.com/Yujia-Yan/Skipping-The-Frame-Level. License: MIT (© 2021 Yujia Yan).

What's in the package

File	Role
transkun.onnx (~53 MB, opset 17)	featuresBatch [1,T,229,6] → (S [T,T,90], ctx [90,T,256]) — the transformer + semi-CRF scorer + backbone features
transkun-heads.onnx (~3.4 MB)	attr [N,768] → (velLogits [N,128], ofRaw [N,4]) — velocity + sub-frame onset/offset heads
freq2mels.f32 [2049,229], windows.f32 [6,4096], symbols.i32 [90], params.json	frozen front-end constants
LICENSE.transkun	upstream MIT license
export_transkun.py, export_transkun_heads.py	regeneration scripts (need the transkun PyTorch package)

The decode spec (DECODE_SPEC.md) documents the mel front end, the S layout, the 90-track symbol map ([-64, -67, 21..108] = sustain/soft pedal + MIDI 21–108), the semi-CRF viterbiBackward, the 16 s/8 s segment stitching, and the attribute heads (velocity = argmax; ofValue = ContinuousBernoulli mean).

Reference decoder + validation

The reference decoder is the C# implementation in audio-claudio (mel front end, SemiCrfViterbi, TranskunTranscriber). It is validated note-identical to the native transkun CLI (PyTorch): on the test clips it reaches 100% note-level F1 at ±25 ms with exact velocity on every note — the export + decode spec reproduce the reference implementation, not merely approximate it.

Pipeline (how to run)

audio (mono, 44.1 kHz)
  → mel front end (framing 4096/1024, 6 windows, rfft ortho, freq2mels, log-norm) → featuresBatch
  → transkun.onnx → (S, ctx)
  → semi-CRF viterbiBackward(S) → per-track note intervals, over 16 s/8 s stitched segments
  → gather ctx at interval endpoints → transkun-heads.onnx → velocity + sub-frame onset/offset
  → notes (+ sustain/soft pedal from tracks 0/1)

See the repo's decode spec and TranskunTranscriber for the exact arithmetic (segment padding, forcedStartPos carry, merge, resolveOverlapping).