cstr/tromr-GGUF

Polyphonic-TrOMR — GGUF (for CrispEmbed)

GGUF conversion of Polyphonic-TrOMR (NetEase, Apache-2.0), an Optical Music Recognition model: a staff-notation image is transcribed into three parallel token streams (rhythm / pitch / lift) which are merged into symbolic music notation, e.g.

clef-F4+keySignature-CM+note-E3_eighth.|note-C4_eighth.+note-E3_sixteenth|note-D4_sixteenth+…

Runs with CrispEmbed — pure C/C++ ggml inference, no Python/PyTorch at runtime, Metal/CUDA/Vulkan capable.

Architecture

• Encoder — timm hybrid ViT: ResNetV2 backbone (StdConv2dSame + GroupNorm, layers [2,3,7], 1→64→256→512→1024, /16) → 1×1 HybridEmbed projection (1024→256) → 4-block ViT (dim 256, 8 heads, cls token, custom 2D positional index).
• Decoder — x-transformers: 12 sublayers (self-attn → cross-attn → GLU-FF) with SIGLU attn-on-attn gating and GEGLU feed-forward, 4 parallel heads (rhythm 260 / pitch 71 / lift 7 / note 2). Autoregressive over the three streams, greedy argmax.

Files

File	Precision	Size
tromr-f32.gguf	F32	86 MB
tromr-q8_0.gguf	Q8_0 (ResNet backbone kept F16 — cast to F16 in-engine anyway)	31 MB

Both decode byte-identically to the reference model on the repository's own example photos. Validated vs the original PyTorch implementation: every stage cosine = 1.0 (backbone, ViT context, all 12 decoder blocks, all 4 logit heads), 100% per-position argmax agreement under teacher forcing.

Usage

# CLI (architecture is auto-detected from the GGUF)
crispembed -m tromr-q8_0.gguf --ocr score.jpg

// Dart / Flutter
final omr = CrispEmbedOmr('tromr-q8_0.gguf');
final score = omr.recognizeFile('score.jpg');

Feed a reasonably cropped single staff-system image (a plain photo works well — this model is robust on real-world/camera input).

Attribution & license

• Model: Polyphonic-TrOMR, NetEase — https://github.com/NetEase/Polyphonic-TrOMR (Apache-2.0; the img2score_epoch47.pth weights are committed in that repo).
• Paper: TrOMR: Transformer-Based Polyphonic Optical Music Recognition, arXiv:2308.09370.
• This GGUF conversion is redistributed under the same Apache-2.0 license.