AMALIA-SFT-FALA — ONNX (speech-llm)

ONNX export of amalia-llm/AMALIA-SFT-FALA for onnx-asr with the speech-llm model type (requires the speech-llm model-family patches, see TigreGotico/onnx-asr PR #3).

A SLAM-ASR model for European Portuguese: a fine-tuned Whisper large-v3 encoder, a concat projector, and the 9B AMALIA language model.

Three-graph layout: encoder.onnx (30 s of audio -> 300 projected audio embeddings), embed_tokens.onnx, and a merged prefill+decode decoder.onnx with a 42-layer KV cache. The prompt token ids are baked into config.json, so no tokenizer is needed at runtime. The audio embeddings come before the prompt, which is the SLAM-ASR layout. fp32 output matches the native PyTorch checkpoint character-for-character on a FLEURS Portuguese clip.

Hardware requirements — read this first

This is a 9B model in fp32. The graphs total about 36 GB on disk, and loading them needs about 36 GB of memory. On a 12-core CPU it transcribes 11.8 s of audio in 76 s (RTFx 0.16), so a GPU execution provider is needed for anything interactive. Plan for a GPU with 40 GB or more, or expect very slow CPU inference.

An int8 build of every graph is included: encoder_int8.onnx, embed_tokens_int8.onnx and decoder_int8.onnx + decoder_int8.onnx_data. The int8 set is 9.8 GB instead of 36 GB, it loads into about 10 GB of memory, and it transcribes the same 11.8 s clip in 18 s instead of 76 s on the same 12-core CPU (RTFx 0.67).

The weights are per-tensor symmetric int8, the same scheme onnxruntime.quantize_dynamic uses. int8 does change the transcript. On the test clip fp32 gives "Objetivo principal da ciência é entender como o mundo funciona através do método científico esse método que origina e desenvolve as ciências." and int8 gives "Objetivo principal da ciência é entender como o mundo funciona através do método científico, esse método que origina e exige ciências." Use fp32 when accuracy matters more than speed.

Usage

import onnx_asr
model = onnx_asr.load_model("speech-llm", "path/to/this/repo")
print(model.recognize("audio_16khz.wav"))

# or the int8 build, which is four times faster and needs a quarter of the memory
model = onnx_asr.load_model("speech-llm", "path/to/this/repo", quantization="int8")

Files

File	Size
`encoder.onnx` + `encoder.onnx_data`	2.5 GB
`encoder_int8.onnx`	0.6 GB
`embed_tokens.onnx` + `embed_tokens.onnx_data`	2.0 GB
`embed_tokens_int8.onnx`	0.5 GB
`decoder.onnx` + `decoder.onnx_data`	33 GB
`decoder_int8.onnx` + `decoder_int8.onnx_data`	8.0 GB