Field report: benchmarked on real French meetings vs human reference — shipped as a backend (+ one failure mode you may want to know about)
Thanks for this release — a field report from real-world use, since your benchmarks are mostly ZH/EN and we can add a French data point.
Setup: eight 5-minute windows from a real 3.6-hour French committee meeting (narrowband, up to 9 speakers per window), scored against a professionally produced verbatim transcript. Transformers path, greedy, default prompt, bf16 on one RTX 3090.
Results: 0.413 mean WER — the best text score of our whole benchmark (17 configurations, including Voxtral Mini 3B at 0.427 and whisper large-v3 at 0.437, same scorer and reference). No loops, no language drift, 65-104s per window. Speaker counts: exact on 2/8 windows (9/9, 5/5), close on most (5/6, 6/7), under-counted on chaotic overlapping openings (3/8, 6/11). The "50+ languages" claim holds for French in our data. It's also the only unified transcribe+diarize model we've tested that completes real 5-minute windows without repetition loops — that robustness is the headline for us.
One failure mode worth knowing: on one window out of eight, the model silently skipped 22 seconds mid-monologue (50 words), then resumed verbatim on the next turn. Timestamps stay monotonic, so nothing looks anomalous — invisible to WER-style evaluation, only caught by reading against the reference. It's easy to detect downstream (inter-segment gap threshold) and we shipped that guard with our integration. If the Pro variant addresses omission specifically, that would be worth documenting.
The model shipped as an experimental backend in TranscrIA v0.3.5
( https://github.com/Martossien/transcria/releases/tag/v0.3.5 self-hosted meeting transcription, Apache-2.0)
since our stack is on Transformers 4.x, it runs in an isolated pip install --target site via a subprocess worker, which may be a useful pattern for others in the same situation. Full anonymized write-up: STT benchmark on real French meetings. ( https://github.com/Martossien/transcria/blob/main/docs/STT_BENCHMARK_REAL_MEETINGS.md )
Question for the team: any recommended strategy for long-form (1-3h) audio — chunked windows with overlap, or is there a native path planned?
Thank you for evaluating our model. The issue you raised is indeed very important.
Our model currently supports end-to-end inference on audio up to 90 minutes in a single pass. For longer recordings, we plan to release a practical solution in the future.
Thank you again for your evaluation, feedback, and recognition of our model.