BuzzASR

/

hungarian

Like 2

Follow

lemn-lab 21

Épp most futtattam egy tesztet. A Saját sarpba/whisper-hu-large-v3-turbo-finetuned modellem, csak egy lehelletnyivel teljesít gyengébben cserébe kb 2,5* gyorsabb.

modell WER % CER % RTF (3090) megjegyzés
BuzzASR/hungarian (fp16) 9,96 1,84 34,1× legjobb pontosság
nvidia/parakeet-tdt-0.6b-v3 (bf16) 10,45 1,99 43,8× natív szó-időbélyeg
sarpba/whisper-hu-large-v3-turbo-finetuned (fp16) 10,85 1,94 79,9×
nvidia/canary-1b-v2 (bf16) 14,54 3,10 150,3× leggyorsabb; HF-portban nincs időbélyeg
openai/whisper-large-v3 (fp16) 17,80 4,14 22,9×
openai/whisper-large-v3-turbo (fp16) 19,96 4,68 81,6×

q8-ban semmít sem veszít szinte a pontosságából.

Szia 🇭🇺 @spy76573 ! Thank you so much, this made my day! :)

Running fully offline on a phone for everyday Hungarian voice input is exactly the kind of use we hoped for. And nice work getting the model down to just 1.1 GB with Q5_K_M!

Two quick questions:

1. When you say 0.5× the duration of the speech, does a 10-second clip take roughly 5 seconds to transcribe, or closer to 20?
2. Would you be open to sharing your GGUF conversion workflow and a few details about your Offline Voice Input setup?

We're exploring extending BuzzASR with quantized, edge-deployable checkpoints so people can run offline ASR directly on their phones and laptops, and your experience and feedback would be incredibly valuable for that.

Also, if you're attending EMNLP, feel free to reach out to Catherine Arnett (@catherinearnett ), our co-author, who'll be there in person. Unfortunately I couldn't make it this year, but I hope to visit Hungary someday!

Best,
Shivam

@sarpba Szia🇭🇺!! Thank you for running the comparison, really appreciated!

It's great to hear that Q8 loses almost nothing in accuracy, since we're exploring releasing official quantized checkpoints for on-device use. Thank you for sharing that!! However, part of your table seems to have come through garbled on our end. Could you share it again (Hungarian is perfectly fine), along with which test set, metrics and hardware you used?

Also, if you're attending EMNLP, feel free to reach out to Catherine Arnett (@catherinearnett ), our co-author, who'll be there in person. Unfortunately I couldn't make it this year, but I hope to visit Hungary someday!

Best,
Shivam

@vamsin07
Hello. My small test bellow:
sarpba/hu-asr-benchmark-cv22-test

@sarpba Thank you so much for running these benchmarks and sharing the evaluation dataset! :)

It's really encouraging to see BuzzASR/hungarian performing so competitively, especially against newer ASR architectures. And congratulations on your own Hungarian Whisper Turbo fine-tune! The accuracy–speed tradeoff is quite impressive.

I also appreciate the effort you've put into making the evaluation reproducible, including the normalization details and test-set provenance.

I'm currently exploring the possibility of extending BuzzASR with quantized, edge-deployable checkpoints for offline inference on phones and laptops. I am really curious about your observation on Q8 version preserving accuracy .

Would you be interested in exploring FP16 vs. Q8/Q5_K_M evaluations for BuzzASR/hungarian using your benchmark? It could help us understand the accuracy–efficiency tradeoffs and potentially establish a reproducible evaluation setup for future quantized releases.

Thanks again for taking the time to benchmark our work. Really appreciate it!

Best,
Shivam

Two quick questions:

1. When you say 0.5× the duration of the speech, does a 10-second clip take roughly 5 seconds to transcribe, or closer to 20?
2. Would you be open to sharing your GGUF conversion workflow and a few details about your Offline Voice Input setup?

1. I conducted several tests using various apps capable of running this ASR model. I observed significant differences in speed, both in terms of model loading and processing. Currently, the app "Scrib" (available via F-Droid) seems to be the best. After restarting my phone, I transcribed a previously recorded 28-second audio message. The model took 3 seconds to load, and the recognition process took 19.8 seconds—roughly 0.71 times the duration of the audio. I cannot really test real-time voice-to-text dictation effectively; while the accuracy is fine, the processing and text display occur continuously—roughly sentence by sentence—during the dictation process. Once I finish dictating, I have to wait about another 10 seconds for the final word to appear in the input field. This is perfectly acceptable.
   But
   there is a complete lack of punctuation—no commas or periods. Everything I say or transcribe appears as a continuous, unformatted block of text.

2. I’m a total IT novice; I only do this as a hobby.
   But good old ChatGPT patiently guided me through the whole process. Plus, I don’t really know English, but that’s not an issue for AI, of course.
   The main steps were:

• Installing Python.
• Cloning the whisper.cpp and transcribe.cpp repositories via Git, and installing dependencies (transformers, torch, gguf).
• Installing uv, then using it to download the BuzzASR/hungarian model and convert it to the GGUF format.
• Installing CMake using winget, then using it to compile transcribe-quantize.exe.
• Finally, quantizing the model with the command: transcribe-quantize.exe buzzasr-hungarian-f32.gguf buzzasr-hungarian-q5_k_m.gguf --quant Q5_K_M.

Just a heads-up: I was encountering almost all of these programs for the very first time. Needless to say, this entire 6-hour-long procedure was riddled with various unexpected errors—all of which the free version of ChatGPT handled perfectly.

I use it daily because the phone keyboard drives me crazy. For me, it takes much longer to fix all the typos than it does to enter text using this method.

Most likely, I won’t either this year. 😃

Épp most futtattam egy tesztet. A Saját sarpba/whisper-hu-large-v3-turbo-finetuned modellem, csak egy lehelletnyivel teljesít gyengébben cserébe kb 2,5* gyorsabb.

sarpba/whisper-hu-large-v3-turbo-finetuned (fp16) 10,85 1,94 79,9×

Kösz a tippet, ha időm engedi lehet, hogy futok egy kört telefonra quantálással.

Okay, another test—unfortunately, some errors have surfaced. This is a recording of me reading from a free Hungarian book:

Original text (https://mek.oszk.hu/29800/29853/):

Tengerparti népek fiainak nagy utakat tenni — nem sokat
jelent. Akárhány londoni ember nyári szabadságát úgy tölti el, hogy
hajóra ül és elutazik Ausztráliáig meg vissza. Bali szigetén egy
amerikaival akadtam össze, aki akkor utazta körül hetedszer a
földet.

Nálunk, magyaroknál más a helyzet. Tengerünk nincs, orszá-
gunk tőkeszegény. És ha az utolsó 10—15 esztendőben sikerült is
többezer honfitársunknak , filléres társas utazás" keretében eljutni
Európa különböző országaiba, mondjuk Afrika partjaitól az Északi-
tengerig, azért egy nagyobb szabású út legtöbb honfitársunk szá-
mára ma is az elérhetetlen álmok közé tartozik.

Hozzám nagyon jó volt a Gondviselés: lehetővé tette szá-
momra, hogy pontosan egy esztendeig utazva, tel-
jesen megkerüljem földünket. (Gondviselésről beszé-
lek, hiszen Lenau szerint is a gondviselő Isten ajándéka, ha valaki
utazni mehet: Wem Gott will rechte Gunst erweisen, den schickt Er
in die weite Welt.)

Kell-e mondanom, hogy kimondhatatlanul szerencsésnek érez-
tem magamat, s még ma is, hosszú idő után, csak a legnagyobb
gyönyörűséggel tudok visszagondolni utam mindenegyes részletére.

Recognized text:

a tengerparti népek fiainak nagy utakat tenni nem sokat jelent akárhány londoni ember nyári szabadságát úgy tölti el hogy hajóra ül és elutazik ausztráliaig meg visszabali szigetén egy amerikaihova akadtam össze aki akkor utazta körül hetedszer a földet nálunk magyaroknál más a helyzettengerünk nincs országunk tőke szegény és az utolsó tizenöt esztendőben sikerült is több ezer honfitársunknak féléres társas utazás keretében eljutni európa különböző országaiba mondjuk afrika partjáritól az északi tengerig azért egy nagyobb szabású út legtöbb honfitársunk számára ma is az elérhetetlen álmok közé tartozikhozzám nagyon jó volt a gondviselés lehetővé tette számomra hogy pontosan egy esztendeig utazva teljesen megkerüljem a fölünket gondviselésről beszélek hiszen lenaus szerint is a gondviselő isten szándéka ha valaki utazni mehetVem Gott will recht de Gunst erweisen den Schicktern in die weite WeltKellemondanom hogy kimondatlanul szerencsésnek éreztem magamat s még ma is hosszú idő után csak a legnagyobb gyönyörűséggel tudok visszangondolni utam minden egyes részletére

The speech recognition is actually quite good for an offline mobile-based model, but there isn't a single comma, period, question mark, etc. And yet, I made a conscious effort to speak clearly, with proper phrasing and correct intonation. If that were the issue, I would expect it to have "forgotten" the punctuation only occasionally.
Words get run together—most notably where a comma or a period should be.
I tested this using these two programs:
https://f-droid.org/hu/packages/org.scrib.transcriber/
and
https://play.google.com/store/apps/details?id=dev.notune.transcribe

Both produced the exact same errors.

@spy76573 Thank you so much for the detailed answers and this excellent test! Having the original text next to the output makes it very easy to analyze. 🙏

Transcribing a 28-second clip in about 20 seconds, fully offline on a phone, is faster than real time, which is great to hear. Thanks also for the Scrib tip, and honestly, getting from installing Python to a quantized GGUF model on your phone in 6 hours as a hobbyist is impressive!

You've found two separate things:

1. No punctuation or capitalization: you're right, it isn't about how you speak. We trained the models on lowercased transcripts with punctuation removed, so the model never learned to output punctuation. That's how we scoped out our experiments but is fixable either by training on punctuated transcripts or by adding a small punctuation and capitalization step after recognition.

2. Words merged at sentence boundaries ("visszabali", "helyzettengerünk", "tartozikhozzám", "mehetVem"): since this happens in both apps, it's likely on the model side. Whisper normally starts each chunk of transcribed audio with a space so apps can join chunks cleanly, and our fine-tuning on short single-sentence clips may have changed that. We'll look into it on our side.

Would you be willing to share the audio recording of your book reading? Since the text is a free public book, it would be the perfect test case: we can run it through both BuzzASR and the original Whisper model and see exactly where the words get merged. You could upload it here or send it however is easiest for you.

Apart from those two issues, the recognition itself looks very good, with only a handful of real word errors in the whole passage. We're also exploring releasing official quantized/distilled versions so people don't have to go through all those setup steps, and your feedback is exactly what we need for that.

Thanks again for taking the time to test this so carefully!

Shivam