PINEAPPLEANANAS/ClaraVox-models

ClaraVox models

The two models behind ClaraVox, an app that turns atypical speech into clear spoken English.

Folder	What it is	Base model
whisper-sap-v3/	Speech to text, fine-tuned on 384 h of atypical speech from 648 speakers (used by the app)	openai/whisper-small
whisper-sap-v2/	Second speech run, 82 h	openai/whisper-small
whisper-sap-v1/	First speech run, 57 h subset	openai/whisper-small
gec-v2/	Grammar correction that keeps the speaker's meaning (prefix Fix grammar: )	google/flan-t5-base

Speech model results

Word error rate on held-out speakers from the Speech Accessibility Project (never seen in training, 300 clips per group):

Model	ALS	Cerebral palsy	Down syndrome	Parkinson's	Stroke	All
whisper-small, no training	76.3%	48.4%	46.5%	22.1%	42.2%	43.3%
whisper-sap-v1	16.4%	22.8%	14.3%	16.6%	32.5%	19.7%
whisper-sap-v2	16.6%	23.2%	13.8%	15.2%	31.1%	19.0%
whisper-sap-v3	14.1%	19.6%	11.6%	11.9%	36.8%	17.4%

The test set has only 2–6 speakers per group, so the per-group numbers are rough.

On a larger test set of 62 held-out speakers (1,500 clips), v2 scores 14.0% and v3 12.5% (ALS 7.0%, cerebral palsy 17.4%, Down syndrome 19.5%, Parkinson's 7.2%, stroke 17.6%).

Use

from transformers import WhisperProcessor, WhisperForConditionalGeneration, AutoTokenizer, AutoModelForSeq2SeqLM
repo = "PINEAPPLEANANAS/ClaraVox-models"
proc = WhisperProcessor.from_pretrained(repo, subfolder="whisper-sap-v3")
asr = WhisperForConditionalGeneration.from_pretrained(repo, subfolder="whisper-sap-v3")
tok = AutoTokenizer.from_pretrained(repo, subfolder="gec-v2")
gec = AutoModelForSeq2SeqLM.from_pretrained(repo, subfolder="gec-v2")

Training data and credits

• Speech Accessibility Project, University of Illinois Urbana-Champaign. Used under a Data Transfer and Use Agreement. These models are Results under that agreement and contain no SAP audio or transcripts.
• TORGO dysarthric speech database, University of Toronto.
• TalkBank corpora and JFLEG (grammar model).

Code and full training details: https://github.com/TanayMangal/Clara-Vox