Package: whisperx
This repository provides fast automatic speech recognition (70x realtime with large-v2) with word-level timestamps and speaker diarization.⚡️ Batched inference for 70x realtime transcription using whisper large-v2🪶faster-whisperbackend, requires <8GB gpu memory for large-v2 with beam_size=5🎯 Accurate word-level timestamps using wav2vec2 alignment👯‍♂️ Multispeaker ASR using speaker diarization frompyannote-audio(speaker ID labels)🗣️ VAD preprocessing, reduces hallucination & batching with no WER degradationWhisperis an ASR modeldeveloped by OpenAI, trained on a large dataset of diverse audio. Whilst it does produces highly accurate transcriptions, the corresponding timestamps are at the utterance-level, not per word, and can be inaccurate by several seconds. OpenAI's whisper does not natively support batching.Phoneme-Based ASRA suite of models finetuned to recognise the smallest unit of speech distinguishing one word from another, e.g. the element p in "tap". A popular example model iswav2vec2.0.Forced Alignmentrefers to the process by which orthographic transcriptions are aligned to audio recordings to automatically generate phone level segmentation.Voice Activity Detection (VAD)is the detection of the presence or absence of human speech.Speaker Diarizationis the process of partitioning an audio stream containing human speech into homogeneous segments according to the identity of each speaker.New🚨1st place atEgo4d transcription challenge🏆WhisperXaccepted at INTERSPEECH 2023v3 transcript segment-per-sentence: using nltk sent_tokenize for better subtitlting & better diarizationv3 released, 70x speed-up open-sourced. Using batched whisper withfaster-whisperbackend!v2 released, code cleanup, imports whisper library VAD filtering is now turned on by default, as in the paper.Paper drop🎓👨‍🏫! Please see ourArxiV preprintfor benchmarking and details of WhisperX. We also introduce more efficient batch inference resulting in large-v2 with *60-70x REAL TIME speed.Setup ⚙️Tested for PyTorch 2.0, Python 3.10 (use other versions at your own risk!)GPU execution requires the NVIDIA libraries cuBLAS 11.x and cuDNN 8.x to be installed on the system. Please refer to theCTranslate2 documentation.1. Create Python3.10 environmentconda create --name whisperx python=3.10conda activate whisperx2. Install PyTorch, e.g. for Linux and Windows CUDA11.8:conda install pytorch==2.0.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidiaSee other methodshere.3. Install this repopip install git+https://github.com/m-bain/whisperx.gitIf already installed, update package to most recent commitpip install git+https://github.com/m-bain/whisperx.git --upgradeIf wishing to modify this package, clone and install in editable mode:$ git clone https://github.com/m-bain/whisperX.git
$ cd whisperX
$ pip install -e .You may also need to install ffmpeg, rust etc. Follow openAI instructions herehttps://github.com/openai/whisper#setup.Speaker DiarizationToenable Speaker Diarization, include your Hugging Face access token (read) that you can generate fromHereafter the--hf_tokenargument and accept the user agreement for the following models:SegmentationandSpeaker-Diarization-3.1(if you choose to use Speaker-Diarization 2.x, follow requirementshereinstead.)NoteAs of Oct 11, 2023, there is a known issue regarding slow performance with pyannote/Speaker-Diarization-3.0 in whisperX. It is due to dependency conflicts between faster-whisper and pyannote-audio 3.0.0. Please seethis issuefor more details and potential workarounds.Usage 💬 (command line)EnglishRun whisper on example segment (using default params, whisper small) add--highlight_words Trueto visualise word timings in the .srt file.whisperx examples/sample01.wavResult usingWhisperXwith forced alignment to wav2vec2.0 large:https://user-images.githubusercontent.com/36994049/208253969-7e35fe2a-7541-434a-ae91-8e919540555d.mp4Compare this to original whisper out the box, where many transcriptions are out of sync:https://user-images.githubusercontent.com/36994049/207743923-b4f0d537-29ae-4be2-b404-bb941db73652.movFor increased timestamp accuracy, at the cost of higher gpu mem, use bigger models (bigger alignment model not found to be that helpful, see paper) e.g.whisperx examples/sample01.wav --model large-v2 --align_model WAV2VEC2_ASR_LARGE_LV60K_960H --batch_size 4To label the transcript with speaker ID's (set number of speakers if known e.g.--min_speakers 2--max_speakers 2):whisperx examples/sample01.wav --model large-v2 --diarize --highlight_words TrueTo run on CPU instead of GPU (and for running on Mac OS X):whisperx examples/sample01.wav --compute_type int8Other languagesThe phoneme ASR alignment model islanguage-specific, for tested languages these models areautomatically picked from torchaudio pipelines or huggingface.
Just pass in the--languagecode, and use the whisper--model large.Currently default models provided for{en, fr, de, es, it, ja, zh, nl, uk, pt}. If the detected language is not in this list, you need to find a phoneme-based ASR model fromhuggingface model huband test it on your data.E.g. Germanwhisperx --model large-v2 --language de examples/sample_de_01.wavhttps://user-images.githubusercontent.com/36994049/208298811-e36002ba-3698-4731-97d4-0aebd07e0eb3.movSee more examples in other languageshere.Python usage  🐍importwhisperximportgcdevice="cuda"audio_file="audio.mp3"batch_size=16# reduce if low on GPU memcompute_type="float16"# change to "int8" if low on GPU mem (may reduce accuracy)# 1. Transcribe with original whisper (batched)model=whisperx.load_model("large-v2",device,compute_type=compute_type)# save model to local path (optional)# model_dir = "/path/"# model = whisperx.load_model("large-v2", device, compute_type=compute_type, download_root=model_dir)audio=whisperx.load_audio(audio_file)result=model.transcribe(audio,batch_size=batch_size)print(result["segments"])# before alignment# delete model if low on GPU resources# import gc; gc.collect(); torch.cuda.empty_cache(); del model# 2. Align whisper outputmodel_a,metadata=whisperx.load_align_model(language_code=result["language"],device=device)result=whisperx.align(result["segments"],model_a,metadata,audio,device,return_char_alignments=False)print(result["segments"])# after alignment# delete model if low on GPU resources# import gc; gc.collect(); torch.cuda.empty_cache(); del model_a# 3. Assign speaker labelsdiarize_model=whisperx.DiarizationPipeline(use_auth_token=YOUR_HF_TOKEN,device=device)# add min/max number of speakers if knowndiarize_segments=diarize_model(audio)# diarize_model(audio, min_speakers=min_speakers, max_speakers=max_speakers)result=whisperx.assign_word_speakers(diarize_segments,result)print(diarize_segments)print(result["segments"])# segments are now assigned speaker IDsDemos 🚀If you don't have access to your own GPUs, use the links above to try out WhisperX.Technical Details 👷‍♂️For specific details on the batching and alignment, the effect of VAD, as well as the chosen alignment model, see the preprintpaper.To reduce GPU memory requirements, try any of the following (2. & 3. can affect quality):reduce batch size, e.g.--batch_size 4use a smaller ASR model--model baseUse lighter compute type--compute_type int8Transcription differences from openai's whisper:Transcription without timestamps. To enable single pass batching, whisper inference is performed--without_timestamps True, this ensures 1 forward pass per sample in the batch. However, this can cause discrepancies the default whisper output.VAD-based segment transcription, unlike the buffered transcription of openai's. In Wthe WhisperX paper we show this reduces WER, and enables accurate batched inference--condition_on_prev_textis set toFalseby default (reduces hallucination)Limitations ⚠️Transcript words which do not contain characters in the alignment models dictionary e.g. "2014." or "£13.60" cannot be aligned and therefore are not given a timing.Overlapping speech is not handled particularly well by whisper nor whisperxDiarization is far from perfect (working on this with custom model v4 -- see contact me).Language specific wav2vec2 model is neededContribute 🧑‍🏫If you are multilingual, a major way you can contribute to this project is to find phoneme models on huggingface (or train your own) and test them on speech for the target language. If the results look good send a pull request and some examples showing its success.Bug finding and pull requests are also highly appreciated to keep this project going, since it's already diverging from the original research scope.TODO 🗓Multilingual initAutomatic align model selection based on language detectionPython usageIncorporating  speaker diarizationModel flush, for low gpu mem resourcesFaster-whisper backendAdd max-line etc. see (openai's whisper utils.py)Sentence-level segments (nltk toolbox)Improve alignment logicupdate examples with diarization and word highlightingSubtitle .ass output <- bring this back (removed in v3)Add benchmarking code (TEDLIUM for spd/WER & word segmentation)Allow silero-vad as alternative VAD optionImprove diarization (word level).Harder than first thought...Contact/Support 📇Contactmaxhbain@gmail.comfor queries. WhisperX v4 development is underway with with siginificantly improved diarization. To support v4 and get early access, get in touch.Buy Me A CoffeeAcknowledgements 🙏This work, and my PhD, is supported by theVGG (Visual Geometry Group)and the University of Oxford.Of course, this is builds onopenAI's whisper.
Borrows important alignment code fromPyTorch tutorial on forced alignmentAnd uses the wonderful pyannote VAD / Diarizationhttps://github.com/pyannote/pyannote-audioValuable VAD & Diarization Models from [pyannote audio][https://github.com/pyannote/pyannote-audio]Great backend fromfaster-whisperandCTranslate2Those who havesupported this work financially🙏Finally, thanks to the OScontributorsof this project, keeping it going and identifying bugs.CitationIf you use this in your research, please cite the paper:@article{bain2022whisperx,title={WhisperX: Time-Accurate Speech Transcription of Long-Form Audio},author={Bain, Max and Huh, Jaesung and Han, Tengda and Zisserman, Andrew},journal={INTERSPEECH 2023},year={2023}}