Package: insanely-fast-whisper
Insanely Fast WhisperAn opinionated CLI to transcribe Audio files w/ Whisper on-device! Powered by 🤗Transformers,Optimum&flash-attnTL;DR- Transcribe150minutes (2.5 hours) of audio in less than98seconds - withOpenAI's Whisper Large v3. Blazingly fast transcription is now a reality!⚡️Not convinced? Here are some benchmarks we ran on a Nvidia A100 - 80GB 👇Optimisation typeTime to Transcribe (150 mins of Audio)large-v3 (Transformers) (fp32)~31 (31 min 1 sec)large-v3 (Transformers) (fp16+batching [24]+bettertransformer)~5 (5 min 2 sec)large-v3 (Transformers) (fp16+batching [24]+Flash Attention 2)~2 (1 min 38 sec)distil-large-v2 (Transformers) (fp16+batching [24]+bettertransformer)~3 (3 min 16 sec)distil-large-v2 (Transformers) (fp16+batching [24]+Flash Attention 2)~1 (1 min 18 sec)large-v2 (Faster Whisper) (fp16+beam_size [1])~9.23 (9 min 23 sec)large-v2 (Faster Whisper) (8-bit+beam_size [1])~8 (8 min 15 sec)P.S. We also ran the benchmarks on aGoogle Colab T4 GPUinstance too!P.P.S. This project originally started as a way to showcase benchmarks for Transformers, but has since evolved into a lightweight CLI for people to use. This is purely community driven. We add whatever community seems to have a strong demand for!🆕 Blazingly fast transcriptions via your terminal! ⚡️We've added a CLI to enable fast transcriptions. Here's how you can use it:Installinsanely-fast-whisperwithpipx(pip install pipxorbrew install pipx):pipxinstallinsanely-fast-whisperNote: Due to a dependency ononnxruntime, Python 3.12 is currently not supported. You can force a Python version (e.g. 3.11) by adding--python python3.11to the command.Run inference from any path on your computer:insanely-fast-whisper--file-name<filenameorURL>Note: if you are running on macOS, you also need to add--device-id mpsflag.🔥 You can runWhisper-large-v3w/Flash Attention 2from this CLI too:insanely-fast-whisper--file-name<filenameorURL>--flashTrue🌟 You can rundistil-whisperdirectly from this CLI too:insanely-fast-whisper--model-namedistil-whisper/large-v2--file-name<filenameorURL>Don't want to installinsanely-fast-whisper? Just usepipx run:pipxruninsanely-fast-whisper--file-name<filenameorURL>[!NOTE]
The CLI is highly opinionated and only works on NVIDIA GPUs & Mac. Make sure to check out the defaults and the list of options you can play around with to maximise your transcription throughput. Runinsanely-fast-whisper --helporpipx run insanely-fast-whisper --helpto get all the CLI arguments along with their defaults.CLI OptionsTheinsanely-fast-whisperrepo provides an all round support for running Whisper in various settings. Note that as of today 26th Nov,insanely-fast-whisperworks on both CUDA and mps (mac) enabled devices.-h, --help            show this help message and exit
  --file-name FILE_NAME
                        Path or URL to the audio file to be transcribed.
  --device-id DEVICE_ID
                        Device ID for your GPU. Just pass the device number when using CUDA, or "mps" for Macs with Apple Silicon. (default: "0")
  --transcript-path TRANSCRIPT_PATH
                        Path to save the transcription output. (default: output.json)
  --model-name MODEL_NAME
                        Name of the pretrained model/ checkpoint to perform ASR. (default: openai/whisper-large-v3)
  --task {transcribe,translate}
                        Task to perform: transcribe or translate to another language. (default: transcribe)
  --language LANGUAGE   
                        Language of the input audio. (default: "None" (Whisper auto-detects the language))
  --batch-size BATCH_SIZE
                        Number of parallel batches you want to compute. Reduce if you face OOMs. (default: 24)
  --flash FLASH         
                        Use Flash Attention 2. Read the FAQs to see how to install FA2 correctly. (default: False)
  --timestamp {chunk,word}
                        Whisper supports both chunked as well as word level timestamps. (default: chunk)
  --hf_token
                        Provide a hf.co/settings/token for Pyannote.audio to diarise the audio clipsFrequently Asked QuestionsHow to correctly install flash-attn to make it work withinsanely-fast-whisper?Make sure to install it viapipx runpip insanely-fast-whisper install flash-attn --no-build-isolation. Massive kudos to @li-yifei for helping with this.How to solve anAssertionError: Torch not compiled with CUDA enablederror on Windows?The root cause of this problem is still unknown, however, you can resolve this by manually installing torch in the virtualenv likepython -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121. Thanks to @pto2k for all tdebugging this.How to avoid Out-Of-Memory (OOM) exceptions on Mac?Thempsbackend isn't as optimised as CUDA, hence is way more memory hungry. Typically you can run with--batch-size 4without any issues (should use roughly 12GB GPU VRAM). Don't forget to set--device mps.How to use Whisper without a CLI?All you need to run is the below snippet:pip install --upgrade transformers optimum accelerateimporttorchfromtransformersimportpipelinefromtransformers.utilsimportis_flash_attn_2_availablepipe=pipeline("automatic-speech-recognition",model="openai/whisper-large-v3",# select checkpoint from https://huggingface.co/openai/whisper-large-v3#model-detailstorch_dtype=torch.float16,device="cuda:0",# or mps for Mac devicesmodel_kwargs={"use_flash_attention_2":is_flash_attn_2_available()},)ifnotis_flash_attn_2_available():# enable flash attention through pytorch sdpapipe.model=pipe.model.to_bettertransformer()outputs=pipe("<FILE_NAME>",chunk_length_s=30,batch_size=24,return_timestamps=True,)outputsAcknowledgementsOpenAI Whisperteam for open sourcing such a brilliant check point.Hugging Face Transformers team, specificallyArthur,Patrick,Sanchit&Yoach(alphabetical order) for continuing to maintain Whisper in Transformers.Hugging FaceOptimumteam for making the BetterTransformer API so easily accessible.Patrick Arminiofor helping me tremendously to put together this CLI.Community showcase@ochen1 created a brilliant MVP for a CLI here:https://github.com/ochen1/insanely-fast-whisper-cli(Try it out now!)@arihanv created an app (Shush) using NextJS (Frontend) & Modal (Backend):https://github.com/arihanv/Shush(Check it outtt!)@kadirnar created a python package on top of the transformers with optimisations:https://github.com/kadirnar/whisper-plus(Go go go!!!)