Package: whisper-live
WhisperLiveA nearly-live implementation of OpenAI's Whisper.This project is a real-time transcription application that uses the OpenAI Whisper model
to convert speech input into text output. It can be used to transcribe both live audio
input from microphone and pre-recorded audio files.InstallationInstall PyAudio and ffmpegbashscripts/setup.shInstall whisper-live from pippipinstallwhisper-liveSetting up NVIDIA/TensorRT-LLM for TensorRT backendPlease followTensorRT_whisper readmefor setup ofNVIDIA/TensorRT-LLMand for building Whisper-TensorRT engine.Getting StartedThe server supports two backendsfaster_whisperandtensorrt. If runningtensorrtbackend followTensorRT_whisper readmeRunning the ServerFaster Whisperbackendpython3run_server.py--port9090\--backendfaster_whisper# running with custom modelpython3run_server.py--port9090\--backendfaster_whisper-fw"/path/to/custom/faster/whisper/model"TensorRT backend. Currently, we recommend to only use the docker setup for TensorRT. FollowTensorRT_whisper readmewhich works as expected. Make sure to build your TensorRT Engines before running the server with TensorRT backend.# Run English only modelpython3run_server.py-p9090\-btensorrt\-trt/home/TensorRT-LLM/examples/whisper/whisper_small_en# Run Multilingual modelpython3run_server.py-p9090\-btensorrt\-trt/home/TensorRT-LLM/examples/whisper/whisper_small\-mRunning the ClientInitializing the client:fromwhisper_live.clientimportTranscriptionClientclient=TranscriptionClient("localhost",9090,lang="en",translate=False,model="small",use_vad=False,)It connects to the server running on localhost at port 9090. Using a multilingual model, language for the transcription will be automatically detected. You can also use the language option to specify the target language for the transcription, in this case, English ("en"). The translate option should be set toTrueif we want to translate from the source language to English andFalseif we want to transcribe in the source language.Trancribe an audio file:client("tests/jfk.wav")To transcribe from microphone:client()To transcribe from a HLS stream:client(hls_url="http://as-hls-ww-live.akamaized.net/pool_904/live/ww/bbc_1xtra/bbc_1xtra.isml/bbc_1xtra-audio%3d96000.norewind.m3u8")Browser ExtensionsRun the server with your desired backend as shownhere.Transcribe audio directly from your browser using our Chrome or Firefox extensions. Refer toAudio-Transcription-ChromeandAudio-Transcription-Firefoxfor setup instructions.Whisper Live Server in DockerGPUFaster-Whisperdockerrun-it--gpusall-p9090:9090ghcr.io/collabora/whisperlive-gpu:latestTensorRT. FollowTensorRT_whisper readmein order to setup docker and use TensorRT backend. We provide a pre-built docker image which has TensorRT-LLM built and ready to use.CPUdockerrun-it-p9090:9090ghcr.io/collabora/whisperlive-cpu:latestNote: By default we use "small" model size. To build docker image for a different model size, change the size in server.py and then build the docker image.Future WorkAdd translation to other languages on top of transcription.TensorRT backend for Whisper.ContactWe are available to help you with both Open Source and proprietary AI projects. You can reach us via the Collabora website orvineet.suryan@collabora.comandmarcus.edel@collabora.com.Citations@article{Whispertitle={RobustSpeechRecognitionviaLarge-ScaleWeakSupervision},url = {https://arxiv.org/abs/2212.04356},author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},publisher = {arXiv},year = {2022},}@misc{SileroVAD,author={SileroTeam},title = {Silero VAD: pre-trained enterprise-grade Voice Activity Detector (VAD), Number Detector and Language Classifier},year = {2021},publisher = {GitHub},journal = {GitHub repository},howpublished = {\url{https://github.com/snakers4/silero-vad}},email = {hello@silero.ai}}