Package: scraibe
ScrAIbe: Streamlined Conversation Recording with Automated Intelligence Based EnvironmentScrAIbeis a state-of-the-art,PyTorchbased multilingual speech-to-text framework to generate fully automated transcriptions.Beyond transcription, ScrAIbe supports advanced functions, such as speaker diarization and speaker recognition.Designed as a comprehensive AI toolkit, it uses multiple AI models:whisper: A general-purpose speech recognition model.payannote-audio: An open-source toolkit for speaker diarization.The framework utilizes a PyanNet-inspired pipeline, with thePyannotelibrary for speaker diarization andVoxCelebfor speaker embedding.During post-diarization, each audio segment is processed by the OpenAIWhispermodel, in a transformer encoder-decoder structure. Initially, a CNN mitigates noise and enhances speech. Before transcription,VoxLinguaidentifies the language segment, facilitating Whisper's role in both transcription and text translation.The following graphic illustrates the whole pipeline:InstallScrAIbe:The following command will pull and install the latest commit from this repository, along with its Python dependencies.pip install scraibePython version: Python 3.8PyTorch version: Python 1.11.0CUDA version: Cuda-toolkit 11.3.1Important: For thePyannotemodel, you need to be granted access to Hugging Face.
Check thePyannote model pageto get access to the model.Additionally, you need to generate aHugging Face token.UsageWe've developed ScrAIbe with several access points to cater to diverse user needs.Python usageIt enables full control over the functionalities as well as process customization.fromscraibeimportScraibemodel=Scraibe(use_auth_token="hf_yourhftoken")text=model.autotranscribe("audio.wav")print(f"Transcription:\n{text}")TheScraibeClass is taking care of the models being properly loaded. Therefore, you can choose the otherwhispermodels using thewhisper_modelkeyword.
You can also change thepyannotediarization model using thedia_modelkeyword.As input,autoranscribeaccepts every format which is compatible withffmgeg. Examples therefore are.mp4 .mp3 .wav .ogg .flacand many more.To further control the pipeline ofScrAIbeyou can parse almost any keyword you also cloud parsed towardswhisperorpyannoteif you need more option, try to check out the documentations  tows two Frameworks, you might have a good chance that these keywords will work here as well.
Here's are some examples regarding thediarization(which relies on thepyannotepipeline):num_speakersNumber of speakers in the audio filemin_speakersMinimal Number of speakers in the audio filemax_speakersmaximal Number of speakers in the audio fileThen there are arguments about the transcription process, which uses the "whisper" model.languageSpecify the language (list to supported languages)taskcan be justtranscribeortranslate. Iftranslateis selected, the transcribed audio will be translated to English.For example:text = model.autotranscribe("audio.wav", language="german", num_speakers = 2)Scraibealso contains the option to just do a transcriptiontranscription=model.transcribe("audio.wav")or just do a diarization:diarization=model.diarize("audio.wav")Command-line usageNext to the Pyhton interface, you can also run ScrAIbe using the command-line interface:scraibe -f "audio.wav" --hf-token "hf_yourhftoken" --language "german" --num_speakers 2For the full list of options, run:scraibe -hGradio AppThe Gradio App is a user-friendly interface for ScrAIbe. It enables you to run the model without any coding knowledge. Therefore, you can run the app in your browser and upload your audio file, or you can make the Framework avail on your network and run it on your local machine.Running the Gradio App on your local machineTo run the Gradio App on your local machine, just use the following command:scraibe --start_server --port 7860 --hf_token hf_yourhftoken--start_server: Command to start the Gradio App.--port: Flag for connecting the container internal port to the port on your local machine.--hf_token: Flag for entering your personal HuggingFace token in the container.When the app is running, it will show you at which address you can access it.
The default address is:http://127.0.0.1:7860orhttp://0.0.0.0:7860After the app is running, you can upload your audio file and select the desired options.
An example is shown below:Running a Docker containerAnother option to run ScrAIbe is to use a Docker container. This option is especially useful if you want to run the model on a server or if you would like to use the GPU without dealing with CUDA.
After you have installed Docker, you can execute the following commands in the terminal.First, you need to build the Docker image. Therefore, you need to enter your HuggingFace token and the image name.docker build . --build-arg="hf_token=[enter your HuggingFace token] " -t scraibeAfter the image is built, you can run the container with the following command:sudo docker run -it  -p 7860:7860  --name [container name][image name]  --hf_token [enter your HuggingFace token] --start_server-p: Flag for connecting the container internal port to the port on your local machine.--hf_token: Flag for entering your personal HuggingFace token in the container.--start_server: Command to start the Gradio App.Inside the container, thecliis used. Therefore, you can use the same commands as in the command-line interface.Enabling GPU usageTo use the GPU, ensure your Docker installation supports GPU usage.
For further information, check:https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#dockerTo enable GPU usage, you need to add the following flag to thedocker runcommand:docker run -it  -p 7860:7860 --gpus 'all,capabilities=utility'  --name [container name][image name]  --hf_token [enter your HuggingFace token] --start_serverFor further guidance, check:https://blog.roboflow.com/use-the-gpu-in-docker/DocumentationFor further insights, check thedocumentation page.ContributionsWe are happy to have any interest in contributing and about feedback: In order to do that, create an issue with your feedback or feel free to contact us.RoadmapThe following milestones are planned for further releases of ScrAIbe:Model quantizationQuantization to empower memory and computational efficiency.Model fine-tuningIn order to be able to cover a variety of linguistic phenomena.For example, currently ScrAIbe is able to transcribe word by word, but ignores filler words or speech pauses.
These phenomena can be addressed by fine-tuning with the corresponding data.Implementation of LLMsOne example is the implementation of a summarization or extraction model, which enables ScrAIbe to automatically summarize or retrieve the key information out of a generated transcription, which could be the minutes of a meeting.Executable for WindowsContactFor queries contactJacob SchmiederLicenseScrAIbe is licensed under GNU General Public License.AcknowledgmentsSpecial thanks go to the KIDA project and the BMEL (Bundesministerium für Ernährung und Landwirtschaft), especially to the AI Consultancy Team.