Package: go-subtitle
Go SubtitleQuick start「go_subtitle」是一个从音频或视频中生成字幕的开源项目「go_subtitle」 is an open source project that generates subtitles from audio or videoSetup项目需要运行在包含python3.8.16与ffmpeg4.2.2或更高版本的环境中The project needs runs in an environment containingpython3.8.16andffmpeg4.2.2or later# 查看python版本python--version# 查看ffmpeg版本ffmpeg-version你可以通过下面的命令安装最新的版本You can install the latest version using the following command# 从PyPI安装pipinstall-Ugo_subtitle-ihttps://pypi.org/simple# 从GitHub安装pipinstallgit+https://github.com/RedHeartSecretMan/go_subtitle.gitUsageCommand Line提供两种命令行运行方式，通过指定python模块python -m subtitler [command]或触发关键字go_subtitle [command]Provides two ways to run the command line, by specifying python modulepython -m subtitler [command]or trigger keygo_subtitle [command]# 必须指定输入目录python-msubtiter-i"input_dir"# 可额外指定输出目录（所有路径都可以不使用引号）指定模型类型，指定cpu推理线程数go_subtitle-i./data-o./data/result-mnlarge--threads4你可以命令行快速查看帮助You can quickly view help on the command linego_subtitle-hPython提供了几种简单的使用方法，并且因为项目基于Whisper二次开发所以大多数API可以互相兼容importtorchimportsubtitler# 方式一model=subtitler.load_model("small")avpath=subtitler.get_avpath(["assets/data/test/audios"])audio_path=subtitler.generate_subtitle(model,avpath,**{"task":"translate"})# 方式二# load medium modelmodel=subtitler.load_model("medium")# load audio and pad/trim it to fit 30 secondsaudio=subtitler.load_audio("data/test/audios/Mojito.mp3")audio=subtitler.pad_or_trim(audio)# make log-Mel spectrogram and move to the same device as the modelmel=subtitler.log_mel_spectrogram(audio).to(model.device)# detect the spoken language_,probs=model.detect_language(mel)print(f"Detected language:{max(probs,key=probs.get)}")# decode the audiooptions={"language":max(probs,key=probs.get)}options["fp16"]=Trueiftorch.cuda.is_available()elseFalseoptions=subtitler.DecodingOptions(**options)result=subtitler.decode(model,mel,options)# print the recognized textprint(result.text)# 方式三# load large modelmodel=subtitler.load_model("large")# gain audio stream, load audio and pad/trim it to fit 30 secondsaudio_path=subtitler.extract_audio("data/test/videos/接触.mp4")audio=subtitler.load_audio(audio_path)audio=subtitler.pad_or_trim(audio)# make log-Mel spectrogram and move to the same device as the modelmel=subtitler.log_mel_spectrogram(audio).to(model.device)# detect the spoken language_,probs=model.detect_language(mel)print(f"Detected language:{max(probs,key=probs.get)}")# decode the audiooptions={"language":max(probs,key=probs.get)}options["fp16"]=Trueiftorch.cuda.is_available()elseFalseoptions=subtitler.DecodingOptions(**options)result=subtitler.decode(model,mel,options)# print the recognized textprint(result.text)Planning一、目前翻译功能的目标语言只支持英语，考虑支持中文二、目前缺乏高级功能，后续考虑加入内容摘要功能以及类似沐神的视频自剪辑功能Reference[Blog][Paper][Model card][Colab example]Whisper是一种通用的语音识别模型。它是在各种音频的大型数据集上训练的，也是一个多任务模型，可以执行多语言语音识别以及语音翻译和语言识别Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multi-task model that can perform multilingual speech recognition as well as speech translation and language identificationApproach一个Transformer序列到序列模型训练各种语音处理任务，包括多语言语音识别、语音翻译、口语识别和语音活动检测。所有这些任务都被联合表示为由解码器预测的令牌序列，允许单个模型取代传统语音处理管道的许多不同阶段。多任务训练格式使用一组特殊的标记作为任务说明符或分类目标A Transformer sequence-to-sequence model is trained on various speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. All of these tasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing for a single model to replace many different stages of a traditional speech processing pipeline. The multitask training format uses a set of special tokens that serve as task specifiers or classification targetsAvailable models and languages共有五种尺寸的多模态语言模型，模型尺寸越大准确性越高，速度也越慢。尺寸较小的四种语言模型，还有英文单模态版本，处理英文内容时只需使用base.en模型即可，即准确又快There are a total of five sizes of multi-modal language models, the larger the size of the model, the higher the accuracy, the slower the speed. The smaller size of the four language models, as well as the English single-modal version, only need to use thebase.enmodel when processing English content, that is accurate and fastSizeParametersEnglish-only modelMultilingual modelRequired VRAMRelative speedtiny39 Mtiny.entiny~1 GB~32xbase74 Mbase.enbase~1 GB~16xsmall244 Msmall.ensmall~2 GB~6xmedium769 Mmedium.enmedium~5 GB~2xlarge1550 M\large~10 GB1xlarge模型有两个版本large-v1与large-v2。下图显示了使用large-v2模型处理Fleurs数据集时按语言划分的单词错误率Thelargemodel has two versions large-v1 and  large-v2. The figure below shows word error rates(WER) by language when using the large-v2 model for the Fleurs dataset