Package: ffsubsync
FFsubsyncLanguage-agnostic automatic synchronization of subtitles with video, so that
subtitles are aligned to the correct starting point within the video.Turn this:Into this:Helping DevelopmentPlease considersupporting Ukrainerather than donating directly to this project. That said, at the request of
some, you can now help cover my coffee expenses using the Github Sponsors
button at the top, or using the below Paypal Donate button:InstallFirst, make sure ffmpeg is installed. On MacOS, this looks like:brew install ffmpeg(Windows users: make sureffmpegis on your path and can be referenced
from the command line!)Next, grab the package (compatible with Python >= 3.6):pip install ffsubsyncIf you want to live dangerously, you can grab the latest version as follows:pip install git+https://github.com/smacke/ffsubsync@latestUsageffs,subsyncandffsubsyncall work as entrypoints:ffs video.mp4 -i unsynchronized.srt -o synchronized.srtThere may be occasions where you have a correctly synchronized srt file in a
language you are unfamiliar with, as well as an unsynchronized srt file in your
native language. In this case, you can use the correctly synchronized srt file
directly as a reference for synchronization, instead of using the video as the
reference:ffsubsync reference.srt -i unsynchronized.srt -o synchronized.srtffsubsyncuses the file extension to decide whether to perform voice activity
detection on the audio or to directly extract speech from an srt file.Sync IssuesIf the sync fails, the following recourses are available:Try to sync assuming identical video / subtitle framerates by passing--no-fix-framerate;Try passing--gssto usegolden-section searchto find the optimal ratio between video and subtitle framerates (by default,
only a few common ratios are evaluated);Try a value of--max-offset-secondsgreater than the default of 60, in the
event that the subtitles are out of sync by more than 60 seconds (empirically
unlikely in practice, but possible).Try--vad=auditoksinceauditokcan
sometimes work better in the case of low-quality audio than WebRTC's VAD.
Auditok does not specifically detect voice, but instead detects all audio;
this property can yield suboptimal syncing behavior when a proper VAD can
work well, but can be effective in some cases.If the sync still fails, consider trying one of the following similar tools:sc0ty/subsync: does speech-to-text and looks for matching word morphemeskaegi/alass: rust-based subtitle synchronizer with a fancy dynamic programming algorithmtympanix/subsync: neural net based approach that optimizes directly for alignment when performing speech detectionoseiskar/autosubsync: performs speech detection with bespoke spectrogram + logistic regressionpums974/srtsync: similar approach to ffsubsync (WebRTC's VAD + FFT to maximize signal cross correlation)Speedffsubsyncusually finishes in 20 to 30 seconds, depending on the length of
the video. The most expensive step is actually extraction of raw audio. If you
already have a correctly synchronized "reference" srt file (in which case audio
extraction can be skipped),ffsubsynctypically runs in less than a second.How It WorksThe synchronization algorithm operates in 3 steps:Discretize both the video file's audio stream and the subtitles into 10ms
windows.For each 10ms window, determine whether that window contains speech.  This
is trivial to do for subtitles (we just determine whether any subtitle is
"on" during each time window); for the audio stream, use an off-the-shelf
voice activity detector (VAD) like
the one built intowebrtc.Now we have two binary strings: one for the subtitles, and one for the
video.  Try to align these strings by matching 0's with 0's and 1's with
1's. We score these alignments as (# video 1's matched w/ subtitle 1's) - (#
video 1's matched with subtitle 0's).The best-scoring alignment from step 3 determines how to offset the subtitles
in time so that they are properly synced with the video. Because the binary
strings are fairly long (millions of digits for video longer than an hour), the
naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we
use the fact that "scoring all alignments" is a convolution operation and can
be implemented with the Fast Fourier Transform (FFT), bringing the complexity
down to O(n log n).LimitationsIn most cases, inconsistencies between video and subtitles occur when starting
or ending segments present in video are not present in subtitles, or vice versa.
This can occur, for example, when a TV episode recap in the subtitles was pruned
from video. FFsubsync typically works well in these cases, and in my experience
this covers >95% of use cases. Handling breaks and splits outside of the beginning
and ending segments is left to future work (see below).Future WorkBesides general stability and usability improvements, one line
of work aims to extend the synchronization algorithm to handle splits
/ breaks in the middle of video not present in subtitles (or vice versa).
Developing a robust solution will take some time (assuming one is possible).
See#10for more details.HistoryThe implementation for this project was started during HackIllinois 2019, for
which it received anHonorable Mention(ranked in the top 5 projects,
excluding projects that won company-specific prizes).CreditsThis project would not be possible without the following libraries:ffmpegand theffmpeg-pythonwrapper, for extracting raw audio from videoVAD fromwebrtcand thepy-webrtcvadwrapper, for speech detectionsrtfor operating onSRT filesnumpyand, indirectly,FFTPACK, which powers the FFT-based algorithm for fast scoring of alignments between subtitles (or subtitles and video)Other excellent Python libraries likeargparse,rich, andtqdm, not related to the core functionality, but which enable much better experiences for developers and users.LicenseCode in this project isMIT licensed.