Package: resuber
ReSuberReSuber is an automatic tool to re-synchronize any SRT subtitles, using its corresponding vocal audio WAV stream from a movie.It uses machine learning techniques to perform language agnostic re-synchronization, by checking the signal correlation between the vocal audio stream and the corrected subtitle signal.ReSuber also provide different utilities, including vocal audio extraction, subtitle/video file merging into containers and automatic translation of SRT subtitles with the Google Cloud Translation API (no API key required).LicenseUsageThe main tool of this toolbox,resuber, re-synchronize any.srtsubtitle file, given its corresponding.wavaudio vocal file from a movie.
For the complete list of tools, you should readthis paragraph.First useGiven any directory with the following file-structure (and same name):|-examples
  |  |-movie_example.fr.srt
  |  |-movie_example.eng.srt
  |  |-movie_example.wavSimply runresuberfrom within the directory:cd examples
resuberAfter completion, each subtitle will be re-synchronized and saved into new files*.resubed.srt:|-examples
  |  |-movie_example.fr.srt
  |  |-movie_example.eng.srt
  |  |-movie_example.wav
  |  |-movie_example.fr.srt.resubed.srt
  |  |-movie_example.eng.srt.resubed.srtIf you don't have the audio track for your movie, followthis section.Input subtitle language extensions must followISO_639-1.Advanced useArgumentsusage: resuber [-h] [--debug] [--input-dir INPUT_DIR]
               [--output-dir OUTPUT_DIR] [--recursive]
               [--subtitles [SUBTITLES [SUBTITLES ...]]]
               [--vocals [VOCALS [VOCALS ...]]] [--refine {no,mask,sample}]
               [--fs FS] [--start START] [--end END]
               [--range-weight [RANGE_WEIGHT [RANGE_WEIGHT ...]]]
               [--range-offset [RANGE_OFFSET [RANGE_OFFSET ...]]]
               [--fix-weight] [--fix-offset] [--max-shift MAX_SHIFT]
               [--min-clusters-distance MIN_CLUSTERS_DISTANCE]
               [--encoding ENCODING] [--version]

optional arguments:
  -h, --help            show this help message and exit
  --debug               enable debugging information (default: False)
  --input-dir INPUT_DIR
                        input directory to the SRT subtitle and WAV audio
                        file(s) (default: .)
  --output-dir OUTPUT_DIR
                        output dir for the corrected SRT subtitle file
                        (default: input_dir)
  --recursive           allow recursive search if vocals and/or subtitles are
                        not specified (default: False)
  --subtitles [SUBTITLES [SUBTITLES ...]]
                        input filename(s) to the SRT subtitle file(s) per WAV
                        vocal file (default: ./*.srt)
  --vocals [VOCALS [VOCALS ...]]
                        input filename(s) to the WAV vocal file(s) (default:
                        ./*.wav)
  --refine {no,mask,sample}
                        mask (cluster-wise), sample-wise or no non-linear
                        refinement of the subtitle signal (default: 'no')
  --fs FS               sampling rate in Hz (default: 100 Hz)
  --start START         Minimum timestamp to process with format 'h:m:s.ms'
                        (default: '0:0:0.0')
  --end END             Maximum timestamp to process with format 'h:m:s.ms
                        (default: '24:60:60.99')
  --range-weight [RANGE_WEIGHT [RANGE_WEIGHT ...]]
                        range allowed for the weight parameter during rough
                        exploration (default: [-1e-2, 1e-2])
  --range-offset [RANGE_OFFSET [RANGE_OFFSET ...]]
                        range allowed in ms for the offset parameter during
                        rough exploration (default: [-5000., 5000.])
  --fix-weight          disable the optimization of the weight parameter
                        (default: False)
  --fix-offset          disable the optimization of the offset parameter
                        (default: False)
  --max-shift MAX_SHIFT
                        if non-linear refinement is allowed, define the
                        maximum acceptable shift in ms (default: 500 ms)
  --min-clusters-distance MIN_CLUSTERS_DISTANCE
                        if masked non-linear refinement is allowed, specify
                        minimal distance allowed between clusters in ms
                        (default: 10000 ms)
  --encoding ENCODING   encoding for subtitles (default: utf-8)
  --version             show program's version number and exitCustomize the AIresubermakes the assumption that the subtitles are "linearly" de-synchronized with the audio (i.e. it exists a function such as y = α x + β). This is typically the case for subtitles that were created from a movie with a different framerate.There are several parameters that allows you to have some control over the fitting algorithm, more specifically the scaling α and offset parameter β.You can first change the range for the initial parameter search with--range-weightand--range-offset. This can be usefull if you know approximately how your inputs are de-synchronized.It is also possible to disable the gradient-descent loop with--fix-weightand--fix-offset. In this case, the algorithm simply make a standard search in the parameter space (defined by--range-*).Though I would recommend to usepysubs2for that, you can also edit the subtitle yourself and completely disable the AI. For example if you know that your subtitle have a fixed delay of +200ms, use:resuber --range-weight -0.0001 0.0001 --range-offset -200.1 -199.9 --fix-weight --fix-offsetDebuggingI advice you to enable the--debugargument, it will create a new folderresuber-debugin which you will find usefull files.|-resuber-debug
  |  |-movie_example.fr
  |  |  |-data.html
  |  |  |-loss.html
  |  |  |-cost_2d.html
  |  |-movie_example.eng_resubed.txt
  |  |-movie_example.eng
  |  |  |-data.html
  |  |  |-loss.html
  |  |  |-cost_2d.html
  |  |-movie_example.fr_resubed.txt*_resubed.txtis the processed subtitle converted into a.txtAudacitylabel file. This is really usefull if you want to load the audio track, and view the subtitles.*.[lang]folders contains intermediate and output from the algorithm:cost_2d.htmlcorrespond to the initial affine parameters (scale and offset) before the gradient-descent. Ideally, the shape of the cost should be gaussian-like (if) linear de-synchronization).data.htmlis a truncated view of the input subtitle signal (blue), target audio (green) and re-synchronized subtitle signal (orange). Re-synchronized subtitle signal should be as much close as possible to the target audio.loss.htmlshow the variation of the fitness (correlation) at each iteration. Ideally, it should be an asymptotic decreasing function.Non-linear refinement (WIP)Whereresuberworks relatively well for linear de-synchronization, it still has trouble for non-linear de-synchronization.If that is the case, you can try to use the--refineargument. Withmask,resuberwill decompose the signal into multiple clusters, and fit the best delay for each cluster (typically the case for anime song opening with varying lengths).samplemeans that each value will have a different delay.Optionnal toolsReSuber is a software with different tools, each serving its own purpose.
Instead of creating a big pipeline (and messing up with dependencies), I made multiple tools that are independent between each other.
To know which tool is best suited for your case, select one of the following section.>> I don't have the audio track from my movie>> I have an original (correct) subtitle, and I want to synchronize another one>> I want to translate my subtitle>> I have a subtitle that I want to merge with my movie fileI don't have the audio track from my movieSpleeterfromDeezeris a deep-learning tool that separates vocal track from an audio track.
This is obviously really usefull here, since the music and sound effects "pollutes" the signal and makes the correlation between the subtitle and audio signal a mess.I made a tool calledspleeter2resuberthat:take the audio track from the.mp4,.mkvor.avimovie fileextract the (human) vocal from the audio withspleeter.After installingspleeterandffmpeg, just runspleeter2resuberinside any folder with:|-examples
  |  |-movie_example.mkv
  |  |-movie_example.frIt will create a new folderresuber, ready to be use. Check also the sub-folderresuber/spleeterfor debuging.I have an original (correct) subtitle, and I want to synchronize another oneIf that is the case and you were not successfull withresuber(for example, the other subtitle is non-linearly dependent on the audio) then you have another solution.You can useresuber-movethat will take the reference timestamp from the (correct) subtitle, and do a nearest neighbor with your other (de-synchronized) subtitle.
That way you will have the text from the (de-synchronized) subtitle, with approximately the reference timestamp of the (correct) subtitle.Given this folder:|-examples
  |  |-movie_example.fr.srt (original)
  |  |-movie_example.eng.srt (de-synchronized)
  |  |-movie_example.wavRun:resuber-move --ref-lang en --tgt-time frYou have several other options to play with.usage: resuber-move [-h] [--input-dir INPUT_DIR]
                    [--ref-lang REFERENCE_LANGUAGE]
                    [--tgt-time TARGET_TIMESTAMPS] [--encoding ENCODING]
                    [--min-dist MIN_DIST] [--drop-far DROP_FAR]

optional arguments:
  -h, --help            show this help message and exit
  --input-dir INPUT_DIR
  --ref-lang REFERENCE_LANGUAGE
  --tgt-time TARGET_TIMESTAMPS
  --encoding ENCODING
  --min-dist MIN_DIST   Maximum delay (ms) to be considered as a near neighbor
                        (default: 2000ms).
  --drop-far DROP_FAR   Drop subtitles that are too far.I want to translate my subtitleHopefully, if you are lazy like me there is thegoogle translate API.With these files:|-examples
  |  |-movie_example.fr.srt
  |  |-movie_example.wavTranslate from french subtitlemovie_example.fr.srtto englishmovie_example.eng.srtwith:resuber-translate --ref-lang fr --tgt-lang enGoogle imposes several limitations with the number of requests (limit of 100 per hour). I managed to process the translation batch by batch to reduce the burden, but be careful to not run this too often (or use a VPN...).usage: resuber-translate [-h] [--input-dir INPUT_DIR]
                         [--ref-lang REFERENCE_LANGUAGE]
                         [--tgt-lang TARGET_LANGUAGE] [--encoding ENCODING]

optional arguments:
  -h, --help            show this help message and exit
  --input-dir INPUT_DIR
  --ref-lang REFERENCE_LANGUAGE
  --tgt-lang TARGET_LANGUAGE
  --encoding ENCODINGI have a subtitle that I want to merge with my movie fileFor this you can usemkvmerge, but I also made a home-made toolresuber-mergethat requiresffmpegto be installed.Given a folder with a movie file (.mp4,.mkvor.avi) and subtitle:|-examples
  |  |-movie_example.fr.srt
  |  |-movie_example.mp4You can merge the subtitle inside a video.mkvfile with:resuber-mergeMake sure to check all the options.usage: resuber-merge [-h] [--input-dir INPUT_DIR]
                     [--output_container OUTPUT_CONTAINER]

optional arguments:
  -h, --help            show this help message and exit
  --input-dir INPUT_DIR
  --output_container OUTPUT_CONTAINERInstallationpippython3 -m pip install resubersrcgit clone https://github.com/polak0v/ReSuber
cd ReSuber
make installUnit testingYou will needpytest, and then run:pytests tests/SupportIf you like my work, you are welcome to donate some bitcoins. :)BTC: 12eAEKU4rgvhLCxvdkxKJYocJdEFRyNrta