Package: vidify-audiosync
Vidify AudiosyncExtension forVidifytosynchronizethe audio fromYouTubeand what'splaying on your computer.InstallationAudiosync is currently only available on Linux. The requirements are:pulseaudioand libpulse.ffmpeg(must be available in the user's path): a software suite to both download the song and record the system's audio.FFTW: the fastest library to compute the discrete Fourier Transform (DFT), which is the most resource-heavy calculation made in this module.You can install the module with pip:pip3 install vidify-audiosync --user.It's also available on the AUR for Arch Linux users asvidify-audiosync.
And in theGURU overlayfor Gentoo Linux users asmedia-video/vidify-audiosync:eselect repository enable guru && emerge --sync guru && emerge vidify-audiosyncUsageThis README is a guide oriented for developing. Please check out theVidify guidefor more information about how to use it with Vidify.Audiosync's main function isaudiosync.run(title: str) -> int, bool. It will return the displacement between the two audio sources (positive or negative), which will only be valid if the returned boolean is true.titleis the track's title to search for in YouTube.After this function has been called, its progress can be monitored and controlled with other exported functions. Here's a brief description for all of them:audiosync.status() -> str: returns the current job's status as a string.audiosync.resume() -> None: continue the audiosync job. This has no effect if it's not paused.audiosync.pause() -> None: pause the audiosync job.audiosync.abort() -> None: abort the audiosync job.audiosync.setup(stream_name: str) -> None: attempts to initialize a dedicated PulseAudio sink to record more easily the audio directly from the music player stream.audiosync.get_debug() -> bool: obtain the current logging levelaudiosync.set_debug(do_debug: bool) -> None: configure the logging levelThis interface is also available from the C library. You can read more details about these exported functions in theinclude/audiosync.h header, and its implementation insrc/audiosync.c.There are 2 apps to try:apps/main.c: used to debug more easily with C. You can run it with:mkdirbuildcdbuild
mkdirimages
cmake..
make-j4
./apps/main"SONG NAME"apps/main.py: equivalent toapps/main.c, but in Python. You can simply usepython main.py "SONG NAME".How it worksI'll try to explain it as clearly as possible, since this took me a lot of effort to understand without prior knowledge about the mathematics behind it. If someone with a better understanding of the calculations performed in this module considers that the explanation could be improved, pleasecreate an issueto let me know.The algorithm insidesrc/cross_correlation.ccalculates the lag between the audio sources (across-correlation). This is used in many mathematical disciplines for different purposes, includingsignal processing. The cross-correlation function describes the points in time where the signals are best aligned:The graph above indicates that the provided signals are most likely to be aligned at the maximum point (at about ~18,000 frames, taking the absolute value of the function). There are many different ways to calculate the cross-correlation of two signals. In this case, precision is sacrificed for speed, because it's going to be calculated multiple times at real-time for fairly big samples (up to ~1,440,000 frames).TheDiscrete Fourier Transformis vital in the used method, which determines the frequency content of the signal.Letrfftbe theFast Fourier Transform(an algorithm to calculate the DFT) in the real domain,irfftthe Inverse Fast Fourier Transform in the real domain, andconjthe conjugate of the imaginary results from the FFT, the formula is:irfft(rfft(source) * conj(rfft(sample)))There are two different audio tracks: the source, and the sample. In this case, the source is the downloaded YouTube audio, and the sample is the recorded audio. Thus, the results will indicate the sample's delay in respect to the source. This is because in Vidify, the offset has to be applied to the music video, which started playing as soon as the new song was detected, and has the same delay as the desktop audio.Because it's unknown which track is the one that's delayed, acircular cross-correlationhas to be performed, rather than a regular cross-correlation. So before applying the formula, one of the signals is filled with zeroes to size 2*N. In this case, the sample is the one filled with zeroes, because it's the one that takes the most to be obtained, since it has to be recorded in real-time. The downloaded audio is usually completed before recording the full interval.After calculating the cross-correlation, a coefficient is needed to determine how accurate the obtained results are, since the provided tracks could be different, in which case no displacement should be applied. ThePearson correlation coefficientwill return a value between -1 and 1, where 1 is total positive linear correlation, 0 is no linear correlation, and −1 is total negative linear correlation. The function calculates is the positive linear correlation, so the closer this coefficient is to 1, the more accurately the signals are aligned. Knowing this, the module will return the first value that exceedsMIN_CONFIDENCE, declared in theaudiosync.hfile.Before applying the coefficient formula, both tracks have to be aligned with the result obtained from the cross correlation. There are many different ways to align the tracks, discussedherein detail. The current method shifts the sample track, and cuts the useless parts of the array filled with zeroes. While this can both improve performance, and obtain more accurate results, it might result in incorrect coefficients due to the result's size being too small. Do note that the alignment isn't actually performed, the Pearson Coefficient is just calculated with two offsets to avoid callingmemmove(see#30for more).Finally, the module will return the lag between the two audio sources if it's confident enough, or otherwise 0.Another important part of the module is the concurrency. Both audio tracks have to be continuously downloaded and recorded while the algorithm is running every N seconds. This means that there are 3 main threads in this program:The main thread: launches and controls the download and capture threads, and runs the algorithm.The download thread: downloads the song with ffmpeg.The audio capture thread: records the desktop audio with ffmpeg.To keep this module somewhat real-time, the algorithm is run in intervals. After one of the threads has successfully obtained the data in the current interval, it sends a signal to the main thread, which is waiting until both threads are done with it. When both signals are recevied, the algorithm is run. If the results obtained are good enough (they have a confidence higher thanMIN_CONFIDENCE), the main thread sets a variable that indicates the rest of the threads to stop, so that it can return the obtained value. Otherwise, it continues to the next interval.DevelopingYou can run the project's tests with:mkdir build && cd build
cmake .. -DBUILD_TESTING=YES
make
make testUseexport CFLAGS="-DPLOT=YESto enable debugging and save plots into the images directory. You'll needgnuplotinstalled for that, and a directory namedimages.-DCMAKE_BUILD_TYPE=Debugenables Address Sanitizer and more helpfuldebug flags.It's recommended to use Docker to run it in a containerized environment. A simple usage example would besudo docker build -t audiosync . && sudo docker run -t audiosync.Feel free to open up an issue or PR in case you have problems with the module or want to contribute. Do take in mind that this project's current status is still very early, so it's not too stable.Documentation links:PulseAudio:reference,docs,guide.