Become a video analysis expert using python | by Kunal Dhariwal | Towards Data Science

The task of generating highlights for any sport is quite tedious and difficult. It involves a lot of tech and other software, hardware needs. You also need to invest in a lot of time to do it. If you’re into such a video company or department then it is okay, but if you’re a normal user just like me, you wouldn’t want to do it that way. I’ll be using python to get this done in just a few steps. There won’t be any external software involved. Just pure python and Also, we don’t have to rely on machine learning or deep learning techniques to do it. We will be using speech analysis for this task.
When something exciting happens during a game, there is a rise in the commentator’s voice. Let’s take cricket for example. Whenever a batsman hits a boundary or a bowler takes a wicket, there is a rise in the commentator’s voice. Both the audience and the commentators have high pitch during that event. We can use these changes in audio to capture interesting moments from a video.
The process we’ll be following is added down below:
Input the video file
Extract the audio
Break the audio into chunks
Compute short-time energy of every chunk
Classify every chunk as excitement or not (based on a threshold value)
Merge all the excitement-clips to form the video highlights
Generate the final video highlights
Remember, We won’t use any external software here.
1. Download the video input: https://bit.ly/2lKfVa6
You can choose any other video of your favourite sport too.
2. Extract the audio:
For extracting the audio and other operations on the video file, we’ll be using a package called moviepy.
a. Install the package.
pip install moviepy
b. Input the file path in this code.
# AudioExtract.pyimport moviepy.editor as mpclip = mp.VideoFileClip("filepath//videoplayback.mp4").subclip(1, 1380)clip.audio.write_audiofile("filepath//audio.wav")
In the above code, mp.VideoFileClip is where we feed in our video file.
subclip, we specify the the time limit for which we need the audio. *Not required if you want full clip audio*
audio.write_audiofile, we specify the output path and file name here.
That’s it, We have extracted the audio in 2 simple steps.
3. Break the audio into chunks:
We’ll be using the audio that we just extracted in step 2.
librosa is a python package for music and audio analysis.
import librosaimport IPython.display as ipdfilename = "filepath\\audio.wav"# loading the file with a sampling ratex, sr = librosa.load(filename, sr=16000)# To get duration of the audio clip in minutesint(librosa.get_duration(x, sr) / 60)# Dividing into chunks of 5 seconds max_slice = 10window_length = max_slice * sr# Playing the audio chunka = x[21 * window_length:22 * window_length]ipd.Audio(a, rate=sr)
Here, I’ve set the sampling rate as 16000, you can set it as per your convenience and requirement. Now, I have divided the audio into chunks of 10 seconds each, since we want to find out whether a particular audio chunk contains a rise in the audio voice or not. You can set time limit as per your choice, Try different time limits for different outputs. Also, we’ll play the audio chunk just for a check.
4. Compute short-time energy of every chunk:
import numpy as nps_energy = np.array([sum(abs(x[i:i + window_length] ** 2)) for i in range(0, len(x), window_length)])
The energy or power of an audio signal refers to the loudness of the sound. It is computed by the sum of the square of the amplitude of an audio signal in the time domain. When energy is computed for a chunk of an entire audio signal, then it is known as Short Time Energy.
Let’s plot it:
import matplotlib.pyplot as pltplt.hist(s_energy)plt.show()
5. Classify every chunk as excitement or not (based on a threshold value):
import pandas as pddf = pd.DataFrame(columns=['energy', 'start', 'end'])thresh = 180row_index = 0for i in range(len(s_energy)):    value = energy[i]    if value >= thresh:        i = np.where(s_energy == value)[0]        df.loc[row_index, 'energy'] = value        df.loc[row_index, 'start'] = i[0] * 10        df.loc[row_index, 'end'] = (i[0] + 1) * 10        row_index = row_index + 1
I am creating a dataframe based on the energy data. I chose the threshold as 180, you can choose it as per your choice and by seeing the graph above. And then merge consecutive time intervals of audio clips into one:
temp = []i = 0j = 0n = len(df) - 2m = len(df) - 1while (i <= n):    j = i + 1    while (j <= m):        if (df['end'][i] == df['start'][j]):            df.loc[i, 'end'] = df.loc[j, 'end']            temp.append(j)            j = j + 1        else:            i = j            breakdf.drop(temp, axis=0, inplace=True)print(df)
6. Merge all the excitement-clips to form the video highlights
start = np.array(df['start'])end = np.array(df['end'])for i in range(len(df)):    if i != 0:        start_lim = start[i] - 10    else:        start_lim = start[i]    end_lim = end[i]    filename = "highlight" + str(i + 1) + ".mp4"    ffmpeg_extract_subclip("filepath//videofile.mp4", start_lim, end_lim, targetname=filename)
This code will generate the highlights from each of the chunk based on the amplitude and then it will save these files in your system. You are not done yet, You have to merge all 14 files into one to actually create the highlights.
7. Generate the final video highlights
# Final_joined.pyfrom moviepy.editor import VideoFileClip, concatenate_videoclipsclip1 = VideoFileClip("filename\\highlight1.mp4")clip2 = VideoFileClip("filename\\highlight2.mp4")clip3 = VideoFileClip("filename\\highlight3.mp4")clip4 = VideoFileClip("filename\\highlight5.mp4")clip5 = VideoFileClip("filename\\highlight6.mp4")clip6 = VideoFileClip("filename\\highlight7.mp4")clip7 = VideoFileClip("filename\\highlight8.mp4")clip8 = VideoFileClip("filename\\highlight9.mp4")clip9 = VideoFileClip("filename\\highlight10.mp4")clip10 = VideoFileClip("filename\\highlight11.mp4")clip11= VideoFileClip("filename\\highlight12.mp4")clip12 = VideoFileClip("filename\\highlight13.mp4")clip13= VideoFileClip("filename\\highlight14.mp4")clip14= VideoFileClip("filename\\highlight4.mp4")final_clip = concatenate_videoclips([clip1,clip2,clip3,clip4,clip5,clip6,clip7,clip8,clip9,clip9,clip10,clip11,clip12,clip13,clip14])final_clip.write_videofile("filepath\\Final_Highlights.mp4")
We are using concatenate to merge all the files into one. After this the final file will be generated.
Complete code: https://bit.ly/2lE55mi
You can get the video demo here: https://bit.ly/2lDEd5V
If you encounter any error or need any help, you can always comment or ping me on LinkedIn.
LinkedIn: https://bit.ly/2u4YPoF
Reference: https://bit.ly/2kry0tB
I hope that this has helped you to enhance your knowledge base :)
Follow me for more!
Thanks for your read and valuable time!