Extract Keywords from Audio files with Natural Language processing(NLP) | by Manmohan Singh | Towards Data Science

The latest version of HuggingFace transformers introduces a model, Wav2Vec 2.0, which has the potential to solve audio-related Natural Language Processing (NLP) tasks. And now, one of the tasks you can solve is how to extract keywords from audio.
A Wav2Vec 2.0 model is an automatic speech recognition model released by the Facebook AI team. When the model is trained on 10 minutes of transcribed speech and 53k hours of unlabeled speech, it has a word error rate (WER) of 8.6% on noisy speech and 5.2% on clean speech. The performance of Wav2Vec 2.0 is checked on the standard LibriSpeech benchmark.
Learn more about the Wav2Vec 2.0 here.
Other NLP tasks that you can solve are automatic summarization, translation, named entity recognition, relationship extraction, sentiment analysis, speech recognition, topic segmentation, etc.
In this article, the NLP task that we will solve is to fetch text from audio and then extract the text’s keywords.
Let’s break these tasks into 3 parts.
1. Audio Preprocessing
2. Predict text from audio
3. Extract keywords from the text
The most frequently used audio format are mp3, mp4, and m4a. But, Wav2Vec 2.0 model is trained on wav format audio. So, it takes a wav format as input. I have used a Pydub library to convert formats that are not a wav format.
Follow the below code to convert audio files into Wav format.
from pydub import AudioSegmentm4a_audio = AudioSegment.from_file(r”dnc-2004-speech.mp3", format=”mp3")m4a_audio.export(“dnc-2004-speech_converted.wav”, format=”wav”)
My laptop can only process 3–4 minutes length of audio at a time. Any audio of more than 3–4 minutes size will throw an out-of-memory error while processing.
I have used the speech of the former president of the United States, Mr. Obama. The audio length is 12 minutes. So, I have broken the 12 minutes length audio into 4 parts to avoid this error, each audio has 3 minutes length. After that, I have used the model to fetch the text from each audio element. And finally, I have combined the text of all audio into one single paragraph.
Follow this code to break the large audio files into smaller ones.
With this, the audio preprocessing part is completed. We will now feed these files into the model to predict the text.
So, let’s execute the second step.
In this step, you will use Wav2vec 2.0 model to get the text from audio. You can achieve this by looping the audio files that earlier we have converted into smaller audio files.
In each loop, predict the text from audio and store it into the collection_of_text variable. At last, add all the text in the collection_of_text variable and store it in the final_complete_speech variable.
You can use the transformers library to import the pre-trained model facebook/wav2vec2-base-960h. This model is fine-tuned on 960 hours of Librispeech speech audio. Others models are also available with or without fine-tuned or with different training hours.
Visit here to download other models.
Also, these models are trained on 16Khz frequency audio. So, input audio should also have a 16 Khz frequency.
Follow this code to complete the second step.
With this, the second part is completed. In the last part, you can extract the keywords from the final_complete_speech variable.
You can use varieties of python libraries to complete this step. But most frequently used python libraries are Spacy, Rake, Bert, and Yake. I prefer spacy for keyword extraction because it is very simple to use. With Spacy, you can complete the third step in only 4 lines of code.
Also, you can refer to this article to use other keyword extraction methods.
Follow this code to extract keywords from the text.
# use this code to install Spacy, Scispacy and en_core_sci_scibert model# pip install -U pip setuptools wheel# pip install -U spacy# pip install scispacy# pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.4.0/en_core_sci_scibert-0.4.0.tar.gzimport spacynlp = spacy.load(“en_core_sci_scibert”)doc = nlp(final_complete_speech.lower())print(doc.ents)
With the improvement in the Transformers library, the NLP field is also growing. Wav2Vec 2.0 model is one of the significant successes for the Facebook AI team and HuggingFace Team. Now, you can successfully extract the transcript from the audio with very low WER.
You can solve automatic summarization, translation, named entity recognition, relationship extraction, sentiment analysis, speech recognition, topic segmentation, etc., NLP task related to audio.
Hopefully, this article will help you to solve the Keyword Extraction task related to audio.