Analyzing historical speeches using Amazon Transcribe and Comprehend
By: Gabriel Dos Santos Goncalves
Tags: Sentiment Analysis, Machine Learning, Boto3, NLP, AWS

4. Performing the analysis

Downloading the videos from Youtube

The first step in our analysis is to download the audio files from the five speeches mentioned in the introduction from Youtube. To organize our analysis and make it reproducible we are going to create a Pandas Dataframe to store the data generated during the analysis.

To start, we created a dictionary with the name of each speaker as key and the respective Youtube URLs video as value (1). Next, we iterated over the values and executed youtube-dl command using os.system to download each file to our local folder (2). On the steps (3) we create the Dataframe from the dictionary and a new column with the name of each downloaded file (4).

Audio transcription with Amazon Transcribe

Now that we have the audio files for the analysis, we just need to upload it to an S3 bucket to make the transcription to text. On step (5) we used boto3 to create a new S3 bucket and uploaded the audio files to it (6). In order to start the automatic speech recognition, we need the path to each file located on the new S3 bucket. So by using the S3 path convention we defined the path for each file and save it to our Dataframe (7). On step (8), we create a function called start_transcription to call AWS API using boto3, and invoked it iteratively for each speech (9). We also defined on the function start_transcription to output the results as a JSON file on our S3 bucket.

Comparing the audio transcription with the original text using spaCy

When the Amazon Transcribe jobs are done, you can access the results by reading the JSON file that was saved on the S3 bucket (10). In order to evaluate the quality of the transcription generated by Amazon Transcribe, we are going to use the original texts stored on the JSON file (11). The original transcripts were copied from the link provided on the Art of Manliness article, and new line characters were removed. Some parts of the original text were removed to match exactly the content of the audio, as some were not the complete speech. Before doing any text comparison we are going to remove the stop words, punctuations, pronouns and perform lemmatization on tokens (12). To establish the similarity between the texts we are going to use spaCy method for semantic similarity (13). It’s not a literal comparison of texts, as it is based on Word2vec, meaning we are comparing the percentage of each word over the whole sentence/text.

The results for word vector similarity are plotted in Figure 7. We can see that based on this metric we can say that Amazon Transcribe is doing a good job transcribing the audio to text, as all the transcription were more than 99% similar to the original one.

Figure 7. Word Vectors similarity for the speeches.

As word vectors similarity is based on the representation of each word on text, it can give us a false perception that two texts are alike. In order to have a better metric on the text difference, we are going to use Word Movers Distance to compare the original transcriptions to the one generated using Amazon Transcribe. To estimate this distance we need the library wmd that can be integrated with spaCy (14). For more information on estimating text similarities, please take a look at Adrien Sieg great article.

Figure 8. Word Movers Distance for each analyzed speech.

As we can see in Figure 8, there are some transcriptions that show a larger distance between the original and the audio transcription, with Churchill’s and MacArthur’s speeches being the most evident.

In order to compare the structure of the original transcripts to the ones generated by Amazon Transcribe, we are going to use a spaCy visualizer for entity recognition called displacy (15).

Displacy offers visual identification of words and their respective class, making it easier to compare the structure of your text (Figure 9).