A quick tutorial to AWS Transcribe with Python | by Viet Hoang Tran Duong | Towards Data Science

This past summer, I was working on some products that involve Speech-to-text mechanisms, and I think it’s best to use existed APIs for these purposes. For this post, I want to share my little experience working with these APIs with those who want to try out these wonderful technologies. Hope you guys find it helpful :)
Oh! And here is the quick content I will cover:- Setting Up: General packages and initializes essential functions- Single Speaker Files- Multiple Speakers Files- Accessing and Uploading files through Colab and Google Drive to S3 Storage- Creating a vocabulary to enhance transcription accuracy
Link to the code: Google Colab (here), Gist (here), or Github(here).
Speech-to-text is a promising technology, not by itself as a product, but more on its underlying applications to many other products. As our reading speed is much faster than listening speed, reading the transcription saves more time than listening to audio with similar content. Google’s new flagship phone: Pixel 4, introduces the Recording app that performs real-time transcription! Promising products could be transcribing meetings (Zoom already offers this feature), lectures (text transcripts), and many more.
Over the year, Speech-to-text has been a quite mature field. As this challenge is more horizontal than vertical, companies with massive amounts of data from various input sources win. Without surprise, big corporations like Amazon, Google, IBM, Microsoft are leaders in providing transcription services on their cloud.
Each product has its pros and cons and might suit your purpose differently. I strongly recommend trying out all these services and select the one best perform on your desired use case. For this post, I focus on Amazon Transcription Service because of its rich output: a JSON file with all the timestamps and other information, which is super useful!
(Small hype for my (hopefully) next post: Google Cloud (when performing single speaker transcription): the output does not have any punctuations. I am hoping to write an RNN model to add punctuation to enrich the output. But this is in the future. Let’s go back to AWS for now!)
I will go through the steps using Google Colab. The link to the full code is here. Let’s do this!
!pip install boto3import pandas as pdimport timeimport boto3
Boto is the AWS software development kit for Python. More info on Boto 3 documentation can be found here.
And we also need the access key to our account on AWS. If you haven’t created an account yet, please do so (it’s free to create and have Free Tier is you don’t use too much)! When you have your account, here’s how to get your personal access key (if you already have your access key, then feel free to use it):
Step 1: Go to the AWS Management Console page.
Step 2: Click on your username on the top right, choose “My Security Credentials.”
Step 3: Choose “Access keys (access key ID and secret access key.”
Step 4: Create new keys, and remember to save it!
Step 5: Add to our code: Initialize the transcription job.
transcribe = boto3.client('transcribe',aws_access_key_id = #insert your access key ID here,aws_secret_access_key = # insert your secret access key hereregion_name = # region: usually, I put "us-east-2"
Also, we need to create/connect our Amazon S3 Storage.AWS Transcribe will transcribe files from your S3 Storage. This is very handy because you can store files into Amazon S3 and directly process them from the cloud. Read how to create your S3 Bucket here.
Try to upload a random audio/video file to S3 Storage, and let’s try the transcription service! These are the values we
job_uri: S3 access link, which is usually “s3://bucket_name/” + audio_file_name (e.g. “s3://viethoangtranduong/aws.wav”)
job_name: For each transcription call, we need a job name. For this case, I use the audio file name itself. We can also use a hash function to automate the system.Notes: the job will crash if there already existed a job with the same name. The possible ways to avoid these problems are:- A hash function to encode both the audio file name and the timestamp of the job (this would avoid duplicates, even with files with the same names)- A key generator database: If we use base 62 (as we want to avoid “/” and “+”), then we can have 626 = 56.8 B unique codes (which should be enough). We can use the unused keys for each job.We can have 2 databases to store used and unused keys. Each time an unused key is used, we move it to the other database. We must keep track of the file name and the matched keys for future traversal. Using this method, we can further develop into link shorteners for transcription files.
file_format: the file format. AWS can handle most files like .mp3, .wav, or even videos like .mp4
For easy cases, I created a function call check_job_name to handle duplicate job names.
def check_job_name(job_name):  job_verification = True  # all the transcriptions  existed_jobs = transcribe.list_transcription_jobs()  for job in existed_jobs['TranscriptionJobSummaries']:    if job_name == job['TranscriptionJobName']:      job_verification = False      break   if job_verification == False:    command = input(job_name + " has existed. \nDo you want to override the existed job (Y/N): ")    if command.lower() == "y" or command.lower() == "yes":                transcribe.delete_transcription_job(TranscriptionJobName=job_name)    elif command.lower() == "n" or command.lower() == "no":      job_name = input("Insert new job name? ")      check_job_name(job_name)    else:      print("Input can only be (Y/N)")      command = input(job_name + " has existed. \nDo you want to override the existed job (Y/N): ")  return job_name
For single speaker files
def amazon_transcribe(audio_file_name):  job_uri = # your S3 access link  # Usually, I put like this to automate the process with the file name  # "s3://bucket_name" + audio_file_name    # Usually, file names have spaces and have the file extension like .mp3  # we take only a file name and delete all the space to name the job  job_name = (audio_file_name.split('.')[0]).replace(" ", "")    # file format    file_format = audio_file_name.split('.')[1]    # check if name is taken or not  job_name = check_job_name(job_name)  transcribe.start_transcription_job(      TranscriptionJobName=job_name,      Media={'MediaFileUri': job_uri},      MediaFormat = file_format,      LanguageCode='en-US')    while True:    result = transcribe.get_transcription_job(TranscriptionJobName=job_name)    if result['TranscriptionJob']['TranscriptionJobStatus'] in ['COMPLETED', 'FAILED']:        break    time.sleep(15)  if result['TranscriptionJob']['TranscriptionJobStatus'] == "COMPLETED":    data = pd.read_json(result['TranscriptionJob']['Transcript']['TranscriptFileUri'])  return data['results'][1][0]['transcript']
Because the transcription might take time, we created a while loop to wait until it is done (rerun every 15 seconds).The last “if” statement is extracting the specific transcript from the JSON file. I will discuss how to extract timestamp at the end of the post.
For multiple speakers files
For AWS Transcribe for multiple speakers, the maximum speakers it can detect is 10. I will take 2 arguments as input this time: audio_file_name and the max_speakers. I strongly recommend having the max_speakers value already to enhance the accuracy for AWS, possibly. However, you can also leave it blank.
def amazon_transcribe(audio_file_name, max_speakers = -1):    if max_speakers > 10:    raise ValueError("Maximum detected speakers is 10.")    job_uri = "s3 bucket link" + audio_file_name  job_name = (audio_file_name.split('.')[0]).replace(" ", "")    # check if name is taken or not  job_name = check_job_name(job_name)    if max_speakers != -1:      transcribe.start_transcription_job(        TranscriptionJobName=job_name,        Media={'MediaFileUri': job_uri},        MediaFormat=audio_file_name.split('.')[1],        LanguageCode='en-US',        Settings = {'ShowSpeakerLabels': True,                  'MaxSpeakerLabels': max_speakers})    else:      transcribe.start_transcription_job(        TranscriptionJobName=job_name,        Media={'MediaFileUri': job_uri},        MediaFormat=audio_file_name.split('.')[1],        LanguageCode='en-US',        Settings = {'ShowSpeakerLabels': True})  while True:    result = transcribe.get_transcription_job(TranscriptionJobName=job_name)    if result['TranscriptionJob']['TranscriptionJobStatus'] in ['COMPLETED', 'FAILED']:      break    time.sleep(15)  if result['TranscriptionJob']['TranscriptionJobStatus'] == 'COMPLETED':    data = pd.read_json(result['TranscriptionJob']['Transcript']['TranscriptFileUri'])  return result
This time, the output is not the transcript anymore but will be a file result (in Python, it’s a dictionary data type).
data = pd.read_json(result['TranscriptionJob']['Transcript']['TranscriptFileUri'])transcript = data['results'][2][0]['transcript']
This code will give you the raw transcription (without speakers label): the result will be similar to inputting these files to the single speaker model.
How to add speaker labels?
Now, we will read the JSON file in the “TranscriptFileUri.” As we are using Google Colab, I will also demonstrate how to access files inside specific folders.Assuming we already have it in a Folder: Colab Notebooks/AWS Transcribe reader: here’s how to access it.
from google.colab import driveimport sysimport os drive.mount('/content/drive/')sys.path.append("/content/drive/My Drive/Colab Notebooks/AWS Transcribe reader")os.chdir("/content/drive/My Drive/Colab Notebooks/AWS Transcribe reader")
Now, we need to process the JSON output from AWS Transcribe. The code below will provide a .txt file with [timestamp, speaker label, content].When inputting the “filename.json” file, expect the “filename.txt” file for the full transcript.
import jsonimport datetimeimport time as ptime def read_output(filename):  # example filename: audio.json    # take the input as the filename    filename = (filename).split('.')[0]    # Create an output txt file  print(filename+'.txt')  with open(filename+'.txt','w') as w:    with open(filename+'.json') as f:      data=json.loads(f.read())      labels = data['results']['speaker_labels']['segments']      speaker_start_times={}        for label in labels:        for item in label['items']:          speaker_start_times[item['start_time']] = item['speaker_label']      items = data['results']['items']      lines = []      line = ''      time = 0      speaker = 'null'      i = 0            # loop through all elements      for item in items:        i = i+1        content = item['alternatives'][0]['content']      # if it's starting time        if item.get('start_time'):          current_speaker = speaker_start_times[item['start_time']]      # in AWS output, there are types as punctuation        elif item['type'] == 'punctuation':          line = line + content        # handle different speaker        if current_speaker != speaker:          if speaker:            lines.append({'speaker':speaker, 'line':line, 'time':time})          line = content          speaker = current_speaker          time = item['start_time']        elif item['type'] != 'punctuation':          line = line + ' ' + content      lines.append({'speaker': speaker, 'line': line,'time': time})             # sort the results by the time      sorted_lines = sorted(lines,key=lambda k: float(k['time']))      # write into the .txt file      for line_data in sorted_lines:        line = '[' + str(datetime.timedelta(seconds=int(round(float(line_data['time']))))) + '] ' + line_data.get('speaker') + ': ' + line_data.get('line')        w.write(line + '\n\n')
Then, in the same folder, the file “filename.txt” will appear, which contains all the transcripts.
Bonus 1: Accessing and uploading files directly to S3 storage
Uploading files to AWS S3 Storage will definitely automate a lot of processes.
# define AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and bucket_name# bucket_name: name of s3 storage folders3 = boto3.client('s3',   aws_access_key_id = AWS_ACCESS_KEY_ID,  aws_secret_access_key = AWS_SECRET_ACCESS_KEY,  region_name = "us-east-2")s3.upload_file(file_name, bucket_name, file_name)
Bonus 2: Why don’t we create a vocabulary to enhance accuracy?
We can manually upload the vocabulary into the AWS Transcribe service through the console management. The accepted files are .csv or .txt.However, if we want to automate this process with Python, it will be a bit trickier. Here is one way I found: AWS accepts a specific type of input through Python, which is a DataFrame with 4 columns: [‘Phrases’, ‘IPA,’ ‘SoundsLike,’ ‘DisplayAs’] convert into .txt file. For more information on the columns' meaning and custom vocabulary, read here.
def vocab_name(custom_name):  vocab = pd.DataFrame([['Los-Angeles', np.nan, np.nan, "Los Angeles"], ["F.B.I.", "ɛ f b i aɪ", np.nan, "FBI"], ["Etienne", np.nan, "eh-tee-en", np.nan]], columns=['Phrase', 'IPA', 'SoundsLike', 'DisplayAs'])  vocab.to_csv(custom_name+'.csv', header=True, index=None, sep='\t')  import csv  import time  csv_file = 'custom_name+'.csv  txt_file = 'custom_name+'.txt  with open(txt_file, "w") as my_output_file:    with open(csv_file, "r") as my_input_file:      my_output_file.write(" ".join(row)+'\n') for row in csv.reader(my_input_file)]    my_output_file.close()  ptime.sleep(30) # wait for the file to finish  bucket_name = #name of the S3 bucket  s3.upload_file(txt_file, bucket_name, txt_file)  ptime.sleep(60)  response = transcribe.create_vocabulary(    VocabularyName= custom_name,    LanguageCode='en-US',    VocabularyFileUri = "your s3 link" + txt_file)    # the link usually is bucketname.region.amazonaws.com# after running vocab_name, we can check the status through this line# if it's ready, the VocabularyState will be 'READY'transcribe.list_vocabularies()
The uploading and adding the vocabulary took quite a bit of time. This might not be the optimal approach, but it works (among trying a list of string, lists of lists, but none work so far).If you found a way to automate this even further, please comment! I would love to discuss and learn more.
Here is a quick tutorial on AWS Transcribe. I hope you found it useful! Also, I would love to hear your thoughts and discuss them even further. (And I might write about adding punctuations to Google Cloud little-punctuations transcript, so I hope that I don’t get lazy).
Link to the code: Google Colab (here), Gist (here), or Github(here).And feel free to contact me through linkedin.com/in/viethoangtranduong/or comment in this post, and I’ll try my best to reply asap!
P.s: real-time transcription is much better than transcribing a similar file.
ReferencesA Five Minute Overview of AWS TranscribeAWS Transcribe to Docx.