Automating Text Extraction and Data Preprocessing using AWS Textract, Lambda & DynamoDB | by Saleh Alkhalifa | Towards Data Science

One of the most important parts of a data scientists job is automating processes — this can reduce the burden of time-consuming tasks and free you up for more important matters. In this tutorial, we will cover an end-to-end AWS solution for automating document text extraction. We will take raw and unstructured documents and organize them for parsing.
Create two S3 BucketsCreate a Textract Lambda Function & IAM RoleCreate a DynamoDB TableCreate a DynamoDB Lambda Function & IAM RoleConclusion
Create two S3 Buckets
Create a Textract Lambda Function & IAM Role
Create a DynamoDB Table
Create a DynamoDB Lambda Function & IAM Role
Conclusion
You will need to create two S3 buckets — one will function as the input bucket, and the other as the output bucket. You can easily consolidate the two into one bucket, however, recursion can be problematic with lambda processes like these — therefore we will use a two bucket solution for this tutorial.
As you create the resources needed for this project, please ensure that the specified region remains the same. If you select ‘us-east-2’ for the S3 buckets, this will also need to be done for all other resources. Go ahead and create a new bucket by navigating to the S3 console and clicking on ‘Create Bucket’. You can name the bucket anything you wish, however, it is recommended that you stick to a specific naming convention— in our case, it will be “s3-json-input-bucket”.
Once completed, you should see your two S3 buckets listed under the S3 AWS console.
With the S3 buckets now in place, let us go ahead and create our Lambda functions. Navigate to the Lambda Console in AWS, and click “Create Function”.
Select the “Use a blueprint” option, and search for “Blueprint name: s3-get-object-python”. Click configure so that we can edit some settings.
Go ahead and enter a function name. Best practice is to label the lambda function based on its purpose. In this case, we will call the function “textract-s3-to-json-function”. Next, create a new role for the function called “textract-s3-to-json-role”. Consistency in naming helps you remember the purpose of each resource when working alone, or can help others easily work on a project when working as a team. At the bottom of the page, select the specific S3 bucket of interest — in our case that would be s3-json-input-bucket” which we created earlier. We will be using this input bucket as the ‘drop zone’ for new PDFs. Once a file is dropped in, the lambda function will be triggered to ignite the Textract script.
You can limit the files this function will be applied to by either a prefix (i.e. only look at files in a specific directory), or by suffix (i.e. only look at files ending with .pdf). These are optional parameters and can be added later if needed. Next, go ahead and click “Create Function”.
We will be connecting this function to our other AWS resources using Boto — the official AWS python SDK. In order to get this SDK working in our Lambda function we will need to upload it as a layer. On the left hand side of the screen, select “Create Layer” from the menu.
Create a layer by uploading a zip file which you can retrieve from my Github. This will allow the function to use Boto. Once uploaded, go ahead and click “Create”. Navigate back to the lambda function, and at the bottom of the screen, click “Add Layer”.
In the code section, go ahead and replace the code with the code in my GitHub file. This code snippet will take in any incoming documents in the first S3 bucket, run them through AWS textract, and subsequently write them back to the second S3 bucket. Note that there are two fields in this Github file that will need to be replaced:
ARN Role (Found on line 27).
Output Bucket Name (Found on line 111).
You can replace the output bucket name with the bucket name you selected in step one. The ARN role however will need to be replaced with the ARN role written at the top of the Role page in the IAM Console. This will ensure that the proper permissions associated with the role are usable by the function.
Some PDFs will need to be analyzed longer than others. In order to ensure that the PDFs do not timeout in the middle of the process, you will need to extend the timeout limit. The larger the document, the longer the analysis time — this setting will be highly dependent on the documents submitted.
With the function completely set, we can now edit the role to ensure all necessary permissions are configured. Navigate to the IAM Console in AWS, and search for the role we previously created called “textract-s3-to-json-role”.
The ARN role will be noted at the top of this page, which is the same ARN role you needed earlier when configuring the Python code.
Navigate to the permissions section and add the other resources we will need:
In addition to the two that are already specified, we will need “AmazonS3FullAccess”, and “AmazonDynamoDBFullAccess”. Go ahead and attach those policies.
On the next tab over called “Trust Relationships”, ensure that the relationship is as follows:
{ “Version”: “2012–10–17”, “Statement”: [ { “Effect”: “Allow”, “Principal”: { “Service”: “lambda.amazonaws.com” }, “Action”: “sts:AssumeRole” } ]}
With these two steps completed, you can now drop pdf files into the “input” bucket, which will automatically textract the files and upload them to the “output” bucket as a JSON. Pretty neat huh?
With the textract portion completed, let us now focus on getting the table set up. Navigate the DynamoDB console in AWS and click “Create Table”.
Enter a table name and a primary key. In our case, we will call the primary key “id”. You are free however to rename this field as you see fit with any other unique ID in your data. Go ahead and click ‘create’ at the bottom of the screen.
Once created, you should see the table populate on the right side of the screen.
Similarly to the previous Lambda function, go ahead and repeat the same process of creating the function, adding a boto-layer, and changing the permissions of the associated role. Be aware that there will be some minor differences between this function and the other:
Making sure that the associated trigger is pointing to the output bucket instead. Remember, the idea here is that the output of the first function will be the input to our second one.Making sure that the code associated with this function does not rerun textract, but simply takes the JSON output and runs it through DynamoDB. Copy the code snippet from below, or from my GitHub file.
Making sure that the associated trigger is pointing to the output bucket instead. Remember, the idea here is that the output of the first function will be the input to our second one.
Making sure that the code associated with this function does not rerun textract, but simply takes the JSON output and runs it through DynamoDB. Copy the code snippet from below, or from my GitHub file.
import jsonimport boto3import osimport urllib.parseimport jsonimport uuidfrom decimal import Decimals3_client = boto3.client(‘s3’)dynamodb = boto3.resource(‘dynamodb’)def lambda_handler(event, context):     print(“Starting Transfer:”)    bucket = event[‘Records’][0][‘s3’][‘bucket’][‘name’]    json_file_name = urllib.parse.unquote_plus(event[‘Records’][0][‘s3’][‘object’][‘key’], encoding=’utf-8')    json_object = s3_client.get_object(Bucket = bucket, Key = json_file_name)    jsonFileReader = json_object[‘Body’].read()    jsonDict = json.loads(jsonFileReader, parse_float=Decimal)    table = dynamodb.Table(‘s3-json-textract-table’)     finalObject = jsonDict[0]    finalObject[“key”] = json_file_name    finalObject[‘id’] = str(uuid.uuid4())    table.put_item(Item = jsonDict[0])
3. Finally, ensuring that the IAM role for this function includes permissions for DynamoDB, and S3 Full Access. You can add these roles by navigating to the IAM console, searching for the role name, and adding DynamoDB and S3 under the permissions section.
With that, you should now be all set! As you populate the input S3 bucket with PDF files, this will trigger the first lambda function to apply the Textract OCR on your file, and send the output in JSON format to the output bucket. Once in the output bucket, the second lambda function will be triggered and transfer the newly created JSON file into DynamoDB. While this tutorial was catered heavily towards PDF files, Textract is also able to handle images. Go ahead and try this yourself!