Load Apache Access Logs from AWS S3 bucket to AWS RDS-MYSql instance using PYSpark and AWS Glue Job | ETL Pipeline.
By: Sarthak Dhawan
Tags: Engineering, Pipeline As Code, Data, Pyspark, Etl

Load Apache Access Logs from AWS S3 bucket to AWS RDS-MYSql instance using PYSpark and AWS Glue Job | ETL Pipeline. Sarthak Dhawan Jun 17·3 min read

With an exponential burst in the size of data and the need of performing different functions on this growing data on a day to day basis with a fast evolving pace we need both automation and speed.

So, in this particular use case I had some Apache Access Logs, which are logged by the Apache Web Server for whosoever accesses the web server. Let us see a sample Apache Access Log :-

13.66.139.0 - - [19/Dec/2020:13:57:26 +0100] "GET /index.php?option=com_phocagallery&view=category&id=1:almhuette-raith&Itemid=53 HTTP/1.1" 200 32653 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"

According to Apache Documentation format of a common Apache Log is :

IP Address Client ID User ID Date and Time of the request Request Line (Method, Path, Query String and Protocol) Status Code Content Size (Size of the content returned to the client)

Whereas in the above example the Sample Log used is a combined Apache Access Log which in addition to these 7 components also include :

Referrer User Agent HTTP Header

So, to parse the above log format into a predefined 9 point columnar format for an RDS Database we need a regular expression to be written to map the data file to a Spark RDD using Spark Context. The regular expression I used is :

regex = '^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] "(\S+) (\S+)\s*(\S*)\s*" (\d{3}) (\S+)'

Lets start building up the Log parser, but first lets understand the basic architecture :

ETL Pipeline Architecture

Import all the necessary libraries :

import boto3

import pandas as pd

import sys

import re

from pyspark.sql import Row

from pyspark.sql import SparkSession

from awsglue.transforms import *

from awsglue.utils import getResolvedOptions

from pyspark.context import SparkContext

from awsglue.context import GlueContext

from awsglue.job import Job

from awsglue.dynamicframe import DynamicFrame

Make an Apache Log Line parser mapper function :

def parseApacheLogLine(logline): match = re.search(regex, logline)

if match is None:

return (Row(

host = '-',

client_identd = '-',

user_id = '-',

date_time = '-',

method = '-',

endpoint = '-',

protocol = '-',

response_code = -1,

content_size = 0

))

size_field = match.group(9)

if size_field == '-':

size = int(0)

else:

size = int(match.group(9))

return (Row(

host = match.group(1),

client_identd = match.group(2),

user_id = match.group(3),

date_time = match.group(4),

method = match.group(5),

endpoint = match.group(6),

protocol = match.group(7),

response_code = int(match.group(8)),

content_size = size

))

Make spark and glue contexts :

sc = SparkContext()

glueContext = GlueContext(sc)

spark = glueContext.spark_session

job = Job(glueContext)

Then after doing all this its time to extract our data, for this we will need to make a boto3 client and extract the data from our S3 bucket :

client = boto3.client('s3')

path= 'YOUR BUCKET PATH'

Bucket Path Format :

s3://bucket_name/file_name

Now using this path we will create a Spark RDD using our mapper function:

parsed_logs = (sc

.textFile(path)

.map(parseApacheLogLine)

.cache())

As we have created a Spark RDD of our data we need to convert it to a data frame so convert it to a Dynamic frame for loading :

logDataset = spark.createDataFrame(parsed_logs)

Now convert this Data frame to AWS GLUE Dynamic Frame :

glue_df=DynamicFrame.fromDF(logDataset, glueContext, "dynaFrame")

Now its time to load our data to our RDS Instance, but before that there are some prerequisites : 1. Make sure your RDS Instance in publicly accessable. 2. Make a Glue Connection to your RDS Instance. 3. Define a Table in your RDS Database for loading the data.

After all these prerequisites load the data into your RDS Instance:

output = glueContext.write_dynamic_frame.from_jdbc_conf

(frame = "<Your Dynamic Frame>",

catalog_connection = "<Your RDS Connection Name>", connection_options =

{"database" : "<Your Database Name >",

"dbtable" : "<Your Table Name>"})

Stop spark context and commit the glue job.

spark.stop()

job.commit()

Now run this script as a custom PYSpark Script in a GLUE Job.

Hope this helps some Data Enthusiasts out there.

Thank you

Sarthak Dhawan