# An Event-Driven Serverless ETL Pipeline on AWS
## 4 Architecture  
We chose the AWS platform for our experimental evaluation for several reasons. Firstly, according to a Canals report, AWS holds the largest market share amongst the major cloud providers [30]. Furthermore, AWS lambdas cold-start times have been proven to be more consistent across multiple invocations [28] and they provide better CPU performance, network bandwidth, and I/O throughput as implementations scale [29]. Other empirical results suggest that the Amazon S3 offers low latency and comparable throughput over other providers [31]. We selected Python3.7 to run our experiments, given that Python is one of the most popular AWS Lambda runtime choices amongst academic studies [32]. Our event-driven ETL pipeline follows a rather basic architecture. The primary aim of this research is to study the behaviour, overheads, and costs in the context of serverless event-driven pipelines, in a manner that is as platform-agnostic as possible: therefore, adopting a straightforward architecture excludes any optimising features AWS may support. Similar to a typical ETL pipeline, the subject architecture consists of a data source, a processing stage, and a data sink. Two additional intermediate stages were implemented for connecting the processing with the source and the sink, primarily for observability and fault tolerance. For reference, we labelled these stages as 'triggering' and 'writing'. Figure 2 gives a high-level overview of the whole architecture, which is outlined here:  
**Data Source:** We use an AWS S3 bucket as a data source. Using S3 is both a cost-effective and scalable choice as it provides low latency and out-of-the-box lambda integration capabilities. While, technically, data could be sent directly to lambda functions while using an API Gateway, S3 acts both as a data source and datalake. Hence, the data persist and they can be processed again in the event of system failure, or act as input for other data analysis pipelines. **Triggering:** S3 supports notifications that directly trigger a lambda function. Notably, this is less desirable in live production environments, as it delegates the burden of implementing a fail-over strategy down to the function. Yet, our implementation uses SNS topics and SQS queues in order to invoke the processing stage. Although this increases architectural and network overhead, it is an essential element of the architecture in order to achieve observability and fault tolerance. More precisely, SQS triggers a lambda function synchronously and, hence, any inputs that resulted in a failed invocation will be forwarded to a dead letter queue. What is more, this architectural design allows for a fanout message forwarding, as opposed to normal S3 notifications. **Data Processing:** we employ stateless lambda functions for data processing. Ensuring that the processing is stateless poses some limitations on the type of operations that can be applied on the data, due to the lack of global context. There are ways to achieve statefulness amongst many lambda functions [33; 34], but these are not within the scope of this work, as they introduce architectural complexities. **Data Writing:** the same as in the triggering stage, data writing provides an extra layer of fault tolerance and failover resolution by using an SQS and a lambda function, which is responsible for persisting the data into DynamoDB. It must be noted that SQS limits message sizes to a maximum of 256 KBs by design; therefore, each lambda transformation worker must chunk the transformation output prior to sending it to the queue. This design permits extremely high parallelisation, as each chunk is written by a separate writer. Furthermore, it opens up the opportunity for applying some common processing logic to all of the data after they have been processed (i.e., validation). **Data Sink:** we use DynamoDB to store the processed data. DynamoDB is a serverless NoSQL database and it fulfils all our experimental requirements.