# An Event-Driven Serverless ETL Pipeline on AWS
## 3 Related Work  
Already, multiple studies have focused on harnessing the power of serverless technologies in order to modernise traditionally tedious tasks. Many evaluated performance across different dimensions, others addressed scalability issues, and others introduced novel paradigms. Hence, the related literature can be categorised into evaluations and applications of serverless technologies, yet there is often an overlap between the two. A seminal example on the application side is PyWren [20], which introduced a MapReduce primitive that was built on top of AWS lambda functions. PyWren provided a simple, \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline  & \multicolumn{1}{c}{**Traditional ETL**} & \multicolumn{1}{c}{**Serverless ETL**} \\ \hline \multirow{3}{*}{**Maintenance**} & Traditional ETL pipelines require regular infrastructure maintenance to ensure the Quality of Service. This is usually managed by another dedicated team. & Infrastructure maintenance and updates is taken care by the serverless components’ vendor. \\ \hline \multirow{3}{*}{**On-Demand**} & Require provisioning and the pricing is the same whether the resources are utilized or not. & Serverless ETL pipelines can be used/created On-Demand which lead to a flexible and dynamic pay per data processed model pricing model. \\ \hline \multirow{3}{*}{**Cost Effectiveness**} & Infrastructural components provisioned for traditional ETL solutions are usually priced at a very high level of utilisation. & On the contrary, Serverless cloud services usually have a very granular pricing which results into high cost effectiveness in the long term. i.e., AWS Lambda functions are now billed per 1 ms of execution. & On the contrary, Serverless cloud services usually have a very granular pricing which results into high cost effectiveness in the long term. i.e., AWS Lambda functions are now billed per 1 ms of execution. & Only have a very granular pricing which results into high cost effectiveness in the long term. i.e., AWS Lambda functions are now billed per 1 ms of execution. & On the contrary, Serverless cloud services usually have a very granular pricing which results into high cost effectiveness in the long term. i.e., AWS Lambda functions are now billed per 1 ms of execution. & Only have a very granular pricing which results into high cost effectiveness in the long term. i.e., AWS Lambda functions are now billed per 1 ms of execution. & Only have a very granular pricing which results into high cost effectiveness in the long term. i.e., AWS Lambda functions are now billed per 1 ms of execution. & Serverless ETL pipelines configuration is still heavily depended on the implementation but there is significantly less technical effort on enabling polyglot implementations. \\ \hline \multirow{3}{*}{**Elasticity**} & Scaling horizontally requires provisioning beforehand and careful planning to ensure the smooth integration of new resources. & Serverless components are very elastic allowing for almost instant horizontal and vertical scaling \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of Traditional vs Serverless ETL solutions. Figure 1: Developer control and serverless computing (adapted from [11]). highly parallel, serverless data processing system, which showcased a performance comparable to Spark. However, PyWren is not event-driven, still requires external orchestration, and only addresses the data processing stage. Similarly, ExCamera [21] applies the same principle in order to leverage extreme parallelism for digital video encoding, but, yet again, also requires a centralized orchestrator for functions. Stanford researchers also worked on a serverless orchestration framework, called gg [22]. It is a more generalisable approach that still requires an external coordinator, yet it only targets to improve highly-parallelised tasks rather than event-driven data processing.