# An Event-Driven Serverless ETL Pipeline on AWS
## 5 Experimental Design  
We design our experiments to investigate several aspects of the system across different data volumes, event frequencies, and processing power. These are critical factors that should be considered when migrating to serverless configurations. Our empirical evaluation pinpoints the I/O and processing overheads at high granularity. In particular, by investigating how the architecture performs with respect to the following elements, we attempt to answer the corresponding questions:  
**Consistency:**: Are pipeline executions consistent?
**Reliability:**: What is the probability that the system delivers the data in its entirety? How often are events lost (if any)?
**Pricing:**: How chargeable elements scale across pipeline runs?
**Performance:**: What is the end to end time of data flow? How do lambda function execution times change across different data volumes and velocities?
**Payload size:**: What is the maximum payload that a pipeline can handle?  
For the initial phase of the experiments, we control the total data size to be sent, the payload size for each event, the available memory for the AWS lambda worker, and the frequency of the events. The event payload size is calculated, such that the sum the payloads equals the total size of data to be sent within an hour of running, based on the given frequency. More formally: let \(f\), \(T\), \(p\), and \(D\) be, respectively: the frequency, the time that the experiment is running for, payload size per event, and total data size to be sent, then:  
\[p=\frac{D}{f\times T}\quad\text{ where }\quad T=60\;\min\]  
We record the relevant metrics for each experiment question across 60 distinct pipeline runs that arise from all possible combinations of the following dependent variables:  
* Total data size to be sent (MB): 1, 10, 100.
* Event frequency (events per minute): 1, 2, 6, 30, 60.
* Lambda worker available memory (MB): 128, 256, 512, 1024.  
A secondary series of experiments was performed, where a single event was sent to the system in order to address event payload size and performance. During this process  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **Data Source** & **Triggering** & **Transformation** & **Data Writing** & **Sink** \\ \cline{2-7}  & **S3** & **SNS** & **SQS** & **Lambda** & **SQS** & **Lambda** & **DynamDB** \\ \hline \multirow{4}{*}{**Active**} & Request (S3 Net) & & & & & & \\ \cline{2-7}  & Data In (S3 In) & Messages (SNSmsg) & Requests (SQSRep) & GB/Seconds (GISSecCost) & Requests (SQSRee) & GB/Seconds (GISSecCost) & Write Units (DBWrites) \\ \cline{2-7}  & Data Out (S3 Out) & & & & & & \\ \hline
**Passive** & Storage & - & Requests & - & Requests & - & Storage \\ \hline \hline \end{tabular}
\end{table}
Table 2: Active and passive chargeable elements of architecture.  
the end-to-end time from source to sink was recorded. Specifically, a total of 32 runs were performed in order to cover the following dependent variables:  
* Event Payload Size (MB) : 1, 5, 10, 20, 40, 60, 80, 100.
* Lambda worker available memory (MB): 128, 256, 512, 1024.  
Because the payload and processing task of the writer function does not vary, the available memory for the writer lambda function is set to 128 MB across all of the experiments. Similarly, the data processing task that the lambda worker carries out remains the same for all experiments. To keep it consistent and quantifiable, we employed a simple pass-through of the data without any extra I/O overhead, which translates to O(N) complexity. Therefore, we can argue that any similar O(N) task can be subjected to our results. All of the Python implementations for the tasks that were carried out by Lambda functions were developed while using Pandas python package v1.1.0 [35] and Python standard libraries.