# Performance Evaluation of Open-Source Serverless Platforms for Kubernetes
## 2 Materials and Methods
### Test and Benchmark  
A single benchmark run consisted of:  
1. Setting up the correct number of function instances
2. Preparing the test data
3. Starting the benchmark and timer
4. Submitting the test data as a function invocations until all test data had been processed
5. Stopping the timer
6. Saving the results  
The coordination of the benchmarks was carried out by a Python program running on the FD VM. The program sends in parallel requests to the HTTP endpoint of the function under test such that there is always enough load without overwhelming the Gateway of the serverless platform. Overall, each benchmark was repeated at least ten times. For the image processing workload, the individual benchmarks were executed with 1, 2, 4, 8, and 16 active function instances in separate benchmark runs. The number of function instances was capped at 16 as each platform runs on four nodes with four vCPU cores each, meaning that with 16 active function instances, every function instance has on average one core to work with. The FD VM only tested the regular variant of this workload and not the NP variant. The test data consisted of 1000 PNG images with a total size of 2.9 GiB taken from test-images.org [15]. The images were from the 16 BIT RGB sampling set ([[https://sourceforge.net/projects/testimages/files/SAMPLING/16](https://sourceforge.net/projects/testimages/files/SAMPLING/16) BIT/RGB]([https://sourceforge.net/projects/testimages/files/SAMPLING/16](https://sourceforge.net/projects/testimages/files/SAMPLING/16) BIT/RGB) accessed on 27 May 2022), with 960 of them having a resolution of 800 \(\times\) 800 and the remaining 40 having a resolution of 600 \(\times\) 600. This amounts to 3 MiB per image on average (1 GiB = 1024 MiB, that is, 2.9 GiB = 2969 MiB and \(\frac{2969}{1000}\) MiB \(\approx\) 3 MiB). The output had a total size of 842 MiB, which on average is 862 KiB per image. If the processing of an image failed, for example due to connection issues, then the image was retried until it could be processed. The bytes reversal workload used 1, 2, 4, 8, and 16 active function instances in separate runs of the parallel variant. The single variant was limited to one and four function instances, as no parallel processing is required to respond to a single request and no performance gains can be expected from using additional instances. The number of requests for the parallel variant was set to 100, and was repeated ten times for each number of bytes. First, 1 B (2\({}^{0}\)) was sent, then 2 B (2\({}^{1}\)), etc, up to 16 MiB (2\({}^{24}\)), for a total of 25,000 requests over 250 test runs. The actual number of requests is likely to be higher, as any failed requestswere retried until they succeeded. The single variant was repeated 1000 times for each number of bytes. Finally, the resulting throughput rates were calculated based on the size of the input data, ignoring variations in the data size of the output as well as any failed requests. Furthermore, each benchmark was run until all of its requests were completed such that all tasks could be finished except for one (possibly the slowest task, on which the entire benchmark has to wait). This effect should be negligible, as the size of the individual tasks was relatively small. Furthermore, the rates at which requests were sent out from the FD were limited in order not to overwhelm the platforms. For the image processing task, this was set as up to two active requests per function instance, while for the byte reversal workload it was set as up to four active requests per function. The rate was set as active requests instead of requests per second, as when using requests per second retries are not accounted for, and over time a backlog of failed requests may form. Limiting based on active requests avoids this, as the number of requests in transit is always known. The experiments were conducted using only a single version for each of the applications used.