# Performance Evaluation of Open-Source Serverless Platforms for Kubernetes
## 3 Results  
### Image Processing  
The benchmark for the image processing workload measured the time required to process 1000 images in each variant and configuration. Analysis was carried out based on the number of images that were processed every second, calculated as \(x=\frac{1000}{t}\), where \(x\) is the number of images processed per second and \(t\) is the time required to process all 1000 images. With 3 MiB per image on average, the data throughput was calculated as \(x_{b}=x\cdot 3\,\mathrm{MiB}\) where \(x_{b}\) is image data in MiB processed per second. Figures 1 and 2 along with Table 1 show the results of our experiments. The figures show how many images per second OpenFaaS and Nuclio were able to process per second for various numbers of function instances as bar plots. The standard deviation of each bar is shown by the small marker on top of the bars. If the marker is not visible, this means that the standard deviation is too low to be shown in the figure. Table 1 notes the specific peak throughput rates for both variants of the workload, including the baseline computed on the FD node alone. In an optimal environment, performance should scale linearly with the number of function instances. With the null-processing variant, the observed scalability is close to this expectation. However, with computation, the performance for Nuclio barely changes between 8 and 16 function instances. A detailed discussion of these results follows. From Figure 1, it can be observed that Nuclio clearly outperforms OpenFaaS for four or more function instances, with OpenFaaS peaking at 4.58 images per second compared to 7.17 images per second, as listed in Table 1. This means that Nuclio processes about 1.5 times more images per second compared to OpenFaaS. Meanwhile, the baseline sits at 12.58 images per second. This difference in performance can most likely be attributed to Nuclio being better suited for computational tasks than OpenFaaS due to different implementations and features. One such feature is Nuclio's internal queue, which gathers requests sent to its HTTP endpoints and forwards them as soon as a worker is available (See Nuclio Documentation: [[https://nuclio.io/docs/latest/concepts/best-practices-and-common-pitfalls/](https://nuclio.io/docs/latest/concepts/best-practices-and-common-pitfalls/)]([https://nuclio.io/docs/latest/concepts/best-practices-and-common-pitfalls/](https://nuclio.io/docs/latest/concepts/best-practices-and-common-pitfalls/)) accessed on 27 May 2022). This reduces the idle time on the workers, as they do not need to wait for a new request to enter the system. While OpenFaaS has a similar queue, it only uses it for its asynchronous endpoints, which were not under test here. Figure 1 shows that OpenFaaS and Nuclio reach their highest throughput rate when using sixteen function instances, that is, with one function instance per CPU core. However, the performance increase from eight function instances to sixteen is not as significant as one might expect, especially for Nuclio. This leads to the conclusion that each function instance can effectively support close to two function processes for this workload. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Unit** & **Baseline** & **OpenFaaS** & **Nuclio** & **OpenFaaS NP** & **Nuclio NP** \\ \hline Images/s & 12.58 & 4.58 & 7.17 & 16.04 & 14.93 \\ \hline MiB/s & 37.74 & 13.74 & 21.51 & 48.12 & 44.79 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Peak input processing rates for image processing workload in Images/s and MiB/s with 3 MiB per image on average. Figure 2: Image workload with null processing: OpenFaaS and Nuclio in comparison. The NP variant of the workload shown in Figure 2 paints a somewhat different picture. Here, the throughput rates scale almost proportionally with the number of function instances, and OpenFaaS has a slight edge over Nuclio for sixteen function instances, with 16.04 images processed per second compared to 14.93.