## 5 Evaluation
### Advesities  
Described below, are the main adversities we came across while collecting performance metrics data, together with an explanation on how we managed to overcome them. They are sorted based on the time period in which they occurred:  
1. _Function apps unavailable._ The fastest way of collecting 100 cold start execution records in MsAzure is by cold starting 100 different _function app_ instances. However, while triggering each azure function, we noticed that the last 30 or so functions were responding with a 503 HTTP status code, indicating that the _function app_ service was unavailable. The _function apps_ were redeployed a few times, but the same problem persisted. We also verified that the consumption plan allowed a maximum number of 100 _function apps_[65], meaning that the limit was not being surpassed. Unfortunately, we were unable to determine the exact reason behind this problem. To overcome this issue, we decided to only cold start the first 50 azure functions twice, causing the process of data collection in MsAzure to take the double the amount of time of GCP.
2. _Unregistered executions._ The warm starts evaluation included a total of 800 warm start execution records, meaning that we had to do at least 200 consecutive invocations for each function definition. It turns out that not all warm start executions are registered by Google Cloud Platform metrics. For instance, out of 100 consecutive function invocations, less than 10 got registered by the metrics service. We tried to mitigate this issue by adding a ten-second delay between consecutive function invocations. The purpose of this delay is to give some extra time to the metrics service to register execution metrics. With this strategy, we were able to reduce the number of necessary invocations from \(10\times\) to \(4\times\) the number of desired records. Therefore, in order to generate the 200 execution records, each function had to be warm started around 800 times (\(4\times 200\)_records_). If no delay was applied while generating the 200 execution records, each function would have to be warm started around 2000 times (\(10\times 200\)_records_). On the other hand, the Application Insights, from MsAzure, is capable of registering every function execution time. However, a zero-second delay for warm starts could no longer be used as well. Otherwise, most execution times with a zero-second delay would be much lower than the ones registered using a ten-second delay, which would lead to unfair comparisons between the two providers. To verify this analysis, we made an extra test in MsAzure using a zero-second delay between warm start invocations of the cloud-agnostic function. We came to the conclusion that most execution times ranged from 20 to 40 ms, while with a ten-second delay, as shown in Figure 15, the average execution was around 75 ms, with most execution times ranging from 60 to 80 ms.
3. _Measurements inconsistency._ Latency cannot be ignored when evaluating the performance of FaaS applications. The highest latency is usually experienced in cold starts, during the preparation of the execution environment. For the _search blobs_ use case in particular, the remote calls to the storage service also contribute with some network latency. All things considered, execution times can therefore be volatile, causing the results to be inconsistent when measured at separate times. To increase the reliability of measurements, various tests had to be repeated multiple times on different days during off-peak hours (i.e., between 2 p.m. and 6 p.m.), in an effort to avoid network congestion.