## 5 Evaluation
### Metrics Definition  
Having established the appropriate use case for evaluation, we now have to decide what were the key metrics that could best characterize the performance of the function's deployment and execution. The execution time is commonly recognized as the primary metric for measuring a function's performance. When cold started, the function's execution time includes an extra latency derived from the container's startup process, thus producing higher execution times than warm starts. The execution time was measured in milliseconds, while the second performance metric, the function's memory usage, was measured in megabytes (MB). The memory usage refers to the total amount of memory consumed during the function's execution. When measuring for Google Cloud Platform, both metrics can be programmatically obtained using the _MetricService_ from the Cloud Monitoring gRPC API [58]. Within this service, the operation _ListTimeSeries_ can then be used to capture sets of metrics data that match certain filters, for a given time frame. The following filters need to be specified when capturing the execution time and the memory usage, respectively: _function/execution_times_ and _function/user_memory_bytes_[59]. An extra cloud service is also required when capturing the function's execution time in MsAzure, called the Application Insights. This is a feature of Azure Monitor that provides extensible application performance management and monitoring for live applications, including _function apps_. We created as many Application Insights resources as _function apps_deployed. The _Query_ operation, from the Application Insights REST API [60], can then be invoked to request a set of execution times (_FunctionExecutionTimeMs_) within a given time frame. This is done by sending the following log query as the body of the HTTP request:  
_requests_ | _project timestamp_, _customDimensions['FunctionExecutionTimeMs']_  
Unfortunately, the measurement of memory usage per function execution is not a metric currently available through Azure Monitor. There are several other related metrics [61]:  
* _Working set_--the current amount of memory used by the app (_function app_), in mebibytes (MiB). * _Private bytes_--the current size, in bytes, of memory that the app process has allocated that cannot be shared with other processes. Useful for detecting memory leaks. * _Function execution units_--a combination of execution time and memory usage, measured in MB-milliseconds. Both the _Working set_ and the _Private bytes_ consist of measuring the app's memory as a whole, they are not exclusive to serverless functions, making them inadequate metrics for this study. Additionally, any sort of interaction with the app via its REST APIs, or through the Azure Portal, can cause memory spikes, even when no functions were recently executed. As a consequence, the data collection process would have difficulties in accurately distinguishing the memory consumed during the function's execution time from the one spent in processing secondary app operations. The _Function execution units_ is the one out of the three specified metrics that best meets our needs, for different reasons: (i) gets measured for every function execution, (ii) does not include memory consumption from secondary app operations, and (iii) the calculation formula is defined in official documentation [62]. The _Function execution units_ per function execution are calculated according to Equation (1):  
\[FunctionExecutionUnits=execution\_time\times\textbf{memory\_usage}, \tag{1}\]  
even though this is being applied once every function execution by Azure Monitor, the units are presented as a time-based aggregation, meaning that they cannot be obtained individually for a particular function execution. To be more specific, Azure Monitor aggregates units of function executions made within the same minute, so that it can be obtained as an average, minimum, maximum, sum, or count.