# Cost-Profiling Microservice Applications Using an APM Stack
## 3 Design and Implementation
### Cost Calculation  
The cost for provisioning the complete target cluster can be collected directly from the cloud service provider, either by extracting it from the monthly billing information or directly from an API on the provider side. However, we also know that this information is too coarse-grained to be of actual use in optimizing the cost of the application. Ideally, we want to be able to assign cost down to the level of pods, the smallest unit in Kubernetes clusters, per service deployment and then calculate the cost per (micro)service by summing these individual costs. Establishing this level of granularity starts with a simple observation: _there is a direct relationship between the resources a service has been allocated and the cost we assign to it._ Intuitively, the more resources from a cluster node the pods for a service are allocated, and the more expensive the node is, the more expensive the service is in turn. Conversely, since the cost per node is linear with its provisioning time, then the cost of each service pod is the proportion of the node cost defined by the utilization of the node attributed to the service's pods. We therefore need to connect utilization metrics at the system and pod level to cost metrics. Doing so at the system level, through, e.g., directly monitoring CPU utilization on the node, both generates too many data to retain and does not guarantee the correlation between consumed resources and deployed services in all cases. Instead, we used the Metricbeat data for this purpose; of particular interest are the following fields:  
-kubernetes.pod.name: the (unique) name auto-generated by Kubernetes for each pod; -kubernetes.pod.cpu.usage.pct: the CPU usage as a percentage of the total node CPU; -kubernetes.pod.memory.usage.pct: the memory usage as a percentage of the total node memory; -kubernetes.node.name: The node on which the pod is running; -kubernetes.deployment.name: The deployment of which the pod is part of; -kubernetes.node.labels.node_kubernetes_io/instance-type: A label that is assigned to the Kubernetes node by DigitalOcean. It contains the machine type of the underlying infrastructure and is necessary because, in contrast to the hyperscalers, DigitalOcean does not define a machine.type as part of the metadata for their VMs. -orchestrator.cluster.name: Metadata to distinguish data coming from environments (e.g., development and production) or projects belonging to different clients. The raw Metricbeat data were first temporarily stored in the default Metricbeat Elasticsearch index. Even with the default reporting period of 10 s, retaining Metricbeat data for a long time quickly becomes very storage-intensive. Instead, we used what in Elastic terms is called a _transform_ action, which allowed us to convert existing indices into new, summarized ones that (eventually) could be used instead of the original ones so that the latter can be discarded. To this effect, we configured Elasticsearch to aggregate the data points that were collected for the fields we defined above every 10 s to a single point per minute, the average value, into a new index. This affects mostly the kubernetes.pod.cpu.usage.pct and kubernetes.pod.memory.usage.pct, since these are the ones that are actually the most volatile over time. While this data compression creates a drop in the accuracy of the metrics, it also results into a reduction of the necessary storage capacity by a factor of 6, which, as we will discuss in the following, is a significant factor. The next step in the process uses another transform, in this case an enrichment, to add to the document produced by the previous step containing the aggregated metrics per pod with the cost information at the node level. To achieve this goal, the API of the cloud service provider (in this case, DigitalOcean) is used to retrieve the appropriate data. The result of this step is a document containing both utilization metrics per pod and the billing data per month and hour at the level of the node in which the pod is deployed.