Project: Vector (Observability and Analysis / Observability)
File: 2020-09-28-cloudwatch-metrics.md

This RFC proposes an implementation for a new metrics source to ingest metrics from AWS CloudWatch. The proposed implementation is fairly similar to the one used by telegraf which scrapes `GetMetricData` on a regular interval. It is probably worth reviewing [Amazon CloudWatch Concepts](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html) before looking at this if you are unfamiliar with CloudWatch. This RFC will simply cover a new `awscloudwatchmetrics` source. Users want to collect and forward metrics from AWS CloudWatch to monitor infrastructure and deployed services in AWS. We will add a `awscloudwatchmetrics` source that scrapes the AWS API () on a user-defined interval to collect metrics and forward them through the user's pipeline. The configuration of the new `awscloudwatchmetrics` source will look like: ```toml [sources.cloudwatch] type = "awscloudwatchmetrics" assumerole = "arn:aws:iam::123456789098:role/myrole" # optional, no default endpoints = ["127.0.0.0:5000/path/to/service"] # optional, no default, relevant when regions = [] regions = ["us-east-1"] # required, required when endpoints unspecified; no default period_secs = 300 # period (s) to aggregate metrics over, optional, can be overridden at metric level, default 300 delay_secs = 300 # delay collection by value (s), used to avoid collecting data that has not fully been processed by CloudWatch, optional, default 300 interval_secs = 300 # interval to scrape metrics; should be a multiple of "period"; default 300 metricsrefreshinterval_secs = 900 # interval to refresh available metrics for queried namespaces if globbing or all metrics are used; default 900 request.inflightlimit = 25 # optional, default, requests request.ratelimitduration_secs = 1 # optional, default, seconds request.ratelimitnum = 25 # optional, default request.retry_attempts = 18446744073709551615 # optional, default request.retryinitialbackoff_secs = 1 # optional, default, seconds request.retrymaxduration_secs = 10 # optional, default, seconds request.timeout_secs = 30 # optional, default, seconds [[sources.cloudwatch.metrics]] namespace = "AWS/EC2" # optional; supports globbing names = ["EBSReadOps", "EBSReadBytes", "Network"] # optional; defaults to all metrics in namespace, [""], (refreshed on interval); supports globbing dimensions.InstanceId = "i-05517fbc2e6124dfb" # optional; supported dimensions differ by namespace and metric; supports globbing statistics = [ "average", "sum", "minimum", "maximum", "samplecount" ] # statistics to collect; can also contain extended statistics like p99; default: [ "average", "sum", "minimum", "maximum", "samplecount" ] period_secs = 300 # period (s) to aggregate metrics over; defaults to top-level `period` setting; top-level interval should be a multiple of this and any other defined periods ``` We could alternatively model dimensions as another table: ```toml [[sources.cloudwatch.metrics.dimensions]] key = "InstanceId" value = "i-05517fbc2e6124dfb" ``` NOTE decided against this table representation since we have a few other examples of vector key/value config that use TOML maps: [`add_tags` transform](https://vector.dev/docs/reference/transforms/add_tags/) [`http` sink](https://vector.dev/docs/reference/sinks/http/). To support: globbing metric names fetching all metrics for a namespace globbing of dimension values We will refresh and cache the available metrics on an interval (`metricsrefreshinterval`) for namespaces appearing in the configuration. A wildcard dimension value (`"*"`) will pull and publish discrete metrics for each dimension

This is different than omitting the dimension altogether which would instead aggregate across the values for the dimension. Note that dimensions must match the dimensions that the metrics were published with. For example, if the metric was published two dimensions, availability-zone and load balancer name, then both dimensions must be specified to retrieve the metric. CloudWatch won't automatically aggregate across the other if only one is specified. This source will use the default AWS credential chain similar to the `awscloudwatchmetrics` sink. The output (shown here in prometheus format) will publish metrics in the form of: ```text <namespace><metricname><metricstat>{<dimensionkey>=<dimensionvalue>*} ``` All metrics will be additionally tagged with the region. For example: ```text awsec2cpucreditbalanceaverage{instanceid="i-0db9620c0ee32d463",region="us-east-1"} 576 awsec2cpucreditbalancemaximum{instanceid="i-0db9620c0ee32d463",region="us-east-1"} 576 awsec2cpucreditbalanceminimum{instanceid="i-0db9620c0ee32d463",region="us-east-1"} 576 awsec2cpucreditbalancesamplecount{instance_id="i-0db9620c0ee32d463",region="us-east-1"} 1 awsec2cpucreditbalancesum{instanceid="i-0db9620c0ee32d463",region="us-east-1"} 576 ``` I think we will want to provide example configs that collect common metrics for various namespaces (like EC2, S3, etc.). Broadening AWS platform support. Otherwise, users will need to run `telegraf` or another agent to ingest this data. The additional maintenance burden of a new source. There is alternate API, that we could use, but I think it will result in less overall throughput due to API rate limit restrictions: `GetMetricStatistics` Allows 400 transactions / sec Each transaction can query the 5 standard metrics and up to 10 extended metrics for one metric name / dimension combination This should allow for: `400 5 = 2000` standard metrics / second `GetMetricData` Allows 50 transactions / sec Each transaction can query up to 500 metric name / metric stat (like `average`) / dimension combination This should allow for: `50 500 = 25000` standard metrics / second These are theoretical maxes. They could differ depending on how many metric statistics are queried for each metric. Querying less than the 5 standard ones would further advantage `GetMetricData`. Do we care about providing a different rate limit for refreshing available metrics? It uses a different API (`ListMetrics`) with different limits than `GetMetricData`. Answer: we'll try without for now. [ ] Submit PR with `awscloudwatchmetrics` source without any support for globbing (which would require listing and caching metrics) and only one region [ ] Submit follow-up PR allowing for globbing of metric names [ ] Submit follow-up PR allowing for globbing of dimension value [ ] Submit follow-up PR allowing for globbing of namespace [ ] Submit follow-up PR allowing for multiple regions Support for [metric expressions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html#metric-math-syntax). Filtering metrics by tags. It looks like the logstash input supports this by querying for resources first (e.g. instances with a specific tag) and then fetching metrics for those instances. Backfilling metrics if `vector` is restarted or offline for a period. Additional config examples covering common use-cases; basic examples should be included with the source documentation. Warnings when configured dimensions do not match any metrics in `ListMetrics`. Caching `ListMetrics` output across vector restarts to speed up initial start.