Package: prometheus-ecs-discoverer
Prometheus ECS DiscovererBased onprometheus-ecs-sd,
this project enables your Prometheus todynamically scrape targetsdeployed
in AWS ECS. The discoverer is perfect if you don't have
a service discovery system like Consul in-place. It provides an easy-to-use
alternative and can be used in low- to mid-sized environments withhundreds
of tasksrunning in parallel.Once the discoverer is up and running, any task can be made visible to
Prometheus.What are the advantages of using this project overprometheus-ecs-sd?Modified throttling in combination with the already existing caching allows
the usage in environments with hundreds of tasks.Every target can expose custom labels via its environment variables. This way
you can provide more ways to aggregate metrics. For example by type of API.Deploy as container (provided and can be used directly) or Python package.
No need to manually install boto3 and so on.Instrumented with Prometheus. Allows more insights into the discoverer. You
can monitor how long discovery rounds take and stuff like used memory.
Dashboard already included and ready to use.Extensive testing with high coverage ensures functionality.More configuration options. For example structured logs.SetupAs this project is based onprometheus-ecs-sd, the setup
is very similar / exactly the same. The setup consists out of three parts.
The targets must be prepared by adding environment variables. Next, PromED
itself must be deployed. And finally, the Prometheus configuration must be
updated.Prepare targetsTargets are setup via setting environment variables in the task definitions.Mark container as targetSetPROMETHEUS_TARGETtotrueto make PromED consider the container. This
by itself is already enough to make it work with the configured defaults.
Alternatively you can use thedockerLabelsin the container definition. For
this you have to set the PromED settingMARKER_TYPEtodockerLabelsinstead of the defaultenvironment.Specify metrics endpoint(s)If your metrics are not exposed on the default/metricsendpoint, you can
specifiy the endpoint withPROMETHEUS_ENDPOINT.You can also declare multiple endpoints and different intervals. The supported
intervals are15s,30s,1mand5m. Based on the interval, targets will
end up in different files. The default interval is "generic". Examples for this:5m:/mymetrics,30s:/mymetrics2/mymetrics/mymetrics,30s:/mymetrics2By default, all targets will end up in a single file calledtasks.jsonin the
configured directory (defaults to/tmp).Set custom labels for containerSometimes you might want to add additonal labels to targets to group them.
For example by the used API type (REST vs. GraphQL). This can be done by adding
environment variables to the container definition in the respective task
definition with thePROMETHEUS_LABEL_prefix. For examplePROMETHEUS_LABEL_api_typeorPROMETHEUS_LABEL_foo. Environment variables
set from within the container are not visible to PromED and are ignored."environment":[{"name":"PROMETHEUS_LABEL_foo","value":"bar"},{"name":"PROMETHEUS_LABEL_high","value":"fi"},]There is an alternative for this using thedockerLabelsattribute instead.
PromED always checks both locations. Unlike with the environment variables,
the additional target labels are all contained in one string. You can configure
the label PromED should look for by settingCUSTOM_LABELS_KEY. It works like
this:"dockerLabels":{"promed.custom_labels":"foo=bar, high=fi,what=ever"}The third option is to use theWITH_DOCKER_LABELSsetting to give PromED an
array of labels it should always try to map fromdockerLabelsto target
labels. The PromED config could look like this (notice the config prefix):"environment":{"PROMED_WITH_DOCKER_LABELS":"['com.company.namespace', 'com.company.alias']"}A container that uses these labels:"dockerLabels":{"com.company.namespace":"whatever","com-company-alias":"zoomzoom"}PromED translates it tocom_company_namespaceandcom_company_alias.Customize networkingRegarding networking, all network modes are supported (bridge,hostandawsvpc).IfPROMETHEUS_PORTandPROMETHEUS_CONTAINER_PORTare not set, the script
will pick the first port from the container definition (in awsvpc and host
network mode) or the container host network bindings in bridge mode. On
Fargate, ifPROMETHEUS_PORTis not set, it will default to port80.IfPROMETHEUS_CONTAINER_PORTis set, it will look at the container host
network bindings, and find the entry with a matchingcontainerPort. It will
then use thehostPortfound there as target port. This is useful when the
container port is known, but thehostPortis randomly picked by ECS (by
settinghostPortto0in the task definition).If your container uses multiple ports, it's recommended to specifyPROMETHEUS_PORT(awsvpc,host) orPROMETHEUS_CONTAINER_PORT(bridge).Quoted fromprometheus-ecs-sdcommitece6ca2.Deploy PromEDYou can either use the Python package directly or go for the Docker image.
Please note that just running the package / image is not enough. The enviornment
must be ready for boto3. This includes credentials and approbiate rights.Using package from PyPIAvailable under the nameprometheus_ecs_discoverer. To start PromED:python-mprometheus_ecs_discoverer.runTo configure PromED you can either provide a settings file or use plain
environment variables.  Please seeConfigurationfor more
info. Please seeConfigurationfor more info.Using Docker imageThe imagetrallnag/prometheus_ecs_discoverercan be foundhere.
The recommended way for configuring the image is to use environment variables.You will probably want to run the discoverer in ECS. Here, you don't have to
provide credentials assuming everything is set up correctly. Boto3 will
automatically detect relative credentials URI and retrieve them from AWS.
Nevertheless, the region must be set by you.AWS IAMThe actions that PromED is performing on the AWS API can be found inaws-iam-policy.json.
The allowed actions must be attached to an approbiate role. If you deploy
PromED in ECS, this should look like inaws-iam-ecs-role.json.Configure PrometheusIf you want all your targets to be scraped in the same interval, the following
job is enough.- job_name: 'ecs'
  file_sd_configs:
    - files:
        - /tmp/tasks.json
  relabel_configs:
    - source_labels: [metrics_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)By default, PromED exposes an/metricsendpoint.- job_name: discovery
  static_configs:
    - targets: ["discovery:8080"]ConfigurationAs this project is based onprometheus-ecs-sd,
the configuration is mostly compatible.PromED usesDynconffor config management. There
are two main ways you can configure the application. Either by providing a
custom settings file with the env varSETTINGS_FILES_FOR_DYNACONF(seehere)
or directly setting the respective values via env vars with thePROMED_prefix. All supported settings together with their default values can be foundsettings.toml(click me).Grafana DashboardYou can find the Grafana dashboards for PromED in the documents folder.A second dashboard contains panels with Grafana Alerts enabled.PrerequesitesSeepyproject.toml.DevelopmentDeveloping and building this package on a local machine requiresPython Poetry. I recommend to run Poetry in
tandem withPyenv. Once the repository is
cloned, runpoetry installandpoetry shell. From here you may start the
IDE of your choice.