Package: icinga2-exporter
icinga2-exporterOverviewThe icinga2-exporter utilizes theIcinga 2REST API to fetch service based performance
data and publish it in a way that letsPrometheusscrape the performance data as metrics.The service is based onQuart. Quart's is compatible with Flask but based
on Asyncio.Benefits:Enable advanced queries and aggregation on timeseriesPrometheus based alerting rulesGrafana graphingUtilize investments with Icinga 2 of collecting metricsMetrics namingMetric namesMetrics that is scraped with the icinga2-exporter will have the following name structure:icinga2_<check_command>_<perfname>_<unit>The icinga2 prefix can be changed by the configuration
Unit is only added if it exists on performance dataExample from check commandcheck_pingwill result in two metrics:icinga2_ping_rta_seconds
icinga2_ping_pl_ratioMetric labelsThe icinga2-exporter adds a number of labels to each metrics:hostname - is thehost_namein icinga2service - is thedisplay_namein icinga2Optional icinga2-exporter can be configured to add specific custom variables configured on the host.Note:Icinga 2 supports custom variables that can be complex data structures - but that is NOT currently supported.Labels created from custom variables are all transformed to lowercase.Performance metrics name to labelsAs described above the default naming of the Prometheus name is:icinga2_<check_command>_<perfname>_<unit>For some checks this does not work well like for thediskcheck command where the perfname are the unique mount paths.
For checks like that the where the perfname is defined depending on environment you can change so the perfname instead becomes a label.
This is defined in the configuration like:perfnametolabel:# The command namedisk:# the label name to be usedlabel_name:mountSo if the check command isdiskthe Prometheus metrics will have a format like, depending on other custom variables :icinga2_disk_bytes{hostname="icinga2", service="disk", os="Docker", mount="/var/lib/icinga2"} 48356130816.0If we did not make this translation we would got the following:icinga2_disk_slashvarslashibslashicinga2_bytes{hostname="icinga2", service="disk", os="Docker"} 48356130816.0This would not be good from a cardinality point of view.Scrape durationThe scrape duration is a metrics that is reported for all targets.icinga2_scrape_duration_seconds{hostname="<target>", server="<icinga2_server_url>"} 0.160983Scrape responseWhen requests are made to the exporter the following responses are possible:A target that exists - return all metrics and http status 200A target does not exists - return no metrics, empty response, and http status 200The export fail to scrape metrics from icinga2 - return empty response and http status 500In the last scenario the exporter will log the reason for the failed scrape. A failed scrape can
have multiple reasons, for example:The icinga2 server is not respondingNot having valid credentialsRequest to icinga2 timeoutConfigurationicinga2-exporterTheicinga2-exporteris configured by a yaml based configuration file.Example:# Port can be overridden by using -p if running development quart#port: 9638icinga2:# The url to the icinga2 serverurl:https://127.0.0.1:5665user:rootpasswd:cf593406ffcfd2ef# All prometheus metrics will be prefixed with this stringmetric_prefix:icinga2# Example of custom vars that should be added as labels and how to be translatedhost_custom_vars:# Specify which custom_vars to extract from hosts in icinga2-env:# Name of the label in Prometheuslabel_name:environment-site:label_name:dc# This section enable that for specific check commands the perfdata metrics name will not be part of the# prometheus metrics name, instead moved to a label# E.g for the disk command the perfdata name will be set to the label disk like:# icinga2_disk_bytes{hostname="icinga2", service="disk", os="Docker", disk="/var/log/icinga2"}perfnametolabel:# The command namedisk:# the label name to be usedlabel:mountlogger:# Path and name for the log file. If not set send to stdoutlogfile:/var/tmp/icinga2-exporter.log# Log levellevel:INFOWhen running with gunicorn the port is selected by gunicornLoggingThe log stream is configure in the above config. Iflogfileis not set the logs will go to stdout.Logs are formatted as json so its easy to store logs in log servers like Loki and Elasticsearch.Prometheus configurationPrometheus can be used with static configuration or with dynamic file discovery using the projectmonitor-promdiscovery.Please add the job to thescrape_configsin prometheus.yml.The target is thehost_nameconfigured in icinga2.Static configscrape_configs:-job_name:'icinga2'metrics_path:/metricsstatic_configs:-targets:-icinga2-google.serelabel_configs:-source_labels:[__address__]target_label:__param_target-source_labels:[__param_target]target_label:instance-target_label:__address__replacement:localhost:9638File discovery config for usage withmonitor-promdiscoveryscrape_configs:-job_name:'icinga2'scrape_interval:1mmetrics_path:/metricsfile_sd_configs:-files:-'sd/icinga2_sd.yml'relabel_configs:-source_labels:[__address__]target_label:__param_target-source_labels:[__param_target]target_label:instance-target_label:__address__replacement:localhost:9638InstallingCheck out the git repo.Install dependencypip install -r requirements.txtBuild a distributionpython setup.py sdistInstall locallypip install dist/icinga2-exporter-X.Y.Z.tar.gzRunningDevelopmentRun theicinga2-exporterwith the built-in Quart webserver:python -m  icinga2_exporter -f config.ymlTo see all options:python -m  icinga2_exporter -hProduction with hypercorn as ASGI continerHypercorn is the recommended ASGI container for Quart. Install hypercorn with:pip install hypercornRunning with default config.yml. The default location is current directoryhypercorn "icinga2_exporter.main:create_app()Set the path to the configuration file.hypercorn "icinga2_exporter.main:create_app('/etc/icinga2-exporter/config.yml')"Port 8000 is the default port for hypercorn. For more configuration for hypercorn please visithttps://pgjones.gitlab.io/hypercorn/index.htmlTest the connectionCheck if exporter is working.curl -s http://localhost:9638/healthGet metrics for a host where target is a host,host_namethat exists in icinga2curl -s http://localhost:9638/metrics?target=google.seSystem requirementsPython 3.For required packages please reviewrequirements.txt.