Package: icinga-passive
Icinga Passive SenderQuick startSend metrics from arbitrary shell commands into an Icinga2 as easy as:./icinga_passive_sender.py -s 'SERVICE_NAME' --host HOST_NAME -c "df |grep '/$' | awk '{print $3}'"Or with preset commands:./icinga_passive_sender.py -s 'SERVICE_NAME' --host HOST_NAME -p root_diskSet warning and critical thresholds:./icinga_passive_sender.py -s 'SERVICE_NAME' --host HOST_NAME -p root_disk --warn 200000000 --crit 220000000DescriptionIcinga2 monitoring platform has the option to perform the so called "passive"
checks. These are little metrics collected within the target machine itself
and sent to Icinga2 via REST API.This tool allows us to collect an send those metrics and send them to the
Icinga2 REST API as passive data in a convenient way, without worrying about the request syntax.It is specially useful in cases where we cannot (don't want to) install any client on the target machine and we do not want to write our ownCommand + REST APIwrapper. In fact, this whole tool is just (AFAIK) the easiest tool to send the STDOUT of a shell command to the Icinga2 API.Icinga2 Server setupAPI userFirst of all, make sure thatapifeature is enabled inicinga2 feature list.We need a user to communicate with the Icinga2 API. For best practices, we can create an special API user to perform these passive check uploads, with just the single required permission to upload the data.For that, we can include the following block into/etc/icinga2/conf.d/api-users.conf:object ApiUser "passive" {
    permissions = [ "actions/process-check-result" ]
    password = "WRITE_SOME_GOOD_PASSWORD_HERE"
}... and off course, after every configuration change, reload theicinga2service.This way, if the credentials get compromised, in theory, the worst thing that could happen is that someone overloads or system with useless data.Passive serviceThe most common Icinga services are usuallyactiveones, which means that the Icinga server triggers locally some plugin (script or executable), which collects some information about some given host.If we want to collect information from inside a target host, we can setup the "Icinga2 agent" on the target, and let Icinga2 server trigger the data collection plugin locally on the target. This is good for example, for checking CPU load, disk usage, ...But what if for some reason, we cannot install the Icinga2 agent or some other supported client on the target host? ... In that case, we can still inject our information directly onto the Icinga2 server by using apassive service.On a passive service, the Icinga server will have a "passive" role, and just wait to receive the information from somewhere else (i.e. the REST API).Here is an example for a passive service configuration:template Service "passive-service" {
  import "generic-service"
  check_command = "passive"
  enable_active_checks = false
  enable_passive_checks = true
  check_interval = 1d
}It is just like a regular service, but it contains theenable_passive_checksset asTrueand thecheck_commandispassive(from the ITL). I put a quite longcheck_intervalin some cases, in order to avoid the service complaining because it did not get any update recently.Finally, just apply the service to some host, group or creativeassignrule:apply Service "TEST service" {
  import "passive-service"
  assign where host.name == "WEB_SERVER"
}Client setupThe client setup is straight-forward. It will need to have Python (2 or 3) installed (the tool works on both), and some basic shell from where to retrieve the actual information to gather.The first time the tool is run, it will as for the login credentials and store them in~/.icinga_api_creds, as:./icinga_passive_sender.py -s 'TEST service' --host WEB_SERVER -p root_disk
Enter the API endpoint for Icinga2, for example:
  "https://icinga2.example.com:5665"
Icinga2 endpoint: "https://icinga2.fritz.box:5665"
Enter the username for the Icinga2 API: "passive"
Enter the password for the Icinga2 API: "WRITE_THE_GOOD_PASSWORD_FROM_ABOVE"After that, the current user on the (target) system, will be able to send data to the REST API.TroubleshootingThere are many things that might go wrong. I figured out some of those and wrote an option-tor--testwhich will check the three most probably issues:Check the connect to the REST API:Ensures the credentials are correctChecks if we have minimum permissions to upload the dataIn case the API user has permissions to request host objects:Check that the requested host is defined in the Icinga2 serverIn case the API user has permissions to request service objects:Ensure that the given service exists, it is assigned to the target host and that it accepts passive dataCommand presetsThe original idea was to provide a way to easily turn a command into an API request. But sometimes, comming up with a command to retrieve a given parameter might be tricky, so I wrote a file calledlib_presets.pyin which I wrote some convenient presets to use instead of custom commands.Presets are a quick way to get a value by sacrifying flexibiilty. For example, for active checks, there is thecheck_diskplugin, with tons of options to chose disks, partitions, paths, ignores, ... Most of the times, I am happy to know how full is my root partition, so theroot_diskpreset will retrieve that (on Unix based systems).In order to list the available presets:./icinga_passive_sender.py -s 'TEST service' --host rasplex -p root_disk --list_presets
- total_cpu:  Total percentage of CPU capacity used
 # grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage }'

- lm_cpu_temp:  CPU temperature in Celsius from lm-sensors
 # sensors -u | grep -A1 'Package id 0:' |tail -1 | awk '{print $NF}'

- root_disk:  The the amount of used bytes on the main/root partition
 # df |grep '/$' | awk '{print $3}'

- total_mem:  Total memory RAM in "active" state
 # awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.0f\n", ((total - free) - (buffers + cached)) / 1024}' /proc/meminfoWhich means, these two commands are the same:./icinga_passive_sender.py -s 'TEST service' --host WEB_SERVER -c "df |grep '/$' | awk '{print \$3}'"and./icinga_passive_sender.py -s 'TEST service' --host WEB_SERVER -p root_disk