Project: Icinga (Observability and Analysis / Observability)
File: 07-agent-based-monitoring.md

If the remote services are not directly accessible through the network, a local agent installation exposing the results to check queries can become handy. Prior to installing and configuration an agent service, evaluate possible options based on these requirements: Security (authentication, TLS certificates, secure connection handling, etc.) Connection direction Master/satellite can execute commands directly or Agent sends back passive/external check results Availability on specific OS types and versions Packages available Configuration and initial setup Updates and maintenance, compatibility Available agent types: on Linux/Unix and Windows on Linux/Unix on Linux/Unix and hardware as passive check results for passive external check results and on Windows For the most common setups on Linux/Unix and Windows, we recommend to setup the Icinga agent in a . Key benefits: Directly integrated into the distributed monitoring stack of Icinga Works on Linux/Unix and Windows Secure communication with TLS Connection can be established from both sides. Once connected, command execution and check results are exchanged. Master/satellite connects to agent Agent connects to parent satellite/master Same configuration language and binaries Troubleshooting docs and community best practices Follow the setup and configuration instructions . On Windows hosts, the Icinga agent can query a local NSClient++ service for additional checks in case there are no plugins available. Tip This is the recommended way for systems where the Icinga agent is not available Be it specific hardware architectures, old systems or forbidden to install an additional software. This method uses the SSH service on the remote host to execute an arbitrary plugin command line. The output and exit code is returned and used by the core. The `checkbyssh` plugin takes care of this. It is available in the package. For your convenience, the Icinga template library provides the CheckCommand already. SSH key pair for the Icinga daemon user. In case the user has no shell, temporarily enable this. When asked for a passphrase, do not set it and press enter. ```bash sudo su - icinga ssh-keygen -b 4096 -t rsa -C "icinga@$(hostname) user for checkbyssh" -f $HOME/.ssh/id_rsa ``` On the remote agent, create the icinga user and generate a temporary password. ```bash useradd -m icinga passwd icinga ``` Copy the public key from the Icinga server to the remote agent, e.g. with `ssh-copy-id` or manually into `/home/icinga/.ssh/authorized_keys`. This will ask for the password once. ```bash sudo su - icinga ssh-copy-id -i $HOME/.ssh/id_rsa icinga@ssh-agent1.localdomain ``` After the SSH key is copied, test at the connection at least once and accept the host key verification. If you forget about this step, checks will become UNKNOWN later. ```bash ssh -i $HOME/.ssh/id_rsa icinga@ssh-agent1.localdomain ``` After the SSH key login works, disable the previously enabled

Remote agent user's password with `passwd -l icinga` Local icinga user terminal Also, ensure that the permissions are correct for the `.ssh` directory as otherwise logins will fail. `.ssh` directory: 700 `.ssh/id_rsa.pub` public key file: 644 `.ssh/id_rsa` private key file: 600 First, create a host object which has SSH configured and enabled. Mark this e.g. with the custom variable `agent_type` to later use this for service apply rule matches. Best practice is to store that in a specific template, either in the static configuration or inside the Director. ``` template Host "ssh-agent" { check_command = "hostalive" vars.agent_type = "ssh" vars.os_type = "linux" } object Host "ssh-agent1.localdomain" { import "ssh-agent" address = "192.168.56.115" } ``` Example for monitoring the remote users: ``` apply Service "users" { checkcommand = "byssh" vars.bysshcommand = [ "/usr/lib/nagios/plugins/check_users" ] // Follows the same principle as with command arguments, e.g. for ordering vars.byssharguments = { "-w" = { value = "$users_wgreater$" // Can reference an existing custom variable defined on the host or service, evaluated at runtime } "-c" = { value = "$users_cgreater$" } } vars.users_wgreater = 3 vars.users_cgreater = 5 assign where host.vars.ostype == "linux" && host.vars.agenttype == "ssh" } ``` A more advanced example with better arguments is shown in . The SNMP daemon runs on the remote system and answers SNMP queries by plugin scripts. The package provides the `check_snmp` plugin binary, but there are plenty of for specific use cases already around, for example monitoring Cisco routers. The following example uses the CheckCommand and sets the `snmp_oid` custom variable. A service is created for all hosts which have the `snmp-community` custom variable. ``` template Host "snmp-agent" { check_command = "hostalive" vars.agent_type = "snmp" vars.snmp_community = "public-icinga" } object Host "snmp-agent1.localdomain" { import "snmp-agent" } ``` ``` apply Service "uptime" { import "generic-service" check_command = "snmp" vars.snmp_oid = "1.3.6.1.2.1.1.3.0" vars.snmp_miblist = "DISMAN-EVENT-MIB" assign where host.vars.agenttype == "snmp" && host.vars.snmpcommunity != "" } ``` If no `snmp_miblist` is specified, the plugin will default to `ALL`. As the number of available MIB files on the system increases so will the load generated by this plugin if no `MIB` is specified. As such, it is recommended to always specify at least one `MIB`. Additional SNMP plugins are available using the . For network monitoring, community members advise to use for example. SNMP Traps can be received and filtered by using and specific trap handlers passing the check results to Icinga 2. Following the SNMPTT documentation and the Icinga external command syntax found we can create generic services that can accommodate any number of hosts for a given scenario. A simple example might be monitoring host reboots indicated by an SNMP agent reset. Building the event to auto reset after dispatching a notification is

Setup the manual check parameters to reset the event from an initial unhandled state or from a missed reset event. Add a directive in `snmptt.conf` ``` EVENT coldStart .1.3.6.1.6.3.1.1.5.1 "Status Events" Normal FORMAT Device reinitialized (coldStart) EXEC echo "[$@] PROCESSSERVICECHECK_RESULT;$A;Coldstart;2;The snmp agent has reinitialized." >> /var/run/icinga2/cmd/icinga2.cmd SDESC A coldStart trap signifies that the SNMPv2 entity, acting in an agent role, is reinitializing itself and that its configuration may have been altered. EDESC ``` Define the `EVENT` as per your need. Construct the `EXEC` statement with the service name matching your template applied to your n hosts. The host address inferred by SNMPTT will be the correlating factor. You can have snmptt provide host names or ip addresses to match your Icinga convention. Note Replace the deprecated command pipe EXEC statement with a curl call to the REST API action . Add an `EventCommand` configuration object for the passive service auto reset event. ``` object EventCommand "coldstart-reset-event" { command = [ ConfigDir + "/conf.d/custom/scripts/coldstartresetevent.sh" ] arguments = { "-i" = "$service.state_id$" "-n" = "$host.name$" "-s" = "$service.name$" } } ``` Create the `coldstartresetevent.sh` shell script to pass the expanded variable data in. The `$service.state_id$` is important in order to prevent an endless loop of event firing after the service has been reset. ```bash SERVICESTATEID="" HOST_NAME="" SERVICE_NAME="" show_help() { cat <<-EOF Usage: ${0##*/} [-h] -n HOSTNAME -s SERVICENAME Writes a coldstart reset event to the Icinga command pipe. -h Display this help and exit. -i SERVICESTATEID The associated service state id. -n HOST_NAME The associated host name. -s SERVICE_NAME The associated service name. EOF } while getopts "hi:n:s:" opt; do case "$opt" in h) show_help exit 0 ;; i) SERVICESTATEID=$OPTARG ;; n) HOST_NAME=$OPTARG ;; s) SERVICE_NAME=$OPTARG ;; '?') show_help exit 0 ;; esac done if [ -z "$SERVICESTATEID" ]; then show_help printf "\n Error: -i required.\n" exit 1 fi if [ -z "$HOST_NAME" ]; then show_help printf "\n Error: -n required.\n" exit 1 fi if [ -z "$SERVICE_NAME" ]; then show_help printf "\n Error: -s required.\n" exit 1 fi if [ "$SERVICESTATEID" -gt 0 ]; then echo "[`date +%s`] PROCESSSERVICECHECKRESULT;$HOSTNAME;$SERVICE_NAME;0;Auto-reset (`date +"%m-%d-%Y %T"`)." >> /var/run/icinga2/cmd/icinga2.cmd fi ``` Note Replace the deprecated command pipe EXEC statement with a curl call to the REST API action . Finally create the `Service` and assign it: ``` apply Service "Coldstart" { import "generic-service-custom" check_command = "dummy" event_command = "coldstart-reset-event" enable_notifications = 1 enableactivechecks = 0 enablepassivechecks = 1 enable_flapping = 0 volatile = 1 enable_perfdata = 0 vars.dummy_state = 0 vars.dummy_text = "Manual reset." vars.sla = "24x7" assign where (host.vars.os == "Linux" || host.vars.os == "Windows") } ``` A more complex example might be passing dynamic data from a traps varbind list for a backup scenario where the backup software dispatches status updates. By utilizing active and passive checks, the older freshness concept can be

By defining the active check as a hard failed state, a missed backup can be reported. As long as the most recent passive update has occurred, the active check is bypassed. Add a directive in `snmptt.conf` ``` EVENT enterpriseSpecific <YOUR OID> "Status Events" Normal FORMAT Enterprise specific trap EXEC echo "[$@] PROCESSSERVICECHECK_RESULT;$A;$1;$2;$3" >> /var/run/icinga2/cmd/icinga2.cmd SDESC An enterprise specific trap. The varbinds in order denote the Icinga service name, state and text. EDESC ``` Define the `EVENT` as per your need using your actual oid. The service name, state and text are extracted from the first three varbinds. This has the advantage of accommodating an unlimited set of use cases. Note Replace the deprecated command pipe EXEC statement with a curl call to the REST API action . Create a `Service` for the specific use case associated to the host. If the host matches and the first varbind value is `Backup`, SNMPTT will submit the corresponding passive update with the state and text from the second and third varbind: ``` object Service "Backup" { import "generic-service-custom" host_name = "host.domain.com" check_command = "dummy" enable_notifications = 1 enableactivechecks = 1 enablepassivechecks = 1 enable_flapping = 0 volatile = 1 maxcheckattempts = 1 check_interval = 87000 enable_perfdata = 0 vars.sla = "24x7" vars.dummy_state = 2 vars.dummy_text = "No passive check result received." } ``` Whenever the remote agent cannot run the Icinga agent, or a backup script should just send its current state after finishing, you can use the as secure transport and send . Use the API action to send the external passive check result. You can either use `curl` or implement the HTTP requests in your preferred programming language. Examples for API clients are available in . Feeding check results from remote hosts requires the host/service objects configured on the master/satellite instance. works on both Windows and Linux platforms and is well known for its magnificent Windows support. There are alternatives like the WMI interface, but using `NSClient++` will allow you to run local scripts similar to check plugins fetching the required output and performance counters. Tip Best practice is to use the Icinga agent as secure execution bridge (`checknt` and `checknrpe` are considered insecure) and query the NSClient++ service . You can use the `check_nt` plugin from the Monitoring Plugins project to query NSClient++. Icinga 2 provides the for this: Example: ``` object Service "disk" { import "generic-service" host_name = "remote-windows-host" check_command = "nscp" vars.nscp_variable = "USEDDISKSPACE" vars.nscp_params = "c" vars.nscp_warn = 70 vars.nscp_crit = 80 } ``` For details on the `NSClient++` configuration please refer to the . The most popular plugin is . Check WMI Plus uses the Windows Management Interface (WMI) to check for common services (cpu, disk, sevices, eventlog) on Windows machines. It requires the open source wmi client for Linux. Community examples: