Smokeping

Smokeping
allows you to probe a list of servers, store that data using
RRDtool, and generate statistical charts based on RRDtool's
output. Smokeping consists of two parts. A daemon runs in the
background pinging and collecting data at set intervals. A web
interface displays that data in the form of graphs.
This wiki page covers a basic setup of the smokeping daemon and the CGI webinterface.
Installation
This section covers the installation of
Smokeping
using the
smokeping
package. FastCGI on Apache will be setup as described in
Apache and FastCGI
.
The smokeping package consists of two parts:
The smokeping daemon and configs in
/etc/smokeping/
. This daemon performs the monitoring.
The smokeping "htdocs" in
/srv/http/smokeping
. These will be used by the web interface.
In addition to the
smokeping
package, you will need:
A tool that smokeping can use for monitoring.
fping
is the simplest and default method for simple ping probes.
apache
and
mod_fcgid
AUR
for the web interface if you are using Apache.
fcgiwrap
and start and enable fcgiwrap.socket if you are using Nginx
An image cache directory that the FastCGI script can write to, e.g.
/srv/smokeping/imgcache
A data directory that the smokeping daemon can write to, and the FastCGI script can read, e.g.
/srv/smokeping/data
To ensure that the main configuration file is readable by the smokeping daemon.
Optional Prerequisites
If you want to use other probes such as the DNS or http probe you will
need other packages as shown below. The configuration of these will is not covered by this wiki page.
Probe
Package Needed
Curl
curl
DNS
bind
(for the dig utility)
EchoPing
echoping
AUR
SSH
openssh
TelnetIOSPing
perl-net-telnet
AnotherDNS
perl-net-dns
LDAP
perl-ldap
LDAP (tls)
perl-io-socket-ssl
Authen
perl-authen-radius
Configuration
Smokeping requires you to edit a few files. The unedited files end
with the
.dist
extension. Rename the
.dist
files in
/etc/smokeping
to remove the suffix. The
find
command does this and
prints out each file that is being renamed and needs editing:
# cd /etc/smokeping
# find . -name '*.dist' -print -execdir sh -c 'mv {} $(basename {} .dist)' \;
# mv /srv/http/smokeping/smokeping.fcgi.dist /srv/http/smokeping/smokeping.fcgi
Editing the configuration file
Next, edit the
/etc/smokeping/config
file; this is smokeping's main configuration file. A brief description of the sections is followed by a complete example.
Note:
most paths need changing, potentially from
/usr/etc/
to
/etc/smokeping
, but confirm each one.
Paths for the various
.dist
files can be found with
pacman -Ql smokeping
The
General
section of the
/etc/smokeping/config
file is the easiest to edit. Personalize the
top of the configuration file to match your information. The comments describe each field. Note that if you do not have the
sendmail
program installed (ie from postfix or sendmail) then use something else instead like
/bin/false
. The file you specify must exist or smokeping will error out.
The
Alerts
section does not require any changes for this example.
The
Database
section does not require any changes.
In the
Presentation
section the path to the template filename needs to be updated, (ie
/etc/smokeping/basepage.html
)
The
Probes
section specifies which probes are active. By default only the
FPing
probe is enabled. This section does not require any changes.
The
Slaves
section is not needed in this minimal example, so it can be either commented out or removed. Note that if you use the
smokeping_secrets
setting in the
Slaves
section, you will have to make that file unreadable to the rest of the world, or else smokeping will error:
chmod 600 /etc/smokeping/smokeping_secrets
.
The
Targets
section specifies which hosts will be probed (pinged in our example). Customize it so that it probes the host(s) you would like to collect statistics on, as shown in the example below.
You can learn more about the Smokeping configuration file with the examples at
https://oss.oetiker.ch/smokeping/doc/smokeping_examples.en.html
/etc/smokeping/config
*** General ***

owner     = Your Name Here                            # your name
contact   = your.email@host.bla                       # your email
mailhost  = your.smtp.server.bla                      # your mail server
sendmail  = /bin/false                                # where the sendmail program is
imgcache  = /srv/smokeping/imgcache                   # filesystem directory where we store files
imgurl    = imgcache                                  # URL directory to find them
datadir   = /srv/smokeping/data                       # where we share data between the daemon & webapp
piddir    = /var/run                                  # filesystem directory to store PID file
cgiurl    = http://localhost/smokeping/smokeping.fcgi  # exterior URL
smokemail = /etc/smokeping/smokemail   
tmail     = /etc/smokeping/tmail
syslogfacility = local0
# each probe is now run in its own process
# disable this to revert to the old behaviour
# concurrentprobes = no

*** Alerts ***
to = alertee@address.somewhere
from = smokealert@company.xy

+someloss
type = loss
# in percent
pattern = >0%,*12*,>0%,*12*,>0%
comment = loss 3 times  in a row

*** Database ***

step     = 300
pings    = 20

# consfn mrhb steps total

AVERAGE  0.5   1  1008
AVERAGE  0.5  12  4320
    MIN  0.5  12  4320
    MAX  0.5  12  4320
AVERAGE  0.5 144   720
    MAX  0.5 144   720
    MIN  0.5 144   720

*** Presentation ***

template = /etc/smokeping/basepage.html

+ charts

menu = Charts
title = The most interesting destinations
++ stddev
sorter = StdDev(entries=>4)
title = Top Standard Deviation
menu = Std Deviation
format = Standard Deviation %f

++ max
sorter = Max(entries=>5)
title = Top Max Roundtrip Time
menu = by Max
format = Max Roundtrip Time %f seconds

++ loss
sorter = Loss(entries=>5)
title = Top Packet Loss
menu = Loss
format = Packets Lost %f

++ median
sorter = Median(entries=>5)
title = Top Median Roundtrip Time
menu = by Median
format = Median RTT %f seconds

+ overview 

width = 600
height = 50
range = 10h

+ detail

width = 600
height = 200
unison_tolerance = 2
nodata_color = ffb0b0
loss_background = yes

"Last 3 Hours"    3h
"Last 30 Hours"   30h
"Last 10 Days"    10d
"Last 400 Days"   400d

*** Probes ***

+ FPing

binary = /usr/bin/fping

*** Targets ***

probe = FPing

menu = Top
title = Network Latency Grapher
remark = Welcome to the SmokePing website of Arch User. \
         Here you will learn all about the latency of our network.

+ targets
menu = Targets
title = Targets

++ CloudflareDNS

menu = Cloudflare DNS
title = Cloudflare DNS server
host = 1.1.1.1

++ GoogleDNS

menu = Google DNS
title = Google DNS server
host = 8.8.8.8

++ MultiHost

menu = Multihost example
title = CloudflareDNS and Google DNS
host = /targets/CloudflareDNS /targets/GoogleDNS
Notes on the smokeping configuration file syntax
Each
+
character defines a section in the hierarchy. Spaces are not allowed in the section names. Period and forward slashes should probably also be avoided in section names. This is probably because the RRD files are stored under the data directory with the same exact names as the sections.
In the
Targets
section, you can define
host
as either a real host name or the path to another section to generate a multiple host chart, as shown in the
MultiHost
example above.
Setup the rest of the system
Setup the extra directories referenced by the configuration file:
# mkdir -p /srv/smokeping/data
# mkdir -p /srv/smokeping/imgcache
# chown -R smokeping:smokeping /srv/smokeping
# chown -R http:http /srv/smokeping/imgcache
# chmod a+rx /srv/smokeping
# chmod -R a+rx /srv/smokeping/data
Since the smokeping configuration is read by both the smokeping daemon and the FastCGI scripts, so it needs to be readable:
# chmod a+rx /etc/smokeping
# chmod a+r /etc/smokeping/config
Notes on Smokeping in Master Slave configuration
A master-slave has one additional wrinkle, though the principles are the same.
For slaves, smokeping IS NOT writing the data to the RRD files - they're getting pushed there by the web-server. So, in master-slave mode you need to be sure that the user your web-server is running as [
nginx
running as user
http
] also has WRITE permissions to the RRD files (under
/srv/smokeping/data
).
When running Smokeping in
Master / Slave
mode, the master has effective permissions of the user the smokeping slave runs as. Do NOT run slaves as root. Read the
security considerations
first.
The
Smokeping FAQ
on their wiki details the complex permission requirements. If your master slave config is still not working, use
setfacl
to provide the minimum required permissions:
On the master, give both `http` and `smokeping` full access (rwx) to `/srv/smokeping/*`
# setfacl -R -m u:http:rwx /srv/smokeping
# setfacl -R -m u:smokeping:rwx /srv/smokeping

Set specific read permissions for files inside `/etc/smokeping`
# setfacl -m u:http:r /etc/smokeping/smokeping_secrets
# setfacl -m u:smokeping:r /etc/smokeping/config /etc/smokeping/smokeping_secrets
Slaves require no config file, they pull their config from the master, and just need to read
/etc/smokeping/smokeping_secrets
:
# /usr/bin/smokeping --master-url=https://smokeping.site/smokeping.fcgi
# setfacl -m u:http:r /etc/smokeping/smokeping_secrets
A sample slave's systemd unit:
[Unit]
Description=Smokeping Slave: my.slave.net
Documentation=man:smokeping_master_slave(7)

[Service]
ExecStart=/usr/sbin/smokeping \
  --master-url=https://smokeping.site/smokeping.fcgi \
  --cache-dir=/var/lib/smokeping \
  --shared-secret=/etc/smokeping/smokeping_secrets \
  --pid-dir=/run/smokeping \
  --slave-name=my.slave.net 
# The --slave-name should be an exact match to both the master's config and smokeping_secrets

ExecReload=/bin/kill -HUP $MAINPID  

[Service]
Type=forking
RuntimeDirectory=smokeping
PIDFile=/run/smokeping/smokeping.pid
User=smokeping
Group=smokeping
Restart=on-failure
Start and enable daemon
Start
and enable
smokeping.service
. Then check that it is running.
Setup web frontend
Apache
Edit
/etc/httpd/conf/httpd.conf
so that it includes:
LoadModule fcgid_module modules/mod_fcgid.so
<IfModule fcgid_module>
  AddHandler fcgid-script .fcgi
</IfModule>

Alias /smokeping/imgcache /srv/smokeping/imgcache
Alias /smokeping /srv/http/smokeping

<Directory "/srv/smokeping/imgcache">
  AllowOverride all
  Require all granted
</Directory>

<Directory "/srv/http/smokeping">
 Options FollowSymLinks ExecCGI
 AllowOverride all
 Require all granted
</Directory>
Start
Apache via the
httpd.service
.
Check that
http://localhost/smokeping/smokeping.fcgi
loads. The first data should appear after a couple of minutes.
If the fonts in the graphs are unreadable, you may need to install the
ttf-dejavu
package.
Caddy
Thanks to the
Caddy community
and with this configuration file
/etc/smokeping/config
and with
enable
fcgiwrap.socket
/etc/caddy/caddy.conf.d/smokeping.conf
smokeping.example.com {

        log stdout
        errors

        tls john@example.com
        root /srv/http/smokeping

        fastcgi / unix:/var/run/fcgiwrap.sock {
                env SCRIPT_FILENAME /srv/http/smokeping/smokeping.fcgi.dist
        }
}

smokeping.example.com/js {
        root /srv/http/smokeping/js
}

smokeping.example.com/css {
        root /srv/http/smokeping/css
}

smokeping.example.com/cache {
        root /var/cache/smokeping
}
An updated version of this Caddy configuration for smokeping -
https://gist.github.com/Strykar/4df1eb8aebc4d5f7039f6045301352c7
Caddy 2
Thanks to the
francislavoie and Caddy community
and to the
Caddy docs
.
Start and enable fcgiwrap.socket!
/etc/caddy/caddy.conf.d/smokeping.conf
smokeping.example.com {

    handle /js/* {
        root * /srv/http/smokeping/
        file_server
    }
    handle /css/* {
        root * /srv/http/smokeping/
        file_server
    }
    handle /imgcache/* {
        root * /srv/http/smokeping/
        file_server
    }
    handle /images/* {
        root * /srv/http/smokeping/
        file_server
    }

    handle {
        root * /srv/http/smokeping/
        reverse_proxy unix//var/run/fcgiwrap.sock {
            transport fastcgi {
                env SCRIPT_FILENAME /srv/http/smokeping/smokeping.fcgi
                split ""
            }
        }
    }
}
Lighttpd
First install
lighttpd
and
fcgi
.
Edit the
Lighttpd
configuration file and ensure you have at least all of the following configuration directives:
/etc/lighttpd/lighttpd.conf
server.document-root    = "/srv/http"
server.follow-symlink   = "enable"

server.modules += ("mod_fastcgi")
fastcgi.server = (
        ".fcgi" =>
        ((
                "bin-path" => "/srv/http/smokeping/smokeping.fcgi",
                "host" => "localhost",
                "port" => 8001,
        )),
)
Start/enable
the
lighttpd.service
Systemd
should show
smokeping_cgi
being managed by
lighttpd.service
.
$ systemctl status lighttpd
● lighttpd.service - Lighttpd Web Server
   Loaded: loaded (/usr/lib/systemd/system/lighttpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-12-26 14:34:42 PST; 1 day 7h ago
  Process: 17117 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
 Main PID: 28321 (lighttpd-angel)
    Tasks: 6 (limit: 4915)
   Memory: 128.9M
   CGroup: /system.slice/lighttpd.service
           ├─17119 /usr/bin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
           ├─17126 /usr/bin/perl /usr/bin/smokeping_cgi /etc/smokeping/config
           ├─17127 /usr/bin/perl /usr/bin/smokeping_cgi /etc/smokeping/config
           ├─17128 /usr/bin/perl /usr/bin/smokeping_cgi /etc/smokeping/config
           ├─17129 /usr/bin/perl /usr/bin/smokeping_cgi /etc/smokeping/config
           └─28321 /usr/bin/lighttpd-angel -D -f /etc/lighttpd/lighttpd.conf
Finally, add a symbolic link to allow images to load:
# ln -s -t /srv/http/smokeping /srv/smokeping/imgcache
Nginx
Ensure that
fcgiwrap.socket
and
nginx.service
are both running via systemctl.
Add a server block for smokeping to
/etc/nginx/nginx.conf
, following is an example for a TLS enabled block.
server {
       server_name smokeping.example.com;
       listen 443 ssl http2;
       listen [::]:443 ssl http2;
       root /srv/http/smokeping/;
       index smokeping.fcgi;
       gzip off;
       access_log /var/log/nginx_smokeping.log combined;
       error_log /var/log/nginx_smokeping.log;
       location ~ \.fcgi$ {
           fastcgi_intercept_errors on;
           include /etc/nginx/fastcgi_params;
           fastcgi_param SCRIPT_FILENAME /srv/http/smokeping/smokeping.fcgi;
           fastcgi_pass unix:/var/run/fcgiwrap.sock;
       }
       location ^~ /imgcache {
           alias /srv/smokeping/imgcache;
           gzip off;
       }
   }
Verify that your configuration is fine via
nginx -t
as the root user and reload the configuration via
nginx -s reload
as the root user.
Advanced Configuration
Smokeping is a powerful tool that can be configured in many ways. You can setup many different types of probes. You can setup slave smokeping servers that can send their statistics and show you probes from other servers.
You can also create your custom probes in perl, see some third-party probes
on their wiki
. These options are currently not covered by this guide, please consult the documentation on the
Smokeping website
instead.
Also see -
https://github.com/oetiker/SmokePing/wiki/FAQ
Notes
Smoketrace (Tr.cgi)
The
SmokeTraceroute
utility is gone since v2.5.0 according to the
release notes
.