Package: caduceus
CaduceusWhat is Caduceus?Caduceus isthat long stick with the intertwined snakesthat Hermes used to carry around.
It is also a service that will notify you if your scheduled tasks/cronjobs fail to run.MotivationYou know how you set all these cronjobs to run, and added fancy error reporting and things, only to realize too late that this doesn't help you at all when the server has been down for a month and nobody noticed?
Caduceus won't let this happen again.Rather than trigger on failure, Caduceus triggers on absence of success.
Services have to actively check in (by visiting a URL), and, if they don't, Caduceus notifies you by email that the task has failed.
If the service starts working again, Caduceus will notify you of that as well.InstallationTo install Caduceus, you can just get it from PyPI:pipinstallcaduceusAlternatively, you can pull the Docker image:dockerpullregistry.gitlab.com/stavros/caduceus:latestUsageTo run Caduceus, you need to configure it.
This is done by placing a file calledcaduceus.tomlin the directory you want to run Caduceus in.
That directory is where the Caduceus SQLite database will be created.If you installed Caduceus from the repo or withpip, just run it:caduceusIt will load the configuration from the file, create its database and start running onhttp://localhost:5000/.To run it via Docker:dockerrun-v(pwd):/caduceusregistry.gitlab.com/stavros/caduceus:latestConfigurationHere's a sample configuration file (which is also available ascaduceus.toml.examplein the repository):[config]# An optional secret key to use for checking in.secret_key="somelongkey"[alerting.email]# Where you want the notification emails sent if services don't check in.recipient_emails=["notifyme@example.com","otherdev@example.com"]# SMTP server configuration, for sending email.from_addr="caduceus@example.com"hostname="example.com"port=25username="myuser"password="mypassword"encryption="none"# Can also be "ssl" or "starttls".[alerting.telegram]apikey="#############:####################"chat_id="99999999"[alerts]# How alerts will be sent by default.default_channels=["telegram"]# Your alerts go here.# An alert needs a short name (here, `cron`), and an interval it needs to check in by.[alerts.cron]every="1h"# You can override the alerting channels per-alert.channels=["email"][alerts.backups]every="1d"channels=["email","telegram"]# For alerts that use email, you can also override the recipient emails.recipient_emails=["thirdemail@example.com"][alerts.alwaysfail]every="1s"# You can tell Caduceus to only notify every minute, instead of every second,# to prevent spam.notify_every="1m"The above config defines three services,raidscrub,backupsandalwaysfail.raidscrubneeds to check in every hour,backupsneeds to check in every day, andalwaysfailneeds to check in every second.
That's why it was called that.However, as emailing you every second would get spammy,notify_everyis set to one minute, so Caduceus will only email you once a minute, even though the alert will be considered failed if it doesn't check in once per second.
You will get an initial email right when the failure is detected (there is a 10 second notification resolution) and then emails every minute after that.Always leave a bit of leeway in your tasks, to account for running time.
If a task starts at midnight one day and runs for an hour, it'll check in at 1am.
If the next day it runs for 61 minutes, it will check in more than a day later, so you'll get a "failed" email.
To avoid that, add an buffer of 10% or so to your alerts.Checking inChecking in is done by retrieving a URL on the server.
The URL for checking in and resetting the alert timer is/reset/<alert name>/.
For example, to check in tobackupsif you haven't specified asecret_key(and if Caduceus is running on example.com), you'd simply do:curlhttp://example.com/reset/backups/If youdidspecify a secret key, just include it:curlhttp://example.com/reset/backups/?key=<yoursecret_key>If your alert is set up for, say, one hour, and your task does not check in, you will get an email one hour after its last checkin, saying "your task has not checked in".
If it still doesn't check in, you'll get another email an hour after that, then an hour after that, etc, until it does, at which point you'll get an email saying that the job is now fine.