Package: daemonocle
daemonocle is a library for creating your own Unix-style daemons written in Python. It solves many
problems that other daemon libraries have and provides some really useful features you don’t often
see in other daemons.Table of ContentsInstallationBasic UsageRationaleThe ProblemThe SolutionOther Useful FeaturesThestatusActionSlightly SmarterrestartActionSelf-ReloadShutdown CallbackNon-Detached ModeFile Descriptor HandlingDetailed UsageActionsIntegration with clickBugs, Requests, Questions, etc.Release Historyv1.2.3 (2021-04-22)v1.2.1 and v1.2.2 (2021-02-16)v1.2.0 (2021-02-15)v1.1.1 (2020-12-31)v1.1.0 (2020-12-16)v1.0.2 (2020-07-12)v1.0.1 (2016-04-17)v1.0.0 (2016-04-17)v0.8 (2014-08-01)v0.7 (2014-06-23)v0.6 (2014-06-10)v0.5 (2014-06-09)v0.4 (2014-05-19)v0.3 (2014-05-14)v0.2 (2014-05-12)v0.1 (2014-05-11)InstallationTo install via pip:pip install daemonocleOr download the source code and install manually:git clone https://github.com/jnrbsn/daemonocle.git
cd daemonocle/
python setup.py installBasic UsageHere’s areally reallybasic example:importsysimporttimeimportdaemonocle# This is your daemon. It sleeps, and then sleeps again.defmain():whileTrue:time.sleep(10)if__name__=='__main__':daemon=daemonocle.Daemon(worker=main,pid_file='/var/run/daemonocle_example.pid',)daemon.do_action(sys.argv[1])And here’s the same example with logging and aShutdown Callback:importloggingimportsysimporttimeimportdaemonocledefcb_shutdown(message,code):logging.info('Daemon is stopping')logging.debug(message)defmain():logging.basicConfig(filename='/var/log/daemonocle_example.log',level=logging.DEBUG,format='%(asctime)s[%(levelname)s]%(message)s',)logging.info('Daemon is starting')whileTrue:logging.debug('Still running')time.sleep(10)if__name__=='__main__':daemon=daemonocle.Daemon(worker=main,shutdown_callback=cb_shutdown,pid_file='/var/run/daemonocle_example.pid',)daemon.do_action(sys.argv[1])And here’s what it looks like when you run it:user@host:~$ python example.py start
Starting example.py ... OK
user@host:~$ python example.py status
example.py -- pid: 1234, status: running, uptime: 1m, %cpu: 0.0, %mem: 0.0
user@host:~$ python example.py stop
Stopping example.py ... OK
user@host:~$ cat /var/log/daemonocle_example.log
2014-05-04 12:39:21,090 [INFO] Daemon is starting
2014-05-04 12:39:21,091 [DEBUG] Still running
2014-05-04 12:39:31,091 [DEBUG] Still running
2014-05-04 12:39:41,091 [DEBUG] Still running
2014-05-04 12:39:51,093 [DEBUG] Still running
2014-05-04 12:40:01,094 [DEBUG] Still running
2014-05-04 12:40:07,113 [INFO] Daemon is stopping
2014-05-04 12:40:07,114 [DEBUG] Terminated by SIGTERM (15)For more details, see theDetailed Usagesection below.RationaleIf you think about it, a lot of Unix daemons don’t really know what the hell they’re doing. Have you
ever found yourself in a situation that looked something like this?user@host:~$ sudo example start
starting example ... ok
user@host:~$ ps aux | grep example
user      1234  0.0  0.0   1234  1234 pts/1    S+   12:34   0:00 grep example
user@host:~$ sudo example start
starting example ... ok
user@host:~$ echo $?
0
user@host:~$ tail -f /var/log/example.log
...Or something like this?user@host:~$ sudo example stop
stopping example ... ok
user@host:~$ ps aux | grep example
user       123  0.0  0.0   1234  1234 ?        Ss   00:00   0:00 /usr/local/bin/example
user      1234  0.0  0.0   1234  1234 pts/1    S+   12:34   0:00 grep example
user@host:~$ sudo example stop
stopping example ... ok
user@host:~$ ps aux | grep example
user       123  0.0  0.0   1234  1234 ?        Ss   00:00   0:00 /usr/local/bin/example
user      1240  0.0  0.0   1234  1234 pts/1    S+   12:34   0:00 grep example
user@host:~$ sudo kill -9 123
...Or something like this?user@host:~$ sudo example status
Usage: example {start|stop|restart}
user@host:~$ ps aux | grep example
...These are just a few examples of unnecessarily common problems. It doesn’t have to be this way.Note:You might be thinking, “Why not just write a smarter start/stop shell script wrapper
for your daemon that checks whether or not it actually started, actually stopped, etc.?”
Seriously?It doesn’t have to be this way.I believe daemons should be more self-aware. They
should handle their own problems most of the time, and your start/stop script should only be a
very thin wrapper around your daemon or simply a symlink to your daemon.The ProblemIf you’ve ever dug deep into the nitty-gritty details of how daemonization works, you’re probably
familiar with thestandard “double fork” paradigmfirst introduced
by W. Richard Stevens in the bookAdvanced Programming in the UNIX Environment. One of the problems with the standard way to implement this is that
if the final child dies immediately when it gets around to doing real work, the original parent
process (the one that actually had control of your terminal) is long gone. So all you know is that
the process got forked, but you have no idea if it actually kept running for more than a fraction of
a second. And let’s face it, one of the most likely times for a daemon to die is immediately after
it starts (due to bad configuration, permissions, etc.).The next problem mentioned in the section above is when you try to stop a daemon, it doesn’t
actually stop, and you have no idea that it didn’t actually stop. This happens when a process
doesn’t respond properly to aSIGTERMsignal. It happens more often than it should. The problem
is not necessarily the fact that it didn’t stop. It’s the fact that you didn’tknowthat it didn’t
stop. The start/stop script knows that it successfully sent the signal and so it assumes success.
This also becomes a problem when yourrestartcommand blindly callsstopand thenstart,
because it will try to start a new instance of the daemon before the previous one has exited.These are the biggest problems most daemons have in my opinion. daemonocle solves these problems and
provides many other “fancy” features.The SolutionThe problem with the daemon immediately dying on startup and you not knowing about it is solved by
having the first child (the immediate parent of the final child) sleep for one second and then callos.waitpid(pid, os.WNOHANG)to see if the process is still running. This is what daemonocle
does. So if you’re daemon dies within one second of starting, you’ll know about it.This problem with the daemon not stopping and you not knowing about it is solved by simply waiting
for the process to finish (with a timeout). This is what daemonocle does. (Note: When a timeout
occurs, it doesn’t try to send aSIGKILL. This is not always what you’d want and often not a
good idea.)Other Useful FeaturesBelow are some other useful features that daemononcle provides that you might not find elsewhere.ThestatusActionThere is astatusaction that not only displays whether or not the daemon is running and its
PID, but also the uptime of the daemon and the % CPU and % memory usage of all the processes in the
same process group as the daemon (which are probably its children). So if you have a daemon that
launches mulitple worker processes, thestatusaction will show the % CPU and % memory usage of
all the workers combined.It might look something like this:user@host:~$ python example.py status
example.py -- pid: 1234, status: running, uptime: 12d 3h 4m, %cpu: 12.4, %mem: 4.5You can even get JSON output if you call the action like this:daemon.do_action('status',json=True)If you use theIntegration with clickdescribed below, this option is available via the--jsonCLI option. You can also just get adictdirectly and programatically without
printing it to STDOUT by callingDaemon.get_status().Slightly SmarterrestartActionHave you ever tried to restart a daemon only to realize that it’s not actually running? Let me
guess: it just gave you an error and didn’t start the daemon. A lot of the time this is not a
problem, but if you’re trying to restart the daemon in an automated way, it’s more annoying to have
to check if it’s running and do either astartorrestartaccordingly. With daemonocle, if
you try to restart a daemon that’s not running, it will give you a warning saying that it wasn’t
running and then start the daemon. This is often what people expect.Self-ReloadDaemons that use daemonocle have the ability to reload themselves by simply callingdaemon.reload()wheredaemonis yourdaemonocle.Daemoninstance. The execution of the
current daemon halts whereverdaemon.reload()was called, and a new daemon is started up to
replace the current one. From your code’s perspective, it’s pretty much the same as a doing arestartexcept that it’s initiated from within the daemon itself and there’s no signal handling
involved. Here’s a basic example of a daemon that watches a config file and reloads itself when the
config file changes:importosimportsysimporttimeimportdaemonocleclassFileWatcher(object):def__init__(self,filename,daemon):self._filename=filenameself._daemon=daemonself._file_mtime=os.stat(self._filename).st_mtimedeffile_has_changed(self):current_mtime=os.stat(self._filename).st_mtimeifcurrent_mtime!=self._file_mtime:self._file_mtime=current_mtimereturnTruereturnFalsedefwatch(self):whileTrue:ifself.file_has_changed():self._daemon.reload()time.sleep(1)if__name__=='__main__':daemon=daemonocle.Daemon(pid_file='/var/run/daemonocle_example.pid')fw=FileWatcher(filename='/etc/daemonocle_example.conf',daemon=daemon)daemon.worker=fw.watchdaemon.do_action(sys.argv[1])Shutdown CallbackYou may have noticed from theBasic Usagesection above that ashutdown_callbackwas defined.
This function gets called whenever the daemon is shutting down in a catchable way, which should be
most of the time except for aSIGKILLor if your server crashes unexpectedly or loses power or
something like that. This function can be used for doing any sort of cleanup that your daemon needs
to do. Also, if you want to log (to the logger of your choice) the reason for the shutdown and the
intended exit code, you can use themessageandcodearguments that will be passed to your
callback (your callback must take these two arguments).Non-Detached ModeThis is not particularly interesting per se, but it’s worth noting that in non-detached mode, your
daemon will do everything else you’ve configured it to do (i.e.setuid,setgid,chroot,
etc.) except actually detaching from your terminal. So while you’re testing, you can get an
extremely accurate view of how your daemon will behave in the wild. It’s also worth noting that
self-reloading works in non-detached mode, which was a little tricky to figure out initially.File Descriptor HandlingOne of the things that daemons typically do is close all open file descriptors and establish new
ones forSTDIN,STDOUT,STDERRthat just point to/dev/null. This is fine most of
the time, but if your worker is an instance method of a class that opens files in its__init__()method, then you’ll run into problems if you’re not careful. This is also a problem if you’re
importing a module that leaves open files behind. For example, importing therandomstandard library module in Python 3
results in an open file descriptor for/dev/urandom.Since this “feature” of daemons often causes more problems than it solves, and the problems it
causes sometimes have strange side-effects that make it very difficult to troubleshoot, this feature
is optional and disabled by default in daemonocle via theclose_open_filesoption.Detailed UsageThedaemonocle.Daemonclass is the main class for creating a daemon using daemonocle. Here’s the
constructor signature for the class:classdaemonocle.Daemon(name=None,worker=None,detach=True,pid_file=None,work_dir='/',stdout_file=None,stderr_file=None,chroot_dir=None,uid=None,gid=None,umask=0o22,close_open_files=False,shutdown_callback=None,stop_timeout=10)And here are descriptions of all the arguments:nameThe name of your program to use in output messages. Default:os.path.basename(sys.argv[0])workerThe function that does all the work for your daemon.detachWhether or not to detach from the terminal and go into the background. SeeNon-Detached Modefor more details. Default:Truepid_fileThe path to a PID file to use. It’s not required to use a PID file, but if you don’t, you won’t
be able to use all the features you might expect. Make sure the user your daemon is running as
has permission to write to the directory this file is in.work_dirThe path to a directory to change to when the daemon starts. Note that a file system cannot be
unmounted if a process has its working directory on that file system. So if you change the
default, be careful about what you change it to. Default:"/"stdout_fileIf provided whendetach=True, the STDOUT stream will be redirected (appended) to the file
at the given path. In non-detached mode, this argument is ignored.New in version 1.1.0.stderr_fileIf provided whendetach=True, the STDERR stream will be redirected (appended) to the file
at the given path. In non-detached mode, this argument is ignored.New in version 1.1.0.chroot_dirThe path to a directory to set as the effective root directory when the daemon starts. The
default is not to do anything.uidThe user ID to switch to when the daemon starts. The default is to not switch users.gidThe group ID to switch to when the daemon starts. The default is to not switch groups.umaskThe file creation mask (“umask”) for the process. Default:0o022close_open_filesWhether or not to close all open files when the daemon detaches. Default:Falseshutdown_callbackThis will get called anytime the daemon is shutting down. It should take amessageand acodeargument. The message is a human readable message that explains why the daemon is
shutting down. It might useful to log this message. The code is the exit code with which it
intends to exit. SeeShutdown Callbackfor more details.stop_timeoutNumber of seconds to wait for the daemon to stop before throwing an error. Default:10ActionsThe default actions arestart,stop,restart, andstatus. You can get a list of
available actions using thedaemonocle.Daemon.list_actions()method. The recommended way to call
an action is using thedaemonocle.Daemon.do_action(action)method. The string name of an action
is the same as the method name except with dashes in place of underscores.If you want to create your own actions, simply subclassdaemonocle.Daemonand add the@daemonocle.expose_actiondecorator to your action method, and that’s it.Here’s an example:importdaemonocleclassMyDaemon(daemonocle.Daemon):@daemonocle.expose_actiondeffull_status(self):"""Get more detailed status of the daemon."""passThen, if you did the basicdaemon.do_action(sys.argv[1])like in all the examples above, you can
call your action with a command likepython example.pyfull-status.Integration with clickdaemonocle also provides an integration withclick, the “Command Line
Interface Creation Kit”. The integration is in the form of a custom command classdaemonocle.cli.DaemonCLIthat you can use in conjunction with the@click.command()decorator
to automatically generate a command line interface with subcommands for all your actions. It also
automatically daemonizes the decorated function. The decorated function becomes the worker, and the
actions are automatically mapped from click to daemonocle.Here’s an example:importtimeimportclickfromdaemonocle.cliimportDaemonCLI@click.command(cls=DaemonCLI,daemon_params={'pid_file':'/var/run/example.pid'})defmain():"""This is my awesome daemon. It pretends to do work in the background."""whileTrue:time.sleep(10)if__name__=='__main__':main()Here are all the help pages for the default actions:user@host:~$ python example.py --help
Usage: example.py [OPTIONS] COMMAND [ARGS]...

  This is my awesome daemon. It pretends to do work in the background.

Options:
  --help  Show this message and exit.

Commands:
  start    Start the daemon.
  stop     Stop the daemon.
  restart  Stop then start the daemon.
  status   Get the status of the daemon.

user@host:~$ python example.py start --help
Usage: example.py start [OPTIONS]

  Start the daemon.

Options:
  --debug  Do NOT detach and run in the background.
  --help   Show this message and exit.

user@host:~$ python example.py stop --help
Usage: example.py stop [OPTIONS]

  Stop the daemon.

Options:
  --timeout INTEGER  Number of seconds to wait for the daemon to stop.
                     Overrides "stop_timeout" from daemon definition.
  --force            Kill the daemon uncleanly if the timeout is reached.
  --help             Show this message and exit.

user@host:~$ python example.py restart --help
Usage: example.py restart [OPTIONS]

  Stop then start the daemon.

Options:
  --timeout INTEGER  Number of seconds to wait for the daemon to stop.
                     Overrides "stop_timeout" from daemon definition.
  --force            Kill the daemon forcefully after the timeout.
  --debug            Do NOT detach and run in the background.
  --help             Show this message and exit.

user@host:~$ python example.py status --help
Usage: example.py status [OPTIONS]

  Get the status of the daemon.

Options:
  --json         Show the status in JSON format.
  --fields TEXT  Comma-separated list of process info fields to display.
  --help         Show this message and exit.Thedaemonocle.cli.DaemonCLIclass also accepts adaemon_classargument that can be a
subclass ofdaemonocle.Daemon. It will use your custom class, automatically create subcommands
for any custom actions you’ve defined, and use the docstrings of the action methods as the help text
just like click usually does.This integration is entirely optional. daemonocle doesn’t enforce any sort of argument parsing. You
can use argparse, optparse, or just plainsys.argvif you want.Starting with version 1.1.0, you can also use a couple different shorter ways of invoking the CLI.Like this:fromdaemonocle.cliimportcli@cli(pid_file='/var/run/example.pid')defmain():"""Do stuff"""...if__name__=='__main__':main()Or like this:fromdaemonocleimportDaemondefmain():"""Do stuff"""...if__name__=='__main__':daemon=Daemon(worker=main,pid_file='/var/run/example.pid')daemon.cli()The above two examples are equivalent. Use whichever way works best for you.Bugs, Requests, Questions, etc.Please create anissue on GitHub.Release Historyv1.2.3 (2021-04-22)Pass args to click from Daemon.cli()Don’t run start hook on reload or when the daemon is already runningv1.2.1 and v1.2.2 (2021-02-16)Minor tweaks to some undocumented experimental featuresv1.2.0 (2021-02-15)Dropped support for Python 2.7 and 3.5 (only >= 3.6 is now supported)Deprecatedshutdown_callbackargument in favor of new hooks system.
Replaceshutdown_callback=funcwithhooks={'shutdown':func}.Added'start'hook that gets called before attempting to start the
daemon. This can be useful for checking things or setting things up.Improvements to some undocumented experimental features. :)v1.1.1 (2020-12-31)Thestopaction will now clean up the PID file if the worker didn’t
do it for some reason.A few bug fixes to some undocumented experimental features. :)v1.1.0 (2020-12-16)Official support for Python 3.9Added back official support for Python 3.5Increased test coverage to over 95%.All tests now pass on macOS (Intel) and the GitHub Actions build now runs on
macOS 10.15 in addition to Ubuntu.Fixed theclose_open_filesoption to be much more reliable and consistent
across different platforms.Fixed a race condition with the self-reload functionality where the PID file
of the parent process was being deleted while the child process was trying to
read it.Addedstdout_fileandstderr_filearguments toDaemon. If these
arguments are provided whendetach=True, STDOUT and STDERR will be
redirected to these files. In non-detached mode, these arguments are ignored.Whenchrootdiris given, all other paths are now always considered
relative to the chroot directory, even with a leading slash.Actions can now take arbitrary arguments, and (on Python 3) CLI options are
auto-generated from the function signature. The auto-generated CLI options
work best when your action’s function signature contains type annotations and
default values where applicable.Addedtimeoutandforcearguments to the built-instopaction,
accessible from the CLI as--timeoutand--force.Addedjsonandfieldsarguments to the built-instatusaction,
accessible from the CLI as--jsonand--fields.Added colored output when the output stream is attached to a terminal.Fixed a bug where the daemon wouldn’t respond properly todocker stopwhen running in a docker container.The worker function can now be a method calledworkeron aDaemonsubclass.Some more secret experimental stuff. :)v1.0.2 (2020-07-12)Official support for Python 2.7, 3.6, 3.7, and 3.8.Fixed bug checking if a stream is a socket on Python 3.8.Closing open files is now more efficient on systems with a very high limit
on the number of open files.Improved detection of running inside a container.v1.0.1 (2016-04-17)No changes in this release. Bumped version only to re-upload to PyPI.v1.0.0 (2016-04-17)Added official support for Python 2.7, 3.3, 3.4, and 3.5.Added a comprehensive suite of unit tests with over 90% code coverage.Dependencies (click and psutil) are no longer pinned to specific versions.Fixed bug withatexithandlers not being called in intermediate processes.Fixed bug when PID file is a relative path.Fixed bug when STDIN doesn’t have a file descriptor number.Fixed bug when running in non-detached mode in a Docker container.A TTY is no longer checked for when deciding how to run in non-detached mode.
The behavior was inconsistent across different platforms.Fixed bug when a process stopped before having chance to check if it stopped.Fixed bug where an exception could be raised if a PID file is already gone
when trying to remove it.Subdirectories created for PID files now respect theumasksetting.The pre-umaskmode for PID files is now0o666instead of0o777,
which will result in a default mode of0o644instead of0o755when
using the defaultumaskof0o22.v0.8 (2014-08-01)Upgraded click to version 2.5.Status action now returns exit code 1 if the daemon is not running.v0.7 (2014-06-23)Fixed bug that was causing an empty PID file on Python 3.Upgraded click to version 2.1.Open file discriptors are no longer closed by default. This functionality is now optional via theclose_open_filesargument toDaemon().Addedis_workerargument toDaemonCLI()as well as thepass_daemondecorator.v0.6 (2014-06-10)Upgraded click to version 2.0.v0.5 (2014-06-09)Fixed literal octal formatting to work with Python 3.v0.4 (2014-05-19)Fixed bug with uptime calculation in status action.Upgraded click to version 0.7.v0.3 (2014-05-14)Reorganized package and cleaned up code.v0.2 (2014-05-12)RenamedDaemon.get_actions()toDaemon.list_actions().Improvements to documentation.Fixed bug with non-detached mode when parent is in the same process group.v0.1 (2014-05-11)Initial release.