SchedMD/slurm — pull request #15 by grondo (2012-03-21 23:30:37+00:00)
Title:  Add spank plugin hooks in slurmd, prolog & epilog

I have a set of patches for SLURM that add the following callbacks to
the spank plugin interface:

slurm_spank_slurmd_init: Called as slurmd(8) starts
slurm_spank_slurmd_exit: Called just before slurmd(8) exit
slurm_spank_job_prolog: Called at the same time as SLURM job prolog
slurm_spank_job_epilog: Called at the same time as SLURM job epilog

In addition, a new function for spank plugins that process options
is added:

spank_err_t spank_option_getopt (spank_t sp, struct spank_option *opt,
char **optargp);

as an alternative to requiring a global 'option handler' script.

The motivation for this functionality is to enable the consilidation
of code that currently must be spread in three different places --
a spank plugin (to provide a new SLURM option), the job prolog script,
and/or the job epilog script.

An example of this would be a plugin which provides an option to disable
memory overcommit on the nodes of a job based on user option. Currently,
we have a spank/lua plugin that exports a --overcommit-memory=never
option which, when used, sets a 'job control environment' variable
that is propagated to the job prolog and epilog. The job prolog and
epilog then *separately* have code to check for the option variable
and take action. After these patches are applied, the code for this
functionality can fit in one self-contained plugin.

The method chosen for loading spank plugins at job prolog/epilog
time is of note. These hooks cannot be called within the address
space of slurmd for several reasons (see description in
commit 4e6a2964), so instead a new "mode" for slurmstepd is used.
When slurmstepd is run in the form

slurmstepd spank [prolog|epilog]

it loads the spank stack and runs relevant plugins for either
the prolog or epilog. This not only allows plugins to re-initialize
at each prolog/epilog run, but also ensures plugins that are updated
at runtime are reloaded at the next prolog or epilog run (akin to
the functionality of the job prolog/epilog scripts). This method
is unusual enough that I wanted to be sure to note it up front
here, in case there are further comments.

Further implementation details are contained within the commit
messages for the 30 or so commits that add this functionality.