Unit tests to exercise job and task mapping.
Layout models and mapping for a job metrics plugin framework.
Implement plugin framework for instrumenting jobs.

The main goal here is collecting metrics, but the included 'env' plugin could be potentially useful for debugging cluster/dependency/job problems as well.

Right now all (cluster) jobs must be instrumented the same way - using a global configuration in job_metrics.xml (see job_metrics.xml.sample for an example of how to layout this file). Ideally, this would be generalized to allow different destinations to target different job metrics (likewise it would be good dependency_resolvers was likewise rewritten).

This commit include two initial plugins with options documented in job_metrics.xml.sample.

The 'core' plugin collects GALAXY_SLOTS as well as start and end time of jobs (in seconds since epoch). This should allow computing the runtime of jobs and core count assigned - providing a much clearer picture of what Galaxy is actually allocating cluster compute cycles to.

The 'env' plugin can collect all environment variables set at the time of job execution or just targetted environment variables. Environment variables like PATH or PYTHONPATH may be useful for debugging and environment variables such as HOSTNAME, PID, and DRM-specific variables like SLURM_JOB_ID could be useful for integrating Galaxy's view of jobs and users with external accounting/metric systems. See job_metrics.xml.sample for how to configure environment variable collection.

Current limitations - This only works with job runners utilizing the job script module, hence it won't yet work with...

 - Local job runner - I do have a downstream fork of Galaxy where I have reworked the local job runner to use the common job script template.

    https://github.com/jmchilton/galaxy-central/commits/local_job_script
    https://github.com/jmchilton/galaxy-central/commit/949db2cd14c7191cedf1febeb5f4a2f53123123e

 - LWR - Changes are straight forward, this code was architected to allow reuse within the LWR (which does utilize the job script module) and file names are consistently prefixed so it will be clear what needs to be unstaged to collect metrics on the Galaxy side of things.

 - CLI runner - CLI runner needs to be reworked to use the job script module anyway so GALAXY_SLOTS works - the LWR version of the CLI runner uses the job script moduule - this work just needs to be back ported to Galaxy.
Collectl based job instrumentation.

If configured, while a job is running collectl will run in record mode (think of this as grabbing top output at specified interval) - either for the whole system or just the job's user.

After the job is complete this whole file can be copied to a specified directory and path saved to be associated with job along with job's PID. This will allow complete time series data of job memory, CPU, IO (limitted without process level I/O logging enabled in kernel) etc... in batch or interactively downstream re-using collectl in playback mode and building a process tree for record PID. Collecting such detailed collectl logs for all jobs on large Galaxy instances could consume hundreds of gigabytes of disk space quickly but would provide

The plugin can also do some of this immediately, and save summary job statistics directly to Galaxy's database - max and mean CPU and memory utilization for instance.

Usage of this plugin requires collectl to be installed both on the Galaxy server and compute nodes. The daemon doesn't need to be running prior to the job and collectl can easily be installed in userland - so it worth considering automating installation and configuration of this dependency if using this plugin.