Layout models and mapping for a job metrics plugin framework.
Implement plugin framework for collecting data about runtime job execution.

An example job_metrics.xml.sample is included that describes which plugins are enabled and how they are configured. This will be updated for each new plugin added. By default not instrumentation or data collection occurs - but if a job_metrics.xml file is present it will serve as the default for all job destination. Additionally, individual job destinations may disable, load a different job metrics file, or define metrics directly in job_conf.xml in an embedded fashion. See comment at top of job_metrics.xml for more information.

This commit include an initial plugin (named 'core') to demonstrate the framework and capture the highest priority data - namely the number of cores allocated to the job and the runtime of the job on the cluster. These two pieces of information alone should provide a much clearer picture of what Galaxy is actually allocating cluster compute cycles to.

Current limitations - This only works with job runners utilizing the job script module, hence it won't yet work with...

 - Local job runner - I do have a downstream fork of Galaxy where I have reworked the local job runner to use the common job script template.

    https://github.com/jmchilton/galaxy-central/commits/local_job_script
    https://github.com/jmchilton/galaxy-central/commit/949db2cd14c7191cedf1febeb5f4a2f53123123e

 - LWR - Changes are straight forward, this code was architected to allow reuse within the LWR (which does utilize the job script module) and file names are consistently prefixed so it will be clear what needs to be unstaged to collect metrics on the Galaxy side of things.

 - CLI runner - CLI runner needs to be reworked to use the job script module anyway so GALAXY_SLOTS works - the LWR version of the CLI runner uses the job script moduule - this work just needs to be back ported to Galaxy.
Implement env job instrumentation plugin.

The 'env' plugin can collect all environment variables set at the time of job execution or just targetted environment variables. Environment variables like PATH or PYTHONPATH may be useful for debugging and environment variables such as HOSTNAME, PID, and DRM-specific variables like SLURM_JOB_ID could be useful for integrating Galaxy's view of jobs and users with external accounting/metric systems. See job_metrics.xml.sample for how to configure environment variable collection.
Implement cpuinfo job instrumentation plugin.
Implement meminfo job instrumentation plugin.
Implement uname job instrumentation plugin.

Gathers information on operating system configuration job is run on - linux only.
Collectl based job instrumentation.

If configured, while a job is running collectl will run in record mode (think of this as grabbing top output at specified interval) - either for the whole system or just the job's user.

After the job is complete this whole file can be copied to a specified directory and path saved to be associated with job along with job's PID. This will allow complete time series data of job memory, CPU, IO (limitted without process level I/O logging enabled in kernel) etc... in batch or interactively downstream re-using collectl in playback mode and building a process tree for record PID. Collecting such detailed collectl logs for all jobs on large Galaxy instances could consume hundreds of gigabytes of disk space quickly but would provide

The plugin can also do some of this immediately, and save summary job statistics directly to Galaxy's database - max and mean CPU and memory utilization for instance.

Usage of this plugin requires collectl to be installed both on the Galaxy server and compute nodes. The daemon doesn't need to be running prior to the job and collectl can easily be installed in userland - so it worth considering automating installation and configuration of this dependency if using this plugin.