Merge pull request #1 from VertebrateResequencing/develop

pulling from vertres develop into wtsi-hgi develop
Merge pull request #2 from wtsi-hgi/develop

pulling develop into feature/runner-checkpointing
Adds +checkpoint option to Runner.pm to enable checkpointing

This is not yet fully implemented in RunnerLSF.pm, but using +checkpoint will
cause jobs to be submitted with `bsub -k` and using cr_run
Adds perl5lib_dependencies to .gitignore
Adds basic LSF/BLCR checkpoint handling, activated by +blcr option
Adds copyright and license header to Runner and RunnerlSF

(as required by GRL software policy)
More changes to checkpoint/restart

In particular, changing to have_chkpnt as opposed to chkpnt_failed.
Adds detection for cpus and adjustment to cpu_time
Fixes prepending cr_run to command line
Fixes checkpoint time calculation from m to s
Reverts farm option 'memory_mb' back to 'memory'
Changes default checkpoint rate to 0.15 s/MB
Returns to Sanger farm3 queue limits

And adds runtime_limit_seconds which can be specified as a limit to enable
use of -W limits in LSF.
Changes new runtime limit to use -We instead of -W

LSF was being to viciously murderous in killing off jobs that were
in the middle of checkpointing.  Now using -We as an estimate, and
killing the job ourselves after the runtime has elapsed and the
latest checkpoint is done.
Fixes regex capture of RUNTIME
Changes minimum checkpoint period to 1 minute
Fixes bug wherein some jobs would be restarted & submitted from start
Improves output and error handling
Adds a check for checkpoints that take too long

Adds another hardcoded farm-specific setting for max checkpoint time per MB
and checks if a job that has gone over the run limit has had a checkpoint in
progress for longer than the maximum time (if so, it kills it).

Fixes a number of output message formatting issues.
Improves performance of jid file processing

Only processes jid file entries and bjobs -l parsing as far back
as is needed.