ENH: Large SGE jobs were overloading on qstat

When large jobs were run with SGE, the accumulation of
qstat queries was causing a massive load on the cluster
server, and was affecting overal system performance.

This modification queries based on userid, stores information
about all that users jobs (both running and recently finished),
subsequent queries are then addressed by looking at the cached
values first, then only updating with a system qstat call
periodically.  This change so that qstat is called on a
more reasonable replication time frame.

User can supply externally supplied version of qmake.

Prevent DOS style of attacks on the batch processing
server by preventing continuous queries by many jobs.

This was affecting the performance of the entire server
and the excess load of querying when jobs were done in the
niavie way was affecting the performance of dispatching new
jobs.

The following two scripts can be used as plugin_args to
provide cached versions of qmake suitable for running
huge jobs.

plugin_args=dict(template=JOB_SCRIPT,
            qsub_args="-S /bin/bash -cwd -pe smp 1-12 -l h_vmem=19G,mem_free=9G -o /dev/null -e /dev/null " + CLUSTER_QUEUE,
            qstatProgramPath=qstat_immediate.sh,
            qstatCachedProgramPath=qstat_cached.sh))

=qstat_cached.sh===================================================
\#!/bin/bash
\# \author Hans J. Johnson
\# This file provides a wrapper around qstat to ensure that
\# the qstat server is not overloaded with too many requests

\#debug_file=/dev/null
debug_file=/tmp/TESTINGLOG
qstat_cache=/tmp/qstat.xml

echo "USING EXTERNAL QSTAT: $@" >> ${debug_file}  2>&1

older_than_60_sec=$( find $(dirname ${qstat_cache}) -maxdepth 1 -name $(basename ${qstat_cache}) -mmin $(echo 5/60 |bc -l) )
if [ -z "${older_than_60_sec}" ]; then
  DoQstatNow=$(dirname ${0})/qstat_immediate.sh
  ${DoQstatNow} $@
else
  echo "using cache $(date)" >> ${debug_file} 2>&1
  cat ${qstat_cache}
fi
===================================================================

=qstat_immediate.sh===============================================
\#!/bin/bash
\# \author Hans J. Johnson
\# This file provides a wrapper around qstat to ensure that
\# the qstat server is not overloaded with too many requests

\#debug_file=/dev/null
debug_file=/tmp/TESTINGLOG
qstat_cache=/tmp/qstat.xml

echo "USING EXTERNAL QSTAT: $@" >> ${debug_file}  2>&1

echo "Refreshing $(date)" >> ${debug_file} 2>&1
cacheUpdated=0;
while [ ${cacheUpdated} -eq 0 ]; do
  if [ ! -f  ${qstat_cache}_lock ]; then
    touch ${qstat_cache}_lock
    DoQstatNow=$(which qstat)
    ${DoQstatNow} $@ > ${qstat_cache}_tmp 2>&1
    mv ${qstat_cache}_tmp ${qstat_cache}
    rm ${qstat_cache}_lock
    let cacheUpdated=1
  else
    echo "Waiting for contention lock $(date)"  >> ${debug_file} 2>&1
    sleep 1
  fi
done

cat ${qstat_cache}
===================================================================
fix: update conda
dbg: conda update
dbg: anaconda to miniconda update
fix: parenthesis fixes
fix: update conda in environment
install ordereddict for py26
fix: moving save from numpy to pickle
install ordereddict for py26
fix: moving save from numpy to pickle
Merge pull request #6 from satra/enh/sge

fix: parenthesis fixes
Merge pull request #797 from satra/fix/conda

fix: update conda
Merge pull request #728 from BRAINSia/EnhanceQSubProcessing

ENH: Large SGE jobs were overloading on qstat
Merge branch 'master' into fix/eddy

* master:
  fix: moving save from numpy to pickle
  install ordereddict for py26
  fix: moving save from numpy to pickle
  install ordereddict for py26
  fix: update conda in environment
  fix: parenthesis fixes
  dbg: anaconda to miniconda update
  dbg: conda update
  fix: update conda
  ENH: Large SGE jobs were overloading on qstat