fix: showq may not be available on all slurm systems
fix: add option for cuda
fix: enable cuda not required - variable set by asking for gres=gpu
Don't call subprocess.Popen twice when output == 'file'.

This fixes the real issue behind https://github.com/nipy/nipype/issues/705
Don't call subprocess.Popen twice when output == 'file'.

This fixes the real issue behind https://github.com/nipy/nipype/issues/705
Merge branch 'master' of github.com:carlohamalainen/nipype

This is for 421601e8b9d3fd259061590a0ed6b962c466cd8c regarding
"Don't call subprocess.Popen twice when output == 'file'."
Merge pull request #789 from carlohamalainen/master

Don't call subprocess.Popen twice when output == 'file'.
ENH: Large SGE jobs were overloading on qstat

When large jobs were run with SGE, the accumulation of
qstat queries was causing a massive load on the cluster
server, and was affecting overal system performance.

This modification queries based on userid, stores information
about all that users jobs (both running and recently finished),
subsequent queries are then addressed by looking at the cached
values first, then only updating with a system qstat call
periodically.  This change so that qstat is called on a
more reasonable replication time frame.

User can supply externally supplied version of qmake.

Prevent DOS style of attacks on the batch processing
server by preventing continuous queries by many jobs.

This was affecting the performance of the entire server
and the excess load of querying when jobs were done in the
niavie way was affecting the performance of dispatching new
jobs.

The following two scripts can be used as plugin_args to
provide cached versions of qmake suitable for running
huge jobs.

plugin_args=dict(template=JOB_SCRIPT,
            qsub_args="-S /bin/bash -cwd -pe smp 1-12 -l h_vmem=19G,mem_free=9G -o /dev/null -e /dev/null " + CLUSTER_QUEUE,
            qstatProgramPath=qstat_immediate.sh,
            qstatCachedProgramPath=qstat_cached.sh))

=qstat_cached.sh===================================================
\#!/bin/bash
\# \author Hans J. Johnson
\# This file provides a wrapper around qstat to ensure that
\# the qstat server is not overloaded with too many requests

\#debug_file=/dev/null
debug_file=/tmp/TESTINGLOG
qstat_cache=/tmp/qstat.xml

echo "USING EXTERNAL QSTAT: $@" >> ${debug_file}  2>&1

older_than_60_sec=$( find $(dirname ${qstat_cache}) -maxdepth 1 -name $(basename ${qstat_cache}) -mmin $(echo 5/60 |bc -l) )
if [ -z "${older_than_60_sec}" ]; then
  DoQstatNow=$(dirname ${0})/qstat_immediate.sh
  ${DoQstatNow} $@
else
  echo "using cache $(date)" >> ${debug_file} 2>&1
  cat ${qstat_cache}
fi
===================================================================

=qstat_immediate.sh===============================================
\#!/bin/bash
\# \author Hans J. Johnson
\# This file provides a wrapper around qstat to ensure that
\# the qstat server is not overloaded with too many requests

\#debug_file=/dev/null
debug_file=/tmp/TESTINGLOG
qstat_cache=/tmp/qstat.xml

echo "USING EXTERNAL QSTAT: $@" >> ${debug_file}  2>&1

echo "Refreshing $(date)" >> ${debug_file} 2>&1
cacheUpdated=0;
while [ ${cacheUpdated} -eq 0 ]; do
  if [ ! -f  ${qstat_cache}_lock ]; then
    touch ${qstat_cache}_lock
    DoQstatNow=$(which qstat)
    ${DoQstatNow} $@ > ${qstat_cache}_tmp 2>&1
    mv ${qstat_cache}_tmp ${qstat_cache}
    rm ${qstat_cache}_lock
    let cacheUpdated=1
  else
    echo "Waiting for contention lock $(date)"  >> ${debug_file} 2>&1
    sleep 1
  fi
done

cat ${qstat_cache}
===================================================================
fix: update conda
dbg: conda update
dbg: anaconda to miniconda update
fix: parenthesis fixes
fix: update conda in environment
install ordereddict for py26
fix: moving save from numpy to pickle
install ordereddict for py26
fix: moving save from numpy to pickle
Merge pull request #6 from satra/enh/sge

fix: parenthesis fixes
Merge pull request #797 from satra/fix/conda

fix: update conda
Merge pull request #728 from BRAINSia/EnhanceQSubProcessing

ENH: Large SGE jobs were overloading on qstat