Fix for Eddy-Wrapper does not work properly #769
Fix version 2 for Eddy-Wrapper does not work properly #769
Fix version 2 for Eddy-Wrapper does not work properly #769
fix: updated with copyfile to ensure the inputs get linked to a local directory when running in a workflow
Merge pull request #1 from satra/fix/eddy

a workflow fix, auto tests and some PEP8 changes
Merge pull request #785 from JensNRAD/eddy_fix

Fix for Eddy-Wrapper does not work properly #769
Don't call subprocess.Popen twice when output == 'file'.

This fixes the real issue behind https://github.com/nipy/nipype/issues/705
Don't call subprocess.Popen twice when output == 'file'.

This fixes the real issue behind https://github.com/nipy/nipype/issues/705
Merge branch 'master' of github.com:carlohamalainen/nipype

This is for 421601e8b9d3fd259061590a0ed6b962c466cd8c regarding
"Don't call subprocess.Popen twice when output == 'file'."
Merge pull request #789 from carlohamalainen/master

Don't call subprocess.Popen twice when output == 'file'.
ENH: Large SGE jobs were overloading on qstat

When large jobs were run with SGE, the accumulation of
qstat queries was causing a massive load on the cluster
server, and was affecting overal system performance.

This modification queries based on userid, stores information
about all that users jobs (both running and recently finished),
subsequent queries are then addressed by looking at the cached
values first, then only updating with a system qstat call
periodically.  This change so that qstat is called on a
more reasonable replication time frame.

User can supply externally supplied version of qmake.

Prevent DOS style of attacks on the batch processing
server by preventing continuous queries by many jobs.

This was affecting the performance of the entire server
and the excess load of querying when jobs were done in the
niavie way was affecting the performance of dispatching new
jobs.

The following two scripts can be used as plugin_args to
provide cached versions of qmake suitable for running
huge jobs.

plugin_args=dict(template=JOB_SCRIPT,
            qsub_args="-S /bin/bash -cwd -pe smp 1-12 -l h_vmem=19G,mem_free=9G -o /dev/null -e /dev/null " + CLUSTER_QUEUE,
            qstatProgramPath=qstat_immediate.sh,
            qstatCachedProgramPath=qstat_cached.sh))

=qstat_cached.sh===================================================
\#!/bin/bash
\# \author Hans J. Johnson
\# This file provides a wrapper around qstat to ensure that
\# the qstat server is not overloaded with too many requests

\#debug_file=/dev/null
debug_file=/tmp/TESTINGLOG
qstat_cache=/tmp/qstat.xml

echo "USING EXTERNAL QSTAT: $@" >> ${debug_file}  2>&1

older_than_60_sec=$( find $(dirname ${qstat_cache}) -maxdepth 1 -name $(basename ${qstat_cache}) -mmin $(echo 5/60 |bc -l) )
if [ -z "${older_than_60_sec}" ]; then
  DoQstatNow=$(dirname ${0})/qstat_immediate.sh
  ${DoQstatNow} $@
else
  echo "using cache $(date)" >> ${debug_file} 2>&1
  cat ${qstat_cache}
fi
===================================================================

=qstat_immediate.sh===============================================
\#!/bin/bash
\# \author Hans J. Johnson
\# This file provides a wrapper around qstat to ensure that
\# the qstat server is not overloaded with too many requests

\#debug_file=/dev/null
debug_file=/tmp/TESTINGLOG
qstat_cache=/tmp/qstat.xml

echo "USING EXTERNAL QSTAT: $@" >> ${debug_file}  2>&1

echo "Refreshing $(date)" >> ${debug_file} 2>&1
cacheUpdated=0;
while [ ${cacheUpdated} -eq 0 ]; do
  if [ ! -f  ${qstat_cache}_lock ]; then
    touch ${qstat_cache}_lock
    DoQstatNow=$(which qstat)
    ${DoQstatNow} $@ > ${qstat_cache}_tmp 2>&1
    mv ${qstat_cache}_tmp ${qstat_cache}
    rm ${qstat_cache}_lock
    let cacheUpdated=1
  else
    echo "Waiting for contention lock $(date)"  >> ${debug_file} 2>&1
    sleep 1
  fi
done

cat ${qstat_cache}
===================================================================