Repository: nipy/nipype

Issue:
sgegraph only submits the next processing node
### Summary
the `SGEGraph` plugin isn't submitting the entire workflow only the next required step.

### Actual behavior

When calling `workflow.run(plugin='SGEGraph', plugin_args = {'dont_resubmit_completed_jobs': True})` only the next processing node is submitted. subsequent calls are needed to iteratively processes subsequent nodes.

`workflow.run(plugin='SGE)` works as expected. submitting one job waiting until completion then submitting the next.

### Expected behavior
I would expect that `workflow.run(plugin='SGEGraph', plugin_args = {'dont_resubmit_completed_jobs': True})` would submit all processing jobs at once with later jobs being held until the earlier required jobs are finished.

### How to replicate the behavior
I'm new to nipype so I'm not sure how to get sample/example data to provide a minimal example but I think that any of the workflows in the example folder would demonstrate this issue.

### Platform details:

<!-- Please run the following code from your shell and place the output between the triple ticks, below.
python -c "import nipype; from pprint import pprint; pprint(nipype.get_info())"
-->

```
200207-13:43:08,275 nipype.utils INFO:
	 No new version available.
{'commit_hash': '792988f19',
 'commit_source': 'installation',
 'networkx_version': '2.4',
 'nibabel_version': '3.0.1',
 'nipype_version': '1.4.1',
 'numpy_version': '1.18.1',
 'pkg_path': '/mnt/isilon/meg_lab/Linux_software/anaconda3_lab/envs/nipype/lib/python3.8/site-packages/nipype',
 'scipy_version': '1.3.2',
 'sys_executable': '/mnt/isilon/meg_lab/Linux_software/anaconda3_lab/envs/nipype/bin/python',
 'sys_platform': 'linux',
 'sys_version': '3.8.1 (default, Jan  8 2020, 22:29:32) \n[GCC 7.3.0]',
 'traits_version': '5.2.0'}
```

### Execution environment

Choose one
- My python environment outside container



Discussion:
The problem was cause by a configuration on cluster. for security reasons LD_LIBRARY_PATH environment variables are stripped from the qsub environment. qsub prints a warning when this happens:

```
removed environment variable LD_LIBRARY_PATH from submit environment - it is considered a security issue
removed environment variable LD_LIBRARY_PATH_modshare from submit environment - it is considered a security issue
Your job 13917966 ("j0_dcm_dir_a0") has been submitted
```
and this breaks the awk syntax

https://github.com/nipy/nipype/blob/b62d80627317eaac2a04a423bad1151a256340b1/nipype/pipeline/plugins/sgegraph.py#L151-L158

so job ids aren't parsed correctly and the submit script crashes without a log message.

I'll hopefully push a PR in a few minutes that modifies the `awk` command. 

Resolving patch:
diff --git a/nipype/pipeline/plugins/sgegraph.py b/nipype/pipeline/plugins/sgegraph.py
--- a/nipype/pipeline/plugins/sgegraph.py
+++ b/nipype/pipeline/plugins/sgegraph.py
@@ -148,7 +148,7 @@ def make_job_name(jobnumber, nodeslist):
                     stdoutFile = ""
                     if self._qsub_args.count("-o ") == 0:
                         stdoutFile = "-o {outFile}".format(outFile=batchscriptoutfile)
-                    full_line = "{jobNm}=$(qsub {outFileOption} {errFileOption} {extraQSubArgs} {dependantIndex} -N {jobNm} {batchscript} | awk '{{print $3}}')\n".format(
+                    full_line = "{jobNm}=$(qsub {outFileOption} {errFileOption} {extraQSubArgs} {dependantIndex} -N {jobNm} {batchscript} | awk '/^Your job/{{print $3}}')\n".format(
                         jobNm=jobname,
                         outFileOption=stdoutFile,
                         errFileOption=stderrFile,


Test patch: