Repository: DataBiosphere/toil

Issue:
How to run toil-cwl-runner on a local HPC with the torque batch system?
I run the following command on the login node：

```shell
export TOIL_TORQUE_ARGS="-N graham-rhapsody-wta -e admin:/public/home/graham/src/rhapsody-wta/logs"
export TOIL_TORQUE_REQS="walltime=48:00:00,mem=192gb,nodes=2:ppn=16"

toil-cwl-runner \
  --batchSystem torque \
  --user-space-docker-cmd=udocker \
  --jobStore file:results/rhapsody-wta-job-store \
  --outdir results \
  --writeLogs logs \
  --logFile cwltoil.log \
  --logLevel DEBUG \
  --retryCount 2 \
  --maxLogFileSize 20000000000 \
  --stats \
  rhapsody-wta-yaml.cwl template_wta.yml
```

I saw this error:

```
Additional Torque resource requirements appended to qsub from TOIL_TORQUE_REQS env. variable: walltime=48:00:00,mem=192gb,nodes=2:ppn=16
GridEngine like batch system failure
Traceback (most recent call last):
  File "/public/home/weiwanqian/miniconda3/envs/bioinfo/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 226, in run
    while self._runStep():
  File "/public/home/weiwanqian/miniconda3/envs/bioinfo/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 215, in _runStep
    activity |= self.createJobs(newJob)
  File "/public/home/weiwanqian/miniconda3/envs/bioinfo/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 121, in createJobs
    subLine = self.prepareSubmission(cpu, memory, jobID, command, jobName)
  File "/public/home/weiwanqian/miniconda3/envs/bioinfo/lib/python3.6/site-packages/toil/batchSystems/torque.py", line 121, in prepareSubmission
    return self.prepareQsub(cpu, memory, jobID) + [self.generateTorqueWrapper(command, jobID)]
  File "/public/home/weiwanqian/miniconda3/envs/bioinfo/lib/python3.6/site-packages/toil/batchSystems/torque.py", line 179, in prepareQsub
    raise ValueError("Incompatible resource arguments ('mem=', 'nodes=', 'ppn='): {}".format(reqlineEnv))
ValueError: Incompatible resource arguments ('mem=', 'nodes=', 'ppn='): walltime=48:00:00,mem=192gb,nodes=2:ppn=16
```

Can't I specify any computer hardware resources in the TOIL_TORQUE_REQS variable?

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-680)
┆Issue Number: TOIL-680



Discussion:
Hello @altairwei 

> Can't I specify any computer hardware resources in the TOIL_TORQUE_REQS variable?

`TOIL_TORQUE_REQS` can not contain `mem`, `nodes`, nor `ppn`; those are generated dynamically by `toil-cwl-runner` based upon the https://www.commonwl.org/v1.0/CommandLineTool.html#ResourceRequirement `ram{Max,Min}`, and `cores{Max,Min}`.

By the way, do any of your workflow steps use MPI?
### Problem overview

I'm trying to run a CWL workflow using `toil-cwl-runner` on an HPC on my local LAN, but there seems to be a problem with `toil`'s job submission, or I'm using it incorrectly. The batch system of the HPC is **torque**, with 16 compute nodes, each with 32 CPUs and 93G of RAM.

I can use `cwltool` or `toil-cwl-runner` to run the CWL workflow successfully on a single compute node without using the `--batchSystem torque` option. If I run `toil-cwl-runner` with `--batchSystem torque` on the login node, the tasks are committed to torque, but each task fails quickly.

I found very little useful information from the toil logs, so I had to ask the nice developers for help.

The operating system information of the HPC is as follows：

```
                 ..                    graham@admin
               .PLTJ.                  ----------------
              <><><><>                 OS: CentOS Linux release 7.4.1708 (Core) x86_64 
     KKSSV' 4KKK LJ KKKL.'VSSKK        Host: I620-G30 Purley 
     KKV' 4KKKKK LJ KKKKAL 'VKK        Kernel: 3.10.0-693.el7.x86_64 
     V' ' 'VKKKK LJ KKKKV' ' 'V        Uptime: 614 days, 8 hours, 55 mins 
     .4MA.' 'VKK LJ KKV' '.4Mb.        Packages: 3462 (rpm) 
   . KKKKKA.' 'V LJ V' '.4KKKKK .      Shell: bash 4.2.46 
 .4D KKKKKKKA.'' LJ ''.4KKKKKKK FA.    Terminal: node 
<QDD ++++++++++++  ++++++++++++ GFD>   CPU: Intel Xeon Gold 5115 (20) @ 2.401GHz 
 'VD KKKKKKKK'.. LJ ..'KKKKKKKK FV     GPU: 03:00.0 ASPEED Technology, Inc. ASPEED Graphics Family 
   ' VKKKKK'. .4 LJ K. .'KKKKKV '      Memory: 21579MiB / 128413MiB 
      'VK'. .4KK LJ KKA. .'KV'
     A. . .4KKKK LJ KKKKA. . .4
     KKA. 'KKKKK LJ KKKKK' .4KK
     KKSSA. VKKK LJ KKKV .4SSKK
              <><><><>
               'MKKM'
                 ''
```

### How to reproduce this problem

I run the following command on the login node：

```shell
export TOIL_TORQUE_ARGS="-N graham-rhapsody-wta -e admin:/public/home/graham/src/rhapsody-wta/logs"
export TOIL_TORQUE_REQS="walltime=48:00:00"

toil-cwl-runner \
  --batchSystem torque \
  --user-space-docker-cmd=udocker \
  --jobStore file:results/rhapsody-wta-job-store \
  --outdir results \
  --writeLogs logs \
  --logFile cwltoil.log \
  --logLevel DEBUG \
  --retryCount 2 \
  --maxLogFileSize 20000000000 \
  --stats \
  rhapsody-wta-yaml.cwl template_wta.yml
```

The contents of `template_wta.yml` are as follows:

```yaml
#!/usr/bin/env cwl-runner

cwl:tool: rhapsody

Reads:
  - class: File
    location: "data/reads/SRR6354915_R1_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354915_R2_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354916_R1_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354916_R2_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354917_R1_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354917_R2_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354918_R1_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354918_R2_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354919_R1_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354919_R2_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354920_R1_001.fastq.gz"
  - class: File
    location: "data/reads/SRR6354920_R2_001.fastq.gz"

Reference_Genome:
  class: File
  location: "data/reference_sequences/Homo_sapiens.GRCh38.tar.gz"

Transcriptome_Annotation:
  class: File
  location: "data/annotations/Homo_sapiens.GRCh38.99.gtf"

Subsample: 0.00001
Subsample_seed: 3445

Label_Version: 1
```

The contents of [`rhapsody-wta-yaml.cwl`](https://github.com/DataBiosphere/toil/files/5328014/rhapsody-wta-yaml.cwl.txt) are attached.

### Problems encountered

The full contents of [`cwltoil.log`](https://github.com/DataBiosphere/toil/files/5328493/cwltoil.log) are in the attach file.

#### 1. Automatic attempts to connect AWS

I want to run toil on my local HPC and don't want to use AWS, but still see the AWS failure log.

```log
Logging to file 'cwltoil.log'.
Total machine size: 20 cores
CPU quota: -1
Path to job store directory is '/public/home/graham/src/rhapsody-wta/results/rhapsody-wta-job-store'.
The workflow ID is: '38652965-9036-40a1-ac10-1d1a386de7e3'
Resolved 'rhapsody-wta-yaml.cwl' to 'file:///public/home/graham/src/rhapsody-wta/rhapsody-wta-yaml.cwl'
Starting new HTTP connection (1): arvados.org:80
http://arvados.org:80 "HEAD /cwl HTTP/1.1" 301 0
http://arvados.org:80 "HEAD /cwl HTTP/1.1" 301 0
rhapsody-wta-yaml.cwl:710:17: Unknown hint http://arvados.org/cwl#RuntimeConstraints
rhapsody-wta-yaml.cwl:941:11: Unknown hint http://arvados.org/cwl#RuntimeConstraints
rhapsody-wta-yaml.cwl:993:11: Unknown hint http://arvados.org/cwl#RuntimeConstraints
rhapsody-wta-yaml.cwl:1073:11: Unknown hint http://arvados.org/cwl#RuntimeConstraints
rhapsody-wta-yaml.cwl:1252:11: Unknown hint http://arvados.org/cwl#RuntimeConstraints
rhapsody-wta-yaml.cwl:1323:11: Unknown hint http://arvados.org/cwl#RuntimeConstraints
rhapsody-wta-yaml.cwl:1788:11: Unknown hint http://arvados.org/cwl#RuntimeConstraints
rhapsody-wta-yaml.cwl:1840:11: Unknown hint http://arvados.org/cwl#RuntimeConstraints
rhapsody-wta-yaml.cwl:1937:11: Unknown hint http://arvados.org/cwl#APIRequirement
rhapsody-wta-yaml.cwl:1938:11: Unknown hint http://arvados.org/cwl#RuntimeConstraints
rhapsody-wta-yaml.cwl:2050:17: Unknown hint http://arvados.org/cwl#RuntimeConstraints
rhapsody-wta-yaml.cwl:2090:17: Unknown hint http://arvados.org/cwl#RuntimeConstraints
Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
Changing event name from before-call.apigateway to before-call.api-gateway
Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
Loading JSON file: /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/boto3/data/s3/2006-03-01/resources-1.json
Looking for credentials via: env
Looking for credentials via: assume-role
Looking for credentials via: assume-role-with-web-identity
Looking for credentials via: sso
Looking for credentials via: shared-credentials-file
Looking for credentials via: custom-process
Looking for credentials via: config-file
Looking for credentials via: ec2-credentials-file
Looking for credentials via: boto-config
Looking for credentials via: container-role
Looking for credentials via: iam-role
Starting new HTTP connection (1): 169.254.169.254:80
Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/toil/lib/memoize.py", line 35, in new_f
    return memory[args]
KeyError: (FileJobStore(/public/home/graham/src/rhapsody-wta/results/rhapsody-wta-job-store),)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/util/retry.py", line 379, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/http/client.py", line 1272, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/awsrequest.py", line 92, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/http/client.py", line 1318, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/http/client.py", line 1267, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/awsrequest.py", line 119, in _send_output
    self.send(msg)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/awsrequest.py", line 203, in send
    return super(AWSConnection, self).send(str)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/http/client.py", line 976, in send
    self.connect()
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7f3042436278>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/utils.py", line 302, in _fetch_metadata_token
    response = self._session.send(request.prepare())
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/httpsession.py", line 287, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Starting new HTTP connection (2): 169.254.169.254:80
Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/toil/lib/memoize.py", line 35, in new_f
    return memory[args]
KeyError: (FileJobStore(/public/home/graham/src/rhapsody-wta/results/rhapsody-wta-job-store),)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/util/retry.py", line 379, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/http/client.py", line 1272, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/awsrequest.py", line 92, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/http/client.py", line 1318, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/http/client.py", line 1267, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/awsrequest.py", line 119, in _send_output
    self.send(msg)
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/awsrequest.py", line 203, in send
    return super(AWSConnection, self).send(str)TOIL WORKER OUTPUT LOG
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/http/client.py", line 976, in send
    self.connect()
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7f3042436898>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/utils.py", line 345, in _get_request
    response = self._session.send(request.prepare())
  File "/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/httpsession.py", line 287, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
Loading JSON file: /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/data/endpoints.json
Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f3048a06f28>
Loading JSON file: /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages/botocore/data/s3/2006-03-01/service-2.json
Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f3048a41f28>
Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f30424acd90>
Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f3048a41d08>
Setting s3 timeout as (60, 60)
```

#### 2. The torque task failed to run

There are three types of toil worker output log.

The first:

```log
[2020-10-05T16:07:21+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2020-10-05T16:07:21+0800] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host admin.
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil] Configuration: {'workflowID': '38652965-9036-40a1-ac10-1d1a386de7e3', 'workflowAttemptNumber': 0, 'jobStore': 'file:/public/home/graham/src/rhapsody-wta/results/rhapsody-wta-job-store', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': True, 'clean': 'never', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'torque', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 20, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '10.10.1.20:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'nodeOptions': None, 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 2, 'enableUnlimitedPreemptableRetries': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 20000000000, 'writeLogs': 'logs', 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.deferred] Running for file /tmp/node-38652965-9036-40a1-ac10-1d1a386de7e3-2628f3d8-c0d4-44cd-910e-e441656d37c7/deferred/funcdoy_idr0
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.worker] Parsed job wrapper
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-CWLWorkflow/instance-hlnz37vd/cleanup/file-1owkt82z/stream /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages toil.cwl.cwltoil True
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.deferred] Running job
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:21+0800] [MainThread] [D] [toil.resource] Module dir is /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages, our prefix is /public/home/graham/miniconda3/envs/bioinfo, virtualenv: True
[2020-10-05T16:07:21+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:22+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:23+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:24+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:24+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:24+0800] [MainThread] [W] [toil.resource] Can't globalize module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T16:07:24+0800] [MainThread] [D] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job files/for-job/kind-CWLWorkflow/instance-hlnz37vd/cleanup/file-1owkt82z/stream used 0.00% (8.0 KB [8192B] used, 2.0 GB [2147483648B] requested) at the end of its run.
[2020-10-05T16:07:24+0800] [MainThread] [D] [toil.deferred] Running own deferred functions
[2020-10-05T16:07:24+0800] [MainThread] [D] [toil.deferred] Out of deferred functions!
[2020-10-05T16:07:24+0800] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
[2020-10-05T16:07:24+0800] [MainThread] [D] [toil.worker] No more jobs can run in series by this worker, it's got 6 children
[2020-10-05T16:07:24+0800] [MainThread] [I] [toil.worker] Worker log can be found at /tmp/node-38652965-9036-40a1-ac10-1d1a386de7e3-2628f3d8-c0d4-44cd-910e-e441656d37c7/tmpbm0kylj9. Set --cleanWorkDir to retain this log
[2020-10-05T16:07:24+0800] [MainThread] [I] [toil.worker] Finished running the chain of jobs on this node, we ran for a total of 2.327002 seconds

```

The second:

```log
[2020-10-05T16:07:35+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2020-10-05T16:07:35+0800] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host admin.
[2020-10-05T16:07:35+0800] [MainThread] [D] [toil] Configuration: {'workflowID': '38652965-9036-40a1-ac10-1d1a386de7e3', 'workflowAttemptNumber': 0, 'jobStore': 'file:/public/home/graham/src/rhapsody-wta/results/rhapsody-wta-job-store', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': True, 'clean': 'never', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'torque', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 20, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '10.10.1.20:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'nodeOptions': None, 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 2, 'enableUnlimitedPreemptableRetries': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 20000000000, 'writeLogs': 'logs', 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
[2020-10-05T16:07:35+0800] [MainThread] [D] [toil.deferred] Running for file /tmp/node-38652965-9036-40a1-ac10-1d1a386de7e3-2628f3d8-c0d4-44cd-910e-e441656d37c7/deferred/funcf_mw1ndl
[2020-10-05T16:07:35+0800] [MainThread] [D] [toil.worker] Parsed job wrapper
[2020-10-05T16:07:35+0800] [MainThread] [D] [toil.worker] Wrapper has no user job to run.
[2020-10-05T16:07:35+0800] [MainThread] [D] [toil.worker] Cleaned up any references to completed successor jobs
[2020-10-05T16:07:35+0800] [MainThread] [D] [toil.worker] No user job to run, so finishing
[2020-10-05T16:07:35+0800] [MainThread] [I] [toil.worker] Worker log can be found at /tmp/node-38652965-9036-40a1-ac10-1d1a386de7e3-2628f3d8-c0d4-44cd-910e-e441656d37c7/tmpoo_9_iah. Set --cleanWorkDir to retain this log
[2020-10-05T16:07:35+0800] [MainThread] [I] [toil.worker] Finished running the chain of jobs on this node, we ran for a total of 0.000154 seconds
```

The second: 

```log
[2020-10-05T15:53:00+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2020-10-05T15:53:00+0800] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host comput23.
[2020-10-05T15:53:00+0800] [MainThread] [D] [toil] Configuration: {'workflowID': 'be3d1639-4f27-461e-9cbc-f11f1e5140d8', 'workflowAttemptNumber': 0, 'jobStore': 'file:/public/home/graham/src/rhapsody-wta/results/rhapsody-wta-job-store', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': True, 'clean': 'never', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'singleMachine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 32, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '127.0.0.1:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'nodeOptions': None, 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 2, 'enableUnlimitedPreemptableRetries': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': True, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 20000000000, 'writeLogs': 'logs', 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
[2020-10-05T15:53:00+0800] [MainThread] [D] [toil.deferred] Running for file /tmp/node-be3d1639-4f27-461e-9cbc-f11f1e5140d8-1ce75288-758a-422d-9b49-440f18459fd0/deferred/funcgbm3owim
[2020-10-05T15:53:00+0800] [MainThread] [D] [toil.worker] Parsed job wrapper
[2020-10-05T15:53:00+0800] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-CWLWorkflow/instance-x29npew7/cleanup/file-qhdohmv3/stream /public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages toil.cwl.cwltoil True
[2020-10-05T15:53:00+0800] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/public/home/graham/miniconda3/envs/bioinfo/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
[2020-10-05T15:53:00+0800] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
[2020-10-05T15:53:00+0800] [MainThread] [D] [toil.deferred] Running job
[2020-10-05T15:53:00+0800] [MainThread] [D] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job files/for-job/kind-CWLWorkflow/instance-x29npew7/cleanup/file-qhdohmv3/stream used 0.00% (32.0 KB [32768B] used, 3.0 GB [3221225472B] requested) at the end of its run.
[2020-10-05T15:53:01+0800] [MainThread] [D] [toil.deferred] Running own deferred functions
[2020-10-05T15:53:01+0800] [MainThread] [D] [toil.deferred] Out of deferred functions!
[2020-10-05T15:53:01+0800] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
[2020-10-05T15:53:01+0800] [MainThread] [D] [toil.worker] The jobGraph has multiple predecessors, we must return to the leader.
[2020-10-05T15:53:01+0800] [MainThread] [I] [toil.worker] Worker log can be found at /tmp/node-be3d1639-4f27-461e-9cbc-f11f1e5140d8-1ce75288-758a-422d-9b49-440f18459fd0/tmpwgaxfhox. Set --cleanWorkDir to retain this log
[2020-10-05T15:53:01+0800] [MainThread] [I] [toil.worker] Finished running the chain of jobs on this node, we ran for a total of 0.280400 seconds

```
Interesting. The only workaround that comes to mind is installing via `pip install toil[cwl]` instead of `pip install toil[all]` in a fresh virtualenv so the `boto3` codepaths aren't even considered. 

@adamnovak @DailyDreaming Do you have any insights here?
Problem 1 (the AWS stuff) is Boto checking to see if you are running inside AWS. It does this when we import it [via here](https://github.com/DataBiosphere/toil/blob/b7de76b5f9086175e179ebd0f763908128a14ab9/src/toil/jobStores/abstractJobStore.py#L219-L244). It seems to insist on printing out all the problems it encounters trying to talk to the AWS metadata service when Toil's log level is debug, even though it often expects to not be able to talk to the AWS metadata service (which is only available inside AWS). We can try to get Boto to not log at debug level when Toil does, but I don't think we want to try and stop it from trying to make the network connection.

The Toil worker logs don't really shed much light on problem 2. I'll check on the main log.

> Interesting. The only workaround that comes to mind is installing via `pip install toil[cwl]` instead of `pip install toil[all]` in a fresh virtualenv so the `boto3` codepaths aren't even considered.

I tried following this approach and the issue about AWS was indeed resolved, but the problem 2 remained.


The jobs that failed are being killed by something external to Toil before they can save their logs where the leader can get at them. There are a lot of `Job failed with exit value -9` (characteristic of manual or automatic immediate forced kill, or the out-of-memory killer) and `No log file is present, despite job failing` lines in the main log, although I did see a `Job failed with exit value -3` (which is SIGQUIT).

This might be due to the CWL workflow not specifying sufficient memory for its tasks, or not specifying sufficient memory for its tasks *plus* the memory overhead of the Toil worker. If the workflow doesn't specify anything, I think we use a default memory requirement which you could override with e.g. `--defaultMemory 192G`. There are similar `--defaultCores` and `--defaultDisk` arguments.

Your workflow looks like it wants to work on a while human genome, but all the places where Toil logs about "memory" in the log you posted seem to indicate that all the jobs are runnign with only 2 GB of memory (which is the default `--defaultMemory`), and I didn't see anything about memory in your CWL file.

I think you need to add appropriate memory requirements to your CWL, or up `--defaultMemory`, so your jobs won't be killed by Torque for hitting their memory limits.
> The contents of [`rhapsody-wta-yaml.cwl`](https://github.com/DataBiosphere/toil/files/5328014/rhapsody-wta-yaml.cwl.txt) are attached.

Have you checked this file? In this CWL, I have specified memory and CPU with `ResourceRequirement`

@adamnovak 
Are you saying that some step in CWL that doesn't specify ResourceRequirement actually requires more memory than `--defaultMemory`, so it gets killed?
http://docs.adaptivecomputing.com/torque/4-2-7/Content/topics/2-jobs/jobExitStatus.htm

I found a page that explains the meaning of the exit status, and the meaning of `-9` seems to be: Could not create/open stdout stderr files
I tried the following command, but it didn't solve the problem:

```shell
toil-cwl-runner \
  --batchSystem torque \
  --user-space-docker-cmd=udocker \
  --defaultMemory 32G \
  --defaultCores 20 \
  --defaultDisk 120G \
  --jobStore file:results/rhapsody-wta-job-store \
  --outdir results \
  --writeLogs logs \
  --logFile cwltoil.log \
  --logLevel DEBUG \
  --retryCount 1 \
  --maxLogFileSize 20000000000 \
  --stats \
  rhapsody-wta-yaml.cwl template_wta.yml
```
@adamnovak 

Since `exit status = -9` is related to `stdout/stderr`, and I noticed `generateTorqueWrapper` function in `src\toil\batchSystems\torque.py`:

```python
        def generateTorqueWrapper(self, command, jobID):
            """
            A very simple script generator that just wraps the command given; for
            now this goes to default tempdir
            """
            stdoutfile = self.boss.formatStdOutErrPath(jobID, 'torque', '$PBS_JOBID', 'std_output')
            stderrfile = self.boss.formatStdOutErrPath(jobID, 'torque', '$PBS_JOBID', 'std_error')

            _, tmpFile = tempfile.mkstemp(suffix='.sh', prefix='torque_wrapper')
            fh = open(tmpFile , 'w')
            fh.write("#!/bin/sh\n")
            fh.write("#PBS -o {}\n".format(stdoutfile))
            fh.write("#PBS -e {}\n".format(stderrfile))
            fh.write("cd $PBS_O_WORKDIR\n\n")
            fh.write(command + "\n")

            fh.close

            return tmpFile
```

I'm guessing there's something wrong with the path generated by `self.boss.formatStdOutErrPath`, so try using `--noStdOutErr` to disable this feature.

My hypothesis turned out to be correct, and I successfully solved problem 2 using the following command:

```shell
export TOIL_TORQUE_ARGS="-N graham-rhapsody-wta"
export TOIL_TORQUE_REQS="walltime=48:00:00"

toil-cwl-runner \
  --batchSystem torque \
  --user-space-docker-cmd=udocker \
  --defaultMemory 32G \
  --defaultCores 20 \
  --defaultDisk 120G \
  --jobStore file:results/rhapsody-wta-job-store \
  --outdir results \
  --writeLogs logs \
  --logFile cwltoil.log \
  --noStdOutErr \
  --logLevel INFO \
  --retryCount 1 \
  --maxLogFileSize 20000000000 \
  --stats \
  rhapsody-wta-yaml.cwl template_wta.yml
```
@mr-c 

> `TOIL_TORQUE_REQS` can not contain `mem`, `nodes`, nor `ppn`; those are generated dynamically by `toil-cwl-runner` based upon the https://www.commonwl.org/v1.0/CommandLineTool.html#ResourceRequirement `ram{Max,Min}`, and `cores{Max,Min}`.
> 
> By the way, do any of your workflow steps use MPI?

I'm new to HPC and I don't know much about MPI, but I don't think the CWL process I want to run uses MPI. However, I have a question, if the `ResourceRequirement` of a CWL step exceeds the hardware resources of a single compute node, will the workflow fail immediately? For example, the HPC normal compute node memory is 96 G, and there is a fat node memory of 6T. Now I have a step in my CWL that requires a minimum memory of 120 G. Will I fail to run this CWL workflow?
>  if the ResourceRequirement of a CWL step exceeds the hardware resources of a single compute node, will the workflow fail immediately?

That step would not get scheduled by your batch system (torque, in your case) on a node that was too small . If no node was available then the workflow would eventually fail, I guess?

Torque should send the big workflow step to the "fat node" automatically as long as they are in the same queue.
> > if the ResourceRequirement of a CWL step exceeds the hardware resources of a single compute node, will the workflow fail immediately?
> 
> That step would not get scheduled by your batch system (torque, in your case) on a node that was too small . If no node was available then the workflow would eventually fail, I guess?
> 
> Torque should send the big workflow step to the "fat node" automatically as long as they are in the same queue.

Thank you for your patience in explaining this! @mr-c 
Hm, that's weird that it can't create the stdout/stderr files. It could be that the system temp directory on the leader doesn't also exist on the worker, or wherever the output and error files are being written from. You could try using `--workDir` to point it at a directory that always exists on all the machines.

Or, it could be that the file names are too long. The log file name we make [here](https://github.com/DataBiosphere/toil/blob/cb1f59af0f60dadd8349edc8ee21292349cdd9e9/src/toil/batchSystems/abstractBatchSystem.py#L374) is going to be pretty long, and it includes the job ID, which in this workflow is something long like `kind-file_public_home_graham_src_rhapsody-wta_rhapsody-wta-yaml.cwl_rhapsody-wta_Internal_Settings_a3c9fc20-5052-4ee2-a916-179e2dcf2688/instance-xi8z242k`.

We may need to drop the length limit we lowered in https://github.com/DataBiosphere/toil/pull/3230 even more, so we can account for all the extra stuff we mix into the log file name. Or, we could remove the job ID from the log file name and just rely on the batch system ID to make it unique.
@adamnovak 
I checked and the directory `/tmp` on the leader node does not match `/tmp` on the worker node. However, setting the `--workDir . /tmp` doesn't work, and the exit status of the job is still `-9`. If I set the `--noStdOutErr` option, will it affect the results I end up getting from the CWL workflow?
@altairwei `--workDir` only takes one item, so use `--workDir . ` or `--workDir $PWD` and not  `--workDir . /tmp`

>   If I set the `--noStdOutErr` option, will it affect the results I end up getting from the CWL workflow?

It will not affect the execution or the results of a CWL workflow, no.
> @altairwei `--workDir` only takes one item, so use `--workDir . ` or `--workDir $PWD` and not `--workDir . /tmp`
> 
> > If I set the `--noStdOutErr` option, will it affect the results I end up getting from the CWL workflow?
> 
> It will not affect the execution or the results of a CWL workflow, no.

I actually use `--workDir ./tmp` instead of `--workDir .  /tmp`
> I actually use `--workDir ./tmp` instead of `--workDir .  /tmp`

Okay.

Looks like `--workDir` must be an absolute path, so you should use `--workDir $PWD/tmp` or similar
> > I actually use `--workDir ./tmp` instead of `--workDir .  /tmp`
> 
> Okay.
> 
> Looks like `--workDir` must be an absolute path, so you should use `--workDir $PWD/tmp` or similar

I run the following command and still get the `Job failed with exit value -9` error.

```shell
toil-cwl-runner \
  --batchSystem torque \
  --user-space-docker-cmd=udocker \
  --defaultMemory 16G \
  --defaultCores 8 \
  --defaultDisk 120G \
  --jobStore file:results/rhapsody-wta-job-store \
  --outdir results \
  --writeLogs logs \
  --logFile logs/cwltoil_$(date +%Y%m%d_%H%M%S).log \
  --workDir $PWD/tmp \
  --logLevel INFO \
  --retryCount 2 \
  --maxLogFileSize 20000000000 \
  --stats \
  rhapsody-wta-yaml.cwl template_wta.yml
```
Seems like you'll need to use the `--noStdOutErr` option until the issue you are experiencing can get fixed. Sorry!
Did you `mkdir $PWD/tmp` first?

On 10/7/20, Michael R. Crusoe <notifications@github.com> wrote:
> Seems like you'll need to use the `--noStdOutErr` option until the issue you
> are experiencing can get fixed. Sorry!
>
> --
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub:
> https://github.com/DataBiosphere/toil/issues/3254#issuecomment-704833800


-- 
---
Adam Novak (He/Him)
Senior Software Engineer
Computational Genomics Lab
UC Santa Cruz Genomics Institute
"Revealing life’s code."

Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5

> Did you `mkdir $PWD/tmp` first?

@adamnovak Yes

I installed the development version of toil from the source code, and will report back when I see exactly what the contents of `-o` and `-e` are in debug mode.
Why do these paths cause failure? They look so normal.

```
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_1_batch_torque_$PBS_JOBID_std_output.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_1_batch_torque_$PBS_JOBID_std_error.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_3_batch_torque_$PBS_JOBID_std_output.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_3_batch_torque_$PBS_JOBID_std_error.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_4_batch_torque_$PBS_JOBID_std_output.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_4_batch_torque_$PBS_JOBID_std_error.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_5_batch_torque_$PBS_JOBID_std_output.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_5_batch_torque_$PBS_JOBID_std_error.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_6_batch_torque_$PBS_JOBID_std_output.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_6_batch_torque_$PBS_JOBID_std_error.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_7_batch_torque_$PBS_JOBID_std_output.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_7_batch_torque_$PBS_JOBID_std_error.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_8_batch_torque_$PBS_JOBID_std_output.log
/tmp/toil_workflow_57b6a086-6d74-4578-8941-0ce2401e1c48_job_8_batch_torque_$PBS_JOBID_std_error.log
```

I tested it and it seems that just using the `$PBS_JOBID` variable causes an error!
If I change the reference format of `PBS_JOBID`:

```python
            stdoutfile = self.boss.formatStdOutErrPath(jobID, 'torque', r'${PBS_JOBID}', 'std_output')
            stderrfile = self.boss.formatStdOutErrPath(jobID, 'torque', r'${PBS_JOBID}', 'std_error')
```

The error code has changed.

```
Job 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#InternalSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_InternalSettings.cwl/instance-k9appcyh with ID kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_InternalSettings.cwl/instance-k9appcyh is completely failed
Job failed with exit value 120: 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#PutativeCellSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_PutativeCellSettings.cwl/instance-jinp7mis
No log file is present, despite job failing: 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#PutativeCellSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_PutativeCellSettings.cwl/instance-jinp7mis
Due to failure we are reducing the remaining retry count of job 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#PutativeCellSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_PutativeCellSettings.cwl/instance-jinp7mis with ID kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_PutativeCellSettings.cwl/instance-jinp7mis to 0
Job 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#PutativeCellSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_PutativeCellSettings.cwl/instance-jinp7mis with ID kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_PutativeCellSettings.cwl/instance-jinp7mis is completely failed
Job failed with exit value 120: 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#MultiplexingSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_MultiplexingSettings.cwl/instance-lvfnv1bw
No log file is present, despite job failing: 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#MultiplexingSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_MultiplexingSettings.cwl/instance-lvfnv1bw
Due to failure we are reducing the remaining retry count of job 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#MultiplexingSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_MultiplexingSettings.cwl/instance-lvfnv1bw with ID kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_MultiplexingSettings.cwl/instance-lvfnv1bw to 0
Job 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#MultiplexingSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_MultiplexingSettings.cwl/instance-lvfnv1bw with ID kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_MultiplexingSettings.cwl/instance-lvfnv1bw is completely failed
Job failed with exit value 120: 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#SubsampleSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_SubsampleSettings.cwl/instance-eq9vdegq
No log file is present, despite job failing: 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#SubsampleSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_SubsampleSettings.cwl/instance-eq9vdegq
Due to failure we are reducing the remaining retry count of job 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#SubsampleSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_SubsampleSettings.cwl/instance-eq9vdegq with ID kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_SubsampleSettings.cwl/instance-eq9vdegq to 0
Job 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#SubsampleSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_SubsampleSettings.cwl/instance-eq9vdegq with ID kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_SubsampleSettings.cwl/instance-eq9vdegq is completely failed
Finished toil run with 5 failed jobs.
Failed jobs at end of the run: 'CWLWorkflow' kind-CWLWorkflow/instance-7_r7n9ze 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#InternalSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_InternalSettings.cwl/instance-k9appcyh 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#SubsampleSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_SubsampleSettings.cwl/instance-eq9vdegq 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#PutativeCellSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_PutativeCellSettings.cwl/instance-jinp7mis 'file:///public/home/graham/src/rhapsody-wta/workflow/rhapsody_wta_1.8.packed.cwl#MultiplexingSettings.cwl' kind-file_public_home_graham_src_rhapsody-wta_workflow_rhapsody_wta_1.8.packed.cwl_MultiplexingSettings.cwl/instance-lvfnv1bw

```

Resolving patch:
diff --git a/src/toil/batchSystems/torque.py b/src/toil/batchSystems/torque.py
--- a/src/toil/batchSystems/torque.py
+++ b/src/toil/batchSystems/torque.py
@@ -201,8 +201,8 @@ def generateTorqueWrapper(self, command, jobID):
             A very simple script generator that just wraps the command given; for
             now this goes to default tempdir
             """
-            stdoutfile = self.boss.formatStdOutErrPath(jobID, 'torque', '$PBS_JOBID', 'std_output')
-            stderrfile = self.boss.formatStdOutErrPath(jobID, 'torque', '$PBS_JOBID', 'std_error')
+            stdoutfile = self.boss.formatStdOutErrPath(jobID, 'torque', r'${PBS_JOBID}', 'std_output')
+            stderrfile = self.boss.formatStdOutErrPath(jobID, 'torque', r'${PBS_JOBID}', 'std_error')
 
             _, tmpFile = tempfile.mkstemp(suffix='.sh', prefix='torque_wrapper')
             fh = open(tmpFile , 'w')


Test patch: