Repository: DataBiosphere/toil

Issue:
CWL: Merging and scattering arrays from the same step : "The given job is already a predecessor of this job"
Hi,

When merging and scattering arrays from the same step (like in the example workflow below) we got the error `The given job is already a predecessor of this job` 

This error is specific to toil and does not happen when using cwltool

Below How The error and How to reproduce it

# Version used

```
jfouret@compute-dedicated:/home/users/jfouret/toil_test$ toil --version
5.2.0a1
jfouret@compute-dedicated:/home/users/jfouret/toil_test$ which cwltool
/exports/./../home/users/jfouret/toil/venv/bin/cwltool
jfouret@compute-dedicated:/home/users/jfouret/toil_test$ which toil-cwl-runner 
/exports/./../home/users/jfouret/toil/venv/bin/toil-cwl-runner
```
 built from 4e7d12a65793c4a4c1c4e2b61109c19ad2756f02

# Test workflow:

```
jfouret@compute-dedicated:/home/users/jfouret/toil_test$ more echo-string.cwl 
cwlVersion: v1.2
class: Workflow

inputs: 
  - id: arrayS
    type: string[]
    default: ['hello','world']

steps:
  hello:
    run:
      class: CommandLineTool
      inputs: 
        s: string
      baseCommand: [echo]
      arguments: [ $(inputs.s)]
      outputs: 
        out: stdout
        err: stderr
    in:
      - id: s
        source: arrayS
    scatter: 
     - s
    out:
      - id: out
      - id: err

  list:
    run:
      class: CommandLineTool
      inputs:
       file: File
      baseCommand: [ ls , -lh]
      arguments: [ $(inputs.file) ]
      outputs: []
    in:
      - id: file
        linkMerge: merge_flattened
        source:
          - hello/out
          - hello/err
    scatter:
      - file
    out: []

outputs: []

requirements:
  - class: ScatterFeatureRequirement
  - class: MultipleInputFeatureRequirement

$namespaces:
  cwltool: "http://commonwl.org/cwltool#"

```

# Error log

```
jfouret@compute-dedicated:/home/users/jfouret/toil_test$ toil-cwl-runner --jobStore file:$PWD/jobstore --logLevel DEBUG --maxCores 16 --maxMemory 50G --strict-memory-limit --force-docker-pull --outdir outdir echo-string.cwl 
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.statsAndLogging] Suppressing the following loggers: {'boto3', 'google', 'cachecontrol', 'websocket', 'humanfriendly', 'botocore', 'oauthlib', 'urllib3', 'requests_oauthlib', 'boto', 'prov', 'rdflib', 'kubernetes', 'bcdocs', 'dill', 'salad', 'docker', 'requests', 'galaxy'}
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.statsAndLogging] Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'.
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.lib.threading] Total machine size: 16 cores
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.lib.threading] CPU quota: -1
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.lib.threading] Total machine size: 16 cores
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.lib.threading] CPU quota: -1
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.jobStores.fileJobStore] Path to job store directory is '/home/users/jfouret/toil_test/jobstore'.
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.jobStores.abstractJobStore] The workflow ID is: '41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd'
[2021-01-06T13:00:04+0100] [MainThread] [I] [cwltool] Resolved 'echo-string.cwl' to 'file:///home/users/jfouret/toil_test/echo-string.cwl'
[2021-01-06T13:00:05+0100] [MainThread] [D] [cwltool] Parsed job order from command line: {
    "id": "echo-string.cwl",
    "arrayS": [
        "hello",
        "world"
    ]
}
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Using the single machine batch system
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Obtained node ID af5ec22510434ef4a5723e75099c2291 from file /var/lib/dbus/machine-id
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Created the workflow directory for this machine at /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291
[2021-01-06T13:00:05+0100] [Thread-1  ] [D] [toil.batchSystems.singleMachine] Started daddy thread.
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.batchSystems.singleMachine] Started in normal mode.
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] User script ModuleDescriptor(dirPath='/home/users/jfouret/toil/src', name='toil.cwl.cwltoil', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it.
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] No user script to auto-deploy.
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Written the environment for the jobs to the environment file
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Caching all jobs in job store
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] 0 jobs downloaded.
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.job] Saving graph of 2 jobs, 2 new
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.job] Processing job 'ResolveIndirect' kind-ResolveIndirect/instance-k4qbud0u
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.job] Processing job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil] Running Toil version 5.2.0a1-4e7d12a65793c4a4c1c4e2b61109c19ad2756f02 on host compute-dedicated.
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil] Configuration: {'workflowID': '41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd', 'workflowAttemptNumber': 0, 'jobStore': 'file:/home/users/jfouret/toil_test/jobstore', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 16, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '10.0.4.50:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 16, 'maxMemory': 53687091200, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.realtimeLogger] Real-time logging disabled
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.toilState] Found job to run: kind-CWLWorkflow/instance-kefe7e_w, with command: True, with checkpoint: False, with  services: False, with no next successors: False
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Found 1 jobs to start and 0 jobs with successors to run
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Checked batch system has no running jobs and no updated jobs
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.serviceManager] Initializing service manager
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Built the jobs list, currently have 1 jobs to update and 0 jobs issued
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Updating status of job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with result status: 0
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.batchSystems.singleMachine] Issuing the command: _toil_worker CWLWorkflow file:/home/users/jfouret/toil_test/jobstore kind-CWLWorkflow/instance-kefe7e_w with memory: 2147483648, cores: 1.0, disk: 2147483648
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Issued job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with job batch system ID: 0 and cores: 1, disk: 2.0 G, and memory: 2.0 G
[2021-01-06T13:00:05+0100] [Thread-1  ] [D] [toil.batchSystems.singleMachine] Launched job 0 as child 17706
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.statsAndLogging] Suppressing the following loggers: {'cachecontrol', 'prov', 'humanfriendly', 'bcdocs', 'urllib3', 'salad', 'websocket', 'dill', 'google', 'requests_oauthlib', 'requests', 'rdflib', 'boto3', 'galaxy', 'botocore', 'kubernetes', 'oauthlib', 'docker', 'boto'}
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Obtained node ID af5ec22510434ef4a5723e75099c2291 from file /var/lib/dbus/machine-id
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/tmp6lq7py0f/worker_log.txt
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Deleting /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funchqj9mn3v
[2021-01-06T13:00:05+0100] [Thread-1  ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/home/users/jfouret/toil_test/jobstore kind-CWLWorkflow/instance-kefe7e_w.
[2021-01-06T13:00:05+0100] [Thread-1  ] [D] [toil.batchSystems.singleMachine] Child 17706 for job 0 succeeded
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.batchSystems.singleMachine] Ran jobID: 0 with exit value: 1
[2021-01-06T13:00:05+0100] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w continues to exist (i.e. has more to do)
[2021-01-06T13:00:05+0100] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:05+0100] [MainThread] [W] [toil.leader] Log from job "kind-CWLWorkflow/instance-kefe7e_w" follows:
=========>
	[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2021-01-06T13:00:05+0100] [MainThread] [I] [toil] Running Toil version 5.2.0a1-4e7d12a65793c4a4c1c4e2b61109c19ad2756f02 on host compute-dedicated.
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil] Configuration: {'workflowID': '41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd', 'workflowAttemptNumber': 0, 'jobStore': 'file:/home/users/jfouret/toil_test/jobstore', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 16, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '10.0.4.50:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 16, 'maxMemory': 53687091200, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Running for file /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funchqj9mn3v
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.worker] Parsed job description
	[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.worker] Working on job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-CWLWorkflow/instance-kefe7e_w/cleanup/file-c5d21eb9af93442f86d7064cf9608268/stream /home/users/jfouret/toil/src toil.cwl.cwltoil False
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/home/users/jfouret/toil/src', name='toil.cwl.cwltoil', fromVirtualEnv=False).
	[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.worker] Loaded body Job('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) from description 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Starting job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w).
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Running job
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Actually running job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w) which wants 2147483648 of our 646134366208 bytes.
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Max space: 646134366208
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total file size: 0
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total job disk requirement size: 2147483648
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total immutable reference size: 0
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Available space with job: 646134366208 bytes
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w used 0.00% (4.0 KB [4096B] used, 2.0 GB [2147483648B] requested) at the end of its run.
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Running own deferred functions
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Out of deferred functions!
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	Traceback (most recent call last):
	  File "/home/users/jfouret/toil/src/toil/worker.py", line 394, in workerScript
	    job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2359, in _runner
	    returnValues = self._run(jobGraph=None, fileStore=fileStore)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2280, in _run
	    return self.run(fileStore)
	  File "/home/users/jfouret/toil/src/toil/cwl/cwltoil.py", line 1660, in run
	    promises[s].addFollowOn(wfjob)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1211, in addFollowOn
	    followOnJob._addPredecessor(self)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1806, in _addPredecessor
	    raise RuntimeError("The given job is already a predecessor of this job")
	RuntimeError: The given job is already a predecessor of this job
	[2021-01-06T13:00:05+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host compute-dedicated
<=========
[2021-01-06T13:00:05+0100] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with ID kind-CWLWorkflow/instance-kefe7e_w to 1
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Added job: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w to active jobs
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.leader] 0 jobs are running, 0 jobs are issued and waiting to run
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Built the jobs list, currently have 1 jobs to update and 0 jobs issued
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Updating status of job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with result status: 1
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.batchSystems.singleMachine] Issuing the command: _toil_worker CWLWorkflow file:/home/users/jfouret/toil_test/jobstore kind-CWLWorkflow/instance-kefe7e_w with memory: 2147483648, cores: 1.0, disk: 2147483648
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Issued job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 2.0 G
[2021-01-06T13:00:05+0100] [Thread-1  ] [D] [toil.batchSystems.singleMachine] Launched job 1 as child 17712
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.statsAndLogging] Suppressing the following loggers: {'rdflib', 'kubernetes', 'galaxy', 'urllib3', 'salad', 'websocket', 'oauthlib', 'boto', 'cachecontrol', 'bcdocs', 'google', 'humanfriendly', 'dill', 'boto3', 'docker', 'requests', 'botocore', 'prov', 'requests_oauthlib'}
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.common] Obtained node ID af5ec22510434ef4a5723e75099c2291 from file /var/lib/dbus/machine-id
[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/tmp5rtu1skc/worker_log.txt
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Deleting /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funcrd9m6h8u
[2021-01-06T13:00:06+0100] [Thread-1  ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/home/users/jfouret/toil_test/jobstore kind-CWLWorkflow/instance-kefe7e_w.
[2021-01-06T13:00:06+0100] [Thread-1  ] [D] [toil.batchSystems.singleMachine] Child 17712 for job 1 succeeded
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.batchSystems.singleMachine] Ran jobID: 1 with exit value: 1
[2021-01-06T13:00:06+0100] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.leader] Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w continues to exist (i.e. has more to do)
[2021-01-06T13:00:06+0100] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:06+0100] [MainThread] [W] [toil.leader] Log from job "kind-CWLWorkflow/instance-kefe7e_w" follows:
=========>
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil] Running Toil version 5.2.0a1-4e7d12a65793c4a4c1c4e2b61109c19ad2756f02 on host compute-dedicated.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil] Configuration: {'workflowID': '41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd', 'workflowAttemptNumber': 0, 'jobStore': 'file:/home/users/jfouret/toil_test/jobstore', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 16, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '10.0.4.50:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 16, 'maxMemory': 53687091200, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running for file /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funcrd9m6h8u
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.worker] Parsed job description
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] Working on job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-CWLWorkflow/instance-kefe7e_w/cleanup/file-c5d21eb9af93442f86d7064cf9608268/stream /home/users/jfouret/toil/src toil.cwl.cwltoil False
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/home/users/jfouret/toil/src', name='toil.cwl.cwltoil', fromVirtualEnv=False).
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] Loaded body Job('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) from description 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Starting job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w).
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running job
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Actually running job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w) which wants 2147483648 of our 646134366208 bytes.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Max space: 646134366208
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total file size: 0
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total job disk requirement size: 2147483648
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total immutable reference size: 0
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Available space with job: 646134366208 bytes
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w used 0.00% (4.0 KB [4096B] used, 2.0 GB [2147483648B] requested) at the end of its run.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running own deferred functions
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Out of deferred functions!
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	Traceback (most recent call last):
	  File "/home/users/jfouret/toil/src/toil/worker.py", line 394, in workerScript
	    job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2359, in _runner
	    returnValues = self._run(jobGraph=None, fileStore=fileStore)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2280, in _run
	    return self.run(fileStore)
	  File "/home/users/jfouret/toil/src/toil/cwl/cwltoil.py", line 1660, in run
	    promises[s].addFollowOn(wfjob)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1211, in addFollowOn
	    followOnJob._addPredecessor(self)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1806, in _addPredecessor
	    raise RuntimeError("The given job is already a predecessor of this job")
	RuntimeError: The given job is already a predecessor of this job
	[2021-01-06T13:00:06+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host compute-dedicated
<=========
[2021-01-06T13:00:06+0100] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with ID kind-CWLWorkflow/instance-kefe7e_w to 0
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.leader] Added job: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w to active jobs
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.leader] Built the jobs list, currently have 1 jobs to update and 0 jobs issued
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.leader] Updating status of job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with result status: 1
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.leader] Found new failed successors: kind-ResolveIndirect/instance-k4qbud0u of job: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:06+0100] [MainThread] [W] [toil.leader] Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w is completely failed
[2021-01-06T13:00:08+0100] [MainThread] [D] [toil.leader] Finished the main loop: no jobs left to run.
[2021-01-06T13:00:08+0100] [MainThread] [D] [toil.serviceManager] Waiting for service manager thread to finish ...
[2021-01-06T13:00:09+0100] [Thread-2  ] [D] [toil.serviceManager] Received signal to quit starting services.
[2021-01-06T13:00:10+0100] [MainThread] [D] [toil.serviceManager] ... finished shutting down the service manager. Took 1.7502992153167725 seconds
[2021-01-06T13:00:10+0100] [MainThread] [D] [toil.statsAndLogging] Waiting for stats and logging collator thread to finish ...
[2021-01-06T13:00:10+0100] [MainThread] [D] [toil.statsAndLogging] ... finished collating stats and logs. Took 0.0006322860717773438 seconds
[2021-01-06T13:00:14+0100] [MainThread] [I] [toil.leader] Finished toil run with 1 failed jobs.
[2021-01-06T13:00:14+0100] [MainThread] [I] [toil.leader] Failed jobs at end of the run: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w

[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.common] Shutting down batch system ...
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.common] Obtained node ID af5ec22510434ef4a5723e75099c2291 from file /var/lib/dbus/machine-id
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.deferred] Cleaning up deferred functions system
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.deferred] Running for file /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funcuzd9xmsm
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.deferred] Deleting /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funcuzd9xmsm
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Connecting to latest caching database cache-0.db for cleanup
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.common] ... finished shutting down the batch system in 0.003482818603515625 seconds.
Traceback (most recent call last):
  File "/exports/./../home/users/jfouret/toil/venv/bin/toil-cwl-runner", line 33, in <module>
    sys.exit(load_entry_point('toil', 'console_scripts', 'toil-cwl-runner')())
  File "/home/users/jfouret/toil/src/toil/cwl/cwltoil.py", line 2250, in main
    outobj = toil.start(wf1)
  File "/home/users/jfouret/toil/src/toil/common.py", line 842, in start
    return self._runMainLoop(rootJobDescription)
  File "/home/users/jfouret/toil/src/toil/common.py", line 1135, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/users/jfouret/toil/src/toil/leader.py", line 262, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/home/users/jfouret/toil_test/jobstore' contains 1 failed jobs: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
Log from job "'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w" follows:
=========>
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil] Running Toil version 5.2.0a1-4e7d12a65793c4a4c1c4e2b61109c19ad2756f02 on host compute-dedicated.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil] Configuration: {'workflowID': '41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd', 'workflowAttemptNumber': 0, 'jobStore': 'file:/home/users/jfouret/toil_test/jobstore', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 16, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '10.0.4.50:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 16, 'maxMemory': 53687091200, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running for file /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funcrd9m6h8u
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.worker] Parsed job description
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] Working on job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-CWLWorkflow/instance-kefe7e_w/cleanup/file-c5d21eb9af93442f86d7064cf9608268/stream /home/users/jfouret/toil/src toil.cwl.cwltoil False
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/home/users/jfouret/toil/src', name='toil.cwl.cwltoil', fromVirtualEnv=False).
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] Loaded body Job('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) from description 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Starting job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w).
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running job
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Actually running job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w) which wants 2147483648 of our 646134366208 bytes.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Max space: 646134366208
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total file size: 0
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total job disk requirement size: 2147483648
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total immutable reference size: 0
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Available space with job: 646134366208 bytes
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w used 0.00% (4.0 KB [4096B] used, 2.0 GB [2147483648B] requested) at the end of its run.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running own deferred functions
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Out of deferred functions!
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	Traceback (most recent call last):
	  File "/home/users/jfouret/toil/src/toil/worker.py", line 394, in workerScript
	    job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2359, in _runner
	    returnValues = self._run(jobGraph=None, fileStore=fileStore)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2280, in _run
	    return self.run(fileStore)
	  File "/home/users/jfouret/toil/src/toil/cwl/cwltoil.py", line 1660, in run
	    promises[s].addFollowOn(wfjob)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1211, in addFollowOn
	    followOnJob._addPredecessor(self)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1806, in _addPredecessor
	    raise RuntimeError("The given job is already a predecessor of this job")
	RuntimeError: The given job is already a predecessor of this job
	[2021-01-06T13:00:06+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host compute-dedicated
<=========
jfouret@compute-dedicated:/home/users/jfouret/toil_test$ cwltool echo-string.cwl 
INFO /exports/./../home/users/jfouret/toil/venv/bin/cwltool 3.0.20201203173111
INFO Resolved 'echo-string.cwl' to 'file:///home/users/jfouret/toil_test/echo-string.cwl'
INFO [workflow ] start
INFO [workflow ] starting step hello
INFO [step hello] start
INFO [job hello] /tmp/vqzk_5ue$ echo \
    hello > /tmp/vqzk_5ue/6ab573038c4ba4db9bd833536c9b2265f08a272e 2> /tmp/vqzk_5ue/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
INFO [job hello] completed success
INFO [step hello] start
INFO [job hello_2] /tmp/lf9i0f73$ echo \
    world > /tmp/lf9i0f73/6ab573038c4ba4db9bd833536c9b2265f08a272e 2> /tmp/lf9i0f73/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
INFO [job hello_2] completed success
INFO [step hello] completed success
INFO [workflow ] starting step list
INFO [step list] start
INFO [job list] /tmp/7ngnbjl2$ ls \
    -lh \
    /tmp/qfcmenae/stgdf1b8826-cfe3-49c8-8131-5bcea19d8b8c/6ab573038c4ba4db9bd833536c9b2265f08a272e
lrwxrwxrwx 1 jfouret employee 54 Jan  6 13:01 /tmp/qfcmenae/stgdf1b8826-cfe3-49c8-8131-5bcea19d8b8c/6ab573038c4ba4db9bd833536c9b2265f08a272e -> /tmp/vqzk_5ue/6ab573038c4ba4db9bd833536c9b2265f08a272e
INFO [job list] completed success
INFO [step list] start
INFO [job list_2] /tmp/hlv_pk4g$ ls \
    -lh \
    /tmp/rpxpvp3o/stg4ffd99b2-612d-4e50-b284-55382a2b45ff/6ab573038c4ba4db9bd833536c9b2265f08a272e
lrwxrwxrwx 1 jfouret employee 54 Jan  6 13:01 /tmp/rpxpvp3o/stg4ffd99b2-612d-4e50-b284-55382a2b45ff/6ab573038c4ba4db9bd833536c9b2265f08a272e -> /tmp/lf9i0f73/6ab573038c4ba4db9bd833536c9b2265f08a272e
INFO [job list_2] completed success
INFO [step list] start
INFO [job list_3] /tmp/gg1rl9l_$ ls \
    -lh \jfouret@compute-dedicated:/home/users/jfouret/toil_test$ toil-cwl-runner --jobStore file:$PWD/jobstore --logLevel DEBUG --maxCores 16 --maxMemory 50G --strict-memory-limit --force-docker-pull --outdir outdir echo-string.cwl 
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.statsAndLogging] Suppressing the following loggers: {'boto3', 'google', 'cachecontrol', 'websocket', 'humanfriendly', 'botocore', 'oauthlib', 'urllib3', 'requests_oauthlib', 'boto', 'prov', 'rdflib', 'kubernetes', 'bcdocs', 'dill', 'salad', 'docker', 'requests', 'galaxy'}
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.statsAndLogging] Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'.
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.lib.threading] Total machine size: 16 cores
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.lib.threading] CPU quota: -1
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.lib.threading] Total machine size: 16 cores
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.lib.threading] CPU quota: -1
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.jobStores.fileJobStore] Path to job store directory is '/home/users/jfouret/toil_test/jobstore'.
[2021-01-06T13:00:04+0100] [MainThread] [D] [toil.jobStores.abstractJobStore] The workflow ID is: '41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd'
[2021-01-06T13:00:04+0100] [MainThread] [I] [cwltool] Resolved 'echo-string.cwl' to 'file:///home/users/jfouret/toil_test/echo-string.cwl'
[2021-01-06T13:00:05+0100] [MainThread] [D] [cwltool] Parsed job order from command line: {
    "id": "echo-string.cwl",
    "arrayS": [
        "hello",
        "world"
    ]
}
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Using the single machine batch system
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Obtained node ID af5ec22510434ef4a5723e75099c2291 from file /var/lib/dbus/machine-id
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Created the workflow directory for this machine at /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291
[2021-01-06T13:00:05+0100] [Thread-1  ] [D] [toil.batchSystems.singleMachine] Started daddy thread.
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.batchSystems.singleMachine] Started in normal mode.
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] User script ModuleDescriptor(dirPath='/home/users/jfouret/toil/src', name='toil.cwl.cwltoil', fromVirtualEnv=False) belongs to Toil. No need to auto-deploy it.
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] No user script to auto-deploy.
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Written the environment for the jobs to the environment file
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Caching all jobs in job store
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] 0 jobs downloaded.
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.job] Saving graph of 2 jobs, 2 new
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.job] Processing job 'ResolveIndirect' kind-ResolveIndirect/instance-k4qbud0u
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.job] Processing job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil] Running Toil version 5.2.0a1-4e7d12a65793c4a4c1c4e2b61109c19ad2756f02 on host compute-dedicated.
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil] Configuration: {'workflowID': '41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd', 'workflowAttemptNumber': 0, 'jobStore': 'file:/home/users/jfouret/toil_test/jobstore', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 16, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '10.0.4.50:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 16, 'maxMemory': 53687091200, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.realtimeLogger] Real-time logging disabled
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.toilState] Found job to run: kind-CWLWorkflow/instance-kefe7e_w, with command: True, with checkpoint: False, with  services: False, with no next successors: False
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Found 1 jobs to start and 0 jobs with successors to run
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Checked batch system has no running jobs and no updated jobs
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.serviceManager] Initializing service manager
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Built the jobs list, currently have 1 jobs to update and 0 jobs issued
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Updating status of job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with result status: 0
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.batchSystems.singleMachine] Issuing the command: _toil_worker CWLWorkflow file:/home/users/jfouret/toil_test/jobstore kind-CWLWorkflow/instance-kefe7e_w with memory: 2147483648, cores: 1.0, disk: 2147483648
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Issued job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with job batch system ID: 0 and cores: 1, disk: 2.0 G, and memory: 2.0 G
[2021-01-06T13:00:05+0100] [Thread-1  ] [D] [toil.batchSystems.singleMachine] Launched job 0 as child 17706
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.statsAndLogging] Suppressing the following loggers: {'cachecontrol', 'prov', 'humanfriendly', 'bcdocs', 'urllib3', 'salad', 'websocket', 'dill', 'google', 'requests_oauthlib', 'requests', 'rdflib', 'boto3', 'galaxy', 'botocore', 'kubernetes', 'oauthlib', 'docker', 'boto'}
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.common] Obtained node ID af5ec22510434ef4a5723e75099c2291 from file /var/lib/dbus/machine-id
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/tmp6lq7py0f/worker_log.txt
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Deleting /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funchqj9mn3v
[2021-01-06T13:00:05+0100] [Thread-1  ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/home/users/jfouret/toil_test/jobstore kind-CWLWorkflow/instance-kefe7e_w.
[2021-01-06T13:00:05+0100] [Thread-1  ] [D] [toil.batchSystems.singleMachine] Child 17706 for job 0 succeeded
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.batchSystems.singleMachine] Ran jobID: 0 with exit value: 1
[2021-01-06T13:00:05+0100] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w continues to exist (i.e. has more to do)
[2021-01-06T13:00:05+0100] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:05+0100] [MainThread] [W] [toil.leader] Log from job "kind-CWLWorkflow/instance-kefe7e_w" follows:
=========>
	[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2021-01-06T13:00:05+0100] [MainThread] [I] [toil] Running Toil version 5.2.0a1-4e7d12a65793c4a4c1c4e2b61109c19ad2756f02 on host compute-dedicated.
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil] Configuration: {'workflowID': '41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd', 'workflowAttemptNumber': 0, 'jobStore': 'file:/home/users/jfouret/toil_test/jobstore', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 16, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '10.0.4.50:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 16, 'maxMemory': 53687091200, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Running for file /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funchqj9mn3v
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.worker] Parsed job description
	[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.worker] Working on job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-CWLWorkflow/instance-kefe7e_w/cleanup/file-c5d21eb9af93442f86d7064cf9608268/stream /home/users/jfouret/toil/src toil.cwl.cwltoil False
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/home/users/jfouret/toil/src', name='toil.cwl.cwltoil', fromVirtualEnv=False).
	[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.worker] Loaded body Job('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) from description 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Starting job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w).
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Running job
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Actually running job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w) which wants 2147483648 of our 646134366208 bytes.
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Max space: 646134366208
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total file size: 0
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total job disk requirement size: 2147483648
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total immutable reference size: 0
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Available space with job: 646134366208 bytes
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w used 0.00% (4.0 KB [4096B] used, 2.0 GB [2147483648B] requested) at the end of its run.
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Running own deferred functions
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Out of deferred functions!
	[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	Traceback (most recent call last):
	  File "/home/users/jfouret/toil/src/toil/worker.py", line 394, in workerScript
	    job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2359, in _runner
	    returnValues = self._run(jobGraph=None, fileStore=fileStore)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2280, in _run
	    return self.run(fileStore)
	  File "/home/users/jfouret/toil/src/toil/cwl/cwltoil.py", line 1660, in run
	    promises[s].addFollowOn(wfjob)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1211, in addFollowOn
	    followOnJob._addPredecessor(self)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1806, in _addPredecessor
	    raise RuntimeError("The given job is already a predecessor of this job")
	RuntimeError: The given job is already a predecessor of this job
	[2021-01-06T13:00:05+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host compute-dedicated
<=========
[2021-01-06T13:00:05+0100] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with ID kind-CWLWorkflow/instance-kefe7e_w to 1
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Added job: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w to active jobs
[2021-01-06T13:00:05+0100] [MainThread] [I] [toil.leader] 0 jobs are running, 0 jobs are issued and waiting to run
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Built the jobs list, currently have 1 jobs to update and 0 jobs issued
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Updating status of job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with result status: 1
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.batchSystems.singleMachine] Issuing the command: _toil_worker CWLWorkflow file:/home/users/jfouret/toil_test/jobstore kind-CWLWorkflow/instance-kefe7e_w with memory: 2147483648, cores: 1.0, disk: 2147483648
[2021-01-06T13:00:05+0100] [MainThread] [D] [toil.leader] Issued job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 2.0 G
[2021-01-06T13:00:05+0100] [Thread-1  ] [D] [toil.batchSystems.singleMachine] Launched job 1 as child 17712
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.statsAndLogging] Suppressing the following loggers: {'rdflib', 'kubernetes', 'galaxy', 'urllib3', 'salad', 'websocket', 'oauthlib', 'boto', 'cachecontrol', 'bcdocs', 'google', 'humanfriendly', 'dill', 'boto3', 'docker', 'requests', 'botocore', 'prov', 'requests_oauthlib'}
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.common] Obtained node ID af5ec22510434ef4a5723e75099c2291 from file /var/lib/dbus/machine-id
[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/tmp5rtu1skc/worker_log.txt
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Deleting /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funcrd9m6h8u
[2021-01-06T13:00:06+0100] [Thread-1  ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/home/users/jfouret/toil_test/jobstore kind-CWLWorkflow/instance-kefe7e_w.
[2021-01-06T13:00:06+0100] [Thread-1  ] [D] [toil.batchSystems.singleMachine] Child 17712 for job 1 succeeded
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.batchSystems.singleMachine] Ran jobID: 1 with exit value: 1
[2021-01-06T13:00:06+0100] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.leader] Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w continues to exist (i.e. has more to do)
[2021-01-06T13:00:06+0100] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:06+0100] [MainThread] [W] [toil.leader] Log from job "kind-CWLWorkflow/instance-kefe7e_w" follows:
=========>
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil] Running Toil version 5.2.0a1-4e7d12a65793c4a4c1c4e2b61109c19ad2756f02 on host compute-dedicated.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil] Configuration: {'workflowID': '41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd', 'workflowAttemptNumber': 0, 'jobStore': 'file:/home/users/jfouret/toil_test/jobstore', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 16, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '10.0.4.50:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 16, 'maxMemory': 53687091200, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running for file /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funcrd9m6h8u
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.worker] Parsed job description
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] Working on job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-CWLWorkflow/instance-kefe7e_w/cleanup/file-c5d21eb9af93442f86d7064cf9608268/stream /home/users/jfouret/toil/src toil.cwl.cwltoil False
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/home/users/jfouret/toil/src', name='toil.cwl.cwltoil', fromVirtualEnv=False).
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] Loaded body Job('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) from description 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Starting job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w).
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running job
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Actually running job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w) which wants 2147483648 of our 646134366208 bytes.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Max space: 646134366208
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total file size: 0
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total job disk requirement size: 2147483648
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total immutable reference size: 0
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Available space with job: 646134366208 bytes
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w used 0.00% (4.0 KB [4096B] used, 2.0 GB [2147483648B] requested) at the end of its run.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running own deferred functions
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Out of deferred functions!
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	Traceback (most recent call last):
	  File "/home/users/jfouret/toil/src/toil/worker.py", line 394, in workerScript
	    job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2359, in _runner
	    returnValues = self._run(jobGraph=None, fileStore=fileStore)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2280, in _run
	    return self.run(fileStore)
	  File "/home/users/jfouret/toil/src/toil/cwl/cwltoil.py", line 1660, in run
	    promises[s].addFollowOn(wfjob)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1211, in addFollowOn
	    followOnJob._addPredecessor(self)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1806, in _addPredecessor
	    raise RuntimeError("The given job is already a predecessor of this job")
	RuntimeError: The given job is already a predecessor of this job
	[2021-01-06T13:00:06+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host compute-dedicated
<=========
[2021-01-06T13:00:06+0100] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with ID kind-CWLWorkflow/instance-kefe7e_w to 0
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.leader] Added job: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w to active jobs
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.leader] Built the jobs list, currently have 1 jobs to update and 0 jobs issued
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.leader] Updating status of job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w with result status: 1
[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.leader] Found new failed successors: kind-ResolveIndirect/instance-k4qbud0u of job: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
[2021-01-06T13:00:06+0100] [MainThread] [W] [toil.leader] Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w is completely failed
[2021-01-06T13:00:08+0100] [MainThread] [D] [toil.leader] Finished the main loop: no jobs left to run.
[2021-01-06T13:00:08+0100] [MainThread] [D] [toil.serviceManager] Waiting for service manager thread to finish ...
[2021-01-06T13:00:09+0100] [Thread-2  ] [D] [toil.serviceManager] Received signal to quit starting services.
[2021-01-06T13:00:10+0100] [MainThread] [D] [toil.serviceManager] ... finished shutting down the service manager. Took 1.7502992153167725 seconds
[2021-01-06T13:00:10+0100] [MainThread] [D] [toil.statsAndLogging] Waiting for stats and logging collator thread to finish ...
[2021-01-06T13:00:10+0100] [MainThread] [D] [toil.statsAndLogging] ... finished collating stats and logs. Took 0.0006322860717773438 seconds
[2021-01-06T13:00:14+0100] [MainThread] [I] [toil.leader] Finished toil run with 1 failed jobs.
[2021-01-06T13:00:14+0100] [MainThread] [I] [toil.leader] Failed jobs at end of the run: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w

[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.common] Shutting down batch system ...
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.common] Obtained node ID af5ec22510434ef4a5723e75099c2291 from file /var/lib/dbus/machine-id
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.deferred] Cleaning up deferred functions system
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.deferred] Running for file /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funcuzd9xmsm
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.deferred] Deleting /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funcuzd9xmsm
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Connecting to latest caching database cache-0.db for cleanup
[2021-01-06T13:00:14+0100] [MainThread] [D] [toil.common] ... finished shutting down the batch system in 0.003482818603515625 seconds.
Traceback (most recent call last):
  File "/exports/./../home/users/jfouret/toil/venv/bin/toil-cwl-runner", line 33, in <module>
    sys.exit(load_entry_point('toil', 'console_scripts', 'toil-cwl-runner')())
  File "/home/users/jfouret/toil/src/toil/cwl/cwltoil.py", line 2250, in main
    outobj = toil.start(wf1)
  File "/home/users/jfouret/toil/src/toil/common.py", line 842, in start
    return self._runMainLoop(rootJobDescription)
  File "/home/users/jfouret/toil/src/toil/common.py", line 1135, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/users/jfouret/toil/src/toil/leader.py", line 262, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/home/users/jfouret/toil_test/jobstore' contains 1 failed jobs: 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
Log from job "'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w" follows:
=========>
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil] Running Toil version 5.2.0a1-4e7d12a65793c4a4c1c4e2b61109c19ad2756f02 on host compute-dedicated.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil] Configuration: {'workflowID': '41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd', 'workflowAttemptNumber': 0, 'jobStore': 'file:/home/users/jfouret/toil_test/jobstore', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': False, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 16, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 10000, 'scale': 1, 'linkImports': True, 'moveExports': True, 'mesosMasterAddress': '10.0.4.50:5050', 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 16, 'maxMemory': 53687091200, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 60, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.0, 'badWorkerFailInterval': 0.01, 'cwl': False}
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running for file /tmp/node-41ac8c0b-29cc-420e-a7f6-7f1cdb8a2afd-af5ec22510434ef4a5723e75099c2291/deferred/funcrd9m6h8u
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.worker] Parsed job description
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] Working on job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-CWLWorkflow/instance-kefe7e_w/cleanup/file-c5d21eb9af93442f86d7064cf9608268/stream /home/users/jfouret/toil/src toil.cwl.cwltoil False
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/home/users/jfouret/toil/src', name='toil.cwl.cwltoil', fromVirtualEnv=False).
	[2021-01-06T13:00:06+0100] [MainThread] [I] [toil.worker] Loaded body Job('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) from description 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Starting job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w).
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running job
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Actually running job ('CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w) with ID (kind-CWLWorkflow/instance-kefe7e_w) which wants 2147483648 of our 646134366208 bytes.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Max space: 646134366208
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total file size: 0
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total job disk requirement size: 2147483648
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Total immutable reference size: 0
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.cachingFileStore] Available space with job: 646134366208 bytes
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.resource] Module dir is /home/users/jfouret/toil/src, our prefix is /home/users/jfouret/toil/venv, virtualenv: True
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job 'CWLWorkflow' kind-CWLWorkflow/instance-kefe7e_w used 0.00% (4.0 KB [4096B] used, 2.0 GB [2147483648B] requested) at the end of its run.
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running own deferred functions
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Out of deferred functions!
	[2021-01-06T13:00:06+0100] [MainThread] [D] [toil.deferred] Running orphaned deferred functions
	Traceback (most recent call last):
	  File "/home/users/jfouret/toil/src/toil/worker.py", line 394, in workerScript
	    job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2359, in _runner
	    returnValues = self._run(jobGraph=None, fileStore=fileStore)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 2280, in _run
	    return self.run(fileStore)
	  File "/home/users/jfouret/toil/src/toil/cwl/cwltoil.py", line 1660, in run
	    promises[s].addFollowOn(wfjob)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1211, in addFollowOn
	    followOnJob._addPredecessor(self)
	  File "/home/users/jfouret/toil/src/toil/job.py", line 1806, in _addPredecessor
	    raise RuntimeError("The given job is already a predecessor of this job")
	RuntimeError: The given job is already a predecessor of this job
	[2021-01-06T13:00:06+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host compute-dedicated
<=========
```

# cwltool output

```
jfouret@compute-dedicated:/home/users/jfouret/toil_test$ cwltool echo-string.cwl 
INFO /exports/./../home/users/jfouret/toil/venv/bin/cwltool 3.0.20201203173111
INFO Resolved 'echo-string.cwl' to 'file:///home/users/jfouret/toil_test/echo-string.cwl'
INFO [workflow ] start
INFO [workflow ] starting step hello
INFO [step hello] start
INFO [job hello] /tmp/vqzk_5ue$ echo \
    hello > /tmp/vqzk_5ue/6ab573038c4ba4db9bd833536c9b2265f08a272e 2> /tmp/vqzk_5ue/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
INFO [job hello] completed success
INFO [step hello] start
INFO [job hello_2] /tmp/lf9i0f73$ echo \
    world > /tmp/lf9i0f73/6ab573038c4ba4db9bd833536c9b2265f08a272e 2> /tmp/lf9i0f73/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
INFO [job hello_2] completed success
INFO [step hello] completed success
INFO [workflow ] starting step list
INFO [step list] start
INFO [job list] /tmp/7ngnbjl2$ ls \
    -lh \
    /tmp/qfcmenae/stgdf1b8826-cfe3-49c8-8131-5bcea19d8b8c/6ab573038c4ba4db9bd833536c9b2265f08a272e
lrwxrwxrwx 1 jfouret employee 54 Jan  6 13:01 /tmp/qfcmenae/stgdf1b8826-cfe3-49c8-8131-5bcea19d8b8c/6ab573038c4ba4db9bd833536c9b2265f08a272e -> /tmp/vqzk_5ue/6ab573038c4ba4db9bd833536c9b2265f08a272e
INFO [job list] completed success
INFO [step list] start
INFO [job list_2] /tmp/hlv_pk4g$ ls \
    -lh \
    /tmp/rpxpvp3o/stg4ffd99b2-612d-4e50-b284-55382a2b45ff/6ab573038c4ba4db9bd833536c9b2265f08a272e
lrwxrwxrwx 1 jfouret employee 54 Jan  6 13:01 /tmp/rpxpvp3o/stg4ffd99b2-612d-4e50-b284-55382a2b45ff/6ab573038c4ba4db9bd833536c9b2265f08a272e -> /tmp/lf9i0f73/6ab573038c4ba4db9bd833536c9b2265f08a272e
INFO [job list_2] completed success
INFO [step list] start
INFO [job list_3] /tmp/gg1rl9l_$ ls \
    -lh \
    /tmp/7e1g8ri7/stgef503449-599e-4250-80f6-618e9e5acb78/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
lrwxrwxrwx 1 jfouret employee 54 Jan  6 13:01 /tmp/7e1g8ri7/stgef503449-599e-4250-80f6-618e9e5acb78/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7 -> /tmp/vqzk_5ue/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
INFO [job list_3] completed success
INFO [step list] start
INFO [job list_4] /tmp/cp5azbdt$ ls \
    -lh \
    /tmp/18e66dp4/stgff198105-01d6-48c5-8f28-1d477c9643c8/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
lrwxrwxrwx 1 jfouret employee 54 Jan  6 13:01 /tmp/18e66dp4/stgff198105-01d6-48c5-8f28-1d477c9643c8/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7 -> /tmp/lf9i0f73/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
INFO [job list_4] completed success
INFO [step list] completed success
INFO [workflow ] completed success
{}
INFO Final process status is success

    /tmp/7e1g8ri7/stgef503449-599e-4250-80f6-618e9e5acb78/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
lrwxrwxrwx 1 jfouret employee 54 Jan  6 13:01 /tmp/7e1g8ri7/stgef503449-599e-4250-80f6-618e9e5acb78/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7 -> /tmp/vqzk_5ue/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
INFO [job list_3] completed success
INFO [step list] start
INFO [job list_4] /tmp/cp5azbdt$ ls \
    -lh \
    /tmp/18e66dp4/stgff198105-01d6-48c5-8f28-1d477c9643c8/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
lrwxrwxrwx 1 jfouret employee 54 Jan  6 13:01 /tmp/18e66dp4/stgff198105-01d6-48c5-8f28-1d477c9643c8/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7 -> /tmp/lf9i0f73/cc7e4becc0c2d09944d5ac1fb18c4ecf874ac2d7
INFO [job list_4] completed success
INFO [step list] completed success
INFO [workflow ] completed success
{}
INFO Final process status is success

```

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-774)
┆Issue Number: TOIL-774



Discussion:
@jfouret Will look into this, thanks for posting the issue.

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -1544,7 +1544,7 @@ def remove_pickle_problems(obj: ProcessType) -> ProcessType:
 
 class CWLWorkflow(Job):
     """
-    Toil Job to convert a CWL workflow grah into a Toil job graph.
+    Toil Job to convert a CWL workflow graph into a Toil job graph.
 
     The Toil job graph will include the appropriate dependencies.
     """
@@ -1587,7 +1587,7 @@ def run(self, file_store: AbstractFileStore):
         all_outputs_fulfilled = False
         while not all_outputs_fulfilled:
             # Iteratively go over the workflow steps, scheduling jobs as their
-            # dependencies can be fufilled by upstream workflow inputs or
+            # dependencies can be fulfilled by upstream workflow inputs or
             # step outputs. Loop exits when the workflow outputs
             # are satisfied.
 
@@ -1597,10 +1597,9 @@ def run(self, file_store: AbstractFileStore):
                 if step.tool["id"] not in jobs:
                     stepinputs_fufilled = True
                     for inp in step.tool["inputs"]:
-                        if "source" in inp:
-                            for s in aslist(inp["source"]):
-                                if s not in promises:
-                                    stepinputs_fufilled = False
+                        for s in aslist(inp.get("source", [])):
+                            if s not in promises:
+                                stepinputs_fufilled = False
                     if stepinputs_fufilled:
                         jobobj = {}
 
@@ -1654,9 +1653,12 @@ def run(self, file_store: AbstractFileStore):
                         connected = False
                         for inp in step.tool["inputs"]:
                             for s in aslist(inp.get("source", [])):
-                                if isinstance(
-                                    promises[s], (CWLJobWrapper, CWLGather)
-                                ) and not promises[s].hasFollowOn(wfjob):
+                                if (
+                                    isinstance(promises[s], (CWLJobWrapper, CWLGather))
+                                    and not promises[s].hasFollowOn(wfjob)
+                                    # promises[s] job has already added wfjob as a followOn prior
+                                    and not wfjob.hasPredecessor(promises[s])
+                                ):
                                     promises[s].addFollowOn(wfjob)
                                     connected = True
                                 if not isinstance(
@@ -1665,9 +1667,8 @@ def run(self, file_store: AbstractFileStore):
                                     promises[s].addChild(wfjob)
                                     connected = True
                         if not connected:
-                            # the workflow step has default inputs only & isn't
-                            # connected to other jobs, so add it as child of
-                            # this workflow.
+                            # Workflow step is default inputs only & isn't connected
+                            # to other jobs, so add it as child of this workflow.
                             self.addChild(wfjob)
 
                         for out in step.tool["outputs"]:
@@ -1804,7 +1805,8 @@ def determine_load_listing(tool: ToilCommandLineTool):
     return load_listing
 
 
-usage_message = '\n\n' + textwrap.dedent(f"""
+usage_message = "\n\n" + textwrap.dedent(
+    f"""
             * All positional arguments [cwl, yml_or_json] must always be specified last for toil-cwl-runner.
               Note: If you're trying to specify a jobstore, please use --jobStore.
     
@@ -1815,7 +1817,10 @@ def determine_load_listing(tool: ToilCommandLineTool):
                            --logInfo \\
                            example.cwl \\
                            example-job.yaml
-            """[1:])
+            """[
+        1:
+    ]
+)
 
 
 def main(args: Union[List[str]] = None, stdout: TextIO = sys.stdout) -> int:
@@ -2120,8 +2125,11 @@ def main(args: Union[List[str]] = None, stdout: TextIO = sys.stdout) -> int:
                     loading_context.fetcher_constructor,
                 )
             except schema_salad.exceptions.ValidationException:
-                print('\nYou may be getting this error because your arguments are incorrect or out of order.' +
-                      usage_message, file=sys.stderr)
+                print(
+                    "\nYou may be getting this error because your arguments are incorrect or out of order."
+                    + usage_message,
+                    file=sys.stderr,
+                )
                 raise
 
             options.tool_help = None
@@ -2174,9 +2182,11 @@ def main(args: Union[List[str]] = None, stdout: TextIO = sys.stdout) -> int:
                 )
             except SystemExit as e:
                 if e.code == 2:  # raised by argparse's parse_args() function
-                    print('\nIf both a CWL file and an input object (YAML/JSON) file were '
-                          'provided, this may be the argument order.' +
-                          usage_message, file=sys.stderr)
+                    print(
+                        "\nIf both a CWL file and an input object (YAML/JSON) file were "
+                        "provided, this may be the argument order." + usage_message,
+                        file=sys.stderr,
+                    )
                 raise
 
             fs_access = cwltool.stdfsaccess.StdFsAccess(options.basedir)
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -66,6 +66,11 @@ def __init__(self, promisingJob, recipientJob=None):
             super().__init__(f"Job {promisingJob.description} cannot promise its return value to non-successor {recipientJob.description}")
 
 
+class ConflictingPredecessorError(Exception):
+    def __init__(self, predecessor: 'Job', successor: 'Job'):
+        super().__init__(f'The given job: "{predecessor.description}" is already a predecessor of job: "{successor.description}".')
+
+
 class TemporaryID:
     """
     Placeholder for a job ID used by a JobDescription that has not yet been
@@ -1200,7 +1205,7 @@ def addFollowOn(self, followOnJob):
         :return: followOnJob
         :rtype: toil.job.Job
         """
-        
+
         assert isinstance(followOnJob, Job)
         
         # Join the job graphs
@@ -1209,9 +1214,13 @@ def addFollowOn(self, followOnJob):
         self._description.addFollowOn(followOnJob.jobStoreID)
         # Record the temporary back-reference
         followOnJob._addPredecessor(self)
-        
+
         return followOnJob
 
+    def hasPredecessor(self, job: 'Job') -> bool:
+        """Check if a given job is already a predecessor of this job."""
+        return job in self._directPredecessors
+
     def hasFollowOn(self, followOnJob):
         """
         Check if given job is already a follow-on of this job.
@@ -1221,7 +1230,7 @@ def hasFollowOn(self, followOnJob):
         :rtype: bool
         """
         return self._description.hasChild(followOnJob.jobStoreID) 
-        
+
     def addService(self, service, parentService=None):
         """
         Add a service.
@@ -1793,17 +1802,10 @@ def check(self):
                 RuntimeError, not return False!
             """
 
-    ####################################################
-    #Private functions
-    ####################################################
-
     def _addPredecessor(self, predecessorJob):
-        """
-        Adds a predecessor job to the set of predecessor jobs. Raises a \
-        RuntimeError if the job is already a predecessor.
-        """
+        """Adds a predecessor job to the set of predecessor jobs."""
         if predecessorJob in self._directPredecessors:
-            raise RuntimeError("The given job is already a predecessor of this job")
+            raise ConflictingPredecessorError(predecessorJob, self)
         self._directPredecessors.add(predecessorJob)
         
         # Record the need for the predecessor to finish


Test patch:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -427,6 +427,9 @@ def test_run_conformance(self, batchSystem=None, caching=False):
                               caching=caching,
                               batchSystem=batchSystem)
 
+
+@needs_cwl
+class CWLSmallTests(ToilTest):
     def test_usage_message(self):
         """
         This is purely to ensure a (more) helpful error message is printed if a user does
@@ -448,3 +451,18 @@ def test_usage_message(self):
             self.assertIn(b'Usage: toil-cwl-runner [options] example.cwl example-job.yaml', stderr)
             self.assertIn(b'All positional arguments [cwl, yml_or_json] '
                           b'must always be specified last for toil-cwl-runner.', stderr)
+
+    def test_workflow_echo_string(self):
+        toil = 'toil-cwl-runner'
+        jobstore = f'--jobStore=file:explicit-local-jobstore-{uuid.uuid4()}'
+        option_1 = '--strict-memory-limit'
+        option_2 = '--force-docker-pull'
+        option_3 = '--clean=always'
+        cwl = os.path.join(self._projectRootPath(), 'src/toil/test/cwl/echo_string.cwl')
+        cmd = [toil, jobstore, option_1, option_2, option_3, cwl]
+        log.debug(f'Now running: {" ".join(cmd)}')
+        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate()
+        assert stdout == b'{}'
+        assert b'Finished toil run successfully' in stderr
+        assert p.returncode == 0
diff --git a/src/toil/test/cwl/echo_string.cwl b/src/toil/test/cwl/echo_string.cwl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/cwl/echo_string.cwl
@@ -0,0 +1,51 @@
+cwlVersion: v1.2
+class: Workflow
+
+inputs:
+  - id: arrayS
+    type: string[]
+    default: ['hello','world']
+
+steps:
+  hello:
+    run:
+      class: CommandLineTool
+      inputs:
+        s: string
+      baseCommand: [echo]
+      arguments: [ $(inputs.s)]
+      outputs:
+        out: stdout
+        err: stderr
+    in:
+      - id: s
+        source: arrayS
+    scatter:
+     - s
+    out:
+      - id: out
+      - id: err
+
+  list:
+    run:
+      class: CommandLineTool
+      inputs:
+       file: File
+      baseCommand: [ ls , -lh]
+      arguments: [ $(inputs.file) ]
+      outputs: []
+    in:
+      - id: file
+        linkMerge: merge_flattened
+        source:
+          - hello/out
+          - hello/err
+    scatter:
+      - file
+    out: []
+
+outputs: []
+
+requirements:
+  - class: ScatterFeatureRequirement
+  - class: MultipleInputFeatureRequirement