Repository: DataBiosphere/toil

Issue:
Toil does user work in local jobs in CWL workflows. Is chaining to blame?
Running Toil version 4.2.0a1-3f4e790c1fa32a2c2257bd5cd203a3b51d5d5661
Batch scheduler: LSF

> Toil is thinking that there are running jobs, but we don’t have them in bqueue

looks like
```
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
```
https://github.com/EBI-Metagenomics/pipeline-v5/blob/master/workflows/subworkflows/raw_reads/functional_annotation_raw.cwl

https://github.com/EBI-Metagenomics/pipeline-v5/blob/master/workflows/conditionals/raw-reads/raw-reads-2.cwl

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-566)
┆Issue Number: TOIL-566



Discussion:
Hi, 

Maybe this also could help:
I killed main job and restarted again

```bash
Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-ERR1697183 kind-CWLWorkflow/instance-2kiod88f.
Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-ERR1697183 kind-CWLWorkflow/instance-3na_vy1m.
Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-ERR1697183 kind-CWLWorkflow/instance-52y0sz_s.
Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-ERR1697183 kind-CWLWorkflow/instance-rold6w0h.
Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-ERR1697183 kind-CWLWorkflow/instance-cl7jd114.
Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-ERR1697183 kind-CWLWorkflow/instance-f136loxc.
Repairing job: kind-CWLWorkflow/instance-2kiod88f
Repairing job: kind-CWLWorkflow/instance-7vr25_79
Repairing job: kind-CWLWorkflow/instance-f136loxc
Repairing job: kind-CWLWorkflow/instance-rold6w0h
Repairing job: kind-CWLWorkflow/instance-52y0sz_s
Repairing job: kind-CWLWorkflow/instance-3na_vy1m
Repairing job: kind-CWLScatter/instance-3bk1d3ps
Repairing job: kind-ResolveIndirect/instance-9xk77evr
Repairing job: kind-CWLWorkflow/instance-c6ut6jip
Repairing job: kind-ResolveIndirect/instance-c6u2scdv
Repairing job: kind-CWLWorkflow/instance-cl7jd114
Repairing job: kind-CWLWorkflow/instance-ie7vccat
```
I see more repaired jobs and lovely ResolveIndirect again... 
```bash
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/Combined_gene_caller/FGS.cwl' run_FGS.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_Combined_gene_caller_FGS.cwl/instance-laepimsn
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/Combined_gene_caller/FGS.cwl' run_FGS.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_Combined_gene_caller_FGS.cwl/instance-ipwnjyyv
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/concatenate.cwl' cat kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_concatenate.cwl/instance-l8rby53j with job batch system ID: 160 and cores: 1, disk: 2.0 G, and memory: 200.0 M
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/concatenate.cwl' cat kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_concatenate.cwl/instance-0cpdzs55 with job batch system ID: 161 and cores: 1, disk: 2.0 G, and memory: 200.0 M
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/concatenate.cwl' cat kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_concatenate.cwl/instance-l8rby53j
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/concatenate.cwl' cat kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_concatenate.cwl/instance-0cpdzs55
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/chunks/fasta_chunker.cwl' split_to_chunks.py kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_chunks_fasta_chunker.cwl/instance-l8j71nqs with job batch system ID: 166 and cores: 1, disk: 2.0 G, and memory: 4.9 G
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/chunks/fasta_chunker.cwl' split_to_chunks.py kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_chunks_fasta_chunker.cwl/instance-l8j71nqs
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/result-file-chunker/result_chunker.cwl' run_result_file_chunker.py kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_result-file-chunker_result_chunker.cwl/instance-uvhh_zl0 with job batch system ID: 170 and cores: 1, disk: 2.0 G, and memory: 1000.0 M
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/result-file-chunker/result_chunker.cwl' run_result_file_chunker.py kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_result-file-chunker_result_chunker.cwl/instance-_71t4a5d with job batch system ID: 171 and cores: 1, disk: 2.0 G, and memory: 1000.0 M
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/result-file-chunker/result_chunker.cwl' run_result_file_chunker.py kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_result-file-chunker_result_chunker.cwl/instance-t7fn0b_l with job batch system ID: 172 and cores: 1, disk: 2.0 G, and memory: 1000.0 M
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/chunks/fasta_chunker.cwl' split_to_chunks.py kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_chunks_fasta_chunker.cwl/instance-kw5wr3r2 with job batch system ID: 173 and cores: 1, disk: 2.0 G, and memory: 4.9 G
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/result-file-chunker/result_chunker.cwl' run_result_file_chunker.py kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_result-file-chunker_result_chunker.cwl/instance-uvhh_zl0
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/result-file-chunker/result_chunker.cwl' run_result_file_chunker.py kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_result-file-chunker_result_chunker.cwl/instance-t7fn0b_l
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/result-file-chunker/result_chunker.cwl' run_result_file_chunker.py kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_result-file-chunker_result_chunker.cwl/instance-_71t4a5d
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/return_directory.cwl' kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_return_directory.cwl/instance-broec61j with job batch system ID: 175 and cores: 1, disk: 2.0 G, and memory: 200.0 M
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/chunks/fasta_chunker.cwl' split_to_chunks.py kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_chunks_fasta_chunker.cwl/instance-kw5wr3r2
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/return_directory.cwl' kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_return_directory.cwl/instance-broec61j
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-j6h66xrk with job batch system ID: 188 and cores: 4, disk: 2.0 G, and memory: 14.6 G
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-pfudkt66 with job batch system ID: 189 and cores: 4, disk: 2.0 G, and memory: 14.6 G
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-u3upm4ut with job batch system ID: 190 and cores: 4, disk: 2.0 G, and memory: 14.6 G
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-_m7ye_9p with job batch system ID: 191 and cores: 4, disk: 2.0 G, and memory: 14.6 G
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-2npt0vua with job batch system ID: 192 and cores: 4, disk: 2.0 G, and memory: 14.6 G
Issued job 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-3abgm9qb with job batch system ID: 193 and cores: 4, disk: 2.0 G, and memory: 14.6 G
12 jobs are running, 0 jobs are issued and waiting to run
12 jobs are running, 0 jobs are issued and waiting to run
12 jobs are running, 0 jobs are issued and waiting to run
Got message from job at time 06-08-2020 16:07:57: Job used more disk than requested. Consider
 modifying the user script to avoid the chance of failure due to incorrectly requested resour
ces. Job files/for-job/kind-CWLScatter/instance-3bk1d3ps/cleanup/file-covrabd_/stream used 11
13.66% (22.3 GB [23915702784B] used, 2.0 GB [2147483648B] requested) at the end of its run.
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-3abgm9qb
11 jobs are running, 0 jobs are issued and waiting to run
11 jobs are running, 0 jobs are issued and waiting to run
Got message from job at time 06-08-2020 18:15:12: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLScatter/instance-3bk1d3ps/cleanup/file-7pjx8yg4/stream used 1114.40% (22.3 GB [23931496960B] used, 2.0 GB [2147483648B] requested) at the end of its run.
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-u3upm4ut
Got message from job at time 06-08-2020 18:18:20: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLScatter/instance-3bk1d3ps/cleanup/file-u9cz7nl5/stream used 1114.41% (22.3 GB [23931824640B] used, 2.0 GB [2147483648B] requested) at the end of its run.
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-pfudkt66
Got message from job at time 06-08-2020 18:22:01: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLScatter/instance-3bk1d3ps/cleanup/file-h6nf4hrh/stream used 1114.38% (22.3 GB [23931038208B] used, 2.0 GB [2147483648B] requested) at the end of its run.
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-2npt0vua
8 jobs are running, 0 jobs are issued and waiting to run
Got message from job at time 06-08-2020 19:44:33: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLScatter/instance-3bk1d3ps/cleanup/file-dh5hp8tb/stream used 1114.38% (22.3 GB [23931038208B] used, 2.0 GB [2147483648B] requested) at the end of its run.
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-_m7ye_9p
Got message from job at time 06-08-2020 19:48:19: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLScatter/instance-3bk1d3ps/cleanup/file-8nyqka6u/stream used 1114.40% (22.3 GB [23931496960B] used, 2.0 GB [2147483648B] requested) at the end of its run.
Job ended: 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/InterProScan/InterProScan-v5-none_docker.cwl#i5' interproscan.sh kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_tools_InterProScan_InterProScan-v5-none_docker.cwl_i5/instance-j6h66xrk
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
6 jobs are running, 1 jobs are issued and waiting to run
Got exit code 1 (indicating failure) from job _toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-ERR1697183 kind-CWLWorkflow/instance-2kiod88f.
```
Steps (split_to_chunks is a common step before 2 scatters) :
CGC-subwf: cnunk fasta-> FGS -> FGS post-processing -> cat ->
-> split_to_chunks -> scatter hmmscan -> unite hmmscan tables
-> split_to_chunks -> scatter IPS -> unite IPS tables

I remember that the Toil CWL runner runs some housekeeping jobs on the leader, instead of sending them to (in this case) LSF. Can you check with `tree` to see if your Toil has any child worker processes when it says it has 6 running jobs? It could be they really are running but are stuck somehow.
@adamnovak I can't check it because I killed that run because I didn't see any child jobs in bjobs queue. That means toil didn't run these jobs at all. When I restarted - it ran these 6 jobs.
@adamnovak Here are some full debug logs from a recent run: https://gist.github.com/mr-c/903d767bef7cd62a529994b6a0aa9a80
If the jobs ended up in the `bjobs` queue on restart, that does suggest that they are meant to be real cluster jobs and not local jobs. If all you saw was the jobs starting and then finishing (i.e. you didn't check the bjobs queue) it could still be that these are local jobs getting stuck.

If they are showing up in the `6 jobs...` message and not when you query the cluster scheduler, and they aren't local jobs, that means that Toil is not querying the cluster scheduler for quite a long time. Maybe its last query time is being set in the future, or set to None, or something, which is breaking the caching logic for the cluster state. That 6 isn't supposed to include jobs that Toil *thinks* it has run, only jobs that it *actually* observed running.
I looked at the logs. It definitely could be the case that these are local jobs that it thinks are still running, as far as I can tell.

I've opened https://github.com/DataBiosphere/toil/pull/3085 which adds some more debug logging that ought to be able to let us tell for sure if these are LSF jobs that are supposedly running (and we aren't actually refreshing our view of the cluster), or if these are local jobs that are supposedly running. I've also identified a possible way that we might have been able to lose the notification that a local job had finished, and set it to fall back on polling if it hasn't heard from any finished local jobs.

@KeteSakharova Can you try the branch at https://github.com/DataBiosphere/toil/tree/issues/3082-wayward-jobs and see if it fixes your problem, or produces a more enlightening debug log? `pip3 install --upgrade git+https://github.com/DataBiosphere/toil.git@a538fa2c24b4f751a04cb6fcbff53d3c15f27218#egg=toil[all]` ought to get it installed.
@adamnovak I hope your extra logging could help to investigate an issue. It is difficult for me to do it alone :( 
I attached one more log of part of our wf with 2 scatters. 
I see the same behaviour. I had to kill main job today because I didn't see any child jobs.
[test_wf.log.log.gz](https://github.com/DataBiosphere/toil/files/4787423/test_wf.log.log.gz)



OK, @KeteSakharova, that log definitely shows that the problem is stuck local jobs, and nothing to do with `bjobs` or LSF.

Look at this excerpt:

```
Issuing local command: _toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-test_fa_wf kind-CWLWorkflow/instance-hco0q1nq with memory: 42949672960, cores: 8, disk: 2147483648
Issued job 'CWLWorkflow' kind-CWLWorkflow/instance-hco0q1nq with job batch system ID: 13 and cores: 8, disk: 2.0 G, and memory: 40.0 G
...
Launched job 13 as child 26921
...
Found 0 jobs running on the batch scheduler and 3 jobs running locally
...
```

It wants to run a command locally (`_toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-test_fa_wf kind-CWLWorkflow/instance-hco0q1nq`). It starts it up as PID 26921, and never reports that that process ends. (Compare to what it says about child PID 65331, which does end.) At the end of the workflow, when you presumably kill the whole workflow, the child process running this job seems to still be alive (as far as Toil knows), and the workflow is still waiting for it. Depending on how long it ought to take and how long you waited for it, it might even be doing useful work; the log doesn't have good timestamps.

I think the way to debug this is to just run that command (`_toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-test_fa_wf kind-CWLWorkflow/instance-hco0q1nq`) yourself, without the leader running. That will run just the offending, stuck job by itself on the local machine. It should print a log, and that log will probably mention another file that it logs to that you would want to look in.

If the command finishes in a timely fashion (or if it complains that the job no longer exists; they go away when they complete), then the problem is in the Toil leader's ability to detect that its child processes are finished.

If the command runs for an unacceptable amount of time, then its own logs may illuminate why that is. You also might be able to run it under a Python debugger to see where it is getting stuck, if you are familiar with one.

You could also try posting the file `kind-CWLWorkflow/instance-hco0q1nq` which can be found in the job store (`find  /nfs/public/release/metagenomics_scratch/pipeline-5/job-store-test_fa_wf | grep instance-hco0q1nq` will probably uncover its full path). That file has the serialized job, and if it is obnoxiously large or looks like it wants to download 10 million input files or something, that could be a plausible explanation for why it is getting stuck. It also can probably be inspected to uncover exactly what part of the CWL workflow it represents. These job files are Python pickle files, and they can be inspected in an interactive Python shell by just reading the file contents and fluffing them up into a Python object with the `pickle` module. So if you send it along we can look at it and see if there's anything obviously wrong with it.
Hi @adamnovak, thank you for good explanation!

I have one question: what are **local jobs** and why some jobs are running with bsub and some locally? How to identify that toil is running local jobs and don't freeze? In such case I will not see child jobs in bjobs queue.

I ran 
```bash
_toil_worker CWLWorkflow file:/nfs/public/release/metagenomics_scratch/pipeline-5/job-store-test_fa_wf kind-CWLWorkflow/instance-hco0q1nq
```
and got this log (it is excpected to be long work, and it is not finished yet). It looks like it ran one of the lost jobs: hmmscan. It is a big surprise for me that it has whole listing of files for agrument **data**. Can that be a problem?
[worker_log.txt](https://github.com/DataBiosphere/toil/files/4792612/worker_log.txt)

With `grep instance-hco0q1nq` I found folder
` ./files/for-job/kind-CWLWorkflow/instance-hco0q1nq/cleanup`
and file
`./jobs/kind-CWLWorkflow/instance-hco0q1nq/job` <- I guess, I need this.

```bash
INFO:rdflib:RDFLib Version: 4.2.2
JobGraph( **{'unitName': 'hmmscan', 'displayName': 'JobGraph', 'jobName': 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/hmmscan/hmmscan.cwl', '_cores': 8.0, '_memory': 42949672960, '_disk': 2147483648, '_preemptable': False, '_config': None, 'jobStoreID': 'kind-CWLWorkflow/instance-hco0q1nq', 'predecessorNumber': 1, 'command': '_toil files/for-job/kind-CWLWorkflow/instance-hco0q1nq/cleanup/file-rkkk6lrp/stream /hps/nobackup/production/metagenomics/software/test-debug-toil/lib/python3.6/site-packages toil.cwl.cwltoil True', 'remainingRetryCount': 2, 'filesToDelete': [], 'predecessorsFinished': set(), 'stack': [[JobNode( **{'unitName': None, 'displayName': 'JobGraph', 'jobName': 'ResolveIndirect', '_cores': 8.0, '_memory': 42949672960, '_disk': 2147483648, '_preemptable': False, '_config': None, 'jobStoreID': 'kind-ResolveIndirect/instance-_i5c5o79', 'predecessorNumber': 1, 'command': None} )], [], [JobNode( **{'unitName': 'hmmscan_tab.py', 'displayName': 'JobGraph', 'jobName': 'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/utils/hmmscan_tab_modification/hmmscan_tab_modification.cwl', '_cores': 1, '_memory': 104857600, '_disk': 2147483648, '_preemptable': False, '_config': None, 'jobStoreID': 'kind-file_hps_nobackup2_production_metagenomics_pipeline_tools-v5_pipeline-v5_utils_hmmscan_tab_modification_hmmscan_tab_modification.cwl/instance-_qmv12_0', 'predecessorNumber': 1, 'command': None} )]], 'logJobStoreFileID': None, 'services': [], 'terminateJobStoreID': None, 'startJobStoreID': None, 'errorJobStoreID': None, 'checkpoint': None, 'checkpointFilesToDelete': None, 'chainedJobs': ["'file:///hps/nobackup2/production/metagenomics/pipeline/tools-v5/pipeline-v5/tools/hmmscan/hmmscan.cwl' hmmscan kind-CWLWorkflow/instance-hco0q1nq"]} )
```
**Local jobs** are jobs that are run on the leader machine, and aren't sent through LSF or whatever other cluster you're using. [There's a list of their names here](https://github.com/DataBiosphere/toil/blob/7db970dfd80f694e58f26c9ad34829750902b78d/src/toil/cwl/cwltoil.py#L87-L92), and they're identified/determined by name; I think the point might be to make sure that Toil jobs that are just part of the CWL interpreter don't appear to users alongside the jobs that are actual work specified by the CWL workflow.

I think something has gone wrong with this system, because it isn't supposed to result in actual user work (like `hmmscan` itself) running on the leader. I think the chaining system (which allows one job to immediately go on to run another later job, if it fits in the same resource allotment, rather than submitting it for scheduling) might be causing trouble here. The internal, local CWL job is creating a job meant to do actual work, which is then getting chained to and run on the leader instead of sent off for scheduling through LSF like it should be. Running the workflow with `--disableChaining` might be a workaround, and we should be able to fix the chaining system to not operate on these internal CWL jobs.

In terms of detecting the legitimate jobs running on the leader, you can use `ps` or `tree` to look for child processes of the toil leader. All the local jobs run in their own processes as the `_toil_worker` executable. So if you see any of those running, you know Toil is trying to do work on the leader and isn't just stuck.



Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -1453,6 +1453,7 @@ def main(args: Union[List[str]] = None, stdout: TextIO = sys.stdout) -> int:
     # Remove cwltool logger's stream handler so it uses Toil's
     cwllogger.removeHandler(defaultStreamHandler)
     config = Config()
+    config.disableChaining = True
     config.cwl = True
     parser = argparse.ArgumentParser()
     addOptions(parser, config)


Test patch: