Repository: DataBiosphere/toil

Issue:
can't kill job with "toil kill"  -- can it work with Slurm?
I am testing Toil will Slurm. I can run the workflow successfully, however, I can't use "toil kill" to stop and kill the job.

Here is my configuration. 
1. example.cwl
```
cwlVersion: v1.0
class: CommandLineTool
baseCommand: sleep
stdout: output.txt
inputs:
  message:
    type: int
    inputBinding:
      position: 1
outputs:
  output:
    type: stdout
```

2. job.yaml
```
message: 1000
```

I run the job this way (/work is the shared storage in my cluster) :
```
toil-cwl-runner --batchSystem Slurm --jobStore /work/test example.cwl job.yaml
```

I can see the job is running now (I changed the value in job.yaml to 10 jus to see if it could run through):
```
$ toil-cwl-runner --batchSystem Slurm --jobStore /work/test example.cwl job.yaml
DEBUG:rdflib:RDFLib Version: 4.2.2
Resolved 'example.cwl' to 'file:///work/example.cwl'
INFO:cwltool:Resolved 'example.cwl' to 'file:///work/example.cwl'
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (16820604928).
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (8945131520).
INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0.
INFO:toil.realtimeLogger:Real-time logging disabled
INFO:toil.leader:Issued job 'file:///work/example.cwl' sleep P/p/jobWczAJP with job batch system ID: 0 and cores: 1, disk: 3.0 G, and memory: 2.0 G
2018-08-30 23:43:38,165 - toil.leader - INFO - Issued job 'file:///work/example.cwl' sleep P/p/jobWczAJP with job batch system ID: 0 and cores: 1, disk: 3.0 G, and memory: 2.0 G
INFO:toil.leader:Job ended successfully: 'file:///work/example.cwl' sleep P/p/jobWczAJP
2018-08-30 23:43:51,335 - toil.leader - INFO - Job ended successfully: 'file:///work/example.cwl' sleep P/p/jobWczAJP
INFO:toil.leader:Finished toil run successfully.
2018-08-30 23:43:52,214 - toil.leader - INFO - Finished toil run successfully.
{
    "output": {
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "basename": "output.txt",
        "nameext": ".txt",
        "nameroot": "output",
        "http://commonwl.org/cwltool#generation": 0,
        "location": "file:///work/output.txt",
        "class": "File",
        "size": 0
    }
INFO:toil.common:Successfully deleted the job store: <toil.jobStores.fileJobStore.FileJobStore object at 0x7f7cbd848590>
}2018-08-30 23:43:53,359 - toil.common - INFO - Successfully deleted the job store: <toil.jobStores.fileJobStore.FileJobStore object at 0x7f7cbd848590>
```

OK, now I changed the value in job.yaml back to 1000 to make sure it will run a while. I run it, and make sure it is in the queue
```
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                44   compute toil_job ec2-user  R       0:08      1 ip-172-31-67-182
```

Then I run 'toil kill'. Here is what i got:
```
$ toil kill --logDebug  file:/work/test
DEBUG:toil.lib.bioio:Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'.
DEBUG:rdflib:RDFLib Version: 4.2.2
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/work/test'.
INFO:toil.utils.toilKill:Starting routine to kill running jobs in the toil workflow: file:/work/test
DEBUG:toil.common:Using the slurm batch system
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (16820604928).
DEBUG:toil.common:Obtained node ID 4de74c5fa16999dce8eb92ed5b85c15a from file /var/lib/dbus/machine-id
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (8945180672).
DEBUG:toil.batchSystems.singleMachine:Setting up the thread pool with 40 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 4.
INFO:toil.utils.toilKill:All jobs SHOULD have been killed
DEBUG:toil.batchSystems.abstractGridEngineBatchSystem:No activity, sleeping for 1s
DEBUG:toil.batchSystems.abstractGridEngineBatchSystem:No activity, sleeping for 1s
DEBUG:toil.batchSystems.abstractGridEngineBatchSystem:No activity, sleeping for 1s
DEBUG:toil.batchSystems.abstractGridEngineBatchSystem:No activity, sleeping for 1s
DEBUG:toil.batchSystems.abstractGridEngineBatchSystem:No activity, sleeping for 1s
```

**From the log, seems toil kill only works with SGE. Is there a way that I can use it with Slurm?**

Then I tested it with local system without Slurm.
First I start the job
```
$ toil-cwl-runner --jobStore /work/local example.cwl job.yaml
DEBUG:rdflib:RDFLib Version: 4.2.2
Resolved 'example.cwl' to 'file:///work/example.cwl'
INFO:cwltool:Resolved 'example.cwl' to 'file:///work/example.cwl'
WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (4).
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (16820604928).
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (8945233920).
INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0.
INFO:toil.realtimeLogger:Real-time logging disabled
INFO:toil.leader:Issued job 'file:///work/example.cwl' sleep 8/f/jobQknI9x with job batch system ID: 0 and cores: 1, disk: 3.0 G, and memory: 2.0 G
2018-08-30 23:49:03,092 - toil.leader - INFO - Issued job 'file:///work/example.cwl' sleep 8/f/jobQknI9x with job batch system ID: 0 and cores: 1, disk: 3.0 G, and memory: 2.0 G
```

then I try to kill it
```
$ toil kill --logDebug /work/local
DEBUG:toil.lib.bioio:Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'.
DEBUG:rdflib:RDFLib Version: 4.2.2
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/work/local'.
INFO:toil.utils.toilKill:Starting routine to kill running jobs in the toil workflow: file:/work/local
DEBUG:toil.common:Using the single machine batch system
WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (4).
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (16820604928).
DEBUG:toil.common:Obtained node ID 4de74c5fa16999dce8eb92ed5b85c15a from file /var/lib/dbus/machine-id
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (8945201152).
DEBUG:toil.batchSystems.singleMachine:Setting up the thread pool with 40 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 4.
INFO:toil.utils.toilKill:All jobs SHOULD have been killed
```

However, the job is still there, nothing happened. 

**Summary: toil kill can not do anyhting. And it stuck there forever.** 

Any help?

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-343)
┆Issue Number: TOIL-343



Discussion:
Will look into this, thanks for posting the issue!

Resolving patch:
diff --git a/src/toil/utils/toilKill.py b/src/toil/utils/toilKill.py
--- a/src/toil/utils/toilKill.py
+++ b/src/toil/utils/toilKill.py
@@ -1,4 +1,4 @@
-# Copyright (C) 2015-2016 Regents of the University of California
+# Copyright (C) 2015-2018 Regents of the University of California
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -12,10 +12,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Kills any running jobs trees in a rogue toil.
-"""
+"""Kills rogue toil processes."""
 from __future__ import absolute_import
 import logging
+import os
+import signal
 
 from toil.lib.bioio import getBasicOptionParser
 from toil.lib.bioio import parseBasicOptions
@@ -34,11 +35,25 @@ def main():
     options = parseBasicOptions(parser)
     config = Config()
     config.setOptions(options)
-    jobStore = Toil.resumeJobStore(config.jobStore)
+    config.jobStore = config.jobStore[5:] if config.jobStore.startswith('file:') else config.jobStore
 
-    logger.info("Starting routine to kill running jobs in the toil workflow: %s", config.jobStore)
-    ####This behaviour is now broken
-    batchSystem = Toil.createBatchSystem(jobStore.config) #This should automatically kill the existing jobs.. so we're good.
-    for jobID in batchSystem.getIssuedBatchJobIDs(): #Just in case we do it again.
-        batchSystem.killBatchJobs(jobID)
-    logger.info("All jobs SHOULD have been killed")
+    # ':' means an aws/google/azure jobstore; use the old (broken?) method
+    if ':' in config.jobStore:
+        jobStore = Toil.resumeJobStore(config.jobStore)
+        logger.info("Starting routine to kill running jobs in the toil workflow: %s", config.jobStore)
+        # TODO: This behaviour is now broken src: https://github.com/DataBiosphere/toil/commit/a3d65fc8925712221e4cda116d1825d4a1e963a1
+        batchSystem = Toil.createBatchSystem(jobStore.config)  # Should automatically kill existing jobs, so we're good.
+        for jobID in batchSystem.getIssuedBatchJobIDs():  # Just in case we do it again.
+            batchSystem.killBatchJobs(jobID)
+        logger.info("All jobs SHOULD have been killed")
+    # otherwise, kill the pid recorded in the jobstore
+    else:
+        pid_log = os.path.join(os.path.abspath(config.jobStore), 'pid.log')
+        with open(pid_log, 'r') as f:
+            pid2kill = f.read().strip()
+        try:
+            os.kill(int(pid2kill), signal.SIGKILL)
+            logger.info("Toil process %s successfully terminated." % str(pid2kill))
+        except OSError:
+            logger.error("Toil process %s could not be terminated." % str(pid2kill))
+            raise


Test patch:
diff --git a/src/toil/test/utils/ABCWorkflowDebug/sleep.cwl b/src/toil/test/utils/ABCWorkflowDebug/sleep.cwl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/utils/ABCWorkflowDebug/sleep.cwl
@@ -0,0 +1,12 @@
+cwlVersion: v1.0
+class: CommandLineTool
+baseCommand: sleep
+stdout: output.txt
+inputs:
+  message:
+    type: int
+    inputBinding:
+      position: 1
+outputs:
+  output:
+    type: stdout
diff --git a/src/toil/test/utils/ABCWorkflowDebug/sleep.yaml b/src/toil/test/utils/ABCWorkflowDebug/sleep.yaml
new file mode 100644
--- /dev/null
+++ b/src/toil/test/utils/ABCWorkflowDebug/sleep.yaml
@@ -0,0 +1 @@
+message: 100
diff --git a/src/toil/test/utils/utilsTest.py b/src/toil/test/utils/utilsTest.py
--- a/src/toil/test/utils/utilsTest.py
+++ b/src/toil/test/utils/utilsTest.py
@@ -21,12 +21,14 @@
 import uuid
 import shutil
 import tempfile
-from subprocess32 import Popen
 import psutil
 import pytest
+import logging
+
+pkg_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))  # noqa
+sys.path.insert(0, pkg_root)  # noqa
 
 import toil
-import logging
 import toil.test.sort.sort
 from toil import subprocess
 from toil import resolveEntryPoint
@@ -208,7 +210,7 @@ def testGetPIDStatus(self):
         jobstoreLoc = os.path.join(os.getcwd(), jobstoreName)
 
         cmd = ['python', '-m', 'toil.test.sort.sort', 'file:' + jobstoreName, '--clean', 'never']
-        wf = Popen(cmd)
+        wf = subprocess.Popen(cmd)
         time.sleep(2)  # Need to let jobstore be created before checking its contents.
         self.assertEqual(ToilStatus.getPIDStatus(jobstoreLoc),'RUNNING')
         wf.wait()
@@ -228,7 +230,7 @@ def testGetStatusFailedToilWF(self):
         jobstoreLoc = os.path.join(os.getcwd(), jobstoreName)
         # --badWorker is set to force failure.
         cmd = ['python', '-m', 'toil.test.sort.sort', 'file:' + jobstoreName, '--clean', 'never', '--badWorker', '1']
-        wf = Popen(cmd)
+        wf = subprocess.Popen(cmd)
         time.sleep(2)  # Needed to let the jobstore be created before checking its contents.
         self.assertEqual(ToilStatus.getStatus(jobstoreLoc), 'RUNNING')
         wf.wait()
@@ -244,7 +246,7 @@ def testGetStatusFailedCWLWF(self):
         jobstoreLoc = os.path.join(os.getcwd(), jobstoreName)
         cmd = ['toil-cwl-runner', '--jobStore', jobstoreLoc, '--clean', 'never', '--badWorker', '1', files[0],
                '--reverse', '--input', files[1]]
-        wf = Popen(cmd)
+        wf = subprocess.Popen(cmd)
         wfRun = psutil.Process(pid=wf.pid)
         time.sleep(2)  # Needed to let the jobstore be created before checking its contents.
         wfRun.suspend()  # This workflow runs so quickly that we need to pause so we can get a 'RUNNING' response.
@@ -264,7 +266,7 @@ def testGetStatusSuccessfulToilWF(self):
         jobstoreName = 'successful-toil-js'
         jobstoreLoc = os.path.join(os.getcwd(), jobstoreName)
         cmd = ['python', '-m', 'toil.test.sort.sort', 'file:' + jobstoreName, '--clean', 'never']
-        wf = Popen(cmd)
+        wf = subprocess.Popen(cmd)
         time.sleep(2)  # Need to let jobstore be created before checking its contents.
         self.assertEqual(ToilStatus.getStatus(jobstoreLoc), 'RUNNING')
         wf.wait()
@@ -281,7 +283,7 @@ def testGetStatusSuccessfulCWLWF(self):
         jobstoreLoc = os.path.join(os.getcwd(), jobstoreName)
         cmd = ['toil-cwl-runner', '--jobStore', jobstoreLoc, '--clean', 'never', files[0], '--reverse', '--input',
                files[1]]
-        wf = Popen(cmd)
+        wf = subprocess.Popen(cmd)
         wfRun = psutil.Process(pid=wf.pid)
         time.sleep(2)  # Needed to let the jobstore be created before checking its contents.
         wfRun.suspend()  # This workflow runs so quickly that we need to pause so we can get a 'RUNNING' response.