Repository: DataBiosphere/toil

Issue:
toil master branch broken without 'cwl' extra
This was a hard-to-catch bug in #1864, which imports cwltoil from the leader. This understandably didn't get caught by the CI tests because they install the cwl extra. The backtrace looks like this:

```
Traceback (most recent call last):
  File "/cluster/home/jcarmstr/venv/bin/cactus", line 11, in <module>
    load_entry_point('progressiveCactus==1.0', 'console_scripts', 'cactus')()
  File "/cluster/home/jcarmstr/venv/lib/python2.7/site-packages/cactus/progressive/cactus_progressive.py", line 440, in main
    halID = toil.start(RunCactusPreprocessorThenProgressiveDown(options, project, memory=configWrapper.getDefaultMemory()))
  File "/cluster/home/jcarmstr/venv/lib/python2.7/site-packages/toil/common.py", line 725, in start
    return self._runMainLoop(rootJobGraph)
  File "/cluster/home/jcarmstr/venv/lib/python2.7/site-packages/toil/common.py", line 989, in _runMainLoop
    from toil.leader import Leader
  File "/cluster/home/jcarmstr/venv/lib/python2.7/site-packages/toil/leader.py", line 41, in <module>
    from toil.cwl import cwltoil
  File "/cluster/home/jcarmstr/venv/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 32, in <module>
    import cwltool.errors
ImportError: No module named cwltool.errors
```

Anyway, no harm done, it should be a really easy fix; I can probably find time to do this over the weekend.
toil master branch broken without 'cwl' extra
This was a hard-to-catch bug in #1864, which imports cwltoil from the leader. This understandably didn't get caught by the CI tests because they install the cwl extra. The backtrace looks like this:

```
Traceback (most recent call last):
  File "/cluster/home/jcarmstr/venv/bin/cactus", line 11, in <module>
    load_entry_point('progressiveCactus==1.0', 'console_scripts', 'cactus')()
  File "/cluster/home/jcarmstr/venv/lib/python2.7/site-packages/cactus/progressive/cactus_progressive.py", line 440, in main
    halID = toil.start(RunCactusPreprocessorThenProgressiveDown(options, project, memory=configWrapper.getDefaultMemory()))
  File "/cluster/home/jcarmstr/venv/lib/python2.7/site-packages/toil/common.py", line 725, in start
    return self._runMainLoop(rootJobGraph)
  File "/cluster/home/jcarmstr/venv/lib/python2.7/site-packages/toil/common.py", line 989, in _runMainLoop
    from toil.leader import Leader
  File "/cluster/home/jcarmstr/venv/lib/python2.7/site-packages/toil/leader.py", line 41, in <module>
    from toil.cwl import cwltoil
  File "/cluster/home/jcarmstr/venv/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 32, in <module>
    import cwltool.errors
ImportError: No module named cwltool.errors
```

Anyway, no harm done, it should be a really easy fix; I can probably find time to do this over the weekend.


Discussion:

Resolving patch:
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -38,7 +38,11 @@
 from bd2k.util.humanize import bytes2human
 
 from toil import resolveEntryPoint
-from toil.cwl import cwltoil
+try:
+    from toil.cwl.cwltoil import CWL_INTERNAL_JOBS
+except ImportError:
+    # CWL extra not installed
+    CWL_INTERNAL_JOBS = ()
 from toil.jobStores.abstractJobStore import NoSuchJobException
 from toil.provisioners.clusterScaler import ClusterScaler
 from toil.serviceManager import ServiceManager
@@ -437,7 +441,7 @@ def innerLoop(self):
                                 "for job %s", jobID)
                 else:
                     if result == 0:
-                        cur_logger = (logger.debug if str(updatedJob.jobName).startswith(cwltoil.CWL_INTERNAL_JOBS)
+                        cur_logger = (logger.debug if str(updatedJob.jobName).startswith(CWL_INTERNAL_JOBS)
                                       else logger.info)
                         cur_logger('Job ended successfully: %s', updatedJob)
                     else:
@@ -536,7 +540,7 @@ def issueJob(self, jobNode):
             # len(jobBatchSystemIDToIssuedJob) should always be greater than or equal to preemptableJobsIssued,
             # so increment this value after the job is added to the issuedJob dict
             self.preemptableJobsIssued += 1
-        cur_logger = (logger.debug if jobNode.jobName.startswith(cwltoil.CWL_INTERNAL_JOBS)
+        cur_logger = (logger.debug if jobNode.jobName.startswith(CWL_INTERNAL_JOBS)
                       else logger.info)
         cur_logger("Issued job %s with job batch system ID: "
                    "%s and cores: %s, disk: %s, and memory: %s",


Test patch: