Repository: DataBiosphere/toil

Issue:
toil-cwl-runner sort of works without 'cwl' extra installed
I neglected to install the 'cwl' extra and was trying to run toil-cwl-runner. I had previously installed cwltool. I got an error from toil code trying to import from cwltool:
```
Traceback (most recent call last):
  File "/cluster/home/barberc/software/anaconda/envs/py3/bin/toil-cwl-runner", line 6, in <module>
    from toil.cwl.cwltoil import main
  File "/cluster/home/barberc/software/anaconda/envs/py3/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 60, in <module>
    from cwltool.process import (shortname, fill_in_defaults, compute_checksums,
ImportError: cannot import name 'collectFilesAndDirs'
```
This symbol at one time existed in cwltool. In fact, I went so far as identifying and installing cwltool==1.0.20180819175200 which was close enough to the required version for toil 3.18.0 (cwltool==1.0.20180820141117) that everything seemed to work.

Apologies if there is not a practical way to do this with setup.py extras, but I am opening this in hopes that there is a simple way I haven't thought of by which to address this. Ideally the cwl-related commands would not be present, or not be on the PATH, or a warning would be issued when running. Another line of argumentation might be: users may be trying toil after experimenting with cwltool, so there is a good chance they have it already installed (likely the wrong version), and the cwl extras are currently not so heavy of dependencies that they couldn't just be moved out of extras and into a base `pip install toil`. I'm not sure if those points are true, but it could be a way around this.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-45)
┆Issue Number: TOIL-45



Discussion:
Does `cwltool` expose a `__VERSION__` or `version` or similar field in its module? We could have the Toil code that uses it check that against the version Toil requires and complain if the user has managed to install the wrong version. But then the user would have a hard time overriding Toil's demanded cwltool version with another one that they know is compatible.
Searching cwltool repo for the current version (1.0.20190228155703) finds nothing in the code, so I don't think they do, though I'm sure if they were asked nicely they could.

It's interesting that setuptools has a syntax for defining relevant extras for console_scripts (see [setuptools docs for extras]()), which is used for the cwl-related scripts in toil (https://github.com/DataBiosphere/toil/blob/0ba88bbdf375df74883c85ce12737b48deea797d/setup.py#L162-L163), but that apparently this is only for limiting dynamic dependencies. I don't know how they implement it, but maybe it could be hooked into to somehow.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -18,6 +18,9 @@
 from setuptools import find_packages, setup
 
 
+cwltool_version = '3.1.20210616134059'
+
+
 def run_setup():
     """
     Calls setup(). This function exists so the setup() invocation preceded more internal
@@ -33,7 +36,7 @@ def run_setup():
     gcs = 'google-cloud-storage==1.6.0'
     gcs_oauth2_boto_plugin = 'gcs_oauth2_boto_plugin==1.14'
     apacheLibcloud = 'apache-libcloud==2.2.1'
-    cwltool = 'cwltool==3.1.20210616134059'
+    cwltool = f'cwltool=={cwltool_version}'
     galaxyToolUtil = 'galaxy-tool-util'
     htcondor = 'htcondor>=8.6.0'
     kubernetes = 'kubernetes>=12.0.1, <13'
@@ -168,7 +171,10 @@ def import_version():
         # Use the template to generate src/toil/version.py
         import version_template
         with NamedTemporaryFile(mode='w', dir='src/toil', prefix='version.py.', delete=False) as f:
-            f.write(version_template.expand_())
+            f.write(version_template.expand_(others={
+                # expose the dependency versions that we may need to access in Toil
+                'cwltool_version': cwltool_version,
+            }))
         os.rename(f.name, 'src/toil/version.py')
 
     # Unfortunately, we can't use a straight import here because that would also load the stuff
diff --git a/src/toil/cwl/__init__.py b/src/toil/cwl/__init__.py
--- a/src/toil/cwl/__init__.py
+++ b/src/toil/cwl/__init__.py
@@ -0,0 +1,37 @@
+# Copyright (C) 2015-2021 Regents of the University of California
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import logging
+from pkg_resources import get_distribution, DistributionNotFound
+from toil.version import cwltool_version
+
+logger = logging.getLogger(__name__)
+
+
+def check_cwltool_version() -> None:
+    """
+    Check if the installed cwltool version matches Toil's expected version. A
+    warning is printed if the versions differ.
+    """
+    try:
+        installed_version = get_distribution("cwltool").version
+
+        if installed_version != cwltool_version:
+            logger.warning(f"You are using cwltool version {installed_version}, which might not be compatible with "
+                           f"version {cwltool_version} used by Toil. You should consider running 'pip install cwltool=="
+                           f"{cwltool_version}' to match Toil's cwltool version.")
+    except DistributionNotFound:
+        logger.warning("cwltool is not installed.")
+
+
+check_cwltool_version()
diff --git a/version_template.py b/version_template.py
--- a/version_template.py
+++ b/version_template.py
@@ -88,7 +88,7 @@ def dockerTag():
     """The primary tag of the Docker image for the appliance. This uniquely identifies the appliance image."""
     return version() + _pythonVersionSuffix()
 
-  
+
 def currentCommit():
     import os
     from subprocess import check_output
@@ -127,12 +127,23 @@ def dirty():
         return False  # In case the git call fails.
 
 
-def expand_(name=None):
+def expand_(name=None, others=None):
+    """
+    Returns a string of all the globals and additional variables passed as the
+    others keyword argument.
+
+    :param str name: If set, only the value of the given symbol is returned.
+    :param dict others: A dictionary of additional variables to be included in
+                        the return value.
+    """
     variables = {k: v for k, v in globals().items()
                  if not k.startswith('_') and not k.endswith('_')}
 
+    if others is not None:
+        variables.update(others)
+
     def resolve(k):
-        v = variables[k]
+        v = variables.get(k, None)
         if callable(v):
             v = v()
         return v


Test patch: