Repository: DataBiosphere/toil

Issue:
TypeError: string indices must be integers
Hi, thanks for the fast response (#2762)! however now it appears a different response arises:

```
tscc-1-34.sdsc.edu 2019-08-14 14:11:52,420 MainThread INFO cwltool: Resolved '/projects/ps-yeolab4/software/eclip/0.5.0/cwl/wf_get_peaks_scatter_se.cwl' to 'file:///projects/ps-yeolab4/software/eclip/0.5.0/cwl/wf_get_peaks_scatter_se.cwl'
tscc-1-34.sdsc.edu 2019-08-14 14:11:56,070 MainThread WARNING rdflib.term: http://schema.org/docs/!DOCTYPE html does not look like a valid URI, trying to serialize this will break.
tscc-1-34.sdsc.edu 2019-08-14 14:11:56,070 MainThread WARNING rdflib.term: http://schema.org/docs/html lang="en" does not look like a valid URI, trying to serialize this will break.
tscc-1-34.sdsc.edu 2019-08-14 14:11:56,589 MainThread INFO rdflib.plugins.parsers.pyRdfa: Current options:
                preserve space                         : True
                output processor graph                 : True
                output default graph                   : True
                host language                          : RDFa Core
                accept embedded RDF                    : False
                check rdfa lite                        : False
                cache vocabulary graphs                : False
                
tscc-1-34.sdsc.edu 2019-08-14 14:12:00,559 MainThread WARNING rdflib.term: http://schema.org/docs/!DOCTYPE html does not look like a valid URI, trying to serialize this will break.
tscc-1-34.sdsc.edu 2019-08-14 14:12:00,560 MainThread WARNING rdflib.term: http://schema.org/docs/html lang="en" does not look like a valid URI, trying to serialize this will break.
tscc-1-34.sdsc.edu 2019-08-14 14:12:00,986 MainThread INFO rdflib.plugins.parsers.pyRdfa: Current options:
                preserve space                         : True
                output processor graph                 : True
                output default graph                   : True
                host language                          : RDFa Core
                accept embedded RDF                    : False
                check rdfa lite                        : False
                cache vocabulary graphs                : False
                
Traceback (most recent call last):
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/bin/cwltoil", line 10, in <module>
    sys.exit(main())
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 1179, in main
    tool = cwltool.load_tool.make_tool(uri, loading_context)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/load_tool.py", line 347, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 285, in toil_make_tool
    return cwltool.workflow.default_make_tool(toolpath_object, loading_context)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 56, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 537, in __init__
    loadingContext.prov_obj))
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 571, in make_workflow_step
    return WorkflowStep(toolpath_object, pos, loadingContext, parentworkflowProv)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 633, in __init__
    toolpath_object["run"], loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/load_tool.py", line 369, in load_tool
    loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/load_tool.py", line 347, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 285, in toil_make_tool
    return cwltool.workflow.default_make_tool(toolpath_object, loading_context)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 56, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 537, in __init__
    loadingContext.prov_obj))
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 571, in make_workflow_step
    return WorkflowStep(toolpath_object, pos, loadingContext, parentworkflowProv)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 633, in __init__
    toolpath_object["run"], loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/load_tool.py", line 369, in load_tool
    loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/load_tool.py", line 347, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 285, in toil_make_tool
    return cwltool.workflow.default_make_tool(toolpath_object, loading_context)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 56, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 537, in __init__
    loadingContext.prov_obj))
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 571, in make_workflow_step
    return WorkflowStep(toolpath_object, pos, loadingContext, parentworkflowProv)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 633, in __init__
    toolpath_object["run"], loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/load_tool.py", line 369, in load_tool
    loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/load_tool.py", line 347, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 285, in toil_make_tool
    return cwltool.workflow.default_make_tool(toolpath_object, loading_context)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 56, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/workflow.py", line 562, in __init__
    static_checker(workflow_inputs, workflow_outputs, step_inputs, step_outputs, param_to_step)
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/checker.py", line 163, in static_checker
    srcsf = sorted([p["pattern"] for p in src.get("secondaryFiles", [])])
  File "/projects/ps-yeolab4/software/eclipconda/envs/toil-3.19.0.ac71bb0/lib/python3.6/site-packages/cwltool/checker.py", line 163, in <listcomp>
    srcsf = sorted([p["pattern"] for p in src.get("secondaryFiles", [])])
TypeError: string indices must be integers
```

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-408)
┆Issue Number: TOIL-408



Discussion:
Hm. This might be a job for @mr-c. Something is wrong with cwltool's ability to read in the tool info.

Those rdflib messages are worrying. Maybe some important server where schemas or tool info comes from is sending an error page instead of data it is supposed to send?

Can you post your workflow file and tool files so we can see if there is something wrong with their syntax somehow?


Hi Adam, thanks for taking a look! Here are the relevant CWL files https://github.com/YeoLab/eclip/tree/master/cwl
@byee4 This is due to an error in cwltool & schema-salad.

https://github.com/common-workflow-language/schema_salad/pull/274
https://github.com/common-workflow-language/cwltool/pull/1175

Schema-salad version 4.5.20190815125611 has been released 
https://pypi.org/project/schema-salad/4.5.20190815125611/

and cwltool version 1.0.20190815141648 has been released with the needed fixes
https://pypi.org/project/cwltool/1.0.20190815141648/

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -34,8 +34,8 @@ def runSetup():
     gcs = 'google-cloud-storage==1.6.0'
     gcs_oauth2_boto_plugin = 'gcs_oauth2_boto_plugin==1.14'
     apacheLibcloud = 'apache-libcloud==2.2.1'
-    cwltool = 'cwltool==1.0.20190228155703'
-    schemaSalad = 'schema-salad<5,>=4.1'
+    cwltool = 'cwltool==1.0.20190815141648'
+    schemaSalad = 'schema-salad<5,>=4.5.20190815125611'
     galaxyLib = 'galaxy-lib==18.9.2'
     htcondor = 'htcondor>=8.6.0'
     dill = 'dill==0.2.7.1'


Test patch:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -48,7 +48,7 @@ def setUp(self):
         self.workDir = os.path.join(self.cwlSpec, 'v1.0')
         # The latest cwl git commit hash from https://github.com/common-workflow-language/common-workflow-language.
         # Update it to get the latest tests.
-        testhash = '1f501e38ff692a408e16b246ac7d64d32f0822c2'
+        testhash = 'a062055fddcc7d7d9dbc53d28288e3ccb9a800d8'
         url = 'https://github.com/common-workflow-language/common-workflow-language/archive/%s.zip' % testhash
         if not os.path.exists(self.cwlSpec):
             urlretrieve(url, 'spec.zip')