Repository: common-workflow-language/cwltool

Issue:
CWL tools with SchemaDefRequirement fail to run in workflows
`test_workflow.cwl`:
```cwl
cwlVersion: v1.0
class: Workflow

inputs:
  - id: hello
    type: Any
outputs: []

steps:
  step:
    id: step
    run: ../common-workflow-language/v1.0/v1.0/schemadef-tool.cwl
    in:
      hello: hello
    out: []
```
`test_inputs.yaml`
```
hello:
  a: one
  b: two
```

```
$ cwltool --debug test_workflow.cwl test_inputs.yaml
/Users/th10/checkouts/cwltool/venv/bin/cwltool 1.0.20180626215339
Resolved 'test_workflow.cwl' to 'file:///Users/th10/checkouts/cwltool/test_workflow.cwl'
Validation failed at
Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 343, in __init__
    raise SchemaParseException(fail_msg)
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 772, in make_avsc_object
    def parse(json_string):
avro.schema.SchemaParseException: Could not make an Avro Schema object from HelloType.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/process.py", line 535, in __init__
    schema.AvroSchemaFromJSONData(self.inputs_record_schema, self.names)
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 744, in make_avsc_object
    elif type in VALID_TYPES:
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 665, in __init__
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 631, in make_field_objects
    fail_msg = 'Field name %s already in use.' % new_field.name
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 346, in __init__
    self.type = type_schema
avro.schema.SchemaParseException: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 509, in __init__
    loadingContext.prov_obj))
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 657, in __init__
    super(WorkflowStep, self).__init__(toolpath_object, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/process.py", line 535, in __init__
    schema.AvroSchemaFromJSONData(self.inputs_record_schema, self.names)
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py", line 157, in __exit__
    raise self.makeError(six.text_type(exc_value))
schema_salad.validate.ValidationException: test_workflow.cwl:11:5: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.
Tool definition failed validation:
test_workflow.cwl:11:5: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.
Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/main.py", line 504, in main
    metadata, uri, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/load_tool.py", line 342, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 51, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 516, in __init__
    raise validate.ValidationException("\n".join(str(v) for v in validation_errors))
schema_salad.validate.ValidationException: test_workflow.cwl:11:5: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.
```
```
$ cwltool --version
/Users/th10/checkouts/cwltool/venv/bin/cwltool 1.0.20180626215339
```
---
This is due to the type of `hello` in the tool being populated to the inputs of the workflow internally, while the `SchemaDefRequirement` isn't. A simple fix would be to propagate this requirement up into the workflow
CWL tools with SchemaDefRequirement fail to run in workflows
`test_workflow.cwl`:
```cwl
cwlVersion: v1.0
class: Workflow

inputs:
  - id: hello
    type: Any
outputs: []

steps:
  step:
    id: step
    run: ../common-workflow-language/v1.0/v1.0/schemadef-tool.cwl
    in:
      hello: hello
    out: []
```
`test_inputs.yaml`
```
hello:
  a: one
  b: two
```

```
$ cwltool --debug test_workflow.cwl test_inputs.yaml
/Users/th10/checkouts/cwltool/venv/bin/cwltool 1.0.20180626215339
Resolved 'test_workflow.cwl' to 'file:///Users/th10/checkouts/cwltool/test_workflow.cwl'
Validation failed at
Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 343, in __init__
    raise SchemaParseException(fail_msg)
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 772, in make_avsc_object
    def parse(json_string):
avro.schema.SchemaParseException: Could not make an Avro Schema object from HelloType.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/process.py", line 535, in __init__
    schema.AvroSchemaFromJSONData(self.inputs_record_schema, self.names)
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 744, in make_avsc_object
    elif type in VALID_TYPES:
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 665, in __init__
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 631, in make_field_objects
    fail_msg = 'Field name %s already in use.' % new_field.name
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 346, in __init__
    self.type = type_schema
avro.schema.SchemaParseException: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 509, in __init__
    loadingContext.prov_obj))
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 657, in __init__
    super(WorkflowStep, self).__init__(toolpath_object, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/process.py", line 535, in __init__
    schema.AvroSchemaFromJSONData(self.inputs_record_schema, self.names)
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py", line 157, in __exit__
    raise self.makeError(six.text_type(exc_value))
schema_salad.validate.ValidationException: test_workflow.cwl:11:5: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.
Tool definition failed validation:
test_workflow.cwl:11:5: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.
Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/main.py", line 504, in main
    metadata, uri, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/load_tool.py", line 342, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 51, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 516, in __init__
    raise validate.ValidationException("\n".join(str(v) for v in validation_errors))
schema_salad.validate.ValidationException: test_workflow.cwl:11:5: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.
```
```
$ cwltool --version
/Users/th10/checkouts/cwltool/venv/bin/cwltool 1.0.20180626215339
```
---
This is due to the type of `hello` in the tool being populated to the inputs of the workflow internally, while the `SchemaDefRequirement` isn't. A simple fix would be to propagate this requirement up into the workflow


Discussion:

Resolving patch:
diff --git a/cwltool/workflow.py b/cwltool/workflow.py
--- a/cwltool/workflow.py
+++ b/cwltool/workflow.py
@@ -243,6 +243,14 @@ def __init__(
         validation_errors = []
         self.tool = toolpath_object = copy.deepcopy(toolpath_object)
         bound = set()
+
+        if self.embedded_tool.get_requirement("SchemaDefRequirement")[0]:
+            if "requirements" not in toolpath_object:
+                toolpath_object["requirements"] = []
+            toolpath_object["requirements"].append(
+                self.embedded_tool.get_requirement("SchemaDefRequirement")[0]
+            )
+
         for stepfield, toolfield in (("in", "inputs"), ("out", "outputs")):
             toolpath_object[toolfield] = []
             for index, step_entry in enumerate(toolpath_object[stepfield]):


Test patch:
diff --git a/tests/test_examples.py b/tests/test_examples.py
--- a/tests/test_examples.py
+++ b/tests/test_examples.py
@@ -1576,3 +1576,14 @@ def test_staging_files_in_any() -> None:
         [get_data("tests/wf/816_wf.cwl"), "--file", get_data("tests/echo-job.yaml")]
     )
     assert err_code == 0
+
+
+def test_custom_type_in_step_process() -> None:
+    """Test that any needed custom types are available when processing a WorkflowStep."""
+    err_code, _, stderr = get_main_output(
+        [
+            get_data("tests/wf/811.cwl"),
+            get_data("tests/wf/811_inputs.yml"),
+        ]
+    )
+    assert err_code == 0
diff --git a/tests/wf/811.cwl b/tests/wf/811.cwl
new file mode 100644
--- /dev/null
+++ b/tests/wf/811.cwl
@@ -0,0 +1,15 @@
+cwlVersion: v1.0
+class: Workflow
+
+inputs:
+  - id: hello
+    type: Any
+outputs: []
+
+steps:
+  step:
+    id: step
+    run: schemadef-tool.cwl
+    in:
+      hello: hello
+    out: []
diff --git a/tests/wf/811_inputs.yml b/tests/wf/811_inputs.yml
new file mode 100644
--- /dev/null
+++ b/tests/wf/811_inputs.yml
@@ -0,0 +1,4 @@
+hello:
+  a: one
+  b: two
+
diff --git a/tests/wf/schemadef-tool.cwl b/tests/wf/schemadef-tool.cwl
new file mode 100644
--- /dev/null
+++ b/tests/wf/schemadef-tool.cwl
@@ -0,0 +1,24 @@
+#!/usr/bin/env cwl-runner
+class: CommandLineTool
+cwlVersion: v1.0
+hints:
+  ResourceRequirement:
+    ramMin: 8
+
+requirements:
+  - $import: schemadef-type.yml
+
+inputs:
+    - id: hello
+      type: "schemadef-type.yml#HelloType"
+      inputBinding:
+        valueFrom: $(self.a)/$(self.b)
+
+outputs:
+    - id: output
+      type: File
+      outputBinding:
+        glob: output.txt
+
+stdout: output.txt
+baseCommand: echo
diff --git a/tests/wf/schemadef-type.yml b/tests/wf/schemadef-type.yml
new file mode 100644
--- /dev/null
+++ b/tests/wf/schemadef-type.yml
@@ -0,0 +1,9 @@
+class: SchemaDefRequirement
+types:
+  - name: HelloType
+    type: record
+    fields:
+      - name: a
+        type: string
+      - name: b
+        type: string