Repository: common-workflow-language/cwltool

Issue:
Any type in inner tool fails to run
`test_workflow.cwl`:
```
class: Workflow
cwlVersion: v1.0
inputs:
  file:
    type: File
outputs: []
steps:
  - id: step
    in:
      file: file
    out: []
    run: test_tool.cwl
```

`test_tool.cwl`
```
cwlVersion: v1.0
class: CommandLineTool
requirements:
  - class: InlineJavascriptRequirement
baseCommand: ['echo']

inputs:
  - id: file
    type: Any
    inputBinding:
      valueFrom: "[$(self)]"

outputs: []

```

`test_inputs.cwl`
```
file:
  class: File
  path: test_file
```

```
$ cwltool --debug test_workflow.cwl test_inputs.yaml
/Users/th10/checkouts/unjsify_cwl/venv/bin/cwltool 1.0.20180703025338
Resolved 'test_workflow.cwl' to 'file:///Users/th10/checkouts/unjsify_cwl/test_workflow.cwl'
[workflow ] initialized from file:///Users/th10/checkouts/unjsify_cwl/test_workflow.cwl
[workflow ] start
[workflow ] starting step step
[job step step] job input {
    "file:///Users/th10/checkouts/unjsify_cwl/test_workflow.cwl#step/file": {
        "class": "File",
        "location": "file:///Users/th10/checkouts/unjsify_cwl/test_file",
        "size": 0,
        "basename": "test_file",
        "nameroot": "test_file",
        "nameext": ""
    }
}
[job step step] evaluated job input to {
    "file:///Users/th10/checkouts/unjsify_cwl/test_workflow.cwl#step/file": {
        "class": "File",
        "location": "file:///Users/th10/checkouts/unjsify_cwl/test_file",
        "size": 0,
        "basename": "test_file",
        "nameroot": "test_file",
        "nameext": ""
    }
}
[step step] start
[job step] initializing from file:///Users/th10/checkouts/unjsify_cwl/test_tool.cwl as part of step step
[job step] {
    "file": {
        "class": "File",
        "location": "file:///Users/th10/checkouts/unjsify_cwl/test_file",
        "size": 0,
        "basename": "test_file",
        "nameroot": "test_file",
        "nameext": ""
    }
}
Unexpected exception
Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 747, in job
    runtimeContext):
  File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 402, in job
    visit_class([builder.files, builder.bindings], ("File", "Directory"), _check_adjust)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 233, in visit_class
    visit_class(d, cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 233, in visit_class
    visit_class(d, cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 230, in visit_class
    visit_class(rec[d], cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 228, in visit_class
    op(rec)
  File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 199, in check_adjust
    builder.pathmapper.mapper(file_o["location"])[1])
  File "/Users/th10/checkouts/cwltool/cwltool/pathmapper.py", line 281, in mapper
    return self._pathmap[src]
KeyError: 'file:///Users/th10/checkouts/unjsify_cwl/test_file'
[step step] Cannot make job: 'file:///Users/th10/checkouts/unjsify_cwl/test_file'

Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 747, in job
    runtimeContext):
  File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 402, in job
    visit_class([builder.files, builder.bindings], ("File", "Directory"), _check_adjust)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 233, in visit_class
    visit_class(d, cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 233, in visit_class
    visit_class(d, cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 230, in visit_class
    visit_class(rec[d], cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 228, in visit_class
    op(rec)
  File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 199, in check_adjust
    builder.pathmapper.mapper(file_o["location"])[1])
  File "/Users/th10/checkouts/cwltool/cwltool/pathmapper.py", line 281, in mapper
    return self._pathmap[src]
KeyError: 'file:///Users/th10/checkouts/unjsify_cwl/test_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 457, in job
    for newjob in step.iterable:
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 392, in try_make_job
    for j in jobs:
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 199, in job
    for j in self.step.job(joborder, output_callback, runtimeContext):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 754, in job
    raise WorkflowException(Text(exc))
cwltool.errors.WorkflowException: 'file:///Users/th10/checkouts/unjsify_cwl/test_file'
[workflow ] completed permanentFail
Removing intermediate output directory /private/tmp/docker_tmpai55fsot
{}
Final process status is permanentFail
```
```
$ cwltool --version
/Users/th10/checkouts/unjsify_cwl/venv/bin/cwltool 1.0.20180703025338
```

Note: this only doesn't work when you set the tool's type to `Any`
Any type in inner tool fails to run
`test_workflow.cwl`:
```
class: Workflow
cwlVersion: v1.0
inputs:
  file:
    type: File
outputs: []
steps:
  - id: step
    in:
      file: file
    out: []
    run: test_tool.cwl
```

`test_tool.cwl`
```
cwlVersion: v1.0
class: CommandLineTool
requirements:
  - class: InlineJavascriptRequirement
baseCommand: ['echo']

inputs:
  - id: file
    type: Any
    inputBinding:
      valueFrom: "[$(self)]"

outputs: []

```

`test_inputs.cwl`
```
file:
  class: File
  path: test_file
```

```
$ cwltool --debug test_workflow.cwl test_inputs.yaml
/Users/th10/checkouts/unjsify_cwl/venv/bin/cwltool 1.0.20180703025338
Resolved 'test_workflow.cwl' to 'file:///Users/th10/checkouts/unjsify_cwl/test_workflow.cwl'
[workflow ] initialized from file:///Users/th10/checkouts/unjsify_cwl/test_workflow.cwl
[workflow ] start
[workflow ] starting step step
[job step step] job input {
    "file:///Users/th10/checkouts/unjsify_cwl/test_workflow.cwl#step/file": {
        "class": "File",
        "location": "file:///Users/th10/checkouts/unjsify_cwl/test_file",
        "size": 0,
        "basename": "test_file",
        "nameroot": "test_file",
        "nameext": ""
    }
}
[job step step] evaluated job input to {
    "file:///Users/th10/checkouts/unjsify_cwl/test_workflow.cwl#step/file": {
        "class": "File",
        "location": "file:///Users/th10/checkouts/unjsify_cwl/test_file",
        "size": 0,
        "basename": "test_file",
        "nameroot": "test_file",
        "nameext": ""
    }
}
[step step] start
[job step] initializing from file:///Users/th10/checkouts/unjsify_cwl/test_tool.cwl as part of step step
[job step] {
    "file": {
        "class": "File",
        "location": "file:///Users/th10/checkouts/unjsify_cwl/test_file",
        "size": 0,
        "basename": "test_file",
        "nameroot": "test_file",
        "nameext": ""
    }
}
Unexpected exception
Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 747, in job
    runtimeContext):
  File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 402, in job
    visit_class([builder.files, builder.bindings], ("File", "Directory"), _check_adjust)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 233, in visit_class
    visit_class(d, cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 233, in visit_class
    visit_class(d, cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 230, in visit_class
    visit_class(rec[d], cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 228, in visit_class
    op(rec)
  File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 199, in check_adjust
    builder.pathmapper.mapper(file_o["location"])[1])
  File "/Users/th10/checkouts/cwltool/cwltool/pathmapper.py", line 281, in mapper
    return self._pathmap[src]
KeyError: 'file:///Users/th10/checkouts/unjsify_cwl/test_file'
[step step] Cannot make job: 'file:///Users/th10/checkouts/unjsify_cwl/test_file'

Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 747, in job
    runtimeContext):
  File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 402, in job
    visit_class([builder.files, builder.bindings], ("File", "Directory"), _check_adjust)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 233, in visit_class
    visit_class(d, cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 233, in visit_class
    visit_class(d, cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 230, in visit_class
    visit_class(rec[d], cls, op)
  File "/Users/th10/checkouts/cwltool/cwltool/utils.py", line 228, in visit_class
    op(rec)
  File "/Users/th10/checkouts/cwltool/cwltool/command_line_tool.py", line 199, in check_adjust
    builder.pathmapper.mapper(file_o["location"])[1])
  File "/Users/th10/checkouts/cwltool/cwltool/pathmapper.py", line 281, in mapper
    return self._pathmap[src]
KeyError: 'file:///Users/th10/checkouts/unjsify_cwl/test_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 457, in job
    for newjob in step.iterable:
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 392, in try_make_job
    for j in jobs:
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 199, in job
    for j in self.step.job(joborder, output_callback, runtimeContext):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 754, in job
    raise WorkflowException(Text(exc))
cwltool.errors.WorkflowException: 'file:///Users/th10/checkouts/unjsify_cwl/test_file'
[workflow ] completed permanentFail
Removing intermediate output directory /private/tmp/docker_tmpai55fsot
{}
Final process status is permanentFail
```
```
$ cwltool --version
/Users/th10/checkouts/unjsify_cwl/venv/bin/cwltool 1.0.20180703025338
```

Note: this only doesn't work when you set the tool's type to `Any`


Discussion:

Resolving patch:
diff --git a/cwltool/builder.py b/cwltool/builder.py
--- a/cwltool/builder.py
+++ b/cwltool/builder.py
@@ -344,6 +344,22 @@ def bind_input(
                     )
                 )
         else:
+            if schema["type"] == "org.w3id.cwl.salad.Any":
+                if isinstance(datum, dict):
+                    if datum.get("class") == "File":
+                        schema["type"] = "org.w3id.cwl.cwl.File"
+                    elif datum.get("class") == "Directory":
+                        schema["type"] = "org.w3id.cwl.cwl.Directory"
+                    else:
+                        schema["type"] = "record"
+                        schema["fields"] = [
+                            {"name": field_name, "type": "Any"}
+                            for field_name in datum.keys()
+                        ]
+                elif isinstance(datum, list):
+                    schema["type"] = "array"
+                    schema["items"] = "Any"
+
             if schema["type"] in self.schemaDefs:
                 schema = self.schemaDefs[cast(str, schema["type"])]
 


Test patch:
diff --git a/tests/test_examples.py b/tests/test_examples.py
--- a/tests/test_examples.py
+++ b/tests/test_examples.py
@@ -1568,3 +1568,11 @@ def test_bad_networkaccess_expr() -> None:
         "Got '42' for expression '${return 42;}" in stderr
     )
     assert err_code == 1
+
+
+def test_staging_files_in_any() -> None:
+    """Confirm that inputs of type File are staged, even if the schema is Any."""
+    err_code, _, stderr = get_main_output(
+        [get_data("tests/wf/816_wf.cwl"), "--file", get_data("tests/echo-job.yaml")]
+    )
+    assert err_code == 0
diff --git a/tests/wf/816_tool.cwl b/tests/wf/816_tool.cwl
new file mode 100644
--- /dev/null
+++ b/tests/wf/816_tool.cwl
@@ -0,0 +1,14 @@
+cwlVersion: v1.0
+class: CommandLineTool
+requirements:
+  - class: InlineJavascriptRequirement
+baseCommand: ['echo']
+
+inputs:
+  - id: file
+    type: Any
+    inputBinding:
+      valueFrom: "[$(self)]"
+
+outputs: []
+
diff --git a/tests/wf/816_wf.cwl b/tests/wf/816_wf.cwl
new file mode 100644
--- /dev/null
+++ b/tests/wf/816_wf.cwl
@@ -0,0 +1,12 @@
+class: Workflow
+cwlVersion: v1.0
+inputs:
+  file:
+    type: File
+outputs: []
+steps:
+  - id: step
+    in:
+      file: file
+    out: []
+    run: 816_tool.cwl