Repository: common-workflow-language/cwltool

Issue:
Step input with multiple sources adds extra value to array
## Expected Behavior
When using `MultipleInputFeatureRequirement` with `merge_nested` the input must be an array consisting of exactly one entry for each input link. It should be the same even if the `type` for the source is a list of two or more items.

## Actual Behavior
If the source has a `type` set as a list of two or more items, it will be added to the result array exactly the same number of times as the length of its `type` list.   

## Workflow Code
```yaml
cwlVersion: v1.0
class: Workflow

requirements:
  - class: SubworkflowFeatureRequirement
  - class: ScatterFeatureRequirement
  - class: StepInputExpressionRequirement
  - class: InlineJavascriptRequirement
  - class: MultipleInputFeatureRequirement

inputs:
  input_1:
    type: int
  input_2:
    type:
    - File
    - int

outputs:
  resut:
    type: int
    outputSource: echo/result

steps:
  echo:
    run: echo.cwl
    in:
      data:
        source: [input_1, input_2]
        valueFrom: |
          ${
              return self[0] + self[1];
          }
    out: [result]

```
## Short Traceback
```pytb
"file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl#echo/data": [
        1, 
        2, 
        2
    ]
}
```

## Full Traceback
```pytb
cwltool --debug workflow.cwl job.yml 
/usr/local/bin/cwltool 1.0.20170525215327
Resolved 'workflow.cwl' to 'file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl'
[workflow workflow.cwl] initialized from file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl
[workflow workflow.cwl] start
[workflow workflow.cwl] starting step echo
[job step echo] job input {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl#echo/data": [
        1, 
        2, 
        2
    ]
}
[job step echo] evaluated job input to {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl#echo/data": 3
}
[step echo] start
[job echo] initializing from file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/echo.cwl as part of step echo
[job echo] {
    "data": 3
}
[job echo] path mappings is {}
[job echo] command line bindings is [
    {
        "position": [
            -1000000, 
            0
        ], 
        "datum": "echo"
    }, 
    {
        "position": [
            1, 
            "data"
        ], 
        "datum": 3
    }
]
[job echo] /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpzDqvrt$ docker \
    run \
    -i \
    --volume=/private/var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpzDqvrt:/private/var/spool/cwl:rw \
    --volume=/private/var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpnYvOeu:/tmp:rw \
    --workdir=/private/var/spool/cwl \
    --read-only=true \
    --log-driver=none \
    --user=501 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/private/var/spool/cwl \
    ubuntu \
    echo \
    3 > /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpzDqvrt/output.txt
[job echo] completed success
[job echo] {
    "result": 3
}
[step echo] produced output {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl#echo/result": 3
}
[step echo] completed success
[workflow workflow.cwl] completed success
[job echo] Removing input staging directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmp1Q0eyL
[job echo] Removing temporary directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpnYvOeu
Removing intermediate output directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmp2OqTZa
{
    "resut": 3
}
Final process status is success
```
## Link to reproduce test
https://github.com/michael-kotliar/cwl_sandbox/tree/master/multiple_input_source

## Your Environment
* cwltool version: 1.0.20170525215327

checkFormat raises exception for workflow input set by default to Null
## Expected Behavior
When optional workflow input `input_file` with a `format` field is not provided and the `default: null` is used the `checkFormat` function shouldn't raise exception when trying to check `format` for this input.

## Actual Behavior
`checkFormat` function raises exception for optional workflow input with `default: null` and `format` fields.
Workflow doesn't fail if:
- remove either `default: null` or `format`
- remove `namespaces` and `schemas` sections

## Workflow Code
```yaml
cwlVersion: v1.0
class: Workflow

inputs:
  input_file:
    type: File?
    default: null
    format: "http://edamontology.org/format_2572"

outputs: []

steps:
  cat:
    run: cat.cwl
    in:
      input_file: input_file
    out: []

$namespaces:
  s: http://schema.org/
$schemas:
- http://schema.org/docs/schema_org_rdfa.html
s:name: "workflow"
```
```yaml
cwlVersion: v1.0
class: CommandLineTool

hints:
  DockerRequirement:
    dockerPull: ubuntu

inputs:
  input_file:
    type:
      - "null"
      - File
    inputBinding:
      position: 1

outputs: []

baseCommand: cat
```
## Full Traceback
```pytb
cwltool --debug workflow.cwl
/usr/local/bin/cwltool 1.0.20170525215327
Resolved 'workflow.cwl' to 'file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/null_default_2/workflow.cwl'
No handlers could be found for logger "rdflib.term"
Parsed job order from command line: {
    "id": "workflow.cwl", 
    "input_file": null
}
Got workflow error
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cwltool/main.py", line 243, in single_job_executor
    for r in jobiter:
  File "/usr/local/lib/python2.7/site-packages/cwltool/workflow.py", line 534, in job
    builder = self._init_job(job_order, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/cwltool/process.py", line 559, in _init_job
    checkFormat(builder.job[d], builder.do_eval(i["format"]), self.formatgraph)
  File "/usr/local/lib/python2.7/site-packages/cwltool/process.py", line 346, in checkFormat
    if "format" not in af:
TypeError: argument of type 'NoneType' is not iterable
Workflow error, try again with --debug for more information:
argument of type 'NoneType' is not iterable
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cwltool/main.py", line 774, in main
    **vars(args))
  File "/usr/local/lib/python2.7/site-packages/cwltool/main.py", line 255, in single_job_executor
    raise WorkflowException(Text(e))
WorkflowException: argument of type 'NoneType' is not iterable
```

## Your Environment
* cwltool version: 1.0.20170525215327

## Link to replicate test
https://github.com/michael-kotliar/cwl_sandbox/tree/master/null_default_2

Step input with multiple sources adds extra value to array
## Expected Behavior
When using `MultipleInputFeatureRequirement` with `merge_nested` the input must be an array consisting of exactly one entry for each input link. It should be the same even if the `type` for the source is a list of two or more items.

## Actual Behavior
If the source has a `type` set as a list of two or more items, it will be added to the result array exactly the same number of times as the length of its `type` list.   

## Workflow Code
```yaml
cwlVersion: v1.0
class: Workflow

requirements:
  - class: SubworkflowFeatureRequirement
  - class: ScatterFeatureRequirement
  - class: StepInputExpressionRequirement
  - class: InlineJavascriptRequirement
  - class: MultipleInputFeatureRequirement

inputs:
  input_1:
    type: int
  input_2:
    type:
    - File
    - int

outputs:
  resut:
    type: int
    outputSource: echo/result

steps:
  echo:
    run: echo.cwl
    in:
      data:
        source: [input_1, input_2]
        valueFrom: |
          ${
              return self[0] + self[1];
          }
    out: [result]

```
## Short Traceback
```pytb
"file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl#echo/data": [
        1, 
        2, 
        2
    ]
}
```

## Full Traceback
```pytb
cwltool --debug workflow.cwl job.yml 
/usr/local/bin/cwltool 1.0.20170525215327
Resolved 'workflow.cwl' to 'file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl'
[workflow workflow.cwl] initialized from file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl
[workflow workflow.cwl] start
[workflow workflow.cwl] starting step echo
[job step echo] job input {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl#echo/data": [
        1, 
        2, 
        2
    ]
}
[job step echo] evaluated job input to {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl#echo/data": 3
}
[step echo] start
[job echo] initializing from file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/echo.cwl as part of step echo
[job echo] {
    "data": 3
}
[job echo] path mappings is {}
[job echo] command line bindings is [
    {
        "position": [
            -1000000, 
            0
        ], 
        "datum": "echo"
    }, 
    {
        "position": [
            1, 
            "data"
        ], 
        "datum": 3
    }
]
[job echo] /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpzDqvrt$ docker \
    run \
    -i \
    --volume=/private/var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpzDqvrt:/private/var/spool/cwl:rw \
    --volume=/private/var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpnYvOeu:/tmp:rw \
    --workdir=/private/var/spool/cwl \
    --read-only=true \
    --log-driver=none \
    --user=501 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/private/var/spool/cwl \
    ubuntu \
    echo \
    3 > /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpzDqvrt/output.txt
[job echo] completed success
[job echo] {
    "result": 3
}
[step echo] produced output {
    "file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/multiple_input_source/workflow.cwl#echo/result": 3
}
[step echo] completed success
[workflow workflow.cwl] completed success
[job echo] Removing input staging directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmp1Q0eyL
[job echo] Removing temporary directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmpnYvOeu
Removing intermediate output directory /var/folders/sd/41rg42_16q72_2yzl_vvgsbw0000gn/T/tmp2OqTZa
{
    "resut": 3
}
Final process status is success
```
## Link to reproduce test
https://github.com/michael-kotliar/cwl_sandbox/tree/master/multiple_input_source

## Your Environment
* cwltool version: 1.0.20170525215327



Discussion:

Resolving patch:
diff --git a/cwltool/process.py b/cwltool/process.py
--- a/cwltool/process.py
+++ b/cwltool/process.py
@@ -344,6 +344,8 @@ def formatSubclassOf(fmt, cls, ontology, visited):
 def checkFormat(actualFile, inputFormats, ontology):
     # type: (Union[Dict[Text, Any], List, Text], Union[List[Text], Text], Graph) -> None
     for af in aslist(actualFile):
+        if not af:
+            continue
         if "format" not in af:
             raise validate.ValidationException(u"Missing required 'format' for File %s" % af)
         for inpf in aslist(inputFormats):
diff --git a/cwltool/workflow.py b/cwltool/workflow.py
--- a/cwltool/workflow.py
+++ b/cwltool/workflow.py
@@ -66,13 +66,13 @@ def match_types(sinktype, src, iid, inputobj, linkMerge, valueFrom):
                 return True
     elif isinstance(src.parameter["type"], list):
         # Source is union type
-        # Check that every source type is compatible with the sink.
+        # Check that at least one source type is compatible with the sink.
         for st in src.parameter["type"]:
             srccopy = copy.deepcopy(src)
             srccopy.parameter["type"] = st
-            if not match_types(st, srccopy, iid, inputobj, linkMerge, valueFrom):
-                return False
-        return True
+            if match_types(sinktype, srccopy, iid, inputobj, linkMerge, valueFrom):
+                return True
+        return False
     elif linkMerge:
         if iid not in inputobj:
             inputobj[iid] = []


Test patch: