Repository: common-workflow-language/cwltool

Issue:
[Regression] --record-container-id causes internal exception
When we run `cwltool` with `--record-container-id` to execute a workflows that use docker, the execution fails due to the internal exception.
This issue was introduced by #835 (commit: 8ddb82d81daf165442d90f73926838972612739d).

## Expected Behavior
```console
$ mkdir -p tmp
$ python -mcwltool.main --record-container-id --outdir $PWD/tmp cwltool/schemas/v1.0/v1.0/search.cwl cwltool/schemas/v1.0/v1.0/search-job.json
...
Final process status is success
```

## Actual Behavior
```console
$ mkdir -p tmp
$ python -mcwltool.main --record-container-id --outdir $PWD/tmp cwltool/schemas/v1.0/v1.0/search.cwl cwltool/schemas/v1.0/v1.0/search-job.json
...
[workflow ] start
[workflow ] starting step index
[step index] start
Got workflow error
Traceback (most recent call last):
  File "/Users/tom-tan/cwltool/cwltool/executors.py", line 162, in run_jobs
    job.run(runtime_context)
  File "/Users/tom-tan/cwltool/cwltool/job.py", line 484, in run
    runtime = self.create_runtime(env, runtimeContext)
  File "/Users/tom-tan/cwltool/cwltool/docker.py", line 337, in create_runtime
    cidfile_dir = runtimeContext.cidfile_
AttributeError: 'RuntimeContext' object has no attribute 'cidfile_'
Workflow error, try again with --debug for more information:
'RuntimeContext' object has no attribute 'cidfile_'
```


## Workflow Code
[search.cwl in this repository](https://github.com/common-workflow-language/cwltool/blob/master/cwltool/schemas/v1.0/v1.0/search.cwl) is an example that can reproduce this issue.
I also confirm that it can be reproduced with another workflow that uses `DockerRequirement`.

## Full Traceback
```pytb
/Users/tom-tan/cwltool/cwltool/main.py 1.0.20180720074456
Resolved 'cwltool/schemas/v1.0/v1.0/search.cwl' to 'file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/search.cwl'
Workflow checker warning:
cwltool/schemas/v1.0/v1.0/search.cwl:108:9:  Sink 'file'
cwltool/schemas/v1.0/v1.0/search.cwl:57:7:     expects secondaryFiles: ['.idx1', '^.idx2',
                                               '$(self.basename).idx3', '${ return
                                               self.basename+".idx4"; }',
                                               '$(self.nameroot).idx6$(self.nameext)', '${ return
                                               [self.basename+".idx7"]; }', '_idx8'] but
cwltool/schemas/v1.0/v1.0/search.cwl:103:13:   source 'result' has secondaryFiles ['.idx1',
                                               '^.idx2', '$(self.basename).idx3', '${ return
                                               self.basename+".idx4"; }', '$({"path":
                                               self.path+".idx5", "class": "File"})',
                                               '$(self.nameroot).idx6$(self.nameext)', '${ return
                                               [self.basename+".idx7", inputs.secondfile]; }',
                                               '_idx8'].
                                               To fix, add secondaryFiles: ['.idx1', '^.idx2',
                                               '$(self.basename).idx3', '${ return
                                               self.basename+".idx4"; }',
                                               '$(self.nameroot).idx6$(self.nameext)', '${ return
                                               [self.basename+".idx7"]; }', '_idx8'] to definition of
                                               'result'.
[workflow ] initialized from file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/search.cwl#main
[workflow ] start
[workflow ] {
    "infile": {
        "class": "File",
        "location": "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/whale.txt",
        "size": 1111,
        "basename": "whale.txt",
        "nameroot": "whale",
        "nameext": ".txt"
    },
    "secondfile": {
        "class": "File",
        "location": "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/hello.txt",
        "size": 13,
        "basename": "hello.txt",
        "nameroot": "hello",
        "nameext": ".txt"
    },
    "term": "find"
}
[workflow ] job step file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/search.cwl#main/search not ready
[workflow ] starting step index
[job step index] job input {
    "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/search.cwl#main/index/file": {
        "class": "File",
        "location": "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/whale.txt",
        "size": 1111,
        "basename": "whale.txt",
        "nameroot": "whale",
        "nameext": ".txt"
    },
    "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/search.cwl#main/index/secondfile": {
        "class": "File",
        "location": "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/hello.txt",
        "size": 13,
        "basename": "hello.txt",
        "nameroot": "hello",
        "nameext": ".txt"
    }
}
[job step index] evaluated job input to {
    "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/search.cwl#main/index/file": {
        "class": "File",
        "location": "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/whale.txt",
        "size": 1111,
        "basename": "whale.txt",
        "nameroot": "whale",
        "nameext": ".txt"
    },
    "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/search.cwl#main/index/secondfile": {
        "class": "File",
        "location": "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/hello.txt",
        "size": 13,
        "basename": "hello.txt",
        "nameroot": "hello",
        "nameext": ".txt"
    }
}
[step index] start
[job index] initializing from file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/search.cwl#index as part of step index
[job index] {
    "file": {
        "class": "File",
        "location": "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/whale.txt",
        "size": 1111,
        "basename": "whale.txt",
        "nameroot": "whale",
        "nameext": ".txt"
    },
    "secondfile": {
        "class": "File",
        "location": "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/hello.txt",
        "size": 13,
        "basename": "hello.txt",
        "nameroot": "hello",
        "nameext": ".txt"
    }
}
[job index] path mappings is {
    "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/whale.txt": [
        "/Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/whale.txt",
        "/var/spool/cwl/input.txt",
        "File",
        false
    ],
    "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/index.py": [
        "/Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/index.py",
        "/var/lib/cwl/stg8fac6d5c-2d45-49d6-919f-60abae6f127f/index.py",
        "File",
        true
    ],
    "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/hello.txt": [
        "/Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/hello.txt",
        "/var/lib/cwl/stg6fd9bd87-88b6-4ca0-9cc3-451c217be017/hello.txt",
        "File",
        true
    ]
}
[job index] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "python"
    },
    {
        "position": [
            0,
            "index.py"
        ],
        "datum": {
            "class": "File",
            "location": "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/index.py",
            "basename": "index.py",
            "nameroot": "index",
            "nameext": ".py",
            "path": "/var/lib/cwl/stg8fac6d5c-2d45-49d6-919f-60abae6f127f/index.py",
            "dirname": "/var/lib/cwl/stg8fac6d5c-2d45-49d6-919f-60abae6f127f"
        }
    },
    {
        "valueFrom": "input.txt",
        "position": [
            1,
            0
        ]
    }
]
[job index] initial work dir {
    "file:///Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/whale.txt": [
        "/Users/tom-tan/cwltool/cwltool/schemas/v1.0/v1.0/whale.txt",
        "/var/spool/cwl/input.txt",
        "File",
        true
    ]
}
Got workflow error
Traceback (most recent call last):
  File "/Users/tom-tan/cwltool/cwltool/executors.py", line 162, in run_jobs
    job.run(runtime_context)
  File "/Users/tom-tan/cwltool/cwltool/job.py", line 484, in run
    runtime = self.create_runtime(env, runtimeContext)
  File "/Users/tom-tan/cwltool/cwltool/docker.py", line 337, in create_runtime
    cidfile_dir = runtimeContext.cidfile_
AttributeError: 'RuntimeContext' object has no attribute 'cidfile_'
Workflow error:
'RuntimeContext' object has no attribute 'cidfile_'
Traceback (most recent call last):
  File "/Users/tom-tan/cwltool/cwltool/executors.py", line 162, in run_jobs
    job.run(runtime_context)
  File "/Users/tom-tan/cwltool/cwltool/job.py", line 484, in run
    runtime = self.create_runtime(env, runtimeContext)
  File "/Users/tom-tan/cwltool/cwltool/docker.py", line 337, in create_runtime
    cidfile_dir = runtimeContext.cidfile_
AttributeError: 'RuntimeContext' object has no attribute 'cidfile_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tom-tan/cwltool/cwltool/main.py", line 606, in main
    logger=_logger)
  File "/Users/tom-tan/cwltool/cwltool/executors.py", line 39, in __call__
    return self.execute(*args, **kwargs)
  File "/Users/tom-tan/cwltool/cwltool/executors.py", line 89, in execute
    self.run_jobs(process, job_order_object, logger, runtime_context)
  File "/Users/tom-tan/cwltool/cwltool/executors.py", line 170, in run_jobs
    raise WorkflowException(Text(e))
cwltool.errors.WorkflowException: 'RuntimeContext' object has no attribute 'cidfile_'
```

## Your Environment
* cwltool version: 
```console
$ cwltool --version
/Users/tom-tan/.pyenv/versions/3.6.5/bin/cwltool 1.0.20180720132017
```

## Note
During executing the workflow, I saw the following warning message but it seems not related to this issue.

```console
$ python -mcwltool.main --record-container-id --outdir $PWD/tmp cwltool/schemas/v1.0/v1.0/search.cwl cwltool/schemas/v1.0/v1.0/search-job.json
...
Workflow checker warning:
cwltool/schemas/v1.0/v1.0/search.cwl:108:9:  Sink 'file'
cwltool/schemas/v1.0/v1.0/search.cwl:57:7:     expects secondaryFiles: ['.idx1', '^.idx2',
                                               '$(self.basename).idx3', '${ return
                                               self.basename+".idx4"; }',
                                               '$(self.nameroot).idx6$(self.nameext)', '${ return
                                               [self.basename+".idx7"]; }', '_idx8'] but
cwltool/schemas/v1.0/v1.0/search.cwl:103:13:   source 'result' has secondaryFiles ['.idx1',
                                               '^.idx2', '$(self.basename).idx3', '${ return
                                               self.basename+".idx4"; }', '$({"path":
                                               self.path+".idx5", "class": "File"})',
                                               '$(self.nameroot).idx6$(self.nameext)', '${ return
                                               [self.basename+".idx7", inputs.secondfile]; }',
                                               '_idx8'].
                                               To fix, add secondaryFiles: ['.idx1', '^.idx2',
                                               '$(self.basename).idx3', '${ return
                                               self.basename+".idx4"; }',
                                               '$(self.nameroot).idx6$(self.nameext)', '${ return
                                               [self.basename+".idx7"]; }', '_idx8'] to definition of
                                               'result'.
...
```



Discussion:

Resolving patch:
diff --git a/cwltool/docker.py b/cwltool/docker.py
--- a/cwltool/docker.py
+++ b/cwltool/docker.py
@@ -22,10 +22,10 @@
 from .secrets import SecretStore  # pylint: disable=unused-import
 from .utils import (DEFAULT_TMP_PREFIX, docker_windows_path_adjust, onWindows,
                     subprocess)
-from .context import RuntimeContext
+from .context import RuntimeContext  # pylint: disable=unused-import
 
-found_images = set()  # type: Set[Text]
-found_images_lock = threading.Lock()
+_IMAGES = set()  # type: Set[Text]
+_IMAGES_LOCK = threading.Lock()
 __docker_machine_mounts = None  # type: Optional[List[Text]]
 __docker_machine_mounts_lock = threading.Lock()
 
@@ -91,8 +91,8 @@ def get_image(docker_requirement,     # type: Dict[Text, Text]
                 and "dockerPull" in docker_requirement:
             docker_requirement["dockerImageId"] = docker_requirement["dockerPull"]
 
-        with found_images_lock:
-            if docker_requirement["dockerImageId"] in found_images:
+        with _IMAGES_LOCK:
+            if docker_requirement["dockerImageId"] in _IMAGES:
                 return True
 
         for line in subprocess.check_output(
@@ -132,7 +132,7 @@ def get_image(docker_requirement,     # type: Dict[Text, Text]
             elif "dockerFile" in docker_requirement:
                 dockerfile_dir = str(tempfile.mkdtemp(prefix=tmp_outdir_prefix))
                 with open(os.path.join(
-                        dockerfile_dir, "Dockerfile"), "wb") as dfile:
+                    dockerfile_dir, "Dockerfile"), "wb") as dfile:
                     dfile.write(docker_requirement["dockerFile"].encode('utf-8'))
                 cmd = ["docker", "build", "--tag=%s" %
                        str(docker_requirement["dockerImageId"]), dockerfile_dir]
@@ -171,8 +171,8 @@ def get_image(docker_requirement,     # type: Dict[Text, Text]
                 found = True
 
         if found:
-            with found_images_lock:
-                found_images.add(docker_requirement["dockerImageId"])
+            with _IMAGES_LOCK:
+                _IMAGES.add(docker_requirement["dockerImageId"])
 
         return found
 
@@ -339,7 +339,7 @@ def create_runtime(self, env, runtimeContext):
 
         # add parameters to docker to write a container ID file
         if runtimeContext.record_container_id:
-            cidfile_dir = runtimeContext.cidfile_
+            cidfile_dir = runtimeContext.cidfile_dir
             if cidfile_dir != "":
                 if not os.path.isdir(cidfile_dir):
                     _logger.error("--cidfile-dir %s error:\n%s", cidfile_dir,


Test patch:
diff --git a/tests/test_examples.py b/tests/test_examples.py
--- a/tests/test_examples.py
+++ b/tests/test_examples.py
@@ -3,6 +3,7 @@
 import json
 import logging
 import shutil
+import os
 import sys
 import tempfile
 import unittest
@@ -643,6 +644,19 @@ def test_no_js_console(self):
             self.assertNotIn("[log] Log message", stderr)
             self.assertNotIn("[err] Error message", stderr)
 
+@needs_docker
+class TestRecordContainerId(TestCmdLine):
+    def test_record_container_id(self):
+        test_file = "cache_test_workflow.cwl"
+        cid_dir = tempfile.mkdtemp("cwltool_test_cid")
+        error_code, _, stderr = self.get_main_output(
+            ["--record-container-id", "--cidfile-dir", cid_dir,
+             get_data("tests/wf/" + test_file)])
+        self.assertIn("completed success", stderr)
+        self.assertEqual(error_code, 0)
+        self.assertEqual(len(os.listdir(cid_dir)), 2)
+        shutil.rmtree(cid_dir)
+
 
 @needs_docker
 class TestCache(TestCmdLine):