Repository: DataBiosphere/toil

Issue:
apiDockerCall has a fixed 60 second timeout which is insufficient in some cases
@xchang1 has an AWS instance where starting some particular Docker containers, via Toil, weirdly takes a bit over 2 minutes on the create API call.

I haven't been able to replicate this  by doing anything other than running the particular offending part of a Toil workflow, but it wouldn't be a problem (besides in that it slows things down needlessly) if Toil was willing to wait around for the container to be created.

What Toil actually does is issue the create command, wait for the Docker daemon to finally respond, and *then* fail because the response has taken longer than the `docker` module's default 60 second timeout:

```
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/src/toil_vg/vg_common.py", line 337, in call_with_docker
WARNING:toil.leader:v/r/jobobDgHD        detach=True)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/toil/lib/docker.py", line 260, in apiDockerCall
WARNING:toil.leader:v/r/jobobDgHD        **kwargs)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/docker/models/containers.py", line 708, in run
WARNING:toil.leader:v/r/jobobDgHD        detach=detach, **kwargs)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/docker/models/containers.py", line 759, in create
WARNING:toil.leader:v/r/jobobDgHD        resp = self.client.api.create_container(**create_kwargs)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/docker/api/container.py", line 449, in create_container
WARNING:toil.leader:v/r/jobobDgHD        return self.create_container_from_config(config, name)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/docker/api/container.py", line 459, in create_container_from_config
WARNING:toil.leader:v/r/jobobDgHD        res = self._post_json(u, data=config, params=params)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/docker/api/client.py", line 248, in _post_json
WARNING:toil.leader:v/r/jobobDgHD        return self._post(url, data=json.dumps(data2), **kwargs)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/docker/utils/decorators.py", line 46, in inner
WARNING:toil.leader:v/r/jobobDgHD        return f(self, *args, **kwargs)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/docker/api/client.py", line 185, in _post
WARNING:toil.leader:v/r/jobobDgHD        return self.post(url, **self._set_request_timeout(kwargs))
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/requests/sessions.py", line 581, in post
WARNING:toil.leader:v/r/jobobDgHD        return self.request('POST', url, data=data, json=json, **kwargs)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
WARNING:toil.leader:v/r/jobobDgHD        resp = self.send(prep, **send_kwargs)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
WARNING:toil.leader:v/r/jobobDgHD        r = adapter.send(request, **kwargs)
WARNING:toil.leader:v/r/jobobDgHD      File "/mnt/old/home/ubuntu/toil-vg/toilvenv/local/lib/python2.7/site-packages/requests/adapters.py", line 529, in send
WARNING:toil.leader:v/r/jobobDgHD        raise ReadTimeout(e, request=request)
WARNING:toil.leader:v/r/jobobDgHD    ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
WARNING:toil.leader:v/r/jobobDgHD    ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-16-94
```

(I think the Docker module must have some kind of bug with timeouts on Unix sockets, where the timeout isn't actually checked in real time. But even if it was, the server is taking longer to respond than we are giving it.)

The 60 second timeout can be overridden by passing a `timeout` kwarg to `docker.from_env` inside `apiDockerCall`, but it isn't hooked up to any environment variables (like `DOCKER_CLIENT_TIMEOUT` which is used by the `docker` command). It also isn't exposed as a parameter to `apiDockerCall`. So this makes it impossible to work around this weirdly slow Docker setup on Xian's instance without hacking Toil.

I think we ought to expose the timeout through to the apiDockerCall interface. It also might make sense to set a time longer than 60 seconds as a default in Toil, since we have now encountered in the wild a system that at least sometimes takes more than 60 seconds to do the thing, and since the timeout is not actually in practice protecting us from hanging waiting for an unresponsive or weirdly slow daemon.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-409)
┆Issue Number: TOIL-409



Discussion:
I've commented on a related issue in the `docker` module here: https://github.com/docker/docker-py/issues/2266#issuecomment-522158265

There I have a dump of the traffic between the API client and the daemon; the daemon is just slow to create the container. There's a suggestion in that issue thread that maybe mounting significant amounts of data into the container has something to do with the slow container creation.

Resolving patch:
diff --git a/src/toil/lib/docker.py b/src/toil/lib/docker.py
--- a/src/toil/lib/docker.py
+++ b/src/toil/lib/docker.py
@@ -57,6 +57,7 @@ def apiDockerCall(job,
                   stdout=None,
                   stderr=False,
                   streamfile=None,
+                  timeout=365 * 24 * 60 * 60,
                   **kwargs):
     """
     A toil wrapper for the python docker API.
@@ -128,6 +129,11 @@ def toil_job(job):
                      (some other user : some other user group).
     :param environment: Allows one to set environment variables inside of the
                         container, such as:
+    :param int timeout: Use the given timeout in seconds for interactions with
+                        the Docker daemon. Note that the underlying docker module is
+                        not always able to abort ongoing reads and writes in order
+                        to respect the timeout. Defaults to 1 year (i.e. wait
+                        essentially indefinitely).
     :param kwargs: Additional keyword arguments supplied to the docker API's
                    run command.  The list is 75 keywords total, for examples
                    and full documentation see:
@@ -192,7 +198,7 @@ def toil_job(job):
     assert deferParam in (None, FORGO, STOP, RM), \
         'Please provide a valid value for deferParam.'
 
-    client = docker.from_env(version='auto')
+    client = docker.from_env(version='auto', timeout=timeout)
 
     if deferParam == STOP:
         job.defer(dockerStop, containerName)
@@ -283,14 +289,19 @@ def toil_job(job):
         raise create_api_error_from_http_exception(e)
 
 
-def dockerKill(container_name, gentleKill=False):
+def dockerKill(container_name, gentleKill=False, timeout=365 * 24 * 60 * 60):
     """
     Immediately kills a container.  Equivalent to "docker kill":
     https://docs.docker.com/engine/reference/commandline/kill/
     :param container_name: Name of the container being killed.
     :param client: The docker API client object to call.
+    :param int timeout: Use the given timeout in seconds for interactions with
+                        the Docker daemon. Note that the underlying docker module is
+                        not always able to abort ongoing reads and writes in order
+                        to respect the timeout. Defaults to 1 year (i.e. wait
+                        essentially indefinitely).
     """
-    client = docker.from_env(version='auto')
+    client = docker.from_env(version='auto', timeout=timeout)
     try:
         this_container = client.containers.get(container_name)
         while this_container.status == 'running':
@@ -318,14 +329,19 @@ def dockerStop(container_name):
     dockerKill(container_name, gentleKill=True)
 
 
-def containerIsRunning(container_name):
+def containerIsRunning(container_name, timeout=365 * 24 * 60 * 60):
     """
     Checks whether the container is running or not.
     :param container_name: Name of the container being checked.
     :returns: True if status is 'running', False if status is anything else,
     and None if the container does not exist.
+    :param int timeout: Use the given timeout in seconds for interactions with
+                        the Docker daemon. Note that the underlying docker module is
+                        not always able to abort ongoing reads and writes in order
+                        to respect the timeout. Defaults to 1 year (i.e. wait
+                        essentially indefinitely).
     """
-    client = docker.from_env(version='auto')
+    client = docker.from_env(version='auto', timeout=timeout)
     try:
         this_container = client.containers.get(container_name)
         if this_container.status == 'running':


Test patch: