Repository: DataBiosphere/toil

Issue:
Retry retriable Docker failures (in _fixPermissions, etc)
At medium-large scale, especially if you launch a ton of containers at once, you can end up running up against the containerd launch timeout. This happened especially often previous to the 1.12 series (see docker issue #22226), but I've been getting these even after the AWS provisioner switched over to a CoreOS version containing 1.12.3. (My appliance contains a 1.12.3 client, too). This is obviously a Docker issue, but could Toil retry retriable Docker failures like this? The symptom is that Docker exits with return code 125 (ECANCELLED), and outputs an error message to stderr:

```
30d03dba-1aea-49a1-85cf-197edc816d25    docker: Error response from daemon: containerd: container did not start before the specified timeout.
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:LOG-TO-MASTER: Job 'CactusBarWrapper' 30d03dba-1aea-49a1-85cf-197edc816d25 used 49.73% (7.0 GB [7561211904B] used, 14.2 GB [15204352000B] requested) at the end of its run.
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/mesos/slaves/f15072d4-b9e6-44df-a669-66e986da45f7-S448/frameworks/f15072d4-b9e6-44df-a669-66e986da45f7-0032/executors/toil-9916/runs/51d9f82e-6def-4666-85f4-a95fc40e537c/toil-43764132
-2f0d-41c5-9984-a2f78d9eb7e2/cache-43764132-2f0d-41c5-9984-a2f78d9eb7e2/.cacheLock
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:LOG-TO-MASTER: Successfully deleted local copies of file with ID '3d45f54d-3186-470c-bb60-7b3a4da130f0'.
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Released lock
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/mesos/slaves/f15072d4-b9e6-44df-a669-66e986da45f7-S448/frameworks/f15072d4-b9e6-44df-a669-66e986da45f7-0032/executors/toil-9916/runs/51d9f82e-6def-4666-85f4-a95fc40e537c/toil-43764132
-2f0d-41c5-9984-a2f78d9eb7e2/cache-43764132-2f0d-41c5-9984-a2f78d9eb7e2/.cacheLock
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Released lock
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/mesos/slaves/f15072d4-b9e6-44df-a669-66e986da45f7-S448/frameworks/f15072d4-b9e6-44df-a669-66e986da45f7-0032/executors/toil-9916/runs/51d9f82e-6def-4666-85f4-a95fc40e537c/toil-43764132
-2f0d-41c5-9984-a2f78d9eb7e2/cache-43764132-2f0d-41c5-9984-a2f78d9eb7e2/.cacheLock
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Released lock
30d03dba-1aea-49a1-85cf-197edc816d25    Traceback (most recent call last):
30d03dba-1aea-49a1-85cf-197edc816d25      File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 340, in main
30d03dba-1aea-49a1-85cf-197edc816d25        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
30d03dba-1aea-49a1-85cf-197edc816d25      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1288, in _runner
30d03dba-1aea-49a1-85cf-197edc816d25        returnValues = self._run(jobGraph, fileStore)
30d03dba-1aea-49a1-85cf-197edc816d25      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1235, in _run
30d03dba-1aea-49a1-85cf-197edc816d25        return self.run(fileStore)
30d03dba-1aea-49a1-85cf-197edc816d25      File "/venv/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 773, in run
30d03dba-1aea-49a1-85cf-197edc816d25        
30d03dba-1aea-49a1-85cf-197edc816d25      File "/venv/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 763, in runBarForJob
30d03dba-1aea-49a1-85cf-197edc816d25        def run(self, fileStore):
30d03dba-1aea-49a1-85cf-197edc816d25      File "/venv/lib/python2.7/site-packages/cactus/shared/common.py", line 505, in runCactusBar
30d03dba-1aea-49a1-85cf-197edc816d25        job_name=jobName, fileStore=fileStore, input_size=inputSize)
30d03dba-1aea-49a1-85cf-197edc816d25      File "/venv/lib/python2.7/site-packages/cactus/shared/common.py", line 1006, in cactus_call
30d03dba-1aea-49a1-85cf-197edc816d25        if check_output:
30d03dba-1aea-49a1-85cf-197edc816d25      File "/usr/local/lib/python2.7/dist-packages/toil/lib/docker.py", line 230, in _fixPermissions
30d03dba-1aea-49a1-85cf-197edc816d25        subprocess.check_call(command)
30d03dba-1aea-49a1-85cf-197edc816d25      File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
30d03dba-1aea-49a1-85cf-197edc816d25        raise CalledProcessError(retcode, cmd)
30d03dba-1aea-49a1-85cf-197edc816d25    CalledProcessError: Command '['docker', 'run', '--log-driver=none', '-v', '/var/lib/mesos/slaves/f15072d4-b9e6-44df-a669-66e986da45f7-S448/frameworks/f15072d4-b9e6-44df-a669-66e986da45f7-0032/executors/toil-9916/runs/51d9f82e-6def-4666-85f4-a
95fc40e537c/toil-43764132-2f0d-41c5-9984-a2f78d9eb7e2/tmpx65pGh/2bbeb24b-b855-41b4-a9cf-2a0c35e095a1:/data', '--rm', '--entrypoint=chown', 'quay.io/joelarmstrong/cactus:latest', '-R', '0:0', '/data']' returned non-zero exit status 125
30d03dba-1aea-49a1-85cf-197edc816d25    ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-36-131.us-west-2.compute.internal
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.jobStores.aws.jobStore:Loaded job 30d03dba-1aea-49a1-85cf-197edc816d25
30d03dba-1aea-49a1-85cf-197edc816d25    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarWrapper' 30d03dba-1aea-49a1-85cf-197edc816d25 with ID 30d03dba-1aea-49a1-85cf-197edc816d25 to 1
Added job: 'CactusBarWrapper' 30d03dba-1aea-49a1-85cf-197edc816d25 to active jobs
```

(Incidentally, I don't think _fixPermissions actually needs to be run if Toil is being run in the toil appliance. Maybe it could detect that it's being run as root and not bother changing any permissions?)
Retry retriable Docker failures (in _fixPermissions, etc)
At medium-large scale, especially if you launch a ton of containers at once, you can end up running up against the containerd launch timeout. This happened especially often previous to the 1.12 series (see docker issue #22226), but I've been getting these even after the AWS provisioner switched over to a CoreOS version containing 1.12.3. (My appliance contains a 1.12.3 client, too). This is obviously a Docker issue, but could Toil retry retriable Docker failures like this? The symptom is that Docker exits with return code 125 (ECANCELLED), and outputs an error message to stderr:

```
30d03dba-1aea-49a1-85cf-197edc816d25    docker: Error response from daemon: containerd: container did not start before the specified timeout.
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:LOG-TO-MASTER: Job 'CactusBarWrapper' 30d03dba-1aea-49a1-85cf-197edc816d25 used 49.73% (7.0 GB [7561211904B] used, 14.2 GB [15204352000B] requested) at the end of its run.
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/mesos/slaves/f15072d4-b9e6-44df-a669-66e986da45f7-S448/frameworks/f15072d4-b9e6-44df-a669-66e986da45f7-0032/executors/toil-9916/runs/51d9f82e-6def-4666-85f4-a95fc40e537c/toil-43764132
-2f0d-41c5-9984-a2f78d9eb7e2/cache-43764132-2f0d-41c5-9984-a2f78d9eb7e2/.cacheLock
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:LOG-TO-MASTER: Successfully deleted local copies of file with ID '3d45f54d-3186-470c-bb60-7b3a4da130f0'.
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Released lock
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/mesos/slaves/f15072d4-b9e6-44df-a669-66e986da45f7-S448/frameworks/f15072d4-b9e6-44df-a669-66e986da45f7-0032/executors/toil-9916/runs/51d9f82e-6def-4666-85f4-a95fc40e537c/toil-43764132
-2f0d-41c5-9984-a2f78d9eb7e2/cache-43764132-2f0d-41c5-9984-a2f78d9eb7e2/.cacheLock
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Released lock
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/mesos/slaves/f15072d4-b9e6-44df-a669-66e986da45f7-S448/frameworks/f15072d4-b9e6-44df-a669-66e986da45f7-0032/executors/toil-9916/runs/51d9f82e-6def-4666-85f4-a95fc40e537c/toil-43764132
-2f0d-41c5-9984-a2f78d9eb7e2/cache-43764132-2f0d-41c5-9984-a2f78d9eb7e2/.cacheLock
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.fileStore:CACHE: Released lock
30d03dba-1aea-49a1-85cf-197edc816d25    Traceback (most recent call last):
30d03dba-1aea-49a1-85cf-197edc816d25      File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 340, in main
30d03dba-1aea-49a1-85cf-197edc816d25        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
30d03dba-1aea-49a1-85cf-197edc816d25      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1288, in _runner
30d03dba-1aea-49a1-85cf-197edc816d25        returnValues = self._run(jobGraph, fileStore)
30d03dba-1aea-49a1-85cf-197edc816d25      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1235, in _run
30d03dba-1aea-49a1-85cf-197edc816d25        return self.run(fileStore)
30d03dba-1aea-49a1-85cf-197edc816d25      File "/venv/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 773, in run
30d03dba-1aea-49a1-85cf-197edc816d25        
30d03dba-1aea-49a1-85cf-197edc816d25      File "/venv/lib/python2.7/site-packages/cactus/pipeline/cactus_workflow.py", line 763, in runBarForJob
30d03dba-1aea-49a1-85cf-197edc816d25        def run(self, fileStore):
30d03dba-1aea-49a1-85cf-197edc816d25      File "/venv/lib/python2.7/site-packages/cactus/shared/common.py", line 505, in runCactusBar
30d03dba-1aea-49a1-85cf-197edc816d25        job_name=jobName, fileStore=fileStore, input_size=inputSize)
30d03dba-1aea-49a1-85cf-197edc816d25      File "/venv/lib/python2.7/site-packages/cactus/shared/common.py", line 1006, in cactus_call
30d03dba-1aea-49a1-85cf-197edc816d25        if check_output:
30d03dba-1aea-49a1-85cf-197edc816d25      File "/usr/local/lib/python2.7/dist-packages/toil/lib/docker.py", line 230, in _fixPermissions
30d03dba-1aea-49a1-85cf-197edc816d25        subprocess.check_call(command)
30d03dba-1aea-49a1-85cf-197edc816d25      File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
30d03dba-1aea-49a1-85cf-197edc816d25        raise CalledProcessError(retcode, cmd)
30d03dba-1aea-49a1-85cf-197edc816d25    CalledProcessError: Command '['docker', 'run', '--log-driver=none', '-v', '/var/lib/mesos/slaves/f15072d4-b9e6-44df-a669-66e986da45f7-S448/frameworks/f15072d4-b9e6-44df-a669-66e986da45f7-0032/executors/toil-9916/runs/51d9f82e-6def-4666-85f4-a
95fc40e537c/toil-43764132-2f0d-41c5-9984-a2f78d9eb7e2/tmpx65pGh/2bbeb24b-b855-41b4-a9cf-2a0c35e095a1:/data', '--rm', '--entrypoint=chown', 'quay.io/joelarmstrong/cactus:latest', '-R', '0:0', '/data']' returned non-zero exit status 125
30d03dba-1aea-49a1-85cf-197edc816d25    ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-36-131.us-west-2.compute.internal
30d03dba-1aea-49a1-85cf-197edc816d25    DEBUG:toil.jobStores.aws.jobStore:Loaded job 30d03dba-1aea-49a1-85cf-197edc816d25
30d03dba-1aea-49a1-85cf-197edc816d25    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusBarWrapper' 30d03dba-1aea-49a1-85cf-197edc816d25 with ID 30d03dba-1aea-49a1-85cf-197edc816d25 to 1
Added job: 'CactusBarWrapper' 30d03dba-1aea-49a1-85cf-197edc816d25 to active jobs
```

(Incidentally, I don't think _fixPermissions actually needs to be run if Toil is being run in the toil appliance. Maybe it could detect that it's being run as root and not bother changing any permissions?)


Discussion:

Resolving patch:
diff --git a/src/toil/lib/__init__.py b/src/toil/lib/__init__.py
--- a/src/toil/lib/__init__.py
+++ b/src/toil/lib/__init__.py
@@ -12,3 +12,22 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 from __future__ import absolute_import
+
+import subprocess
+
+FORGO = 0
+STOP = 1
+RM = 2
+
+
+def dockerPredicate(e):
+    """
+    Used to ensure Docker exceptions are retried if appropriate
+
+    :param e: Exception
+    :return: True if e retriable, else False
+    """
+    if not isinstance(e, subprocess.CalledProcessError):
+        return False
+    if e.returncode == 125:
+        return True
diff --git a/src/toil/lib/docker.py b/src/toil/lib/docker.py
--- a/src/toil/lib/docker.py
+++ b/src/toil/lib/docker.py
@@ -14,10 +14,14 @@ def toil_job(job):
 """
 import base64
 import logging
-import subprocess
-import pipes
 import os
+import pipes
+import uuid
+
 from bd2k.util.exceptions import require
+from bd2k.util.retry import retry
+
+from toil.lib import *
 
 _logger = logging.getLogger(__name__)
 
@@ -162,23 +166,20 @@ def _docker(job,
         call = baseDockerCall + [tool] + parameters
     _logger.info("Calling docker with " + repr(call))
 
+    params = {}
     if outfile:
-        subprocess.check_call(call, stdout=outfile)
-        _fixPermissions(tool=tool, workDir=workDir)
+        params['stdout'] = outfile
+    if checkOutput:
+        callMethod = subprocess.check_output
     else:
-        if checkOutput:
-            output = subprocess.check_output(call)
-            _fixPermissions(tool=tool, workDir=workDir)
-            return output
-
-        else:
-            subprocess.check_call(call)
-            _fixPermissions(tool=tool, workDir=workDir)
+        callMethod = subprocess.check_call
 
+    for attempt in retry(predicate=dockerPredicate):
+        with attempt:
+            out = callMethod(call, **params)
 
-FORGO = 0
-STOP = 1
-RM = 2
+    _fixPermissions(tool=tool, workDir=workDir)
+    return out
 
 
 def _dockerKill(containerName, action):
@@ -203,7 +204,9 @@ def _dockerKill(containerName, action):
                          containerName)
             if running and action >= STOP:
                 _logger.info('Stopping container "%s".', containerName)
-                subprocess.check_call(['docker', 'stop', containerName])
+                for attempt in retry(predicate=dockerPredicate):
+                    with attempt:
+                        subprocess.check_call(['docker', 'stop', containerName])
             else:
                 _logger.info('The container "%s" was not found to be running.', containerName)
             if action >= RM:
@@ -212,12 +215,13 @@ def _dockerKill(containerName, action):
                 running = _containerIsRunning(containerName)
                 if running is not None:
                     _logger.info('Removing container "%s".', containerName)
-                    subprocess.check_call(['docker', 'rm', '-f', containerName])
+                    for attempt in retry(predicate=dockerPredicate):
+                        with attempt:
+                            subprocess.check_call(['docker', 'rm', '-f', containerName])
                 else:
                     _logger.info('The container "%s" was not found on the system.  Nothing to remove.',
                                  containerName)
 
-
 def _fixPermissions(tool, workDir):
     """
     Fix permission of a mounted Docker directory by reusing the tool to change ownership.
@@ -228,11 +232,17 @@ def _fixPermissions(tool, workDir):
     :param str tool: Name of tool
     :param str workDir: Path of work directory to recursively chown
     """
+    if os.geteuid() == 0:
+        # we're running as root so this chown is redundant
+        return
+
     baseDockerCall = ['docker', 'run', '--log-driver=none',
                       '-v', os.path.abspath(workDir) + ':/data', '--rm', '--entrypoint=chown']
     stat = os.stat(workDir)
     command = baseDockerCall + [tool] + ['-R', '{}:{}'.format(stat.st_uid, stat.st_gid), '/data']
-    subprocess.check_call(command)
+    for attempt in retry(predicate=dockerPredicate):
+        with attempt:
+            subprocess.check_call(command)
 
 
 def _getContainerName(job):
@@ -248,8 +258,10 @@ def _containerIsRunning(container_name):
     :rtype: bool
     """
     try:
-        output = subprocess.check_output(['docker', 'inspect', '--format', '{{.State.Running}}',
-                                          container_name]).strip()
+        for attempt in retry(predicate=dockerPredicate):
+            with attempt:
+                output = subprocess.check_output(['docker', 'inspect', '--format', '{{.State.Running}}',
+                                                  container_name]).strip()
     except subprocess.CalledProcessError:
         # This will be raised if the container didn't exist.
         _logger.debug("'docker inspect' failed. Assuming container %s doesn't exist.", container_name,


Test patch:
diff --git a/src/toil/test/lib/dockerTest.py b/src/toil/test/lib/dockerTest.py
--- a/src/toil/test/lib/dockerTest.py
+++ b/src/toil/test/lib/dockerTest.py
@@ -2,10 +2,10 @@
 import signal
 import time
 import uuid
-from pwd import getpwuid
 from threading import Thread
 
 import os
+from pwd import getpwuid
 from bd2k.util.files import mkdir_p
 from toil.job import Job
 from toil.leader import FailedJobsException
@@ -176,13 +176,17 @@ def _testDockerPipeChainFn(job):
 
 
 def _testDockerPermissions(job):
-    def ownerName(filename):
-        return getpwuid(os.stat(filename).st_uid).pw_name
-
     testDir = job.fileStore.getLocalTempDir()
     dockerCall(job, tool='ubuntu', workDir=testDir, parameters=[['touch', '/data/test.txt']])
     outFile = os.path.join(testDir, 'test.txt')
     assert os.path.exists(outFile)
-    assert not "root" == ownerName(outFile)
+    assert not ownerName(outFile) == "root"
 
 
+def ownerName(filename):
+    """
+    Determines a given file's owner
+    :param str filename: path to a file
+    :return: name of filename's owner
+    """
+    return getpwuid(os.stat(filename).st_uid).pw_name