Repository: DataBiosphere/toil

Issue:
Restarted job forgets about user script
My script runs fine in a new job store (apart from some failing jobs due to errors from the tools they run), but when I go to restart the job, I get this:

```
WARNING:toil.leader:The jobWrapper seems to have left a log file, indicating failure: 52d0d400_4080_480d_9c53_e7a7624b20e9
WARNING:toil.leader:Reporting file: 52d0d400_4080_480d_9c53_e7a7624b20e9
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.resource:Can't find resource for leader path '/home/anovak/hgvm-graph-bakeoff-evaluations/scripts'
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/anovak/hgvm-graph-bakeoff-evaluations/scripts', name='parallelMappingEvaluation')
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ERROR:toil.job:Failed to import user module ModuleDescriptor(dirPath='/home/anovak/hgvm-graph-bakeoff-evaluations/scripts', name='parallelMappingEvaluation') from sys.path=['/usr/local/bin', '/usr/local/lib/python2.7/dist-packages/pip-8.1.2-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/mesos.interface-0.22.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/mesos-0.22.0-py2.7-linux-x86_64.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/home/anovak/hgvm-graph-bakeoff-evaluations/scripts']
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       Traceback (most recent call last):
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 316, in main
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           job = Job._loadJob(jobWrapper.command, jobStore)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 666, in _loadJob
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           userModule = cls._loadUserModule(userModule)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 640, in _loadUserModule
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           return importlib.import_module(userModule.name)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           __import__(name)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ImportError: No module named parallelMappingEvaluation
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       Exiting the worker because of a failed jobWrapper on host amn2agent3
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host amn2agent3
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 52d0d400_4080_480d_9c53_e7a7624b20e9 to 0
WARNING:toil.leader:Job: 52d0d400_4080_480d_9c53_e7a7624b20e9 is completely failed
```

It looks like Toil is trying to import the user script, and failing. But this only happens with `--restart`; it imports just fine on the first execution.

I'm running with the AzureJobStore and the MesosBatchSystem, if it matters, with `--disableCaching` and with `--workDir tmp`. I'm relying on Toil to distribute the script and a "toillib.py" file that it imports, and that works just fine on the first execution.

Restarted job forgets about user script
My script runs fine in a new job store (apart from some failing jobs due to errors from the tools they run), but when I go to restart the job, I get this:

```
WARNING:toil.leader:The jobWrapper seems to have left a log file, indicating failure: 52d0d400_4080_480d_9c53_e7a7624b20e9
WARNING:toil.leader:Reporting file: 52d0d400_4080_480d_9c53_e7a7624b20e9
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.resource:Can't find resource for leader path '/home/anovak/hgvm-graph-bakeoff-evaluations/scripts'
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/anovak/hgvm-graph-bakeoff-evaluations/scripts', name='parallelMappingEvaluation')
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ERROR:toil.job:Failed to import user module ModuleDescriptor(dirPath='/home/anovak/hgvm-graph-bakeoff-evaluations/scripts', name='parallelMappingEvaluation') from sys.path=['/usr/local/bin', '/usr/local/lib/python2.7/dist-packages/pip-8.1.2-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/mesos.interface-0.22.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/mesos-0.22.0-py2.7-linux-x86_64.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/home/anovak/hgvm-graph-bakeoff-evaluations/scripts']
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       Traceback (most recent call last):
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 316, in main
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           job = Job._loadJob(jobWrapper.command, jobStore)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 666, in _loadJob
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           userModule = cls._loadUserModule(userModule)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 640, in _loadUserModule
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           return importlib.import_module(userModule.name)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           __import__(name)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ImportError: No module named parallelMappingEvaluation
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       Exiting the worker because of a failed jobWrapper on host amn2agent3
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host amn2agent3
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 52d0d400_4080_480d_9c53_e7a7624b20e9 to 0
WARNING:toil.leader:Job: 52d0d400_4080_480d_9c53_e7a7624b20e9 is completely failed
```

It looks like Toil is trying to import the user script, and failing. But this only happens with `--restart`; it imports just fine on the first execution.

I'm running with the AzureJobStore and the MesosBatchSystem, if it matters, with `--disableCaching` and with `--workDir tmp`. I'm relying on Toil to distribute the script and a "toillib.py" file that it imports, and that works just fine on the first execution.



Discussion:
Reproduction according to @cket:

> https://github.com/cket/cgl-docker-lib/tree/reproduce-hotdeploy-bug
> 
> If you clone that, add in your AWS credentials in the test script, and do `make test` in the toil directory it will start a local mini cluster to run the toil script. Once you see that mesos has assigned the task just control-C it. When you restart that run it should fail. 

Reproduction according to @cket:

> https://github.com/cket/cgl-docker-lib/tree/reproduce-hotdeploy-bug
> 
> If you clone that, add in your AWS credentials in the test script, and do `make test` in the toil directory it will start a local mini cluster to run the toil script. Once you see that mesos has assigned the task just control-C it. When you restart that run it should fail. 


Resolving patch:
diff --git a/docker/Dockerfile.py b/docker/Dockerfile.py
--- a/docker/Dockerfile.py
+++ b/docker/Dockerfile.py
@@ -19,14 +19,14 @@
     RUN apt-get update && apt-get install -y {dependencies}
 
     # The stock pip is too old and can't install from sdist with extras
-    RUN pip install --upgrade pip
+    RUN pip install --upgrade pip==8.1.2
+
+    # Include virtualenv, as it is still the recommended way to deploy pipelines
+    RUN pip install virtualenv==15.0.3
 
     # Mesos interface dependency missing on ubuntu
     RUN pip install protobuf==3.0.0
 
-    COPY {options.sdist} .
-    RUN pip install {options.sdist}[aws,mesos,encryption,cwl]
-
     # Move the Mesos module onto the python path
     RUN ln -s /usr/lib/python2.7/site-packages/mesos /usr/local/lib/python2.7/dist-packages/mesos
 
@@ -36,4 +36,8 @@
     # An appliance may need to start more appliances, e.g. when the leader appliance launches the
     # worker appliance on a worker node. To support this, we embed a self-reference into the image:
     ENV TOIL_APPLIANCE_SELF {options.self}
+
+    # This component changes most frequently and keeping it last maximizes Docker cache hits.
+    COPY {options.sdist} .
+    RUN pip install {options.sdist}[aws,mesos,encryption,cwl]
 '''.format(**locals())).lstrip()
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -27,6 +27,7 @@
 
 from toil.lib.bioio import addLoggingOptions, getLogLevelString, setLoggingFromOptions
 from toil.realtimeLogger import RealtimeLogger
+from toil.resource import Resource
 
 logger = logging.getLogger(__name__)
 
@@ -606,7 +607,6 @@ def restart(self):
             self._setBatchSystemEnvVars()
             self._serialiseEnv()
             self._cacheAllJobs()
-
             rootJob = self._jobStore.clean(jobCache=self._jobCache)
             return self._runMainLoop(rootJob)
         finally:
@@ -635,7 +635,7 @@ def getJobStore(cls, locator):
         elif name == 'google':
             from toil.jobStores.googleJobStore import GoogleJobStore
             projectID, namePrefix = rest.split(':', 1)
-            return GoogleJobStore(namePrefix, projectID, config=config)
+            return GoogleJobStore(namePrefix, projectID)
         else:
             raise RuntimeError("Unknown job store implementation '%s'" % name)
 
@@ -669,9 +669,13 @@ def createBatchSystem(config, jobStore=None, userScript=None):
         a user script are given then the user script can be hot deployed into the workflow.
 
         :param toil.common.Config config: the current configuration
-        :param jobStores.abstractJobStore.AbstractJobStore jobStore: an instance of a jobStore
-        :param ModuleDescriptor userScript: a user supplied script to use for hot development
+
+        :param toil.jobStores.abstractJobStore.AbstractJobStore jobStore: an instance of a jobStore
+
+        :param ModuleDescriptor userScript: a handle to the Python module defining the root job
+
         :return: an instance of a concrete subclass of AbstractBatchSystem
+
         :rtype: batchSystems.abstractBatchSystem.AbstractBatchSystem
         """
         kwargs = dict(config=config,
@@ -715,9 +719,21 @@ def createBatchSystem(config, jobStore=None, userScript=None):
         logger.info('Using the %s' %
                     re.sub("([a-z])([A-Z])", "\g<1> \g<2>", batchSystemClass.__name__).lower())
 
-        if jobStore is not None and userScript is not None:
-            if not userScript.belongsToToil and batchSystemClass.supportsHotDeployment():
-                kwargs['userScript'] = userScript.saveAsResourceTo(jobStore)
+        if jobStore is not None:
+            if userScript is not None:
+                if not userScript.belongsToToil and batchSystemClass.supportsHotDeployment():
+                    userScriptResource = userScript.saveAsResourceTo(jobStore)
+                    with jobStore.writeSharedFileStream('userScript') as f:
+                        f.write(userScriptResource.pickle())
+                    kwargs['userScript'] = userScriptResource
+            else:
+                from toil.jobStores.abstractJobStore import NoSuchFileException
+                try:
+                    with jobStore.readSharedFileStream('userScript') as f:
+                        userScriptResource = Resource.unpickle(f.read())
+                    kwargs['userScript'] = userScriptResource
+                except NoSuchFileException:
+                    pass
 
         return batchSystemClass(**kwargs)
 
diff --git a/src/toil/resource.py b/src/toil/resource.py
--- a/src/toil/resource.py
+++ b/src/toil/resource.py
@@ -127,7 +127,7 @@ def register(self):
         """
         Register this resource for later retrieval via lookup(), possibly in a child process.
         """
-        os.environ[self.resourceEnvNamePrefix + self.pathHash] = self._pickle()
+        os.environ[self.resourceEnvNamePrefix + self.pathHash] = self.pickle()
 
     @classmethod
     def lookup(cls, leaderPath):
@@ -147,7 +147,7 @@ def lookup(cls, leaderPath):
             log.warn("Can't find resource for leader path '%s'", leaderPath)
             return None
         else:
-            self = cls._unpickle(s)
+            self = cls.unpickle(s)
             assert self.pathHash == pathHash
             return self
 
@@ -189,11 +189,11 @@ def localDirPath(self):
         rootDirPath = os.environ[self.rootDirPathEnvName]
         return os.path.join(rootDirPath, self.contentHash)
 
-    def _pickle(self):
+    def pickle(self):
         return self.__class__.__module__ + "." + self.__class__.__name__ + ':' + json.dumps(self)
 
     @classmethod
-    def _unpickle(cls, s):
+    def unpickle(cls, s):
         """
         :rtype: Resource
         """


Test patch:
diff --git a/src/toil/test/__init__.py b/src/toil/test/__init__.py
--- a/src/toil/test/__init__.py
+++ b/src/toil/test/__init__.py
@@ -15,17 +15,28 @@
 from __future__ import absolute_import
 import logging
 import os
+import subprocess
 import tempfile
+import threading
 import unittest
 import shutil
 import re
+import uuid
+from abc import ABCMeta, abstractmethod
 from contextlib import contextmanager
 from unittest.util import strclass
 from urllib2 import urlopen
 
+import multiprocessing
+
+import time
 from bd2k.util import less_strict_bool, memoize
 from bd2k.util.files import mkdir_p
+from bd2k.util.iterables import concat
 from bd2k.util.processes import which
+from bd2k.util.threading import ExceptionalThread
+
+from toil.version import version as toil_version
 
 from toil import toilPackageDirPath
 
@@ -152,6 +163,61 @@ def _getTestJobStorePath(self):
         os.rmdir(path)
         return path
 
+    @classmethod
+    def _getSourceDistribution(cls):
+        """
+        Find the sdist tarball for this project, check whether it is up-to date and return the
+        path to it.
+
+        :rtype: str
+        """
+        sdistPath = os.path.join(cls._projectRootPath(), 'dist', 'toil-%s.tar.gz' % toil_version)
+        assert os.path.isfile(
+            sdistPath), "Can't find Toil source distribution at %s. Run 'make sdist'." % sdistPath
+        excluded = set(cls._run('git', 'ls-files', '--others', '-i', '--exclude-standard',
+                                capture=True,
+                                cwd=cls._projectRootPath()).splitlines())
+        dirty = cls._run('find', 'src', '-type', 'f', '-newer', sdistPath,
+                         capture=True,
+                         cwd=cls._projectRootPath()).splitlines()
+        assert all(path.startswith('src') for path in dirty)
+        dirty = set(dirty)
+        dirty.difference_update(excluded)
+        assert not dirty, "Run 'make sdist'. Files newer than %s: %r" % (sdistPath, list(dirty))
+        return sdistPath
+
+    @classmethod
+    def _run(cls, command, *args, **kwargs):
+        """
+        Run a command. Convenience wrapper for subprocess.check_call and subprocess.check_output.
+
+        :param str command: The command to be run.
+
+        :param str args: Any arguments to be passed to the command.
+
+        :param Any kwargs: keyword arguments for subprocess.Popen constructor. Pass capture=True to
+                           have the process' stdout returned.
+
+        :rtype: None|str
+
+        :return: The output of the process' stdout if capture=True was passed, None otherwise.
+        """
+        args = list(concat(command, args))
+        log.info('Running %r', args)
+        capture = kwargs.pop('capture', False)
+        _input = kwargs.pop('input', None)
+        if capture:
+            kwargs['stdout'] = subprocess.PIPE
+        if _input is not None:
+            kwargs['stdin'] = subprocess.PIPE
+        popen = subprocess.Popen(args, **kwargs)
+        stdout, stderr = popen.communicate(input=_input)
+        assert stderr is None
+        if popen.returncode != 0:
+            raise subprocess.CalledProcessError(popen.returncode, args)
+        if capture:
+            return stdout
+
 
 try:
     # noinspection PyUnresolvedReferences
@@ -539,3 +605,136 @@ def tempFileContaining(content, suffix=''):
         yield path
     finally:
         os.unlink(path)
+
+
+class ApplianceTestSupport(ToilTest):
+    """
+    A Toil test that runs a user script on a minimal cluster of appliance containers,
+    i.e. one leader container and one worker container.
+    """
+
+    @contextmanager
+    def _applianceCluster(self, mounts=None, numCores=None):
+        """
+        A context manager for creating and tearing down an appliance cluster.
+
+        :param dict|None mounts: Dictionary mapping host paths to container paths. Both the leader
+               and the worker container will be started with one -v argument per dictionary entry,
+               as in -v KEY:VALUE
+
+        :param int numCores: The number of cores to be offered by the Mesos slave process running
+               in the worker container.
+
+        :rtype: (ApplianceTestSupport.Appliance, ApplianceTestSupport.Appliance)
+
+        :return: A tuple of the form `(leader, worker)` containing the Appliance instances
+                 representing the respective appliance containers
+        """
+        if numCores is None:
+            numCores = multiprocessing.cpu_count()
+        with self.LeaderThread(self, mounts) as leader:
+            with self.WorkerThread(self, mounts, numCores) as worker:
+                yield leader, worker
+
+    class Appliance(ExceptionalThread):
+        __metaclass__ = ABCMeta
+
+        @abstractmethod
+        def _getRole(self):
+            return 'leader'
+
+        @abstractmethod
+        def _containerCommand(self):
+            raise NotImplementedError
+
+        # Lock is used because subprocess is NOT thread safe: http://tinyurl.com/pkp5pgq
+        lock = threading.Lock()
+
+        def __init__(self, outer, mounts):
+            super(ApplianceTestSupport.Appliance, self).__init__()
+            self.outer = outer
+            self.mounts = mounts
+            self.containerName = str(uuid.uuid4())
+            self.popen = None
+
+        def __enter__(self):
+            with self.lock:
+                image = self._getApplianceImage()
+                # Omitting --rm, it's unreliable, see https://github.com/docker/docker/issues/16575
+                args = list(concat('docker', 'run',
+                                   '--net=host',
+                                   '-i',
+                                   '--name=' + self.containerName,
+                                   ['--volume=%s:%s' % mount for mount in self.mounts.iteritems()],
+                                   image,
+                                   self._containerCommand()))
+                log.info('Running %r', args)
+                self.popen = subprocess.Popen(args)
+            self.start()
+            self.__wait_running()
+            return self
+
+        # noinspection PyUnusedLocal
+        def __exit__(self, exc_type, exc_val, exc_tb):
+            try:
+                subprocess.check_call(['docker', 'stop', self.containerName])
+                self.join()
+            finally:
+                subprocess.check_call(['docker', 'rm', '-f', self.containerName])
+            return False  # don't swallow exception
+
+        def __wait_running(self):
+            log.info("Waiting for %s container process to appear. "
+                     "Expect to see 'Error: No such image or container'.", self._getRole())
+            while self.isAlive():
+                try:
+                    running = subprocess.check_output(
+                        ['docker', 'inspect', '--format={{ .State.Running }}',
+                         self.containerName]).strip()
+                except subprocess.CalledProcessError:
+                    pass
+                else:
+                    if 'true' == running:
+                        break
+                time.sleep(1)
+
+        def _getApplianceImage(self):
+            image, tag = os.environ['TOIL_APPLIANCE_SELF'].rsplit(':', 1)
+            return image + '-' + self._getRole() + ':' + tag
+
+        def tryRun(self):
+            self.popen.wait()
+            log.info('Exiting %s', self.__class__.__name__)
+
+        def runOnAppliance(self, *args, **kwargs):
+            # noinspection PyProtectedMember
+            self.outer._run('docker', 'exec', '-i', self.containerName, *args, **kwargs)
+
+        def writeToAppliance(self, path, contents):
+            self.runOnAppliance('tee', path, input=contents)
+
+    class LeaderThread(Appliance):
+        def _getRole(self):
+            return 'leader'
+
+        def _containerCommand(self):
+            return ['--registry=in_memory',
+                    '--ip=127.0.0.1',
+                    '--port=5050',
+                    '--allocation_interval=500ms']
+
+    class WorkerThread(Appliance):
+
+        def __init__(self, outer, mounts, numCores):
+            self.numCores = numCores
+            super(ApplianceTestSupport.WorkerThread, self).__init__(outer, mounts)
+
+        def _getRole(self):
+            return 'worker'
+
+        def _containerCommand(self):
+            return ['--work_dir=/var/lib/mesos',
+                    '--ip=127.0.0.1',
+                    '--master=127.0.0.1:5050',
+                    '--attributes=preemptable:False',
+                    '--resources=cpus(*):%i' % self.numCores]
diff --git a/src/toil/test/provisioners/cgcloud/provisionerTest.py b/src/toil/test/provisioners/cgcloud/provisionerTest.py
--- a/src/toil/test/provisioners/cgcloud/provisionerTest.py
+++ b/src/toil/test/provisioners/cgcloud/provisionerTest.py
@@ -198,21 +198,6 @@ def _runScript(self, toilOptions):
         """
         raise NotImplementedError()
 
-    @classmethod
-    def _run(cls, *args, **kwargs):
-        log.info('Running %r', args)
-        try:
-            capture = kwargs['capture']
-        except KeyError:
-            capture = False
-        else:
-            del kwargs['capture']
-        if capture:
-            return subprocess.check_output(args, **kwargs)
-        else:
-            subprocess.check_call(args, **kwargs)
-            return None
-
     @classmethod
     def _cgcloud(cls, *args):
         if not cls.dryRun:
@@ -240,22 +225,6 @@ def _rsync(cls, role, *args):
     def _leader(cls, *args, **kwargs):
         cls._ssh('toil-leader', *args, **kwargs)
 
-    @classmethod
-    def _getSourceDistribution(cls):
-        sdistPath = os.path.join(cls._projectRootPath(), 'dist', 'toil-%s.tar.gz' % toil_version)
-        assert os.path.isfile(sdistPath), "Can't find Toil source distribution at %s. Run 'make sdist'." % sdistPath
-        excluded = set(cls._run('git', 'ls-files', '--others', '-i', '--exclude-standard',
-                                capture=True,
-                                cwd=cls._projectRootPath()).splitlines())
-        dirty = cls._run('find', 'src', '-type', 'f', '-newer', sdistPath,
-                         capture=True,
-                         cwd=cls._projectRootPath()).splitlines()
-        assert all(path.startswith('src') for path in dirty)
-        dirty = set(dirty)
-        dirty.difference_update(excluded)
-        assert not dirty, "Run 'make sdist'. Files newer than %s: %r" % (sdistPath, list(dirty))
-        return sdistPath
-
 
 class CGCloudRNASeqTest(AbstractCGCloudProvisionerTest):
     def __init__(self, name):
diff --git a/src/toil/test/src/hotDeploymentTest.py b/src/toil/test/src/hotDeploymentTest.py
new file mode 100644
--- /dev/null
+++ b/src/toil/test/src/hotDeploymentTest.py
@@ -0,0 +1,59 @@
+import logging
+from inspect import getsource
+from subprocess import CalledProcessError
+from textwrap import dedent
+
+from bd2k.util.iterables import concat
+
+from toil.test import needs_mesos, ApplianceTestSupport
+
+log = logging.getLogger(__name__)
+
+
+@needs_mesos
+class HotDeploymentTest(ApplianceTestSupport):
+    def setUp(self):
+        logging.basicConfig(level=logging.INFO)
+        super(HotDeploymentTest, self).setUp()
+
+    def test(self):
+        dataDirPath = self._createTempDir(purpose='data')
+        with self._applianceCluster(mounts={dataDirPath: '/data'}) as (leader, worker):
+            leader.runOnAppliance('virtualenv',
+                                  '--system-site-packages',
+                                  '--never-download',
+                                  'venv')
+            leader.runOnAppliance('venv/bin/pip', 'list')
+
+            def userScript():
+                from toil.job import Job
+                from toil.common import Toil
+
+                # noinspection PyUnusedLocal
+                def job(job, disk='10M', cores=1, memory='10M'):
+                    # make job fail during the first invocation
+                    assert job.fileStore.jobStore.config.restart
+
+                if __name__ == '__main__':
+                    options = Job.Runner.getDefaultArgumentParser().parse_args()
+                    with Toil(options) as toil:
+                        if toil.config.restart:
+                            toil.restart()
+                        else:
+                            toil.start(Job.wrapJobFn(job))
+
+            package = 'venv/lib/python2.7/site-packages/foo'
+            leader.runOnAppliance('mkdir', '-p', package)
+            leader.writeToAppliance(package + '/__init__.py', '')
+            script = dedent('\n'.join(getsource(userScript).split('\n')[1:]))
+            leader.writeToAppliance(package + '/bar.py', script)
+            pythonArgs = ['venv/bin/python', '-m', 'foo.bar']
+            toilArgs = ['--logDebug',
+                        '--batchSystem=mesos',
+                        '--mesosMaster=localhost:5050',
+                        '--defaultMemory=10M',
+                        '/data/jobstore']
+            command = concat(pythonArgs, toilArgs)
+            self.assertRaises(CalledProcessError, leader.runOnAppliance, *command)
+            command = concat(pythonArgs, '--restart', toilArgs)
+            leader.runOnAppliance(*command)