Repository: DataBiosphere/toil

Issue:
Restarted job forgets about user script
My script runs fine in a new job store (apart from some failing jobs due to errors from the tools they run), but when I go to restart the job, I get this:

```
WARNING:toil.leader:The jobWrapper seems to have left a log file, indicating failure: 52d0d400_4080_480d_9c53_e7a7624b20e9
WARNING:toil.leader:Reporting file: 52d0d400_4080_480d_9c53_e7a7624b20e9
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.resource:Can't find resource for leader path '/home/anovak/hgvm-graph-bakeoff-evaluations/scripts'
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/anovak/hgvm-graph-bakeoff-evaluations/scripts', name='parallelMappingEvaluation')
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ERROR:toil.job:Failed to import user module ModuleDescriptor(dirPath='/home/anovak/hgvm-graph-bakeoff-evaluations/scripts', name='parallelMappingEvaluation') from sys.path=['/usr/local/bin', '/usr/local/lib/python2.7/dist-packages/pip-8.1.2-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/mesos.interface-0.22.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/mesos-0.22.0-py2.7-linux-x86_64.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/home/anovak/hgvm-graph-bakeoff-evaluations/scripts']
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       Traceback (most recent call last):
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 316, in main
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           job = Job._loadJob(jobWrapper.command, jobStore)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 666, in _loadJob
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           userModule = cls._loadUserModule(userModule)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 640, in _loadUserModule
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           return importlib.import_module(userModule.name)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           __import__(name)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ImportError: No module named parallelMappingEvaluation
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       Exiting the worker because of a failed jobWrapper on host amn2agent3
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host amn2agent3
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 52d0d400_4080_480d_9c53_e7a7624b20e9 to 0
WARNING:toil.leader:Job: 52d0d400_4080_480d_9c53_e7a7624b20e9 is completely failed
```

It looks like Toil is trying to import the user script, and failing. But this only happens with `--restart`; it imports just fine on the first execution.

I'm running with the AzureJobStore and the MesosBatchSystem, if it matters, with `--disableCaching` and with `--workDir tmp`. I'm relying on Toil to distribute the script and a "toillib.py" file that it imports, and that works just fine on the first execution.

Restarted job forgets about user script
My script runs fine in a new job store (apart from some failing jobs due to errors from the tools they run), but when I go to restart the job, I get this:

```
WARNING:toil.leader:The jobWrapper seems to have left a log file, indicating failure: 52d0d400_4080_480d_9c53_e7a7624b20e9
WARNING:toil.leader:Reporting file: 52d0d400_4080_480d_9c53_e7a7624b20e9
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.resource:Can't find resource for leader path '/home/anovak/hgvm-graph-bakeoff-evaluations/scripts'
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/anovak/hgvm-graph-bakeoff-evaluations/scripts', name='parallelMappingEvaluation')
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ERROR:toil.job:Failed to import user module ModuleDescriptor(dirPath='/home/anovak/hgvm-graph-bakeoff-evaluations/scripts', name='parallelMappingEvaluation') from sys.path=['/usr/local/bin', '/usr/local/lib/python2.7/dist-packages/pip-8.1.2-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/mesos.interface-0.22.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/mesos-0.22.0-py2.7-linux-x86_64.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/home/anovak/hgvm-graph-bakeoff-evaluations/scripts']
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       Traceback (most recent call last):
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 316, in main
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           job = Job._loadJob(jobWrapper.command, jobStore)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 666, in _loadJob
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           userModule = cls._loadUserModule(userModule)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 640, in _loadUserModule
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           return importlib.import_module(userModule.name)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:         File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:           __import__(name)
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ImportError: No module named parallelMappingEvaluation
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       Exiting the worker because of a failed jobWrapper on host amn2agent3
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host amn2agent3
WARNING:toil.leader:52d0d400_4080_480d_9c53_e7a7624b20e9:       WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 52d0d400_4080_480d_9c53_e7a7624b20e9 to 0
WARNING:toil.leader:Job: 52d0d400_4080_480d_9c53_e7a7624b20e9 is completely failed
```

It looks like Toil is trying to import the user script, and failing. But this only happens with `--restart`; it imports just fine on the first execution.

I'm running with the AzureJobStore and the MesosBatchSystem, if it matters, with `--disableCaching` and with `--workDir tmp`. I'm relying on Toil to distribute the script and a "toillib.py" file that it imports, and that works just fine on the first execution.



Discussion:
Reproduction according to @cket:

> https://github.com/cket/cgl-docker-lib/tree/reproduce-hotdeploy-bug
> 
> If you clone that, add in your AWS credentials in the test script, and do `make test` in the toil directory it will start a local mini cluster to run the toil script. Once you see that mesos has assigned the task just control-C it. When you restart that run it should fail. 

Reproduction according to @cket:

> https://github.com/cket/cgl-docker-lib/tree/reproduce-hotdeploy-bug
> 
> If you clone that, add in your AWS credentials in the test script, and do `make test` in the toil directory it will start a local mini cluster to run the toil script. Once you see that mesos has assigned the task just control-C it. When you restart that run it should fail. 


Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -27,6 +27,7 @@
 
 from toil.lib.bioio import addLoggingOptions, getLogLevelString, setLoggingFromOptions
 from toil.realtimeLogger import RealtimeLogger
+from toil.resource import Resource
 
 logger = logging.getLogger(__name__)
 
@@ -579,7 +580,6 @@ def restart(self):
             self._setBatchSystemEnvVars()
             self._serialiseEnv()
             self._cacheAllJobs()
-
             rootJob = self._jobStore.clean(jobCache=self._jobCache)
             return self._runMainLoop(rootJob)
         finally:
@@ -619,7 +619,7 @@ def getJobStore(locator):
         elif name == 'google':
             from toil.jobStores.googleJobStore import GoogleJobStore
             projectID, namePrefix = rest.split(':', 1)
-            return GoogleJobStore(namePrefix, projectID, config=config)
+            return GoogleJobStore(namePrefix, projectID)
         else:
             raise RuntimeError("Unknown job store implementation '%s'" % name)
 
@@ -636,9 +636,13 @@ def createBatchSystem(config, jobStore=None, userScript=None):
         a user script are given then the user script can be hot deployed into the workflow. 
 
         :param toil.common.Config config: the current configuration
-        :param jobStores.abstractJobStore.AbstractJobStore jobStore: an instance of a jobStore
-        :param ModuleDescriptor userScript: a user supplied script to use for hot development
+
+        :param toil.jobStores.abstractJobStore.AbstractJobStore jobStore: an instance of a jobStore
+
+        :param ModuleDescriptor userScript: a handle to the Python module defining the root job
+
         :return: an instance of a concrete subclass of AbstractBatchSystem
+
         :rtype: batchSystems.abstractBatchSystem.AbstractBatchSystem
         """
         kwargs = dict(config=config,
@@ -682,9 +686,21 @@ def createBatchSystem(config, jobStore=None, userScript=None):
         logger.info('Using the %s' %
                     re.sub("([a-z])([A-Z])", "\g<1> \g<2>", batchSystemClass.__name__).lower())
 
-        if jobStore is not None and userScript is not None:
-            if not userScript.belongsToToil and batchSystemClass.supportsHotDeployment():
-                kwargs['userScript'] = userScript.saveAsResourceTo(jobStore)
+        if jobStore is not None:
+            if userScript is not None:
+                if not userScript.belongsToToil and batchSystemClass.supportsHotDeployment():
+                    userScriptResource = userScript.saveAsResourceTo(jobStore)
+                    with jobStore.writeSharedFileStream('userScript') as f:
+                        f.write(userScriptResource.pickle())
+                    kwargs['userScript'] = userScriptResource
+            else:
+                from toil.jobStores.abstractJobStore import NoSuchFileException
+                try:
+                    with jobStore.readSharedFileStream('userScript') as f:
+                        userScriptResource = Resource.unpickle(f.read())
+                    kwargs['userScript'] = userScriptResource
+                except NoSuchFileException:
+                    pass
 
         return batchSystemClass(**kwargs)
 
diff --git a/src/toil/resource.py b/src/toil/resource.py
--- a/src/toil/resource.py
+++ b/src/toil/resource.py
@@ -127,7 +127,7 @@ def register(self):
         """
         Register this resource for later retrieval via lookup(), possibly in a child process.
         """
-        os.environ[self.resourceEnvNamePrefix + self.pathHash] = self._pickle()
+        os.environ[self.resourceEnvNamePrefix + self.pathHash] = self.pickle()
 
     @classmethod
     def lookup(cls, leaderPath):
@@ -147,7 +147,7 @@ def lookup(cls, leaderPath):
             log.warn("Can't find resource for leader path '%s'", leaderPath)
             return None
         else:
-            self = cls._unpickle(s)
+            self = cls.unpickle(s)
             assert self.pathHash == pathHash
             return self
 
@@ -189,11 +189,11 @@ def localDirPath(self):
         rootDirPath = os.environ[self.rootDirPathEnvName]
         return os.path.join(rootDirPath, self.contentHash)
 
-    def _pickle(self):
+    def pickle(self):
         return self.__class__.__module__ + "." + self.__class__.__name__ + ':' + json.dumps(self)
 
     @classmethod
-    def _unpickle(cls, s):
+    def unpickle(cls, s):
         """
         :rtype: Resource
         """


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -854,10 +854,7 @@ def testInlinedFiles(self):
                     self.assertEqual(s, f.read())
 
     def testInaccessableLocation(self):
-        # This could break if 1000genomes enables GetBucketLocation in their S3 bucket policy,
-        # see https://rt.sanger.ac.uk/Ticket/Display.html?id=534925. If that happens, we'll have
-        # to set up or mock a bucket that disallows it.
-        url = 's3://1000genomes/20131219.populations.tsv'
+        url = 's3://cgl-toil-tests-disallow-getbucketlocation/README'
         with patch('toil.jobStores.aws.jobStore.log') as mock_log:
             jobStoreID = self.master.importFile(url)
             self.assertTrue(self.master.fileExists(jobStoreID))