Repository: DataBiosphere/toil

Issue:
Passing a Job class to another job to execute later sometimes crashes under hot-deployment
When a job B is passed a Job instance C created in a previous job A, then adds that Job (C) as a successor, C can sometimes fail to import itself when run under hot-deployment. Specifically, it will fail if run on a different machine than job A. That description might be a bit confusing, so consider this pseudocode:

```python
class A(Job):
    def run(self, fileStore):
         self.addChild(B(C('foo')))

class B(Job):
    def __init__(self, job):
         self.successorJob = job

    def run(self, fileStore):
         self.addChild(self.successorJob)

class C(Job):
    def run(self, fileStore):
         self.logToMaster('hi')
```

I'm aware that the "correct" way of doing this is to set C as a child of B during A's run method. In my case, though, B is actually a pipeline of a few jobs that handle constructing and initializing a Service, and needs to get run in quite a few places--which can lead to a lot of duplicated code if its structure had to be redefined each time. I think it can't be encapsulated, unfortunately, because the service needs to still be running during job C and its successors.

I think even though this is a really esoteric use-case, with a good workaround, it's worth fixing  because it works perfectly without hot-deployment, and it's pretty difficult to track down where things went wrong given just the error message.

The import failure looks something like this:

```
abdfc318-2389-4d66-b853-16ed091e8489    DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/tmp/tmp2qLnZC/74d6192189c43680877e98559bfad773', name='cactus.pip
eline.cactus_workflow', fromVirtualEnv=False).
abdfc318-2389-4d66-b853-16ed091e8489    WARNING:toil.resource:Can't find resource for leader path '/tmp/tmp2qLnZC/74d6192189c43680877e98559bfad773/cactus'
abdfc318-2389-4d66-b853-16ed091e8489    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/tmp/tmp2qLnZC/74d6192189c43680877e98559bfad773', name='c
actus.pipeline.cactus_workflow', fromVirtualEnv=False)
abdfc318-2389-4d66-b853-16ed091e8489    ERROR:toil.resource:Failed to import user module ModuleDescriptor(dirPath='/tmp/tmp2qLnZC/74d6192189c43680877e98559bfad773', na
me='cactus.pipeline.cactus_workflow', fromVirtualEnv=False) from sys.path (['/usr/local/bin', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/l
ib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/t
mp/tmp2qLnZC/74d6192189c43680877e98559bfad773']).
abdfc318-2389-4d66-b853-16ed091e8489    Traceback (most recent call last):
abdfc318-2389-4d66-b853-16ed091e8489      File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 325, in main
abdfc318-2389-4d66-b853-16ed091e8489        job = Job._loadJob(jobGraph.command, jobStore)
abdfc318-2389-4d66-b853-16ed091e8489      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 865, in _loadJob
abdfc318-2389-4d66-b853-16ed091e8489        userModule = cls._loadUserModule(userModule)
abdfc318-2389-4d66-b853-16ed091e8489      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 842, in _loadUserModule
abdfc318-2389-4d66-b853-16ed091e8489        return userModule.load()
abdfc318-2389-4d66-b853-16ed091e8489      File "/usr/local/lib/python2.7/dist-packages/toil/resource.py", line 576, in load
abdfc318-2389-4d66-b853-16ed091e8489        return importlib.import_module(module.name)
abdfc318-2389-4d66-b853-16ed091e8489      File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
abdfc318-2389-4d66-b853-16ed091e8489        __import__(name)
abdfc318-2389-4d66-b853-16ed091e8489    ImportError: No module named cactus.pipeline.cactus_workflow
```

The root cause is that the user module is "globalized" only during serialization of successors at the end of the worker process. That globalization relies on local state, so it only works if run on the node where the job was created. I think a decent solution would be to globalize the user module during Job.__init__ instead--but maybe there's a good reason for not doing that. I'll take a stab at it and see if it works.
Passing a Job class to another job to execute later sometimes crashes under hot-deployment
When a job B is passed a Job instance C created in a previous job A, then adds that Job (C) as a successor, C can sometimes fail to import itself when run under hot-deployment. Specifically, it will fail if run on a different machine than job A. That description might be a bit confusing, so consider this pseudocode:

```python
class A(Job):
    def run(self, fileStore):
         self.addChild(B(C('foo')))

class B(Job):
    def __init__(self, job):
         self.successorJob = job

    def run(self, fileStore):
         self.addChild(self.successorJob)

class C(Job):
    def run(self, fileStore):
         self.logToMaster('hi')
```

I'm aware that the "correct" way of doing this is to set C as a child of B during A's run method. In my case, though, B is actually a pipeline of a few jobs that handle constructing and initializing a Service, and needs to get run in quite a few places--which can lead to a lot of duplicated code if its structure had to be redefined each time. I think it can't be encapsulated, unfortunately, because the service needs to still be running during job C and its successors.

I think even though this is a really esoteric use-case, with a good workaround, it's worth fixing  because it works perfectly without hot-deployment, and it's pretty difficult to track down where things went wrong given just the error message.

The import failure looks something like this:

```
abdfc318-2389-4d66-b853-16ed091e8489    DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/tmp/tmp2qLnZC/74d6192189c43680877e98559bfad773', name='cactus.pip
eline.cactus_workflow', fromVirtualEnv=False).
abdfc318-2389-4d66-b853-16ed091e8489    WARNING:toil.resource:Can't find resource for leader path '/tmp/tmp2qLnZC/74d6192189c43680877e98559bfad773/cactus'
abdfc318-2389-4d66-b853-16ed091e8489    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/tmp/tmp2qLnZC/74d6192189c43680877e98559bfad773', name='c
actus.pipeline.cactus_workflow', fromVirtualEnv=False)
abdfc318-2389-4d66-b853-16ed091e8489    ERROR:toil.resource:Failed to import user module ModuleDescriptor(dirPath='/tmp/tmp2qLnZC/74d6192189c43680877e98559bfad773', na
me='cactus.pipeline.cactus_workflow', fromVirtualEnv=False) from sys.path (['/usr/local/bin', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/l
ib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/t
mp/tmp2qLnZC/74d6192189c43680877e98559bfad773']).
abdfc318-2389-4d66-b853-16ed091e8489    Traceback (most recent call last):
abdfc318-2389-4d66-b853-16ed091e8489      File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 325, in main
abdfc318-2389-4d66-b853-16ed091e8489        job = Job._loadJob(jobGraph.command, jobStore)
abdfc318-2389-4d66-b853-16ed091e8489      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 865, in _loadJob
abdfc318-2389-4d66-b853-16ed091e8489        userModule = cls._loadUserModule(userModule)
abdfc318-2389-4d66-b853-16ed091e8489      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 842, in _loadUserModule
abdfc318-2389-4d66-b853-16ed091e8489        return userModule.load()
abdfc318-2389-4d66-b853-16ed091e8489      File "/usr/local/lib/python2.7/dist-packages/toil/resource.py", line 576, in load
abdfc318-2389-4d66-b853-16ed091e8489        return importlib.import_module(module.name)
abdfc318-2389-4d66-b853-16ed091e8489      File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
abdfc318-2389-4d66-b853-16ed091e8489        __import__(name)
abdfc318-2389-4d66-b853-16ed091e8489    ImportError: No module named cactus.pipeline.cactus_workflow
```

The root cause is that the user module is "globalized" only during serialization of successors at the end of the worker process. That globalization relies on local state, so it only works if run on the node where the job was created. I think a decent solution would be to globalize the user module during Job.__init__ instead--but maybe there's a good reason for not doing that. I'll take a stab at it and see if it works.


Discussion:
@joelarmstrong let me know if you want me a look at this too
Thanks, I've got a one-line fix that I think works--but need to test more thoroughly. It just globalizes the module during `Job.__init__`, which shouldn't cause any problems, since it already works that way in the function-wrapping and service jobs. Hope to have a PR soon, once I can confirm it solves this.
@joelarmstrong let me know if you want me a look at this too
Thanks, I've got a one-line fix that I think works--but need to test more thoroughly. It just globalizes the module during `Job.__init__`, which shouldn't cause any problems, since it already works that way in the function-wrapping and service jobs. Hope to have a PR soon, once I can confirm it solves this.

Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -285,7 +285,7 @@ def __init__(self, memory=None, cores=None, disk=None, preemptable=None, unitNam
         # Note that self.__module__ is not necessarily this module, i.e. job.py. It is the module
         # defining the class self is an instance of, which may be a subclass of Job that may be
         # defined in a different module.
-        self.userModule = ModuleDescriptor.forModule(self.__module__)
+        self.userModule = ModuleDescriptor.forModule(self.__module__).globalize()
         # Maps index paths into composite return values to lists of IDs of files containing
         # promised values for those return value items. An index path is a tuple of indices that
         # traverses a nested data structure of lists, dicts, tuples or any other type supporting


Test patch: