Repository: DataBiosphere/toil

Issue:
User script hot deployment not functioning
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    ---JOBTREE WORKER OUTPUT LOG---
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    Traceback (most recent call last):
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:      File "/usr/local/lib/python2.7/dist-packages/jobTree/worker.py", line 267, in main
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:        jobStore=jobStore)
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:      File "/usr/local/lib/python2.7/dist-packages/jobTree/target.py", line 714, in _execute
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:        returnValues = self.run(fileStore)
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:      File "/usr/local/lib/python2.7/dist-packages/jobTree/target.py", line 818, in run
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:        userFunction = self._getUserFunction()
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:      File "/usr/local/lib/python2.7/dist-packages/jobTree/target.py", line 791, in _getUserFunction
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:        return getattr(importlib.import_module(userFunctionModule.name), self.userFunctionName)
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:      File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:        **import**(name)
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    ImportError: No module named testUserScript
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    Exiting the worker because of a failed job on host ip-172-31-21-198
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    ERROR:**main**:Exiting the worker because of a failed job on host ip-172-31-21-198
WARNING:jobTree.leader:2fa1bc1d-7dec-4b88-814f-42bfcc159a2b:    WARNING:jobTree.job:Due to failure we are reducing the remaining retry count of job 2fa1bc1d-7dec-4b88-814f-42bfcc159a2b to 0



Discussion:
There are multiple implementations of startJobTree(). Only the stack.startJobTree() calls getUserScript, while the target.Runner.startJobTree() does not take user script into account at all, so it defaults to None.

Why are there multiple implementations, and when should each be used?

After adding the call to getUserScript() to target.runner.startJobTree(), it appears that hot deployment works successfully for the first task. However, on the child task I get an error when trying to load the job's pickle file. It appears that the user module is pickled in _serializeFirstJob(), but I don't see any similar functions for the subsequent jobs. I suspect that the user Module is just not being included in the subsequent job's pickle file, which causes them to try to import whatever the next token in the command is.

There should be no stack class, the only way to run the jobTree should be
through Target.Runner.startJobTree()

On Tue, Jul 14, 2015 at 2:24 PM, Christopher Ketchum <
notifications@github.com> wrote:

> There are multiple implementations of startJobTree(). Only the
> stack.startJobTree() calls getUserScript, while the
> target.Runner.startJobTree() does not take user script into account at all,
> so it defaults to None.
> 
> Why are there multiple implementations, and when should each be used?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/BD2KGenomics/slugflow/issues/167#issuecomment-121393613
> .

I have been testing with a script using functions as target. When trying to load the target on the worker, the "userModule" is reported as jobtree.target, and the "targetClass" is TargetFunctionWrappingTarget. This working for the very first task, but failing with the message "ImportError: No module named helloWorld" when trying to unpickle on the subsequent task. It appears as though the user module is not added to the path until the target is being run, but it first has to be unpickled , which requires that it be on sys.path.


Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -342,7 +342,7 @@ def startToil(job, options):
             Raises an exception if the given toil already exists.
             """
             setLoggingFromOptions(options)
-            with setupToil(options) as (config, batchSystem, jobStore):
+            with setupToil(options, userScript=job.getUserScript()) as (config, batchSystem, jobStore):
                 jobStore.clean()
                 if "rootJob" not in config.attrib: #No jobs have yet been run
                     # Setup the first batchjob.


Test patch: