Repository: DataBiosphere/toil

Issue:
Importing job functions does not forward descendants
If a job function is imported from a toil script module using `from module import job_function` syntax, the child and follow-on jobs for `job_function` aren't imported.

Pip installed toil-scripts not working on Mesos cluster
On a toil-cluster, I was unable to get toil-scripts to run unless I pip installed toil-scripts on both the leader and the worker _outside_ of an venv. 

This issue is blocked by #929, in a sense, since installation within a venv is preferred.



Discussion:
Using `import module` and `module.job_function` also has this problem.

@almussel Do you happen to have a stack trace for this?

With `from toil_scripts.batch_alignment.bwa_alignment import download_shared_files` in the adam-gatk pipeline, toil gives the following error:

```
---TOIL WORKER OUTPUT LOG---
Next available file descriptor: 8
DEBUG:toil.worker:Next available file descriptor: 8
DEBUG:toil.jobStores.aws.jobStore:Loaded job 9beff25f-f9e2-4e6b-a0f3-509bacc6d9d4
Parsed jobWrapper
DEBUG:toil.worker:Parsed jobWrapper
DEBUG:toil.jobStores.aws.jobStore:Updating job 9beff25f-f9e2-4e6b-a0f3-509bacc6d9d4
Got a command to run: _toil b1d196fd-70e9-4cdb-a932-efdbe2b11da0 /home/mesosbox/toil-scripts/src toil_scripts.adam_gatk_pipeline.align_and_call
DEBUG:toil.worker:Got a command to run: _toil b1d196fd-70e9-4cdb-a932-efdbe2b11da0 /home/mesosbox/toil-scripts/src toil_scripts.adam_gatk_pipeline.align_and_call
DEBUG:toil.jobStores.aws.jobStore:Reading FileInfo(fileID=u'b1d196...e2b11da0', ownerID='d8f28a5b-9b9e-4b10-8c2c-5910389ed12a', encrypted=False, version='', previousVersion='', conte
nt='\x80\x02ctoil.jo...\x07_kwargsq"}ub.', _numContentChunks=1) bound to <toil.jobStores.aws.jobStore.AWSJobStore object at 0x7fb06721ae10> into stream.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 328, in main
    fileStore=fileStore)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1380, in _execute
    returnValues = self._run(jobWrapper, fileStore)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1369, in _run
    return self.run(fileStore)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1490, in run
    userFunction = self._getUserFunction()
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1466, in _getUserFunction
    return getattr(userFunctionModule, self.userFunctionName)
AttributeError: 'module' object has no attribute 'download_url_job'
Exiting the worker because of a failed jobWrapper on host ip-172-31-25-86
ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-25-86
DEBUG:toil.jobStores.aws.jobStore:Loaded job 9beff25f-f9e2-4e6b-a0f3-509bacc6d9d4
WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 9beff25f-f9e2-4e6b-a0f3-509bacc6d9d4 to 0
```

Thanks for the traceback @almussel ! How are you calling the script? We handle hot deployment differently depending on if you use 'python userScript.py' vs 'python module.userScript' to execute.

Also, are you installing toil-scripts on every node in the cluster?

I'm executing using `python -m module.userScript`, and I'm only cloning toil scripts on the leader. 

I'm trying to reproduce this with module structure based on the toil scripts module but I'm running into a separate hot deployment error. Can I see your exact invocation? I think I'm running my test slightly differently from how you are

I'm doing this in the adam_gatk_pipeline. Currently all the jobs are being imported as `from module import *`. I ran into the error when I replaced them with the actual jobs called in the pipeline, like `from toil_scripts.batch_alignment.bwa_alignment import download_shared_files`. You could be getting something different because the imported jobs are encapsulated.

Thanks, so are you invoking it with `python -m toil_scripts.adam_gatk_pipeline.align_and_call`?

Yep, that's correct.

Do you have the stack trace for when toil-scripts was installed only on the leader?

> Do you have the stack trace for when toil-scripts was installed only on the leader?

Apologies, was in the process of writing out the reproduction steps (should have mentioned that). These are the errors i get when toil-scripts is installed _outside_ of a venv.

**Laptop**

```
cgcloud create-cluster --instance-type m3.large --leader-instance-type m3.large --spot-bid .50 --num-workers 1 toil

cgcloud ssh -a toil-leader
```

**Leader**

```
sudo pip install --pre toil-scripts
```

Exit and rejoin as mesosbox with `cgcloud ssh toil-leader`
Setup config with `toil-rnaseq generate-config`.
Confirm local run works: 

```
toil-rnaseq run ./jobStore --samples s3://cgl-pipeline-inputs/rnaseq_cgl/ci/chr6_sample.tar.gz
```

Run succeeds.  Run with mesos:

```
toil-rnaseq run aws:us-west-2:jvivian-piptest-mesos-1 --batchSystem mesos --mesosMaster mesos-master:5050 --samples s3://cgl-pipeline-inputs/rnaseq_cgl/ci/chr6_sample.tar.gz
```

Stacktrace

```
INFO:toil.lib.bioio:Logging set at level: INFO
INFO:toil.common:Using the mesos batch system
I0603 17:33:01.320348  3966 sched.cpp:164] Version: 0.25.0
I0603 17:33:01.326961  3976 sched.cpp:262] New master detected at master@172.31.26.230:5050
I0603 17:33:01.327152  3976 sched.cpp:272] No credentials provided. Attempting to register without authentication
INFO:toil.common:Written the environment for the jobs to the environment file
INFO:toil.job:Downloading entire JobStore
I0603 17:33:01.466910  3980 sched.cpp:641] Framework registered with 05640aa0-073a-4366-871f-c69c040850ad-0000
INFO:toil.job:0 jobs downloaded.
INFO:toil.jobStores.aws.sdbUtils:... got NoSuchDomain, trying again in 5s ...
INFO:toil.leader:(Re)building internal scheduler state
INFO:toil.leader:Checked batch system has no running jobs and no updated jobs
INFO:toil.leader:Found 1 jobs to start and 0 jobs with successors to run
INFO:toil.leader:Starting the main loop
INFO:toil.batchSystems.mesos.batchSystem:Preparing to launch Mesos task 0 using offer 05640aa0-073a-4366-871f-c69c040850ad-O2...
INFO:toil.batchSystems.mesos.batchSystem:...launching Mesos task 0
WARNING:toil.leader:The jobWrapper seems to have left a log file, indicating failure: ffc8b284-49aa-4e54-a207-8592ba8fe65f
WARNING:toil.leader:Reporting file: ffc8b284-49aa-4e54-a207-8592ba8fe65f
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:   ---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:   Traceback (most recent call last):
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:     File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 273, in main
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:       fileStore=fileStore)
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:     File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1294, in _execute
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:       returnValues = self.run(fileStore)
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:     File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1398, in run
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:       userFunction = self._getUserFunction()
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:     File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1374, in _getUserFunction
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:       return getattr(userFunctionModule, self.userFunctionName)
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:   AttributeError: 'module' object has no attribute 'download_sample'
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:   Exiting the worker because of a failed jobWrapper on host ip-172-31-25-177
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:   ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-25-177
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:   WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job ffc8b284-49aa-4e54-a207-8592ba8fe65f to 0
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:   Exception in thread Thread-11:
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:   Traceback (most recent call last):
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:     File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:       self.run()
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:     File "/usr/lib/python2.7/threading.py", line 754, in run
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:       self.__target(*self.__args, **self.__kwargs)
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:     File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 744, in asyncUpdate
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:       raise RuntimeError("The termination flag is set, exiting before update")
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:   RuntimeError: The termination flag is set, exiting before update
WARNING:toil.leader:ffc8b284-49aa-4e54-a207-8592ba8fe65f:   
WARNING:toil.leader:Job: ffc8b284-49aa-4e54-a207-8592ba8fe65f is completely failed
INFO:toil.leader:Only failed jobs and their dependents (1 total) are remaining, so exiting.
INFO:toil.leader:Finished the main loop
INFO:toil.leader:Waiting for stats and logging collator process to finish ...
INFO:toil.leader:... finished collating stats and logs. Took 0.336694002151 seconds
INFO:toil.batchSystems.mesos.batchSystem:Stopping Mesos driver
I0603 17:33:16.184043  3966 sched.cpp:1771] Asked to stop the driver
I0603 17:33:16.184151  3973 sched.cpp:1040] Stopping framework '05640aa0-073a-4366-871f-c69c040850ad-0000'
INFO:toil.batchSystems.mesos.batchSystem:Joining Mesos driver
INFO:toil.batchSystems.mesos.batchSystem:Joined Mesos driver
Traceback (most recent call last):
  File "/usr/local/bin/toil-rnaseq", line 9, in <module>
    load_entry_point('toil-scripts==2.0a2.dev179', 'console_scripts', 'toil-rnaseq')()
  File "/usr/local/lib/python2.7/dist-packages/toil_scripts/rnaseq_cgl/rnaseq_cgl_pipeline.py", line 638, in main
    Job.Runner.startToil(Job.wrapJobFn(map_job, download_sample, samples, config), args)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 445, in startToil
    return mainLoop(config, batchSystem, jobStore, rootJob, jobCache=jobCache)
  File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 533, in mainLoop
    raise FailedJobsException( config.jobStore, totalFailedJobs )
toil.leader.FailedJobsException: The job store 'aws:us-west-2:jvivian-piptest-mesos-1' contains 1 failed jobs
```

The attribute error: `AttributeError: 'module' object has no attribute 'download_sample'` references the first job function in the pipeline script.  Interestingly, the Toil.Runner is passed `map_job()` as the root job, which is imported from lib. Despite being the root job, the script doesn't complain about finding that.

If i install toil-scripts on the worker, I get a different error:

```
ERROR:toil.leader:Failed to unpickle root job return value
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 525, in mainLoop
    rootJobReturnValue = cPickle.load(fH)
EOFError
```

`map_job()` has no return value and neither does the the job function passed to it, `download_sample()`

I'd assume there's an issue with the script or `map_job()` if local mode didn't run fine. 

Again, great description. I wonder if this is the same as #831. If you have a chance, could you try importing `download_sample` right below where `map_job` is imported? It might still fail, but complaining about a different function.

> If you have a chance, could you try importing download_sample right below where map_job is imported?

`download_sample` is in the pipeline script so I can't import it.

`INFO:toil.job:User Module: toil_scripts.lib.jobs`

Interestingly, I couldn't import the `download_sample` function from rnaseq_cgl inside of lib/jobs.py

```
    from toil_scripts.rnaseq_cgl.rnaseq_cgl_pipeline import download_sample
ImportError: cannot import name download_sample
```

I directly copied and pasted the function into the toil_scripts/lib/jobs.py, but then I got an error about a missing Namespace attribute (which is assigned in `main` of the rnaseq_cgl_pipeline script). 

```
INFO:toil.job:User Module: toil_scripts.lib.jobs
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 273, in main
fileStore=fileStore)
File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1294, in _execute
returnValues = self.run(fileStore)
File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1401, in run
rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
File "/usr/local/lib/python2.7/dist-packages/toil_scripts/lib/jobs.py", line 38, in download_sample
AttributeError: 'Namespace' object has no attribute 'maxCores'
```


Resolving patch:
diff --git a/src/toil/__init__.py b/src/toil/__init__.py
--- a/src/toil/__init__.py
+++ b/src/toil/__init__.py
@@ -38,13 +38,20 @@ def resolveEntryPoint(entryPoint):
     return value may be an absolute or a relative path.
     """
     if hasattr(sys, 'real_prefix'):
-        # Inside a virtualenv we will use absolute paths to the entrypoints. For clusters this
-        # means that if Toil is installed in a virtualenv on the leader, it must be installed in
-        # a virtualenv located at the same path on the worker.
         path = os.path.join(os.path.dirname(sys.executable), entryPoint)
-        assert os.path.isfile(path)
-        assert os.access(path, os.X_OK)
-        return path
+        # Inside a virtualenv we try to use absolute paths to the entrypoints. 
+        if os.path.isfile(path):
+            # If the entrypoint is present, Toil must have been installed into the virtualenv (as 
+            # opposed to being included via --system-site-packages). For clusters this means that 
+            # if Toil is installed in a virtualenv on the leader, it must be installed in
+            # a virtualenv located at the same path on each worker as well.
+            assert os.access(path, os.X_OK)
+            return path
+        else:
+            # For virtualenv's that have the toil package directory on their sys.path but whose 
+            # bin directory lacks the Toil entrypoints, i.e. where Toil is included via 
+            # --system-site-packages, we rely on PATH just as if we weren't in a virtualenv. 
+            return entryPoint
     else:
         # Outside a virtualenv it is hard to predict where the entry points got installed. It is
         # the reponsibility of the user to ensure that they are present on PATH and point to the
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -2012,6 +2012,7 @@ def _loadJob(cls, command, jobStore):
         commandTokens = command.split()
         assert "_toil" == commandTokens[0]
         userModule = ModuleDescriptor(*(commandTokens[2:]))
+        logger.debug('Loading user module %s.', userModule)
         userModule = cls._loadUserModule(userModule)
         pickleFile = commandTokens[1]
         if pickleFile == "firstJob":
@@ -2035,8 +2036,10 @@ def _unpickle(cls, userModule, fileHandle):
 
         def filter_main(module_name, class_name):
             if module_name == '__main__':
+                logger.debug('Getting %s from user module __main__ (%s).', class_name, userModule)
                 return getattr(userModule, class_name)
             else:
+                logger.debug('Getting %s from module %s.', class_name, module_name)
                 return getattr(importlib.import_module(module_name), class_name)
 
         unpickler.find_global = filter_main
@@ -2505,6 +2508,9 @@ def __init__(self, userFunction, *args, **kwargs):
         self._kwargs=kwargs
 
     def _getUserFunction(self):
+        logger.debug('Loading user function %s from module %s.',
+                     self.userFunctionName,
+                     self.userFunctionModule) 
         userFunctionModule = self._loadUserModule(self.userFunctionModule)
         return getattr(userFunctionModule, self.userFunctionName)
 
@@ -2599,7 +2605,8 @@ def __init__(self, service):
         self.jobWrapper = None
 
     def run(self, fileStore):
-        #Unpickle the service
+        # Unpickle the service
+        logger.debug('Loading service module %s.', self.serviceModule)
         userModule = self._loadUserModule(self.serviceModule)
         service = self._unpickle( userModule, BytesIO( self.pickledService ) )
         #Start the service
diff --git a/src/toil/resource.py b/src/toil/resource.py
--- a/src/toil/resource.py
+++ b/src/toil/resource.py
@@ -411,8 +411,8 @@ def localize(self):
         else:
             def stash(tmpDirPath):
                 # Save the original dirPath such that we can restore it in globalize()
-                with open(os.path.join(tmpDirPath, '.original'), 'w') as f:
-                    f.write(json.dumps(self))
+                with open(os.path.join(tmpDirPath, '.dirPath'), 'w') as f:
+                    f.write(self.dirPath)
 
             resource.download(callback=stash)
             return self.__class__(dirPath=resource.localDirPath, name=self.name)
@@ -431,8 +431,8 @@ def globalize(self):
         Reverse the effect of localize().
         """
         try:
-            with open(os.path.join(self.dirPath, '.original')) as f:
-                return self.__class__(*json.loads(f.read()))
+            with open(os.path.join(self.dirPath, '.dirPath')) as f:
+                dirPath = f.read()
         except IOError as e:
             if e.errno == errno.ENOENT:
                 if self._runningOnWorker():
@@ -440,6 +440,9 @@ def globalize(self):
                 return self
             else:
                 raise
+        else:
+            return self.__class__( dirPath=dirPath, name=self.name )
+
 
     @property
     def _resourcePath(self):


Test patch: