Repository: DataBiosphere/toil

Issue:
Imported files in the same directory as the user script not available on Mesos nodes
I have a particular user script that imports my own library code I have written for running on Toil, called `toillib.py`, which I keep in the same directory as the user script. I import it thusly:

```
from toillib import *
```

On the master, my script can import `toillib.py` just fine, but when running on Mesos on Azure, when there is no shared filesystem and Mesos has to take care of copying the user script out to the nodes, `toillib.py` appears to be missing when the workers start up:

```
INFO:toil.batchSystems.mesos.batchSystem:Preparing to launch Mesos task 0 using offer 20151120-222317-83886090-5050-35148-O5...
INFO:toil.batchSystems.mesos.batchSystem:...launching Mesos task 0
WARNING:toil.leader:The jobWrapper seems to have left a log file, indicating failure: 46c1efec_0849_4ce3_b74d_c74f1f0602a4
WARNING:toil.leader:Reporting file: 46c1efec_0849_4ce3_b74d_c74f1f0602a4
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       ---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       Traceback (most recent call last):
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:         File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 
266, in main
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:           job = Job._loadJob(jobWrapper.command, jobStore)
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 936, in _loadJob
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:           userModule = cls._loadUserModule(userModule)
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 914, in _loadUserModule
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:           return importlib.import_module(userModule.name)
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:         File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:           __import__(name)
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:         File "/tmp/tmpflu3Jl/5b3f3d5ba2a95124e308998df2ab7bb2/parallelMappingEvaluation.py", line 16, in <module>
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:           from toillib import *
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       ImportError: No module named toillib
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       Exiting the worker because of a failed jobWrapper on host c4agent1
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host c4agent1
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 46c1efec_0849_4ce3_b74d_c74f1f0602a4 to 0
WARNING:toil.leader:Job: 46c1efec_0849_4ce3_b74d_c74f1f0602a4 is completely failed
```

Imported files in the same directory are supposed to be brought along by the resource system and be accessible on the workers, right?

Imported files in the same directory as the user script not available on Mesos nodes
I have a particular user script that imports my own library code I have written for running on Toil, called `toillib.py`, which I keep in the same directory as the user script. I import it thusly:

```
from toillib import *
```

On the master, my script can import `toillib.py` just fine, but when running on Mesos on Azure, when there is no shared filesystem and Mesos has to take care of copying the user script out to the nodes, `toillib.py` appears to be missing when the workers start up:

```
INFO:toil.batchSystems.mesos.batchSystem:Preparing to launch Mesos task 0 using offer 20151120-222317-83886090-5050-35148-O5...
INFO:toil.batchSystems.mesos.batchSystem:...launching Mesos task 0
WARNING:toil.leader:The jobWrapper seems to have left a log file, indicating failure: 46c1efec_0849_4ce3_b74d_c74f1f0602a4
WARNING:toil.leader:Reporting file: 46c1efec_0849_4ce3_b74d_c74f1f0602a4
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       ---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       Traceback (most recent call last):
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:         File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 
266, in main
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:           job = Job._loadJob(jobWrapper.command, jobStore)
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 936, in _loadJob
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:           userModule = cls._loadUserModule(userModule)
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:         File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 914, in _loadUserModule
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:           return importlib.import_module(userModule.name)
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:         File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:           __import__(name)
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:         File "/tmp/tmpflu3Jl/5b3f3d5ba2a95124e308998df2ab7bb2/parallelMappingEvaluation.py", line 16, in <module>
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:           from toillib import *
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       ImportError: No module named toillib
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       Exiting the worker because of a failed jobWrapper on host c4agent1
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host c4agent1
WARNING:toil.leader:46c1efec_0849_4ce3_b74d_c74f1f0602a4:       WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 46c1efec_0849_4ce3_b74d_c74f1f0602a4 to 0
WARNING:toil.leader:Job: 46c1efec_0849_4ce3_b74d_c74f1f0602a4 is completely failed
```

Imported files in the same directory are supposed to be brought along by the resource system and be accessible on the workers, right?



Discussion:
Yes. But we may have to help Toil a little bit. I needs to recognize that the user script and the imported module are part of a package hierarchy and it needs to be able to identify the hierarchy. Can you send an `ls -lR` of your working directory on the leader? I will work on replicating this issue.

s/identify the hierarchy/identify the root of the hierarchy/

This is what it looks like:

```
anovak@c4master1:~/hgvm-graph-bakeoff-evalutations$ ls -lr
total 112
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:32 vg_graphs
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:52 scripts
-rw-rw-r-- 1 anovak anovak    15 Nov 20 22:32 requirements.txt
-rw-rw-r-- 1 anovak anovak  7205 Nov 20 22:32 README.md
-rw-rw-r-- 1 anovak anovak 51746 Nov 20 23:10 log.txt
-rw-rw-r-- 1 anovak anovak  1106 Nov 20 22:32 LICENSE
-rw-rw-r-- 1 anovak anovak     0 Nov 20 22:32 __init__.py
-rw-rw-r-- 1 anovak anovak  8817 Nov 20 22:32 graph_servers.tsv
-rw-rw-r-- 1 anovak anovak  8823 Nov 20 22:32 graph_servers_high_coverage.tsv
-rw-rw-r-- 1 anovak anovak  7050 Nov 20 22:34 graph_servers_azure.tsv
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:32 data
anovak@c4master1:~/hgvm-graph-bakeoff-evalutations$ ls -lR
.:
total 112
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:32 data
-rw-rw-r-- 1 anovak anovak  7050 Nov 20 22:34 graph_servers_azure.tsv
-rw-rw-r-- 1 anovak anovak  8823 Nov 20 22:32 graph_servers_high_coverage.tsv
-rw-rw-r-- 1 anovak anovak  8817 Nov 20 22:32 graph_servers.tsv
-rw-rw-r-- 1 anovak anovak     0 Nov 20 22:32 __init__.py
-rw-rw-r-- 1 anovak anovak  1106 Nov 20 22:32 LICENSE
-rw-rw-r-- 1 anovak anovak 51746 Nov 20 23:10 log.txt
-rw-rw-r-- 1 anovak anovak  7205 Nov 20 22:32 README.md
-rw-rw-r-- 1 anovak anovak    15 Nov 20 22:32 requirements.txt
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:52 scripts
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:32 vg_graphs

./data:
total 15484
-rw-rw-r-- 1 anovak anovak 15803376 Nov 20 22:32 alignments.tar.gz
-rw-rw-r-- 1 anovak anovak    46913 Nov 20 22:32 genes.tar.gz

./scripts:
total 372
-rwxrwxr-x 1 anovak anovak  9930 Nov 20 22:32 barchart.py
-rwxrwxr-x 1 anovak anovak 19631 Nov 20 22:32 boxplot.py
-rwxrwxr-x 1 anovak anovak   403 Nov 20 22:32 call_snps.sh
-rwxrwxr-x 1 anovak anovak 13592 Nov 20 22:32 callVariants.py
-rwxrwxr-x 1 anovak anovak 12824 Nov 20 22:32 clusterGraphs.py
-rwxrwxr-x 1 anovak anovak  9026 Nov 20 22:32 collateStatistics.py
-rwxrwxr-x 1 anovak anovak 19840 Nov 20 22:32 compareCalls.py
-rwxrwxr-x 1 anovak anovak   465 Nov 20 22:32 compare_graphs.sh
-rwxrwxr-x 1 anovak anovak  2671 Nov 20 22:32 compare_snps.sh
-rwxrwxr-x 1 anovak anovak 32025 Nov 20 22:32 getAltReads.py
-rwxrwxr-x 1 anovak anovak 27144 Nov 20 22:32 graphEval.py
-rw-rw-r-- 1 anovak anovak 10164 Nov 20 22:32 heatmap.py
-rwxrwxr-x 1 anovak anovak 18029 Nov 20 22:32 histogram.py
-rw-rw-r-- 1 anovak anovak     0 Nov 20 22:32 __init__.py
-rwxrwxr-x 1 anovak anovak  5890 Nov 20 22:32 parallelAzureDownloader.py
-rwxrwxr-x 1 anovak anovak 46598 Nov 20 22:52 parallelMappingEvaluation.py
-rwxrwxr-x 1 anovak anovak  5232 Nov 20 22:32 plotMappingEvaluation.sh
-rwxrwxr-x 1 anovak anovak  5254 Nov 20 22:32 presplots.sh
-rwxrwxr-x 1 anovak anovak  3322 Nov 20 22:32 redoStats.py
-rwxrwxr-x 1 anovak anovak 13278 Nov 20 22:32 scatter.py
-rwxrwxr-x 1 anovak anovak 14199 Nov 20 22:32 smartSam2Fastq.py
-rwxrwxr-x 1 anovak anovak   959 Nov 20 22:32 toilAzureCleanup.sh
-rw-rw-r-- 1 anovak anovak 25245 Nov 20 22:32 toillib.py
-rw-rw-r-- 1 anovak anovak 21136 Nov 20 22:39 toillib.pyc
-rwxrwxr-x 1 anovak anovak   159 Nov 20 22:32 vcfCountSnps.sh
-rwxrwxr-x 1 anovak anovak   243 Nov 20 22:32 vgHasPath.sh

./vg_graphs:
total 4
-rwxrwxr-x 1 anovak anovak 767 Nov 20 22:32 make_graphs.sh
```

It may be relevant that I'm running my script as `scripts/parallelMappingEvaluation.py`, and it's importing `toillib.py` next to the executing file, but not directly from the working directory.

Try running the script as `PYTHONPATH=. python -m scripts.parallelMappingEvaluation` and changing the import to `from scripts.toillib import *`

I can reproduce this. The above workaround doesn't seem to work. Stay tuned.

Yes. But we may have to help Toil a little bit. I needs to recognize that the user script and the imported module are part of a package hierarchy and it needs to be able to identify the hierarchy. Can you send an `ls -lR` of your working directory on the leader? I will work on replicating this issue.

s/identify the hierarchy/identify the root of the hierarchy/

This is what it looks like:

```
anovak@c4master1:~/hgvm-graph-bakeoff-evalutations$ ls -lr
total 112
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:32 vg_graphs
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:52 scripts
-rw-rw-r-- 1 anovak anovak    15 Nov 20 22:32 requirements.txt
-rw-rw-r-- 1 anovak anovak  7205 Nov 20 22:32 README.md
-rw-rw-r-- 1 anovak anovak 51746 Nov 20 23:10 log.txt
-rw-rw-r-- 1 anovak anovak  1106 Nov 20 22:32 LICENSE
-rw-rw-r-- 1 anovak anovak     0 Nov 20 22:32 __init__.py
-rw-rw-r-- 1 anovak anovak  8817 Nov 20 22:32 graph_servers.tsv
-rw-rw-r-- 1 anovak anovak  8823 Nov 20 22:32 graph_servers_high_coverage.tsv
-rw-rw-r-- 1 anovak anovak  7050 Nov 20 22:34 graph_servers_azure.tsv
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:32 data
anovak@c4master1:~/hgvm-graph-bakeoff-evalutations$ ls -lR
.:
total 112
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:32 data
-rw-rw-r-- 1 anovak anovak  7050 Nov 20 22:34 graph_servers_azure.tsv
-rw-rw-r-- 1 anovak anovak  8823 Nov 20 22:32 graph_servers_high_coverage.tsv
-rw-rw-r-- 1 anovak anovak  8817 Nov 20 22:32 graph_servers.tsv
-rw-rw-r-- 1 anovak anovak     0 Nov 20 22:32 __init__.py
-rw-rw-r-- 1 anovak anovak  1106 Nov 20 22:32 LICENSE
-rw-rw-r-- 1 anovak anovak 51746 Nov 20 23:10 log.txt
-rw-rw-r-- 1 anovak anovak  7205 Nov 20 22:32 README.md
-rw-rw-r-- 1 anovak anovak    15 Nov 20 22:32 requirements.txt
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:52 scripts
drwxrwxr-x 2 anovak anovak  4096 Nov 20 22:32 vg_graphs

./data:
total 15484
-rw-rw-r-- 1 anovak anovak 15803376 Nov 20 22:32 alignments.tar.gz
-rw-rw-r-- 1 anovak anovak    46913 Nov 20 22:32 genes.tar.gz

./scripts:
total 372
-rwxrwxr-x 1 anovak anovak  9930 Nov 20 22:32 barchart.py
-rwxrwxr-x 1 anovak anovak 19631 Nov 20 22:32 boxplot.py
-rwxrwxr-x 1 anovak anovak   403 Nov 20 22:32 call_snps.sh
-rwxrwxr-x 1 anovak anovak 13592 Nov 20 22:32 callVariants.py
-rwxrwxr-x 1 anovak anovak 12824 Nov 20 22:32 clusterGraphs.py
-rwxrwxr-x 1 anovak anovak  9026 Nov 20 22:32 collateStatistics.py
-rwxrwxr-x 1 anovak anovak 19840 Nov 20 22:32 compareCalls.py
-rwxrwxr-x 1 anovak anovak   465 Nov 20 22:32 compare_graphs.sh
-rwxrwxr-x 1 anovak anovak  2671 Nov 20 22:32 compare_snps.sh
-rwxrwxr-x 1 anovak anovak 32025 Nov 20 22:32 getAltReads.py
-rwxrwxr-x 1 anovak anovak 27144 Nov 20 22:32 graphEval.py
-rw-rw-r-- 1 anovak anovak 10164 Nov 20 22:32 heatmap.py
-rwxrwxr-x 1 anovak anovak 18029 Nov 20 22:32 histogram.py
-rw-rw-r-- 1 anovak anovak     0 Nov 20 22:32 __init__.py
-rwxrwxr-x 1 anovak anovak  5890 Nov 20 22:32 parallelAzureDownloader.py
-rwxrwxr-x 1 anovak anovak 46598 Nov 20 22:52 parallelMappingEvaluation.py
-rwxrwxr-x 1 anovak anovak  5232 Nov 20 22:32 plotMappingEvaluation.sh
-rwxrwxr-x 1 anovak anovak  5254 Nov 20 22:32 presplots.sh
-rwxrwxr-x 1 anovak anovak  3322 Nov 20 22:32 redoStats.py
-rwxrwxr-x 1 anovak anovak 13278 Nov 20 22:32 scatter.py
-rwxrwxr-x 1 anovak anovak 14199 Nov 20 22:32 smartSam2Fastq.py
-rwxrwxr-x 1 anovak anovak   959 Nov 20 22:32 toilAzureCleanup.sh
-rw-rw-r-- 1 anovak anovak 25245 Nov 20 22:32 toillib.py
-rw-rw-r-- 1 anovak anovak 21136 Nov 20 22:39 toillib.pyc
-rwxrwxr-x 1 anovak anovak   159 Nov 20 22:32 vcfCountSnps.sh
-rwxrwxr-x 1 anovak anovak   243 Nov 20 22:32 vgHasPath.sh

./vg_graphs:
total 4
-rwxrwxr-x 1 anovak anovak 767 Nov 20 22:32 make_graphs.sh
```

It may be relevant that I'm running my script as `scripts/parallelMappingEvaluation.py`, and it's importing `toillib.py` next to the executing file, but not directly from the working directory.

Try running the script as `PYTHONPATH=. python -m scripts.parallelMappingEvaluation` and changing the import to `from scripts.toillib import *`

I can reproduce this. The above workaround doesn't seem to work. Stay tuned.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -27,7 +27,7 @@
         'bd2k-python-lib==1.10.dev5'],
     tests_require=[
         'mock==1.0.1',
-        'pytest==2.7.2'],
+        'pytest==2.8.3'],
     test_suite='toil',
     extras_require={
         'mesos': [
@@ -39,7 +39,7 @@
         'encryption': [
             'pynacl==0.3.0'],
         'cwl': [
-            'cwltool>=1.0.20150918080732']},
+            'cwltool==1.0.20151112194920']},
     package_dir={'': 'src'},
     packages=find_packages('src', exclude=['*.test']),
     entry_points={
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -911,7 +911,11 @@ def _loadUserModule(cls, userModule):
             userModule = userModule.localize()
         if userModule.dirPath not in sys.path:
             sys.path.append(userModule.dirPath)
-        return importlib.import_module(userModule.name)
+        try:
+            return importlib.import_module(userModule.name)
+        except ImportError:
+            logger.error('Failed to import user module %r from sys.path=%r', userModule, sys.path)
+            raise
 
     @classmethod
     def _loadJob(cls, command, jobStore):
diff --git a/src/toil/resource.py b/src/toil/resource.py
--- a/src/toil/resource.py
+++ b/src/toil/resource.py
@@ -24,9 +24,10 @@
 from tempfile import mkdtemp
 from urllib2 import urlopen
 import errno
-from zipfile import ZipFile
+from zipfile import ZipFile, PyZipFile
 import sys
 import shutil
+from bd2k.util.iterables import concat
 
 log = logging.getLogger(__name__)
 
@@ -252,16 +253,14 @@ class DirectoryResource(Resource):
 
     @classmethod
     def _load(cls, leaderPath):
+        """
+        :type leaderPath: str
+        """
         bytesIO = BytesIO()
-        with ZipFile(file=bytesIO, mode='w') as zipFile:
-            for dirPath, fileNames, dirNames in os.walk(leaderPath):
-                assert dirPath.startswith(leaderPath)
-                for fileName in fileNames:
-                    filePath = os.path.join(dirPath, fileName)
-                    assert filePath.encode('ascii') == filePath
-                    relativeFilePath = os.path.relpath(filePath, leaderPath)
-                    assert not relativeFilePath.startswith(os.path.sep)
-                    zipFile.write(filePath, relativeFilePath)
+        # PyZipFile compiles .py files on the fly, filters out any non-Python files and
+        # distinguishes between packages and simple directories.
+        with PyZipFile(file=bytesIO, mode='w') as zipFile:
+            zipFile.writepy(leaderPath)
         bytesIO.seek(0)
         return bytesIO
 
@@ -277,7 +276,7 @@ def localPath(self):
         return self.localDirPath
 
 
-class ModuleDescriptor(namedtuple('ModuleDescriptor', ('dirPath', 'name', 'extension'))):
+class ModuleDescriptor(namedtuple('ModuleDescriptor', ('dirPath', 'name'))):
     """
     A path to a Python module decomposed into a namedtuple of three elements, namely
 
@@ -287,14 +286,9 @@ class ModuleDescriptor(namedtuple('ModuleDescriptor', ('dirPath', 'name', 'exten
     - moduleName, the fully qualified name of the module with leading package names separated by
       dot and
 
-    - extension, the file extension of the file containing the module, including the leading period.
-
     >>> import toil.resource
     >>> ModuleDescriptor.forModule('toil.resource') # doctest: +ELLIPSIS
-    ModuleDescriptor(dirPath='/.../src', name='toil.resource', extension='.py')
-
-    Note that the above test only succeeds on py.test. To run with doctest or an IDE you may have
-    to change the assertion to .pyc.
+    ModuleDescriptor(dirPath='/.../src', name='toil.resource')
 
     >>> import subprocess, tempfile, os
     >>> dirPath = tempfile.mkdtemp()
@@ -302,10 +296,8 @@ class ModuleDescriptor(namedtuple('ModuleDescriptor', ('dirPath', 'name', 'exten
     >>> with open(path,'w') as f:
     ...     f.write('from toil.resource import ModuleDescriptor\\n'
     ...             'print ModuleDescriptor.forModule(__name__)')
-    >>> expected = [ str( ModuleDescriptor(dirPath=dirPath, name='foo', extension=extension) )
-    ...     for extension in ('.py', '.pyc') ]
-    >>> subprocess.check_output([ sys.executable, path ]).strip() in expected
-    True
+    >>> subprocess.check_output([ sys.executable, path ]) # doctest: +ELLIPSIS
+    "ModuleDescriptor(dirPath='...', name='foo')\\n"
 
     Now test a collision. As funny as it sounds, the robotparser module is included in the Python
     standard library.
@@ -338,16 +330,28 @@ def forModule(cls, name):
         filePath[-1], extension = os.path.splitext(filePath[-1])
         assert extension in ('.py', '.pyc')
         if name == '__main__':
-            name = filePath.pop()
-            dirPath = os.path.sep.join(filePath)
-            cls._check_conflict(dirPath, name)
+            if module.__package__:
+                # invoked via python -m foo.bar
+                name = [filePath.pop()]
+                for package in reversed(module.__package__.split('.')):
+                    dirPathTail = filePath.pop()
+                    assert dirPathTail == package
+                    name.append(dirPathTail)
+                name = '.'.join(reversed(name))
+                dirPath = os.path.sep.join(filePath)
+            else:
+                # invoked via python foo/bar.py
+                name = filePath.pop()
+                dirPath = os.path.sep.join(filePath)
+                cls._check_conflict(dirPath, name)
         else:
+            # imported as a module
             for package in reversed(name.split('.')):
                 dirPathTail = filePath.pop()
                 assert dirPathTail == package
             dirPath = os.path.sep.join(filePath)
 
-        return cls(dirPath=dirPath, name=name, extension=extension)
+        return cls(dirPath=dirPath, name=name)
 
     @classmethod
     def _check_conflict(cls, dirPath, name):
@@ -359,14 +363,13 @@ def _check_conflict(cls, dirPath, name):
         """
         old_sys_path = sys.path
         try:
-            sys.path = [dir for dir in old_sys_path
-                        if os.path.realpath(dir) != os.path.realpath(dirPath)]
+            sys.path = [d for d in old_sys_path if os.path.realpath(d) != os.path.realpath(dirPath)]
             try:
                 colliding_module = importlib.import_module(name)
             except ImportError:
                 pass
             else:
-                raise RuntimeError(
+                raise ResourceException(
                     "The user module '%s' collides with module '%s from '%s'." % (
                         name, colliding_module.__name__, colliding_module.__file__))
         finally:
@@ -379,13 +382,6 @@ def belongsToToil(self):
         """
         return self.name.startswith('toil.')
 
-    @property
-    def filePath(self):
-        """
-        The full path to the file containing this module
-        """
-        return os.path.join(self.dirPath, *self.name.split('.')) + self.extension
-
     def saveAsResourceTo(self, jobStore):
         """
         Store the file containing this module--or even the Python package directory hierarchy
@@ -400,10 +396,13 @@ def saveAsResourceTo(self, jobStore):
     def localize(self):
         """
         Check if this module was saved as a resource. If it was, return a new module descriptor
-        that points to a local copy of that resource. Should only be called on a worker node.
+        that points to a local copy of that resource. Should only be called on a worker node. On
+        the leader, this method returns this resource, i.e. self.
 
         :rtype: toil.resource.Resource
         """
+        if self._runningOnWorker():
+            log.warn('The localize() method should only be invoked on a worker.')
         resource = Resource.lookup(self._resourcePath)
         if resource is None:
             log.warn("Can't localize module %r", self)
@@ -415,16 +414,28 @@ def stash(tmpDirPath):
                     f.write(json.dumps(self))
 
             resource.download(callback=stash)
-            return self.__class__(dirPath=resource.localDirPath, name=self.name,
-                                  extension=self.extension)
+            return self.__class__(dirPath=resource.localDirPath, name=self.name)
+
+    def _runningOnWorker(self):
+        mainModule = sys.modules.get('__main__')
+        if mainModule:
+            mainModuleFile = mainModule.__file__
+            for extension in self.moduleExtensions:
+                if mainModuleFile.endswith('worker' + extension):
+                    return True
+        return False
 
     def globalize(self):
+        """
+        Reverse the effect of localize().
+        """
         try:
             with open(os.path.join(self.dirPath, '.original')) as f:
                 return self.__class__(*json.loads(f.read()))
         except IOError as e:
             if e.errno == errno.ENOENT:
-                log.warn("Can't globalize module %r.", self)
+                if self._runningOnWorker():
+                    log.warn("Can't globalize module %r.", self)
                 return self
             else:
                 raise
@@ -432,7 +443,39 @@ def globalize(self):
     @property
     def _resourcePath(self):
         """
-        The path to the file or package directory that should be used when shipping this module
+        The path to the directory that should be used when shipping this module and its siblings
         around as a resource.
         """
-        return self.dirPath if '.' in self.name else self.filePath
+        if '.' in self.name:
+            return os.path.join(self.dirPath, self._rootPackage())
+        else:
+            initName = self._initModuleName(self.dirPath)
+            if initName:
+                raise ResourceException(
+                    "Toil does not support loading a user script from a package directory. You "
+                    "may want to remove %s from %s or invoke the user script as a module via "
+                    "'PYTHONPATH=\"%s\" python -m %s.%s'." %
+                    tuple(concat(initName, self.dirPath, os.path.split(self.dirPath), self.name)))
+            return self.dirPath
+
+    moduleExtensions = ('.py', '.pyc', '.pyo')
+
+    @classmethod
+    def _initModuleName(cls, dirPath):
+        for extension in cls.moduleExtensions:
+            name = '__init__' + extension
+            if os.path.exists(os.path.join(dirPath, name)):
+                return name
+        return None
+
+    def _rootPackage(self):
+        try:
+            head, tail = self.name.split('.', 1)
+        except ValueError:
+            raise ValueError('%r is stand-alone module.' % self)
+        else:
+            return head
+
+
+class ResourceException(Exception):
+    pass


Test patch:
diff --git a/src/toil/test/src/resourceTest.py b/src/toil/test/src/resourceTest.py
--- a/src/toil/test/src/resourceTest.py
+++ b/src/toil/test/src/resourceTest.py
@@ -11,48 +11,93 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
 from __future__ import absolute_import
+import importlib
+import os
 
 import sys
+from zipfile import ZipFile
+from bd2k.util.files import mkdir_p
+from io import BytesIO
 
 from mock import MagicMock, patch
 
-from toil.resource import ModuleDescriptor, Resource
+from toil.resource import ModuleDescriptor, Resource, ResourceException
 from toil.test import ToilTest
 
 
-class ResourceTest( ToilTest ):
-    def test( self ):
+class ResourceTest(ToilTest):
+    """
+    Test module descriptors and resources derived from them.
+    """
+
+    def testStandAlone(self):
+        self._testExternal(moduleName='userScript', pyFiles=('userScript.py',
+                                                             'helper.py'))
+
+    def testPackage(self):
+        self._testExternal(moduleName='foo.userScript', pyFiles=('foo/__init__.py',
+                                                                 'foo/userScript.py',
+                                                                 'foo/bar/__init__.py',
+                                                                 'foo/bar/helper.py'))
+
+    def testStandAloneInPackage(self):
+        self.assertRaises(ResourceException,
+                          self._testExternal,
+                          moduleName='userScript',
+                          pyFiles=('__init__.py', 'userScript.py', 'helper.py'))
+
+    def _testExternal(self, moduleName, pyFiles):
+        dirPath = self._createTempDir()
+        pycFiles = set(pyFile + 'c' for pyFile in pyFiles)
+        for relPath in pyFiles:
+            path = os.path.join(dirPath, relPath)
+            mkdir_p(os.path.dirname(path))
+            with open(path, 'w') as f:
+                f.write('pass\n')
+        sys.path.append(dirPath)
+        try:
+            userScript = importlib.import_module(moduleName)
+            try:
+                self._test(userScript.__name__, expectedContents=pycFiles)
+            finally:
+                del userScript
+                del sys.modules[moduleName]
+            self.assertFalse(moduleName in sys.modules)
+        finally:
+            sys.path.remove(dirPath)
+
+    def testBuiltIn(self):
         # Create a ModuleDescriptor for the module containing ModuleDescriptor, i.e. toil.resource
         module_name = ModuleDescriptor.__module__
-        self.assertEquals( module_name, 'toil.resource' )
-        module = ModuleDescriptor.forModule( module_name )
+        self.assertEquals(module_name, 'toil.resource')
+        self._test(module_name, shouldBelongToToil=True)
 
+    def _test(self, module_name, shouldBelongToToil=False, expectedContents=None):
+        module = ModuleDescriptor.forModule(module_name)
         # Assert basic attributes and properties
-        self.assertTrue( module.belongsToToil )
-        self.assertEquals( module.name, module_name )
-        self.assertEquals( module.filePath, sys.modules[ module_name ].__file__ )
-        self.assertTrue( module.filePath.startswith( module.dirPath ) )
-        self.assertTrue( module.dirPath.endswith( '/src' ) )
+        self.assertEqual(module.belongsToToil, shouldBelongToToil)
+        self.assertEquals(module.name, module_name)
+        if shouldBelongToToil:
+            self.assertTrue(module.dirPath.endswith('/src'))
 
         # Before the module is saved as a resource, localize() and globalize() are identity
         # methods. This should log warnings.
-        self.assertIs( module.localize( ), module )
-        self.assertIs( module.globalize( ), module )
-
+        self.assertIs(module.localize(), module)
+        self.assertIs(module.globalize(), module)
         # Create a mock job store ...
-        jobStore = MagicMock( )
+        jobStore = MagicMock()
         # ... to generate a fake URL for the resource ...
         url = 'file://foo.zip'
         jobStore.getSharedPublicUrl.return_value = url
         # ... and save the resource to it.
-        resource = module.saveAsResourceTo( jobStore )
+        resource = module.saveAsResourceTo(jobStore)
         # Ensure that the URL generation method is actually called, ...
-        jobStore.getSharedPublicUrl.assert_called_once_with( resource.pathHash )
+        jobStore.getSharedPublicUrl.assert_called_once_with(resource.pathHash)
         # ... and that ensure that writeSharedFileStream is called.
-        jobStore.writeSharedFileStream.assert_called_once_with( resource.pathHash,
-                                                                isProtected=False )
-
+        jobStore.writeSharedFileStream.assert_called_once_with(resource.pathHash,
+                                                               isProtected=False)
         # Now it gets a bit complicated: Ensure that the context manager returned by the
         # jobStore's writeSharedFileStream() method is entered and that the file handle yielded
         # by the context manager is written to once with the zipped source tree from which
@@ -63,34 +108,36 @@ def test( self ):
         # instead of keyword arguments, and the third 0 selects the first positional, i.e. the
         # contents. This is a bit brittle since it assumes that all the data is written in a
         # single call to write(). If more calls are made we can easily concatenate them.
-        zipFile = file_handle.write.call_args_list[ 0 ][ 0 ][ 0 ]
-        self.assertTrue( zipFile.startswith( 'PK' ) ) # the magic header for ZIP files
-        self.assertEquals( resource.url, url )
+        zipFile = file_handle.write.call_args_list[0][0][0]
+        self.assertTrue(zipFile.startswith('PK'))  # the magic header for ZIP files
+
+        # Check contents if requested
+        if expectedContents is not None:
+            with ZipFile(BytesIO(zipFile)) as _zipFile:
+                self.assertEqual(set(_zipFile.namelist()), expectedContents)
 
+        self.assertEquals(resource.url, url)
         # Now we're on the worker. Prepare the storage for localized resources
-        Resource.prepareSystem( )
+        Resource.prepareSystem()
         # Register the resource for subsequent lookup.
-        resource.register( )
+        resource.register()
         # Lookup the resource and ensure that the result is equal to but not the same as the
         # original resource. Lookup will also be used when we localize the module that was
         # originally used to create the resource.
-        localResource = Resource.lookup( module._resourcePath )
-        self.assertEquals( resource, localResource )
-        self.assertIsNot( resource, localResource )
-
+        localResource = Resource.lookup(module._resourcePath)
+        self.assertEquals(resource, localResource)
+        self.assertIsNot(resource, localResource)
         # Now show that we can localize the module using the registered resource. Set up a mock
         # urlopen() that yields the zipped tree ...
-        mock_urlopen = MagicMock( )
+        mock_urlopen = MagicMock()
         mock_urlopen.return_value.read.return_value = zipFile
-        with patch( 'toil.resource.urlopen', mock_urlopen ):
+        with patch('toil.resource.urlopen', mock_urlopen):
             # ... and use it to download and unpack the resource
-            localModule = module.localize( )
-        # Name and extension should be equal between original and localized resource ...
-        self.assertEquals( module.name, localModule.name )
-        self.assertEquals( module.extension, localModule.extension )
+            localModule = module.localize()
+        # The name should be equal between original and localized resource ...
+        self.assertEquals(module.name, localModule.name)
         # ... but the directory should be different.
-        self.assertNotEquals( module.dirPath, localModule.dirPath )
-
+        self.assertNotEquals(module.dirPath, localModule.dirPath)
         # Show that we can 'undo' localization. This is necessary when the user script's jobs are
         # invoked on the worker where they generate more child jobs.
-        self.assertEquals( localModule.globalize( ), module )
+        self.assertEquals(localModule.globalize(), module)
diff --git a/src/toil/test/src/userDefinedJobArgTypeTest.py b/src/toil/test/src/userDefinedJobArgTypeTest.py
--- a/src/toil/test/src/userDefinedJobArgTypeTest.py
+++ b/src/toil/test/src/userDefinedJobArgTypeTest.py
@@ -15,7 +15,6 @@
 from __future__ import absolute_import
 import sys
 from subprocess import check_call
-
 from toil.job import Job
 from toil.test import ToilTest
 
@@ -54,7 +53,7 @@ def testJobClassFromMain(self):
     def _testFromMain(self):
         testMethodName = self.id().split('.')[-1]
         self.assertTrue(testMethodName.endswith('FromMain'))
-        check_call([sys.executable, __file__, testMethodName[:-8]])
+        check_call([sys.executable, '-m', self.__module__, testMethodName[:-8]])
 
 
 class JobClass(Job):
@@ -90,7 +89,3 @@ def main():
     test.setUpClass()
     test.debug()
     test.tearDownClass()
-
-
-if __name__ == '__main__':
-    main()