Repository: DataBiosphere/toil

Issue:
Error removing toil/src directory from python path
When installed from pip (pip install git+http://github.com/bd2kgenomics/toil) into a virtualenv:

> Added job: b/k/jobHAlTgv to active jobs
> Built the jobs list, currently have 1 jobs to update and 0 jobs issued
> Issuing the command: /home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv with memory: 2147483648, cores: >1, disk: 2147483648
> Issued job with job store ID: b/k/jobHAlTgv and job batch system ID: 98 and cores: 1, disk: >2147483648, and memory: 2147483648
> Acquiring 2147483648 bytes of memory from pool of 9223372036854775807.
> Attempting to acquire 10 threads for 1 cpus submitted
> Acquiring semaphore blockingly.
> Semaphore acquired.
> Executing command: '/home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv'.
> Traceback (most recent call last):
>  File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line >24, in <module>
>    sys.path.remove(os.path.dirname(os.path.abspath(**file**)))
> ValueError: list.remove(x): x not in list
> Got exit code 1 (indicating failure) from command '/home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv'.

Error removing toil/src directory from python path
When installed from pip (pip install git+http://github.com/bd2kgenomics/toil) into a virtualenv:

> Added job: b/k/jobHAlTgv to active jobs
> Built the jobs list, currently have 1 jobs to update and 0 jobs issued
> Issuing the command: /home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv with memory: 2147483648, cores: >1, disk: 2147483648
> Issued job with job store ID: b/k/jobHAlTgv and job batch system ID: 98 and cores: 1, disk: >2147483648, and memory: 2147483648
> Acquiring 2147483648 bytes of memory from pool of 9223372036854775807.
> Attempting to acquire 10 threads for 1 cpus submitted
> Acquiring semaphore blockingly.
> Semaphore acquired.
> Executing command: '/home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv'.
> Traceback (most recent call last):
>  File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line >24, in <module>
>    sys.path.remove(os.path.dirname(os.path.abspath(**file**)))
> ValueError: list.remove(x): x not in list
> Got exit code 1 (indicating failure) from command '/home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv'.



Discussion:
I want to get rid of that hack but for now we should just catch the ValueError and move on. I am puzzled though as to why this started happening. When running `/foo/bar.py` python always adds `/foo` to `sys.path`.

BTW: Not a big fan of installing from git. Not every commit to master passes the tests. Does "pip install --pre toil" not work? That way you'd get a build that is guaranteed to have passed the test suite.

Okay, I'll switch to the pre-release version. 

Actually it looks like this fix won't be good enough. I'm getting the conflict with toil.resource now: 

> n/n/jobZpx21Q:   File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line 221, in main
> n/n/jobZpx21Q:     startClock = getTotalCpuTime()
> n/n/jobZpx21Q:   File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 179, in getTotalCpuTime
> n/n/jobZpx21Q:     return getTotalCpuTimeAndMemoryUsage()[0]
> n/n/jobZpx21Q:   File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 170, in getTotalCpuTimeAndMemoryUsage
> n/n/jobZpx21Q:     me = resource.getrusage(resource.RUSAGE_SELF)
> n/n/jobZpx21Q: AttributeError: 'module' object has no attribute 'getrusage'

Could you please print **file** and sys.path and post the sys.path right before this happens and post the output here?

```
0/H/job6n3SVo:  ---TOIL WORKER OUTPUT LOG---
0/H/job6n3SVo:  Next available file descriptor: 7
0/H/job6n3SVo:  DEBUG:__main__:Next available file descriptor: 7
0/H/job6n3SVo:  Parsed job
0/H/job6n3SVo:  DEBUG:__main__:Parsed job
0/H/job6n3SVo:  INFO:toil.lib.bioio:Path = ['/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7', '/home/alden/progressiveCactus/python/lib/python2.7/plat-x86_64-linux-gnu', '/home/alden/progressiveCactus/python/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/lib/python2.7/lib-old', '/home/alden/progressiveCactus/python/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages', '/home/alden/progressiveCactus/submodules']
0/H/job6n3SVo:  INFO:toil.lib.bioio:__file__ = /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.pyc
0/H/job6n3SVo:  Traceback (most recent call last):
0/H/job6n3SVo:    File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line 221, in main
0/H/job6n3SVo:      startClock = getTotalCpuTime()
0/H/job6n3SVo:    File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 174, in getTotalCpuTime
0/H/job6n3SVo:      return getTotalCpuTimeAndMemoryUsage()[0]
0/H/job6n3SVo:    File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 165, in getTotalCpuTimeAndMemoryUsage
0/H/job6n3SVo:      me = resource.getrusage(resource.RUSAGE_SELF)
0/H/job6n3SVo:  AttributeError: 'module' object has no attribute 'getrusage'
0/H/job6n3SVo:  Exiting the worker because of a failed job on host thinkpad
0/H/job6n3SVo:  ERROR:__main__:Exiting the worker because of a failed job on host thinkpad
0/H/job6n3SVo:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 0/H/job6n3SVo to 0
```

Sorry, I meant printing `sys.path` and `__file__` in worker.py _before_ the `sys.path` modification is done. 

```
>Traceback (most recent call last):
>  File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line >28, in <module>
>    sys.path.remove(os.path.dirname(os.path.abspath(__file__)))
>ValueError: list.remove(x): x not in list
>sys.path: ['/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7', '/home/alden/progressiveCactus/python/lib/python2.7/plat-x86_64-linux-gnu', '/home/alden/progressiveCactus/python/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/lib/python2.7/lib-old', '/home/alden/progressiveCactus/python/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages']
>__file__: /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py
```

I think I know what it is. Can you print the result of `os.path.abspath(file)` as well. I think one of the directories that make up the path   `/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil` is a symlink.

sys.path: ['/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7', '/home/alden/progressiveCactus/python/lib/python2.7/plat-x86_64-linux-gnu', '/home/alden/progressiveCactus/python/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/lib/python2.7/lib-old', '/home/alden/progressiveCactus/python/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages']

os.path.aspath(**file**): /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py

os.path.dirname(os.path.abspath(**file**)): /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil

It seems that `/home/alden/progressiveCactus/python/local` is a symlink to `/home/alden/progressiveCactus/python` 

Could you verify that by running `ls -l /home/alden/progressiveCactus/python`

alden@thinkpad:~$ ls -l /home/alden/progressiveCactus/python
total 20
drwxrwxr-x 2 alden alden 4096 Aug 28 11:36 bin
drwxrwxr-x 2 alden alden 4096 Aug 27 16:50 include
drwxrwxr-x 3 alden alden 4096 Aug 27 16:50 lib
drwxrwxr-x 2 alden alden 4096 Aug 27 16:50 local
drwxrwxr-x 3 alden alden 4096 Aug 27 16:50 share

On Fri, Aug 28, 2015 at 2:35 PM, Hannes Schmidt notifications@github.com
wrote:

> It seems that /home/alden/progressiveCactus/python/local is a symlink to
> /home/alden/progressiveCactus/python
> 
> Could you verify that by running ls -l
> /home/alden/progressiveCactus/python
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/BD2KGenomics/toil/issues/290#issuecomment-135894036.

Just for posterity: Alden and I confirmed that he's got symlinks in /home/alden/progressiveCactus/python/local. Solution will be to apply abspath to each sys.path before comparing and removing.

I want to get rid of that hack but for now we should just catch the ValueError and move on. I am puzzled though as to why this started happening. When running `/foo/bar.py` python always adds `/foo` to `sys.path`.

BTW: Not a big fan of installing from git. Not every commit to master passes the tests. Does "pip install --pre toil" not work? That way you'd get a build that is guaranteed to have passed the test suite.

Okay, I'll switch to the pre-release version. 

Actually it looks like this fix won't be good enough. I'm getting the conflict with toil.resource now: 

> n/n/jobZpx21Q:   File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line 221, in main
> n/n/jobZpx21Q:     startClock = getTotalCpuTime()
> n/n/jobZpx21Q:   File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 179, in getTotalCpuTime
> n/n/jobZpx21Q:     return getTotalCpuTimeAndMemoryUsage()[0]
> n/n/jobZpx21Q:   File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 170, in getTotalCpuTimeAndMemoryUsage
> n/n/jobZpx21Q:     me = resource.getrusage(resource.RUSAGE_SELF)
> n/n/jobZpx21Q: AttributeError: 'module' object has no attribute 'getrusage'

Could you please print **file** and sys.path and post the sys.path right before this happens and post the output here?

```
0/H/job6n3SVo:  ---TOIL WORKER OUTPUT LOG---
0/H/job6n3SVo:  Next available file descriptor: 7
0/H/job6n3SVo:  DEBUG:__main__:Next available file descriptor: 7
0/H/job6n3SVo:  Parsed job
0/H/job6n3SVo:  DEBUG:__main__:Parsed job
0/H/job6n3SVo:  INFO:toil.lib.bioio:Path = ['/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7', '/home/alden/progressiveCactus/python/lib/python2.7/plat-x86_64-linux-gnu', '/home/alden/progressiveCactus/python/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/lib/python2.7/lib-old', '/home/alden/progressiveCactus/python/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages', '/home/alden/progressiveCactus/submodules']
0/H/job6n3SVo:  INFO:toil.lib.bioio:__file__ = /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.pyc
0/H/job6n3SVo:  Traceback (most recent call last):
0/H/job6n3SVo:    File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line 221, in main
0/H/job6n3SVo:      startClock = getTotalCpuTime()
0/H/job6n3SVo:    File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 174, in getTotalCpuTime
0/H/job6n3SVo:      return getTotalCpuTimeAndMemoryUsage()[0]
0/H/job6n3SVo:    File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 165, in getTotalCpuTimeAndMemoryUsage
0/H/job6n3SVo:      me = resource.getrusage(resource.RUSAGE_SELF)
0/H/job6n3SVo:  AttributeError: 'module' object has no attribute 'getrusage'
0/H/job6n3SVo:  Exiting the worker because of a failed job on host thinkpad
0/H/job6n3SVo:  ERROR:__main__:Exiting the worker because of a failed job on host thinkpad
0/H/job6n3SVo:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 0/H/job6n3SVo to 0
```

Sorry, I meant printing `sys.path` and `__file__` in worker.py _before_ the `sys.path` modification is done. 

```
>Traceback (most recent call last):
>  File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line >28, in <module>
>    sys.path.remove(os.path.dirname(os.path.abspath(__file__)))
>ValueError: list.remove(x): x not in list
>sys.path: ['/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7', '/home/alden/progressiveCactus/python/lib/python2.7/plat-x86_64-linux-gnu', '/home/alden/progressiveCactus/python/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/lib/python2.7/lib-old', '/home/alden/progressiveCactus/python/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages']
>__file__: /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py
```

I think I know what it is. Can you print the result of `os.path.abspath(file)` as well. I think one of the directories that make up the path   `/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil` is a symlink.

sys.path: ['/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7', '/home/alden/progressiveCactus/python/lib/python2.7/plat-x86_64-linux-gnu', '/home/alden/progressiveCactus/python/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/lib/python2.7/lib-old', '/home/alden/progressiveCactus/python/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages']

os.path.aspath(**file**): /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py

os.path.dirname(os.path.abspath(**file**)): /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil

It seems that `/home/alden/progressiveCactus/python/local` is a symlink to `/home/alden/progressiveCactus/python` 

Could you verify that by running `ls -l /home/alden/progressiveCactus/python`

alden@thinkpad:~$ ls -l /home/alden/progressiveCactus/python
total 20
drwxrwxr-x 2 alden alden 4096 Aug 28 11:36 bin
drwxrwxr-x 2 alden alden 4096 Aug 27 16:50 include
drwxrwxr-x 3 alden alden 4096 Aug 27 16:50 lib
drwxrwxr-x 2 alden alden 4096 Aug 27 16:50 local
drwxrwxr-x 3 alden alden 4096 Aug 27 16:50 share

On Fri, Aug 28, 2015 at 2:35 PM, Hannes Schmidt notifications@github.com
wrote:

> It seems that /home/alden/progressiveCactus/python/local is a symlink to
> /home/alden/progressiveCactus/python
> 
> Could you verify that by running ls -l
> /home/alden/progressiveCactus/python
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/BD2KGenomics/toil/issues/290#issuecomment-135894036.

Just for posterity: Alden and I confirmed that he's got symlinks in /home/alden/progressiveCactus/python/local. Solution will be to apply abspath to each sys.path before comparing and removing.


Resolving patch:
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -21,7 +21,8 @@
 if __name__ == "__main__":
     # FIXME: Until we use setuptools entry points, this is the only way to avoid a conflict between our own resource.py
     # and Python's
-    sys.path.remove(os.path.dirname(os.path.abspath(__file__)))
+    toilSrcDir = os.path.dirname(os.path.realpath(__file__))
+    sys.path = [directory for directory in sys.path if not os.path.realpath(directory) == toilSrcDir]
 
 import traceback
 import time


Test patch: