Repository: DataBiosphere/toil

Issue:
Getting lstat error from docker on tcga1
_Note_: It is not clear that this issue belongs in Toil.

When I run a docker container from a toil script _on tcga1_ (CentOS), I get a persistent error:

```
lstat /inside/depot/users/jeltje/aligns/tmpVno9c1/localTempDir: permission denied"
```

This kills the job, deleting the tmp dir that had the problem (too fast to have a look at permissions).
This problem does not occur in on my Ubuntu VM.

I created a very simple docker container that creates a file, runs ls -l, deletes the file, then sleeps for 3 minutes. It works fine on the tcga1 command line, but when run via the toil script it dies almost instantly. Apparently it doesn't even get to the docker run (otherwise it would sleep before dying).
[the failing script](http://hgwdev.cse.ucsc.edu/~jeltje/toil/toil_sleepy.py)
[wrapper to run it](http://hgwdev.cse.ucsc.edu/~jeltje/toil/sleepy_sh.txt)
[stderr](http://hgwdev.cse.ucsc.edu/~jeltje/toil/sleepy.err)

I also created a non-docker toil script (bash echoes a statement). This script works fine.
[the working script](http://hgwdev.cse.ucsc.edu/~jeltje/toil/toil_test.py)
[wrapper to run it](http://hgwdev.cse.ucsc.edu/~jeltje/toil/test_sh.txt)
[stderr](http://hgwdev.cse.ucsc.edu/~jeltje/toil/test.err)

bioio.getTempDirectory is redundant
There is nothing in bioio.getTempDirectory that can't be achieved with tempfile.mkdtemp. If a root dir is passed in, we can just use tempfile.mkdtemp( ..., dir=... ).



Discussion:
Try adding

```
subprocess.check_call( [ 'ls', '-ld', work_dir ] )
```

just before

```
    with open(os.path.join(work_dir, uuid + '.bam'), 'w') as f_out:
        subprocess.check_call(docker_cmd + bamsort_command, stdout=f_out)
```

in the failing script. It's definitely a permission problem and we may be able to work around it generally in Toil.

The plot thickens...
From the error log:

```
WARNING:toil.leader:B/S/jobNJ0NCW:      WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/inside/home/jeltje/exp/readalign/git/toil-scripts/batch-alignment', name='toil_sleepy', extension='.py')
WARNING:toil.leader:B/S/jobNJ0NCW:      drwxrwxrwx 2 jeltje protein 2 Oct  1 11:22 /inside/depot/jeltje/aligns/tmpKI8QCz/localTempDir
WARNING:toil.leader:B/S/jobNJ0NCW:      time="2015-10-01T11:22:50-07:00" level="fatal" msg="Error response from daemon: lstat /inside/depot/users/jeltje/aligns/tmpKI8QCz/localTempDir: permission denied"
```

So the directory gets created as me, with full permissions.

I tried to find the lstat command from inside the docker container, but it doesn't come standard (weirdly) and I can't figure out which package to apt-get. Looks like it's docker itself that's doing this.

AHA! `ls -d` one level up:

```
WARNING:toil.leader:1/D/job6Du4tp:      drwx------+ 3 jeltje protein 4 Oct  1 11:42 /inside/depot/jeltje/aligns/tmpZQ4Wde
```

And what user does the docker daemon run as? I should be root, just double-checking.

Correct. User root, group root.

So this could be SELinux getting in the way. Could you send the contents of `/etc/sysconfig/docker` if it exists and is readable? Also, try adding this to your user script, just before the `docker` invocation:

```
subprocess.check_call( [ 'chmod', 'go+rx', work_dir, work_dir + '/..' ] )
```

Leave the `ls` invocation in place for verification that the `chmod` worked.

That totally fixed it!
Docker runs as expected now and creates correct output.

```
# /etc/sysconfig/docker
other_args="--graph=/data/docker"
DOCKER_CERT_PATH=/etc/docker
```

I suppose this is not the preferred fix, though? 

We _could_ ask the admins to add `--selinux-enabled=false` to `/etc/sysconfig/docker` in order to confirm that it is indeed SELinux that is causing this. I doubt they would want to disable it permanently and I doubt they have the time to do so temporarily just for us to confirm a hypothesis. Your call.

Ultimately, we should support running docker under SELinux. We could brute force it by always chmod'ing the work dir and its parent in the above way.

What is `/inside/depot/jeltje/aligns`? Is this a directory that you chose? Is it your home? Is it your `TMPDIR`?

It's my TMPDIR. `inside/depot` has several terabytes free so I can mess around with large bam files to my heart's content.

SElinux was disabled by Erich, but this doesn't fix the problem

```
# /etc/sysconfig/docker
other_args="--graph=/data/docker --selinux-enabled=false"
DOCKER_CERT_PATH=/etc/docker
```

If I leave the chmod command out of my toil script, I get the original problem again:

```
WARNING:toil.leader:i/k/jobIRPbYs:      drwx------+ 3 jeltje protein 4 Oct  2 09:04 /inside/depot/jeltje/aligns/tmpxXVAst
WARNING:toil.leader:i/k/jobIRPbYs:      drwxrwxrwx 2 jeltje protein 2 Oct  2 09:04 /inside/depot/jeltje/aligns/tmpxXVAst/localTempDir
```

So it looks like this a caused by a combination of facts:

0) The file is in a subdirectory of $TMPDIR
1) TMPDIR points at an NFS
2) The NFS is mount with root-squash (hypothesized)
3) Docker is running as root
4) The file, its parent, and its grandparent are owned by a different user (jeltje)
5) The grandparent was created by Python's tempfile.mkdtemp with mode 700

Overriding the mode or moving TMPDIR to a FS on a locally attached drive both fix the problem. 

The question now is: Do we work around it in Toil by overriding the mode of work_dir and its parent to 755? I hesitate due to the specificity of the factors listed above.

Or maybe it should be a note in the toil manual eventually, describing a special case and the chmod solution?

Is it possible to have the toil system check permissions when it creates a tmpdir and issue a (non fatal) warning? That may save the next victim some time.

We'll enforce the mode to 755 for localWorkerTempDir.

Related to #480.

Yes, +1 to cleanup bioio, which is generally full of cruft.


Resolving patch:
diff --git a/src/toil/lib/bioio.py b/src/toil/lib/bioio.py
--- a/src/toil/lib/bioio.py
+++ b/src/toil/lib/bioio.py
@@ -289,33 +289,6 @@ def getTempFile(suffix="", rootDir=None):
         os.chmod(tmpFile, 0777) #Ensure everyone has access to the file.
         return tmpFile
 
-def getTempDirectory(rootDir=None):
-    """
-    returns a temporary directory that must be manually deleted. rootDir will be
-    created if it does not exist.
-    """
-    if rootDir is None:
-        return tempfile.mkdtemp()
-    else:
-        if not os.path.exists(rootDir):
-            try:
-                os.makedirs(rootDir)
-            except OSError:
-                # Maybe it got created between the test and the makedirs call?
-                pass
-            
-        while True:
-            # Keep trying names until we find one that doesn't exist. If one
-            # does exist, don't nest inside it, because someone else may be
-            # using it for something.
-            tmpDir = os.path.join(rootDir, "tmp_" + getRandomAlphaNumericString())
-            if not os.path.exists(tmpDir):
-                break
-                
-        os.mkdir(tmpDir)
-        os.chmod(tmpDir, 0777) #Ensure everyone has access to the file.
-        return tmpDir
-
 def main():
     pass
 
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -26,6 +26,7 @@
     toilSrcDir = os.path.dirname(os.path.realpath(__file__))
     sys.path = [directory for directory in sys.path if not os.path.realpath(directory) == toilSrcDir]
 
+import tempfile
 import traceback
 import time
 import socket
@@ -86,7 +87,6 @@ def main():
     from toil.lib.bioio import setLogLevel
     from toil.lib.bioio import getTotalCpuTime
     from toil.lib.bioio import getTotalCpuTimeAndMemoryUsage
-    from toil.lib.bioio import getTempDirectory
     from toil.lib.bioio import makePublicDir
     from toil.lib.bioio import system
     from toil.common import loadJobStore
@@ -148,8 +148,9 @@ def badWorker():
         
     #Dir to put all the temp files in. If tempRootDir is None, tempdir looks at environment variables to determine
     # where to put the tempDir.
-    localWorkerTempDir = getTempDirectory(tempRootDir)
-    
+    localWorkerTempDir = tempfile.mkdtemp(dir=tempRootDir)
+    os.chmod(localWorkerTempDir, 0755)
+
     ##########################################
     #Setup the logging
     ##########################################


Test patch:
diff --git a/src/toil/test/src/jobTest.py b/src/toil/test/src/jobTest.py
--- a/src/toil/test/src/jobTest.py
+++ b/src/toil/test/src/jobTest.py
@@ -16,7 +16,7 @@
 import os
 import random
 
-from toil.lib.bioio import getTempFile, getTempDirectory
+from toil.lib.bioio import getTempFile
 from toil.job import Job, JobGraphDeadlockException
 from toil.test import ToilTest