Repository: DataBiosphere/toil

Issue:
job.fileStore.readGlobalFile() should have an option to write to a specific temp dir obtained using job.fileStore.getLocalTempDir()
if a tool requires n inputs, we need them all to be in the same dir so the dir can be mounted to /data in a docker image. We cannot assume that job.fileStore.readGlobalFile() as it is right now will read all n files into the same directory



Discussion:
If you need the file in a different place, you should hard-link into that place.

Come to think of it: Why don't we do that in readGlobalFile() as a service. Sounds like most users will run into this problem. @benedictpaten what do you think?

When you use readGlobalFile you get one of 2 return values,
1. The absolute path to the cached file
2. The relative path to the file retrieved from the filestore

This means the user needs to test whether the file path is absolute before making the hard link ( I ran into this when i kept getting  OSError 18 about cross-device links).  I agree with @hannes-ucsc that it would be great if readGlobalFile did something like

```
def readGlobalFile(self, fileStoreID, outfile=None):
    """
    Returns a path to a local copy of the file keyed by fileStoreID. 
    The file will be read only. 
    """
    if fileStoreID in self.filesToDelete:
        raise RuntimeError("Trying to access a file in the jobStore you've deleted: %s" % fileStoreID)
    #When requesting a new file from the jobStore first check if fileStoreID
    #is a key in jobStoreFileIDToCacheLocation.
    if fileStoreID in self.jobStoreFileIDToCacheLocation:
        #IF it is, use the cached item
        #  If outfile is provided, ensure is in the temp dir and then return a har link to it
        if outfile:
            assert outfile.isIn(self. localTempDir), "outfile not in the same filesystem as cache"
            os.link(self.jobStoreFileIDToCacheLocation[fileStoreID], outfile)
            return outfile
        #  Else return the cached item
        else:
            return self.jobStoreFileIDToCacheLocation[fileStoreID]

    else:
        #If it is not in the cache read it from the jobStore to the 
        #desired location
        # if outfile is not provided, get a tempfile
        if not outfile:
            localFilePath = self.getLocalTempFile()
        else:
            localFilePath=os.path.abspath(outfile)
        self.jobStore.readFile(fileStoreID, localFilePath)
        self.jobStoreFileIDToCacheLocation[fileStoreID] = localFilePath
        #Chmod to make file read only
        os.chmod(localFilePath, stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
        return localFilePath
```

Independently of whether readGlobalFile() should offer to hard-link the file, it should always return either absolute or relative paths, but not a mix. If the returned paths are relative, they should be relative to a well-known, fixed directory.

They are currently relative to the jobstore directory.
I'm thinking of doing something like

```
if os.path.exists(readGlobalFile_return_value):
    # Then the file was from the cache
    os.link(readGlobalFile_return_value, outfile)
else:
    # Relative path to jobstoreid
    os.link(os.path.join(jobstore_dir, readGlobalFile_return_value), outfile)
```

The relative / absolute path is a bug. 

The path thing could be solved partially if the specified user path is always a location within a directory of subdirectory returned by getLocalTempDir(), or is a location returned by getLocalTempFile().  

I am going to proceed with adding that as an option, and fix the relative/absolute path bug.


Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -429,41 +429,45 @@ def asyncWrite():
 
         def getLocalTempDir(self):
             """
-            Get a new local temporary directory. This directory will exist for the 
+            Get the absolute path to a new local temporary directory. 
+            This directory will exist for the 
             duration of the job only, and is guaranteed to be deleted once
             the job terminates, removing all files it contains recursively. 
             """
-            return tempfile.mkdtemp(prefix="t", dir=self.localTempDir)
+            return os.path.abspath(tempfile.mkdtemp(prefix="t", dir=self.localTempDir))
         
         def getLocalTempFile(self):
             """
-            Get a local temporary file. This file will exist for the duration of the job only, and
+            Get an absolute path to a local temporary file. 
+            This file will exist for the duration of the job only, and
             is guaranteed to be deleted once the job terminates.
             """
             handle, tmpFile = tempfile.mkstemp(prefix="tmp", 
                                                suffix=".tmp", dir=self.localTempDir)
             os.close(handle)
-            return tmpFile
+            return os.path.abspath(tmpFile)
 
         def writeGlobalFile(self, localFileName, cleanup=False):
             """
-            Takes a file (as a path) and uploads it to to the global file store, returns
-            an ID that can be used to retrieve the file. 
+            Takes a file (as a path) and uploads it to to the global file store, 
+            returns an ID that can be used to retrieve the file. 
             
             If cleanup is True then the global file will be deleted once the job
-            and all its successors have completed running. If not the global file must be deleted
-            manually.
+            and all its successors have completed running. If not the global file 
+            must be deleted manually.
             
-            The write is asynchronous, so further modifications to the file pointed by
-            localFileName will result in undetermined behavior. The local file is safely removed 
-            at the end of the job by placing it in a location returned 
-            by getLocalTempDir.
+            The write is asynchronous, so further modifications during execution 
+            to the file pointed by localFileName will result in undetermined behavior. 
             """
             jobStoreFileID = self.jobStore.getEmptyFileStoreID(None 
                             if not cleanup else self.jobWrapper.jobStoreID)
             self.queue.put((open(localFileName, 'r'), jobStoreFileID))
-            #Now put the file into the cache
-            self.jobStoreFileIDToCacheLocation[jobStoreFileID] = localFileName
+            #Now put the file into the cache if it is a path within localTempDir
+            absLocalFileName = os.path.abspath(localFileName)
+            if absLocalFileName.startswith(self.localTempDir):
+                #Chmod to make file read only
+                os.chmod(absLocalFileName, stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
+                self.jobStoreFileIDToCacheLocation[jobStoreFileID] = absLocalFileName
             return jobStoreFileID
         
         def writeGlobalFileStream(self, cleanup=False):
@@ -478,22 +482,49 @@ def writeGlobalFileStream(self, cleanup=False):
             #TODO: Make this work with the caching??
             return self.jobStore.writeFileStream(None if not cleanup else self.jobWrapper.jobStoreID)
         
-        def readGlobalFile(self, fileStoreID):
+        def readGlobalFile(self, fileStoreID, userPath=None):
             """
-            Returns a path to a local copy of the file keyed by fileStoreID. 
-            The file will be read only. 
+            Returns an absolute path to a local, temporary copy of the file 
+            keyed by fileStoreID. 
+            
+            *The returned file will be read only (have permissions 444).* 
+            
+            :param userPath: a path to the name of file to which the global file will be 
+            copied or hard-linked (see below). userPath must either be: (1) a 
+            file path contained within a directory or, recursively, a subdirectory 
+            of a temporary directory returned by Job.FileStore.getLocalTempDir(), 
+            or (2) a file path returned by Job.FileStore.getLocalTempFile(). 
+            If userPath is specified and this is not true a RuntimeError exception 
+            will be raised. If userPath is specified and the file is already cached, 
+            the userPath file will be a hard link to the actual location, else it 
+            will be an actual copy of the file.  
             """
             if fileStoreID in self.filesToDelete:
                 raise RuntimeError("Trying to access a file in the jobStore you've deleted: %s" % fileStoreID)
+            if userPath != None:
+                userPath = os.path.abspath(userPath) #Make an absolute path
+                #Check it is a valid location
+                if not userPath.startswith(self.localTempDir):
+                    raise RuntimeError("The user path is not contained within the"
+                                       " temporary file hierarchy created by the job."
+                                       " User path: %s, temporary file root path: %s" % 
+                                       (userPath, self.localTempDir))
             #When requesting a new file from the jobStore first check if fileStoreID
             #is a key in jobStoreFileIDToCacheLocation.
             if fileStoreID in self.jobStoreFileIDToCacheLocation:
-                #IF it is, return it
-                return self.jobStoreFileIDToCacheLocation[fileStoreID]   
+                cachedAbsFilePath = self.jobStoreFileIDToCacheLocation[fileStoreID]   
+                #If the user specifies a location and it is not the current location
+                # return a hardlink to the location, else return the original location
+                if userPath == None or userPath == cachedAbsFilePath:
+                    return cachedAbsFilePath
+                if os.path.exists(userPath):
+                    os.remove(userPath)
+                os.link(cachedAbsFilePath, userPath)
+                return userPath
             else:
                 #If it is not in the cache read it from the jobStore to the 
                 #desired location
-                localFilePath = self.getLocalTempFile()
+                localFilePath = userPath if userPath != None else self.getLocalTempFile()
                 self.jobStore.readFile(fileStoreID, localFilePath)
                 self.jobStoreFileIDToCacheLocation[fileStoreID] = localFilePath
                 #Chmod to make file read only
@@ -616,9 +647,10 @@ def _cleanLocalTempDir(self, cacheSize):
                                   self.jobStoreFileIDToCacheLocation.keys())
             #Total number of bytes stored in cached files
             totalCachedFileSizes = sum(map(lambda x : x[0], cachedFileSizes))
-            #Remove largest files first - this is not obviously best, could do it a different
+            #Remove smallest files first - this is not obviously best, could do it a different
             #way
             cachedFileSizes.sort()
+            cachedFileSizes.reverse()
             #Now do the actual file removal
             while totalCachedFileSizes > cacheSize:
                 fileSize, fileStoreID =  cachedFileSizes.pop()


Test patch:
diff --git a/src/toil/test/src/jobFileStoreTest.py b/src/toil/test/src/jobFileStoreTest.py
--- a/src/toil/test/src/jobFileStoreTest.py
+++ b/src/toil/test/src/jobFileStoreTest.py
@@ -58,7 +58,17 @@ def fileTestJob(job, inputFileStoreIDs, testStrings, chainLength):
         #Load the input jobStoreFileIDs and check that they map to the 
         #same set of random input strings
         if random.random() > 0.666:
-            tempFile = job.fileStore.readGlobalFile(fileStoreID)
+            #Check we get an exception if we try a file out side of the file path
+            try:
+                job.fileStore.readGlobalFile(fileStoreID, userPath="THIS/IS/NOT/A/REAL/PATH")
+            except RuntimeError:
+                pass
+            if random.random() > 0.5:
+                tempFile = job.fileStore.readGlobalFile(fileStoreID)
+            else:
+                userPath = job.fileStore.getLocalTempFile()
+                tempFile = job.fileStore.readGlobalFile(fileStoreID, userPath)
+                assert userPath == tempFile
             with open(tempFile, 'r') as fH:
                 string = fH.readline()
                 assert testStrings[string[:PREFIX_LENGTH]] == string