Repository: DataBiosphere/toil

Issue:
`readGlobalFile(symlink=False)` can still yield a symlink when used with fileJobStore and symlinked imports
The `FileJobStore.readFile()` method will give you a hardlink to the stored file in the job store, if you call it with `symlink=False` and it thinks a hardlink is possible:

https://github.com/DataBiosphere/toil/blob/aa50bbfdef66bd9a861fb889325e476405fe25b6/src/toil/jobStores/fileJobStore.py#L476-L483

But it doesn't check to see if the file in the job store is itself a symlink. So you can end up getting a hardlink to a symlink on platforms that support such a construct, like Linux. Then you will end up with symlinks that you can't pass to Docker calls, even if you wanted real files or hard links to real files, which can.

This affects vg's toil-vg CI process on Toil 5.4.0.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-1028)
┆Issue Number: TOIL-1028



Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -474,6 +474,10 @@ def readFile(self, jobStoreFileID, localFilePath, symlink=False):
                     raise
 
         # If we get here, symlinking isn't an option.
+        # Make sure we are working with the real source path, in case it is a
+        # symlinked import.
+        jobStoreFilePath = os.path.realpath(jobStoreFilePath)
+
         if os.stat(jobStoreFilePath).st_dev == os.stat(localDirPath).st_dev:
             # It is possible that we can hard link the file.
             # Note that even if the device numbers match, we can end up trying


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -1185,6 +1185,45 @@ def test_jobstore_init_preserves_symlink_path(self):
             if original_filestore and os.path.exists(original_filestore):
                 shutil.rmtree(original_filestore)
 
+    @travis_test
+    def test_jobstore_does_not_leak_symlinks(self):
+        """Test that if we link imports into the FileJobStore, we can't get hardlinks to symlinks."""
+
+        temp_dir = None
+        download_dir = None
+
+        try:
+            # Grab a temp directory to make files in. Make sure it's on the
+            # same device as everything else.
+            temp_dir = os.path.abspath(self.namePrefix + '-import')
+            os.mkdir(temp_dir)
+            to_import = os.path.join(temp_dir, 'import-me')
+            with open(to_import, 'w') as f:
+                f.write('test')
+
+            # And a temp directory next to the job store to download to
+            download_dir = os.path.abspath(self.namePrefix + '-dl')
+            os.mkdir(download_dir)
+
+            # Import it as a symlink
+            file_id = self.jobstore_initialized.importFile('file://' + to_import, symlink=True)
+
+            # Take it out as a hard link or copy
+            download_to = os.path.join(download_dir, 'downloaded')
+            self.jobstore_initialized.readFile(file_id, download_to)
+
+            # Make sure it isn't a symlink
+            self.assertFalse(os.path.islink(download_to))
+
+        finally:
+            if temp_dir and os.path.exists(temp_dir):
+                # Clean up temp directory
+                shutil.rmtree(temp_dir)
+            if download_dir and os.path.exists(download_dir):
+                # Clean up download directory
+                shutil.rmtree(download_dir)
+
+
 
 @needs_google
 class GoogleJobStoreTest(AbstractJobStoreTest.Test):