Repository: DataBiosphere/toil

Issue:
Supporting gs urls for running Toil on Slurm
I have a test WDL that outputs the sizes of the input files. It takes an array of file urls and outputs a text file. I'm running this on a Slurm machine. It works fine when I pass the public url of an object in a public google bucket. But when I pass the gs url it says it does not exist. Isn't Toil supposed to work with gs urls?
I checked my access to the gs url:
```
gsutil ls -lh gs://masri/flagger/v0.3.0/chm13v2.0.censat.bed
218.93 KiB  2023-05-14T22:15:02Z  gs://masri/flagger/v0.3.0/chm13v2.0.censat.bed
TOTAL: 1 objects, 224187 bytes (218.93 KiB)
```

Here is the command I'm using for running Toil on Slurm

```
rm -Rf logdump && rm -Rf ./trash/tree && mkdir logdump && time SINGULARITY_CACHEDIR=`pwd`/trash/cache/.singularity/cache MINIWDL__SINGULARITY__IMAGE_CACHE=`pwd`/trash/cache/.cache/miniwdl toil-wdl-runner -e SINGULARITY_CACHEDIR -e MINIWDL__SINGULARITY__IMAGE_CACHE --disableProgress --logDebug --retryCount 0 --jobStore ./trash/tree --outputDirectory ./trash/out --outputFile ./trash/out.json --batchSystem slurm --batchLogsDir `pwd`/logdump `pwd`/get_sizes.wdl `pwd`/inputs_public_url.json 2>&1 | tee log_public_url.txt
```
I've attached the WDL, input json files and the log files I get for each json file; one with gs url and one with public url.
[test_gs_url.zip](https://github.com/DataBiosphere/toil/files/11502940/test_gs_url.zip)

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1337)
┆Issue Number: TOIL-1337



Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/abstractJobStore.py b/src/toil/jobStores/abstractJobStore.py
--- a/src/toil/jobStores/abstractJobStore.py
+++ b/src/toil/jobStores/abstractJobStore.py
@@ -78,6 +78,21 @@ def __init__(self, url: ParseResult) -> None:
         """
         super().__init__("The URL '%s' is invalid." % url.geturl())
 
+class UnimplementedURLException(RuntimeError):
+    def __init__(self, url: ParseResult, operation: str) -> None:
+        """
+        Make a new exception to report that a URL scheme is not implemented, or
+        that the implementation can't be loaded because its dependencies are
+        not installed.
+
+        :param url: The given URL
+        :param operation: Whether we are trying to 'import' or 'export'
+        """
+        super().__init__(
+            f"No available job store implementation can {operation} the URL "
+            f"'{url.geturl()}'. Ensure Toil has been installed "
+            f"with the appropriate extras."
+        )
 
 class NoSuchJobException(Exception):
     """Indicates that the specified job does not exist."""
@@ -339,8 +354,7 @@ def _findJobStoreForUrl(cls, url: ParseResult, export: bool = False) -> 'Abstrac
         for implementation in cls._get_job_store_classes():
             if implementation._supports_url(url, export):
                 return implementation
-        raise RuntimeError("No job store implementation supports %sporting for URL '%s'" %
-                           ('ex' if export else 'im', url.geturl()))
+        raise UnimplementedURLException(url, "export" if export else "import")
 
     # Importing a file with a shared file name returns None, but without one it
     # returns a file ID. Explain this to MyPy.
diff --git a/src/toil/wdl/wdltoil.py b/src/toil/wdl/wdltoil.py
--- a/src/toil/wdl/wdltoil.py
+++ b/src/toil/wdl/wdltoil.py
@@ -46,7 +46,7 @@
 from toil.job import AcceleratorRequirement, Job, JobFunctionWrappingJob, Promise, Promised, accelerators_fully_satisfy, parse_accelerator, unwrap, unwrap_all
 from toil.fileStores import FileID
 from toil.fileStores.abstractFileStore import AbstractFileStore
-from toil.jobStores.abstractJobStore import AbstractJobStore
+from toil.jobStores.abstractJobStore import AbstractJobStore, UnimplementedURLException
 from toil.lib.conversions import convert_units, human2bytes
 from toil.lib.misc import get_user_name
 from toil.lib.threading import global_mutex
@@ -705,9 +705,16 @@ def import_file_from_uri(uri: str) -> str:
             tried.append(candidate_uri)
             try:
                 imported = toil.import_file(candidate_uri)
+            except UnimplementedURLException as e:
+                # We can't find anything that can even support this URL scheme.
+                # Report to the user, they are probably missing an extra.
+                logger.critical('Error: ' + str(e))
+                sys.exit(1)
             except Exception:
-                # We couldn't try the import. Try the next URL
-                continue
+                # Something went wrong besides the file not being found. Maybe
+                # we have no auth.
+                logger.error("Something went wrong importing %s", candidate_uri)
+                raise
             if imported is None:
                 # Wasn't found there
                 continue
@@ -720,9 +727,7 @@ def import_file_from_uri(uri: str) -> str:
             return pack_toil_uri(imported, file_basename)
 
         # If we get here we tried all the candidates
-        # TODO: Make a more informative message?
-        logger.error('Could not find %s at any of: %s', uri, tried)
-        raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), uri)
+        raise RuntimeError(f"Could not find {uri} at any of: {tried}")
 
     return map_over_files_in_bindings(environment, import_file_from_uri)
 


Test patch: