Repository: pantsbuild/pants

Issue:
Support 'file' scheme for external tool downloads
It would be great if this worked, so I could pull these tools off a shared filesystem.

```
[scc]
url_template = "file:/path/to/scc"
```



Discussion:
We should absolutely do this: thanks for the report!

Aside from adjusting validation, I believe that the implementation would look like adding a conditional to the body of the rule that fetches the tool: https://github.com/pantsbuild/pants/blob/bc31814810a37ba7b9452d460653fff9025d4e21/src/python/pants/core/util_rules/external_tool.py#L244-L246 ... to either `await Get(Digest, DownloadFile(..))`, or `await Get(Snapshot, PathGlobs([f'{relative_file_path}']))` (and then validate that the `Snapshot` captured a single file). This would have some downsides though: it would only support relative file URLs, and it would not be able to validate the file digest (currently).

A better implementation though might be to adjust the rust code to support both schemes: https://github.com/pantsbuild/pants/blob/bc31814810a37ba7b9452d460653fff9025d4e21/src/rust/engine/src/nodes.rs#L734-L749 ... because there is an associated `Digest` to validate, we know that we do not need to "watch" the file path, and can simply use rust's file API to capture it.
That could be a frustrating limitation that the file must be relative to the build root. Often, binaries like scc are installed in a more global location. 

Instead, I suspect we'll want to add back proper support for PathGlobsAndRoot - I think there's an issue tracking it, as we need the functionality already and hack around it by cheating with Path().
@Eric-Arellano : Had the same thoughts. Updated my comment.
If the file: support required size and hash just like the https?: support then there would be no need for path globs, etc - right? In fact, if reqwest supported file: urls then this would all just work.
Beyond how to specify the "URL", another complication is what that URL can point to. Atm, users have no way to change the `generate_exe()` path through the options system - this method defines how to locate the binary within the downloaded file, e.g. `./bin/scc` or `./pex`. So, you could not point to, say, an SCC installed via apt-get, as it would not be the zip file we expect.

I think the solution is simple: add `--exe-template` and `--exe-platform-mapping` options to `TemplatedExternalTool`. @thamenato and I did not do that originally to avoid new complexity, but it could be helpful even without this file scheme added. It would allow you to host a file/archive with a different structure than the default.
> If the file: support required size and hash just like the https?: support then there would be no need for path globs, etc - right? In fact, if reqwest supported file: urls then this would all just work.

Yep, exactly. I had updated my comment to reflect that. I expect that doing this directly in the `DownloadFile` intrinsic on the rust side is the way to go.
> another complication is what that URL can point to...I think the solution is simple

@gatesn let me know if you'd be interested in adding this change, and I'd be happy to write up some instructions. With that added, the only blocker is the Rust changes Stu is mentioning. If not, I'll try to add it late next week.
Does rust not support opening arbitrary URLs? E.g. Java and Python allow globally registering support for a scheme.

urllib.open for instance works with http or file, and probably a bunch of other things.
> Does rust not support opening arbitrary URLs? E.g. Java and Python allow globally registering support for a scheme.
>
> urllib.open for instance works with http or file, and probably a bunch of other things.

We use the [Reqwest library](https://github.com/seanmonstar/reqwest) and it does not. A feature request is outstanding though: https://github.com/seanmonstar/reqwest/issues/178

Resolving patch:
diff --git a/src/python/pants/core/util_rules/external_tool.py b/src/python/pants/core/util_rules/external_tool.py
--- a/src/python/pants/core/util_rules/external_tool.py
+++ b/src/python/pants/core/util_rules/external_tool.py
@@ -204,9 +204,10 @@ def register_options(cls, register):
             help=(
                 "URL to download the tool, either as a single binary file or a compressed file "
                 "(e.g. zip file). You can change this to point to your own hosted file, e.g. to "
-                "work with proxies.\n\nUse `{version}` to have the value from --version "
-                "substituted, and `{platform}` to have a value from --url-platform-mapping "
-                "substituted in, depending on the current platform. For example, "
+                "work with proxies or for access via the filesystem through a file:// URL.\n\nUse "
+                "`{version}` to have the value from --version substituted, and `{platform}` to "
+                "have a value from --url-platform-mapping substituted in, depending on the "
+                "current platform. For example, "
                 "https://github.com/.../protoc-{version}-{platform}.zip."
             ),
         )
diff --git a/src/python/pants/engine/fs.py b/src/python/pants/engine/fs.py
--- a/src/python/pants/engine/fs.py
+++ b/src/python/pants/engine/fs.py
@@ -243,7 +243,7 @@ class AddPrefix:
 
 @dataclass(frozen=True)
 class DownloadFile:
-    """Download an asset via a GET request.
+    """Retrieve the contents of a file via an HTTP GET request or directly for local file:// URLs.
 
     To compute the `expected_digest`, manually download the file, then run `shasum -a 256` to
     compute the fingerprint and `wc -c` to compute the expected length of the downloaded file in


Test patch: