Repository: pantsbuild/pants

Issue:
Port pantsd client to rust.
This saves us ~500ms on my machine:
```
$ hyperfine -w 2 './pants --no-anonymous-telemetry-enabled -V' 'nails-client 0.0.0.0:$(cat .pids/1fd8a12b4fc1/pantsd/socket) does-not-matter --no-anonymous-telemetry-enabled -V'
Benchmark #1: ./pants --no-anonymous-telemetry-enabled -V
  Time (mean ± σ):     700.7 ms ±  11.5 ms    [User: 483.5 ms, System: 45.2 ms]
  Range (min … max):   688.7 ms … 722.2 ms    10 runs
 
Benchmark #2: nails-client 0.0.0.0:$(cat .pids/1fd8a12b4fc1/pantsd/socket) does-not-matter --no-anonymous-telemetry-enabled -V
  Time (mean ± σ):     178.1 ms ±   4.6 ms    [User: 2.8 ms, System: 1.5 ms]
  Range (min … max):   173.4 ms … 193.3 ms    16 runs
 
Summary
  'nails-client 0.0.0.0:$(cat .pids/1fd8a12b4fc1/pantsd/socket) does-not-matter --no-anonymous-telemetry-enabled -V' ran
    3.93 ± 0.12 times faster than './pants --no-anonymous-telemetry-enabled -V'
```


Discussion:
And the rust client is as snappy as we'll likely get since:
```
$ hyperfine -w 3 './pants --no-anonymous-telemetry-enabled -V' 'nails-client 127.0.0.1:$(cat .pids/1fd8a12b4fc1/pantsd/socket) does-not-matter --no-anonymous-telemetry-enabled -V' 'ng --nailgun-server 127.0.0.1 --nailgun-port $(cat .pids/1fd8a12b4fc1/pantsd/socket) does-not-matter --no-anonymous-telemetry-enabled -V'
Benchmark #1: ./pants --no-anonymous-telemetry-enabled -V
  Time (mean ± σ):     703.9 ms ±  14.8 ms    [User: 490.4 ms, System: 45.6 ms]
  Range (min … max):   688.6 ms … 739.4 ms    10 runs
 
Benchmark #2: nails-client 127.0.0.1:$(cat .pids/1fd8a12b4fc1/pantsd/socket) does-not-matter --no-anonymous-telemetry-enabled -V
  Time (mean ± σ):     181.9 ms ±   9.9 ms    [User: 2.6 ms, System: 1.9 ms]
  Range (min … max):   174.4 ms … 205.5 ms    16 runs
 
Benchmark #3: ng --nailgun-server 127.0.0.1 --nailgun-port $(cat .pids/1fd8a12b4fc1/pantsd/socket) does-not-matter --no-anonymous-telemetry-enabled -V
  Time (mean ± σ):     220.1 ms ±   4.0 ms    [User: 1.2 ms, System: 1.1 ms]
  Range (min … max):   215.8 ms … 227.0 ms    13 runs
 
Summary
  'nails-client 127.0.0.1:$(cat .pids/1fd8a12b4fc1/pantsd/socket) does-not-matter --no-anonymous-telemetry-enabled -V' ran
    1.21 ± 0.07 times faster than 'ng --nailgun-server 127.0.0.1 --nailgun-port $(cat .pids/1fd8a12b4fc1/pantsd/socket) does-not-matter --no-anonymous-telemetry-enabled -V'
    3.87 ± 0.22 times faster than './pants --no-anonymous-telemetry-enabled -V'
```
#11922 is epic! The client feels snappier, but do we have numbers to prove it? Is there more speed-related work to do? Or is the remaining work for this ticket related to bringing up pantsd?
I just did some casual benchmarking of `./pants` and `./pants help` and I don't see a noticeable improvement. Is that on deck for future work?
> The client feels snappier, but do we have numbers to prove it? Is there more speed-related work to do? 

It should be a lot snappier. Are you using `USE_NATIVE_PANTS=1`?

The numbers are in this PR: https://github.com/pantsbuild/pants/pull/11922#issuecomment-819882757

The client connection now takes a few ms (see this with -ldebug), so we can't win much more there. The remaining ~200ms noop is all in pantsd. Certainly there is work we could do on that side 

> Or is the remaining work for this ticket related to bringing up pantsd?

Yes, exactly.





Oh, hah, I was not using USE_NATIVE_PANTS=1. That is... excellent.
The combination of #18145 and https://github.com/pantsbuild/scie-pants/issues/94 give a natural home for the Pants native client in Pants distributions and execution flows. It would be good to finish off the client in anticipation of this. Towards that end I'll be updating this ticket with the remaining work outline needed to enable fully replacing the Pants python main entrypoint with the native Pants client.
FWIW: I've had `USE_NATIVE_PANTS` set for ... almost two years now, and never noticed an issue with it.
I'll edit this comment as I gather all the required missing elements and cleanups. The lists are not ordered intentionally, but they are labelled for discussion.

The required missing functionality:
+ [ ] 1. Handle launching pantsd when pantsd is not up and pantsd is enabled.

The cleanup tasks:
+ [ ] A. De-dup Pants client logic between Rust and Python in favor of Rust. #11960
+ [ ] B. Use the engine logging crate to setup logging in the Rust pants client. #11963

Notes:
+ 1 could be parametrized such that some information about how to launch Pantsd comes from configuration / the environment. In the pants scie case, there will be a known Python binary the Pants native client knows nothing about but will need to know about to run Pantsd as well as venv information.
+ B. Requires implementation of dict option parsing in Rust at the least - further review is needed.
I actually think the comment above covers things. 1 is the only thing needed to get a prototype of this working and it can probably be implemented robustly by injecting the command line for running the Python Pants client (or that command line's key components) via env var in the lift manifest. The client would then gain the logic currently in the Pants repo `./pants` script - try to connect to Pantsd and if there is a failure, fall back to the python client command line which already knows how to start pantsd and wait for it to be up. This is probably not an ideal end-state - it would likely be good to get the launching of pantsd and wait loop ported into the native client - but that work would be non-disruptive to the release + install process and users of it.

Resolving patch:
diff --git a/build-support/bin/generate_github_workflows.py b/build-support/bin/generate_github_workflows.py
--- a/build-support/bin/generate_github_workflows.py
+++ b/build-support/bin/generate_github_workflows.py
@@ -68,7 +68,7 @@ def hashFiles(path: str) -> str:
 
 
 NATIVE_FILES = [
-    ".pants",
+    "src/python/pants/bin/native_client",
     "src/python/pants/engine/internals/native_engine.so",
     "src/python/pants/engine/internals/native_engine.so.metadata",
 ]


Test patch:
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
--- a/.github/workflows/test.yaml
+++ b/.github/workflows/test.yaml
@@ -58,7 +58,7 @@ jobs:
       uses: actions/cache@v3
       with:
         key: Linux-ARM64-engine-${{ steps.get-engine-hash.outputs.hash }}-v1
-        path: '.pants
+        path: 'src/python/pants/bin/native_client
 
           src/python/pants/engine/internals/native_engine.so
 
@@ -93,7 +93,7 @@ jobs:
       uses: actions/upload-artifact@v3
       with:
         name: native_binaries.${{ matrix.python-version }}.Linux-ARM64
-        path: '.pants
+        path: 'src/python/pants/bin/native_client
 
           src/python/pants/engine/internals/native_engine.so
 
@@ -158,7 +158,7 @@ jobs:
       uses: actions/cache@v3
       with:
         key: Linux-x86_64-engine-${{ steps.get-engine-hash.outputs.hash }}-v1
-        path: '.pants
+        path: 'src/python/pants/bin/native_client
 
           src/python/pants/engine/internals/native_engine.so
 
@@ -193,7 +193,7 @@ jobs:
       uses: actions/upload-artifact@v3
       with:
         name: native_binaries.${{ matrix.python-version }}.Linux-x86_64
-        path: '.pants
+        path: 'src/python/pants/bin/native_client
 
           src/python/pants/engine/internals/native_engine.so
 
@@ -270,7 +270,7 @@ jobs:
       uses: actions/cache@v3
       with:
         key: macOS11-x86_64-engine-${{ steps.get-engine-hash.outputs.hash }}-v1
-        path: '.pants
+        path: 'src/python/pants/bin/native_client
 
           src/python/pants/engine/internals/native_engine.so
 
@@ -305,7 +305,7 @@ jobs:
       uses: actions/upload-artifact@v3
       with:
         name: native_binaries.${{ matrix.python-version }}.macOS11-x86_64
-        path: '.pants
+        path: 'src/python/pants/bin/native_client
 
           src/python/pants/engine/internals/native_engine.so
 
diff --git a/src/rust/engine/options/src/build_root_tests.rs b/src/rust/engine/options/src/build_root_tests.rs
--- a/src/rust/engine/options/src/build_root_tests.rs
+++ b/src/rust/engine/options/src/build_root_tests.rs
@@ -32,7 +32,7 @@ fn test_find_cwd() {
 
   assert_sentinel("BUILDROOT");
   assert_sentinel("BUILD_ROOT");
-  assert_sentinel("pants");
+  assert_sentinel("pants.toml");
 }
 
 #[test]
@@ -44,7 +44,7 @@ fn test_find_subdir() {
   assert!(BuildRoot::find_from(&buildroot_path).is_err());
   assert!(BuildRoot::find_from(&subdir).is_err());
 
-  let sentinel = &buildroot.path().join("pants");
+  let sentinel = &buildroot.path().join("pants.toml");
   fs::write(sentinel, []).unwrap();
   assert_eq!(
     &buildroot_path,