Repository: pantsbuild/pants

Issue:
Text file busy when attempting to execute script written into sandbox
Using a `Process` to invoke an `is_executable=True` script can result in:
```
Error launching process: Os { code: 26, kind: Other, message: "Text file busy" }
```

We have tests that cover this behavior: https://github.com/pantsbuild/pants/blob/49f5a0aa8b68a9f079481d7f6a682b217ad9742b/src/python/pants/engine/process_test.py#L250-L277 ... so it's possible that this is non-deterministic.


Discussion:
Grr this blocks #10442 https://travis-ci.com/github/pantsbuild/pants/jobs/366894459#L776.
To reproduce, apply this diff:

```diff
diff --git a/src/python/pants/engine/process.py b/src/python/pants/engine/process.py
index 2dbb58714..0001946d2 100644
--- a/src/python/pants/engine/process.py
+++ b/src/python/pants/engine/process.py
@@ -336,12 +336,12 @@ class BinaryPaths(EngineAware):
 @rule(desc="Find binary path")
 async def find_binary(request: BinaryPathRequest) -> BinaryPaths:
     # TODO(John Sirois): Replace this script with a statically linked native binary so we don't
-    #  depend on either /bin/bash being available on the Process host.
-    # TODO(#10507): Running the script directly from a shebang sometimes results in a "Text file
-    #  busy" error.
-    script_path = "./script.sh"
+    #  depend on either `/usr/bin/env` or bash being available on the Process host.
+    script_path = "./find_binary.sh"
     script_content = dedent(
         """
+        #!/usr/bin/env bash
+
         set -euo pipefail

         if command -v which > /dev/null; then
@@ -363,7 +363,7 @@ async def find_binary(request: BinaryPathRequest) -> BinaryPaths:
         Process(
             description=f"Searching for `{request.binary_name}` on PATH={search_path}",
             input_digest=script_digest,
-            argv=["/bin/bash", script_path, request.binary_name],
+            argv=[script_path, request.binary_name],
             env={"PATH": search_path},
         ),
     )
```

Then run something like `./pants run build-support/bin/check_headers.py --no-process-execution-use-local-cache --no-pantsd`. The flags at the end are to force you to re-search the `python` binary to use to run the Pex.

I suspect that this only happens on Linux. I haven't been able to get this error locally on macOS. On my Ubuntu VM, I hit a different error that none of the interpreters were discovered in the first place with this change, rather than hitting Error Code 26. In CI, we've hit Error Code 26 a few times.

The current hypothesis is that we're not calling `fsync` enough, which is called `sync` in Rust. See https://doc.rust-lang.org/std/fs/struct.File.html#method.sync_all.

We call `fsync` right now once on the input digest:

https://github.com/pantsbuild/pants/blob/0e1c8de0d630af77c5f71d030e12cfc3e22a9c6a/src/rust/engine/process_execution/src/local.rs#L365-L371

If you follow `materialize_directory`, you'll see that we call `.sync_all()` at the end:

https://github.com/pantsbuild/pants/blob/0e1c8de0d630af77c5f71d030e12cfc3e22a9c6a/src/rust/engine/fs/store/src/lib.rs#L738-L755

The hypothesis is that for some reason that is not enough. We want to experiment if calling `fsync` again right here would fix things:

https://github.com/pantsbuild/pants/blob/0e1c8de0d630af77c5f71d030e12cfc3e22a9c6a/src/rust/engine/process_execution/src/local.rs#L419-L421

(I'm not sure what that snippet will be. Stu had an idea).

It seems likely that this is docker related. This thread is interesting: https://github.com/moby/moby/issues/9547
Will see if the nuclear option (a global `sync`) resolves this.
> It seems likely that this is docker related. 

That sounds plausible, but I'm not sure Docker is it. We saw this error on the `lint` shard _before_ RBE was used, i.e. when running `./pants run build-support/bin/get_rbe_token.py`. That happens before RBE is used. That lint shard does not use Docker and only uses the default Travis environment. (Unless, under-the-hood, Travis is running Docker?)

Resolving patch:
diff --git a/src/python/pants/engine/process.py b/src/python/pants/engine/process.py
--- a/src/python/pants/engine/process.py
+++ b/src/python/pants/engine/process.py
@@ -343,6 +343,8 @@ async def find_binary(request: BinaryPathRequest) -> BinaryPaths:
     script_path = "./script.sh"
     script_content = dedent(
         """
+        #!/usr/bin/env bash
+
         set -euo pipefail
 
         if command -v which > /dev/null; then
@@ -365,7 +367,7 @@ async def find_binary(request: BinaryPathRequest) -> BinaryPaths:
             description=f"Searching for `{request.binary_name}` on PATH={search_path}",
             level=LogLevel.DEBUG,
             input_digest=script_digest,
-            argv=["/bin/bash", script_path, request.binary_name],
+            argv=[script_path, request.binary_name],
             env={"PATH": search_path},
         ),
     )


Test patch:
diff --git a/src/rust/engine/fs/src/tests.rs b/src/rust/engine/fs/src/tests.rs
new file mode 100644
--- /dev/null
+++ b/src/rust/engine/fs/src/tests.rs
@@ -0,0 +1,18 @@
+use crate::RelativePath;
+
+#[test]
+fn relative_path_ok() {
+  assert_eq!(Some("a"), RelativePath::new("a").unwrap().to_str());
+  assert_eq!(Some("a"), RelativePath::new("./a").unwrap().to_str());
+  assert_eq!(Some("a"), RelativePath::new("b/../a").unwrap().to_str());
+  assert_eq!(
+    Some("a/c"),
+    RelativePath::new("b/../a/././c").unwrap().to_str()
+  );
+}
+
+#[test]
+fn relative_path_err() {
+  assert!(RelativePath::new("../a").is_err());
+  assert!(RelativePath::new("/a").is_err());
+}
diff --git a/src/rust/engine/fs/store/src/tests.rs b/src/rust/engine/fs/store/src/tests.rs
--- a/src/rust/engine/fs/store/src/tests.rs
+++ b/src/rust/engine/fs/store/src/tests.rs
@@ -5,6 +5,7 @@ use crate::{
 
 use bazel_protos;
 use bytes::Bytes;
+use fs::RelativePath;
 use futures::compat::Future01CompatExt;
 use hashing::{Digest, Fingerprint};
 use maplit::btreemap;
@@ -1007,7 +1008,7 @@ async fn materialize_missing_file() {
   let store_dir = TempDir::new().unwrap();
   let store = new_local_store(store_dir.path());
   store
-    .materialize_file(file.clone(), TestData::roland().digest(), false, true)
+    .materialize_file(file.clone(), TestData::roland().digest(), false)
     .compat()
     .await
     .expect_err("Want unknown digest error");
@@ -1027,7 +1028,7 @@ async fn materialize_file() {
     .await
     .expect("Error saving bytes");
   store
-    .materialize_file(file.clone(), testdata.digest(), false, true)
+    .materialize_file(file.clone(), testdata.digest(), false)
     .compat()
     .await
     .expect("Error materializing file");
@@ -1049,7 +1050,7 @@ async fn materialize_file_executable() {
     .await
     .expect("Error saving bytes");
   store
-    .materialize_file(file.clone(), testdata.digest(), true, true)
+    .materialize_file(file.clone(), testdata.digest(), true)
     .compat()
     .await
     .expect("Error materializing file");
@@ -1564,3 +1565,89 @@ async fn explicitly_overwrites_already_existing_file() {
   let file_contents = std::fs::read(&file_path).unwrap();
   assert_eq!(file_contents, b"abc123".to_vec());
 }
+
+fn create_empty_directory_materialize_metadata() -> DirectoryMaterializeMetadata {
+  crate::DirectoryMaterializeMetadata {
+    metadata: LoadMetadata::Local,
+    child_directories: Default::default(),
+    child_files: Default::default(),
+  }
+}
+
+fn create_files_directory_materialize_metadata<S: AsRef<str>, F: IntoIterator<Item = S>>(
+  child_files: F,
+) -> DirectoryMaterializeMetadata {
+  crate::DirectoryMaterializeMetadata {
+    child_files: child_files
+      .into_iter()
+      .map(|c| (c.as_ref().to_string(), LoadMetadata::Local))
+      .collect(),
+    ..create_empty_directory_materialize_metadata()
+  }
+}
+
+fn create_directories_directory_materialize_metadata<
+  S: AsRef<str>,
+  F: IntoIterator<Item = (S, DirectoryMaterializeMetadata)>,
+>(
+  child_dirs: F,
+) -> DirectoryMaterializeMetadata {
+  crate::DirectoryMaterializeMetadata {
+    child_directories: child_dirs
+      .into_iter()
+      .map(|(c, d)| (c.as_ref().to_string(), d))
+      .collect(),
+    ..create_empty_directory_materialize_metadata()
+  }
+}
+
+#[test]
+fn directory_materialize_metadata_empty_contains_file_empty() {
+  let md = create_empty_directory_materialize_metadata();
+  assert!(!md.contains_file(&RelativePath::new("").unwrap()));
+}
+
+#[test]
+fn directory_materialize_metadata_empty_contains_file() {
+  let md = create_empty_directory_materialize_metadata();
+  assert!(!md.contains_file(&RelativePath::new("./script.sh").unwrap()));
+}
+
+#[test]
+fn directory_materialize_metadata_contains_file() {
+  let md = create_files_directory_materialize_metadata(vec!["script.sh"]);
+  assert!(md.contains_file(&RelativePath::new("./script.sh").unwrap()));
+  assert!(md.contains_file(&RelativePath::new("script.sh").unwrap()));
+}
+
+#[test]
+fn directory_materialize_metadata_un_normalized_contains_file() {
+  let md = create_files_directory_materialize_metadata(vec!["./script.sh"]);
+  assert!(md.contains_file(&RelativePath::new("./script.sh").unwrap()));
+  assert!(md.contains_file(&RelativePath::new("script.sh").unwrap()));
+}
+
+#[test]
+fn directory_materialize_metadata_contains_subdir_file() {
+  let subdir_md = create_files_directory_materialize_metadata(vec!["script.sh"]);
+  let md = create_directories_directory_materialize_metadata(vec![
+    ("subdir", subdir_md),
+    ("script", create_empty_directory_materialize_metadata()),
+  ]);
+
+  assert!(!md.contains_file(&RelativePath::new("./script.sh").unwrap()));
+  assert!(!md.contains_file(&RelativePath::new("script.sh").unwrap()));
+  assert!(md.contains_file(&RelativePath::new("./subdir/script.sh").unwrap()));
+  assert!(md.contains_file(&RelativePath::new("subdir/script.sh").unwrap()));
+}
+
+#[test]
+fn directory_materialize_metadata_empty_dir() {
+  let md = create_directories_directory_materialize_metadata(vec![(
+    "script",
+    create_empty_directory_materialize_metadata(),
+  )]);
+
+  assert!(!md.contains_file(&RelativePath::new("./script").unwrap()));
+  assert!(!md.contains_file(&RelativePath::new("script").unwrap()));
+}
diff --git a/src/rust/engine/process_execution/src/tests.rs b/src/rust/engine/process_execution/src/tests.rs
--- a/src/rust/engine/process_execution/src/tests.rs
+++ b/src/rust/engine/process_execution/src/tests.rs
@@ -1,4 +1,4 @@
-use crate::{Process, RelativePath};
+use crate::Process;
 use std::collections::hash_map::DefaultHasher;
 use std::hash::{Hash, Hasher};
 use std::time::Duration;
@@ -37,20 +37,3 @@ fn process_equality() {
   assert!(a != d);
   assert!(hash(&a) != hash(&d));
 }
-
-#[test]
-fn relative_path_ok() {
-  assert_eq!(Some("a"), RelativePath::new("a").unwrap().to_str());
-  assert_eq!(Some("a"), RelativePath::new("./a").unwrap().to_str());
-  assert_eq!(Some("a"), RelativePath::new("b/../a").unwrap().to_str());
-  assert_eq!(
-    Some("a/c"),
-    RelativePath::new("b/../a/././c").unwrap().to_str()
-  );
-}
-
-#[test]
-fn relative_path_err() {
-  assert!(RelativePath::new("../a").is_err());
-  assert!(RelativePath::new("/a").is_err());
-}