Repository: pantsbuild/pants

Issue:
Ability to set working dir for `experimental_shell_command`
**Is your feature request related to a problem? Please describe.**

We (unfortunately) have a build script that is hard-coded to write to `../../some/dir`. I tried putting an `experimental_shell_command` into the same directory as the script with `outputs=["../../some/dir"]`, but that failed with error `Relative paths that escape the root are not allowed`.

To get things working, I had to move the `experimental_shell_command` to the root of my repo and set the `command` to `"cd path/to/script/dir && ./script"` (the script unfortunately cares that it's run from the same directory). I was then able to set `outputs=["some/dir/"]`.

**Describe the solution you'd like**

It makes sense to me that shell commands can't write outputs to parent dirs, but it would be nice if there was a cleaner way to say "run this command in a subdir workdir". A `work_directory` field on `experimental_shell_command` would do the trick, I think.

**Describe alternatives you've considered**

Things seem to work fine with the `cd` so we aren't blocked, it's just messy and raised some eyebrows during review.



Discussion:
Mm, yea: this should be straightforward to do... it should map directly to setting a `Process(.., working_directory=..)`, which becomes: https://github.com/pantsbuild/pants/blob/51f067588a1b382ee49940540c6b6cb1e2c0ab2e/src/rust/engine/process_execution/src/lib.rs#L472-L476
You would add a `StringField` to the `experimental_shell_command` target and then simply wire it up. PR welcomed :)
>  PR welcomed :)

It is on my list for when I get time 😅 will poke my team to see if anyone else is interested in getting their feet wet in the core `pants` code
Turns out there is already a `ShellCommandRunWorkdirField` but it's only hooked up to the `experimental_run_shell_command` target 🤯

Will add it to `experimental_shell_command` and see if it Just Works
Hmmm some oddness with the existing field:
1. It's used as a path from the repo root. I had imagined the `workdir` field on `experimental_shell_command` would be relative to the target's location, not always the repo root. What do others think?
2. The rule that sets up the shell process uses presence/absence of `ShellCommandRunWorkdirField` on the target to set whether or not the process should be interactive. I'm guessing the intent was to make `experimental_shell_command` non-interactive, and `experimental_run_shell_command` interactive - assuming we want to keep things that way, we'll need some other marker for interactivity.
Workdirs and output paths for the shell command targets were a bit of a fuzz to implement as I recall, and not all scenarios where clear to me at the time.

I agree with your conclusions made above 👍🏽 
> It's used as a path from the repo root. I had imagined the workdir field on experimental_shell_command would be relative to the target's location, not always the repo root

Yeah, normally this would be relative to the target. Maybe we were thinking it's convenient to let people place it anywhere? But that's probably premature optimization - it doesn't seem like that big of a deal to me for users to need to move the target definition to another BUILD file? They're not changing their own code, only Pants metadata.
I am leaning towards making the default working directory for `experimental_shell_command` (and the new `experimental_run_in_sandbox` from #17716) be the root of the sandbox -- along with the existing build file location variables, that would support setting the workdir to the location of the target for people who need it.

I'm looking at this mostly from the point of view of #17617, which is a real usability annoyance for anyone consuming the outputs of `esc`: you actually have to know the location of the target you're consuming in order to consume its output, or the author of the target has to proactively do a `relocated_files`.

Resolving patch:
diff --git a/src/python/pants/backend/shell/target_types.py b/src/python/pants/backend/shell/target_types.py
--- a/src/python/pants/backend/shell/target_types.py
+++ b/src/python/pants/backend/shell/target_types.py
@@ -377,6 +377,13 @@ class RunInSandboxSourcesField(MultipleSourcesField):
     expected_num_files = 0
 
 
+class ShellCommandIsInteractiveField(MultipleSourcesField):
+    # We use this to determine whether this is an interactive process.
+    alias = "_is_interactive"
+    uses_source_roots = False
+    expected_num_files = 0
+
+
 class RunInSandboxArgumentsField(StringSequenceField):
     alias = "args"
     default = ()
@@ -421,10 +428,16 @@ class ShellCommandLogOutputField(BoolField):
     help = "Set to true if you want the output from the command logged to the console."
 
 
-class ShellCommandRunWorkdirField(StringField):
+class ShellCommandWorkdirField(StringField):
     alias = "workdir"
-    default = "."
-    help = "Sets the current working directory of the command, relative to the project root."
+    default = None
+    help = softwrap(
+        "Sets the current working directory of the command, relative to the project root. If not "
+        "set, use the project root.\n\n"
+        "To specify the location of the `BUILD` file, use `.`. Values beginning with `.` are "
+        "relative to the location of the `BUILD` file.\n\n"
+        "To specify the project/build root, use `/` or the empty string."
+    )
 
 
 class ShellCommandTestDependenciesField(ShellCommandExecutionDependenciesField):
@@ -452,6 +465,7 @@ class ShellCommandTarget(Target):
         ShellCommandTimeoutField,
         ShellCommandToolsField,
         ShellCommandExtraEnvVarsField,
+        ShellCommandWorkdirField,
         EnvironmentField,
     )
     help = softwrap(
@@ -494,6 +508,7 @@ class ShellRunInSandboxTarget(Target):
         ShellCommandTimeoutField,
         ShellCommandToolsField,
         ShellCommandExtraEnvVarsField,
+        ShellCommandWorkdirField,
         EnvironmentField,
     )
     help = softwrap(
@@ -521,7 +536,8 @@ class ShellCommandRunTarget(Target):
         *COMMON_TARGET_FIELDS,
         ShellCommandExecutionDependenciesField,
         ShellCommandCommandField,
-        ShellCommandRunWorkdirField,
+        ShellCommandWorkdirField,
+        ShellCommandIsInteractiveField,
     )
     help = softwrap(
         """
@@ -558,6 +574,7 @@ class ShellCommandTestTarget(Target):
         ShellCommandExtraEnvVarsField,
         EnvironmentField,
         SkipShellCommandTestsField,
+        ShellCommandWorkdirField,
     )
     help = softwrap(
         """
diff --git a/src/python/pants/backend/shell/util_rules/shell_command.py b/src/python/pants/backend/shell/util_rules/shell_command.py
--- a/src/python/pants/backend/shell/util_rules/shell_command.py
+++ b/src/python/pants/backend/shell/util_rules/shell_command.py
@@ -20,15 +20,16 @@
     ShellCommandCommandField,
     ShellCommandExecutionDependenciesField,
     ShellCommandExtraEnvVarsField,
+    ShellCommandIsInteractiveField,
     ShellCommandLogOutputField,
     ShellCommandOutputDependenciesField,
     ShellCommandOutputDirectoriesField,
     ShellCommandOutputFilesField,
     ShellCommandOutputsField,
-    ShellCommandRunWorkdirField,
     ShellCommandSourcesField,
     ShellCommandTimeoutField,
     ShellCommandToolsField,
+    ShellCommandWorkdirField,
 )
 from pants.backend.shell.util_rules.builtin import BASH_BUILTIN_COMMANDS
 from pants.base.deprecated import warn_or_error
@@ -84,7 +85,7 @@ class GenerateFilesFromRunInSandboxRequest(GenerateSourcesRequest):
 class ShellCommandProcessRequest:
     description: str
     interactive: bool
-    working_directory: str
+    working_directory: str | None
     command: str
     timeout: int | None
     tools: tuple[str, ...]
@@ -105,11 +106,13 @@ class ShellCommandProcessFromTargetRequest:
 async def _prepare_process_request_from_target(shell_command: Target) -> ShellCommandProcessRequest:
     description = f"the `{shell_command.alias}` at `{shell_command.address}`"
 
-    interactive = shell_command.has_field(ShellCommandRunWorkdirField)
-    if interactive:
-        working_directory = shell_command[ShellCommandRunWorkdirField].value or ""
-    else:
-        working_directory = shell_command.address.spec_path
+    interactive = shell_command.has_field(ShellCommandIsInteractiveField)
+    working_directory = _parse_working_directory(
+        shell_command[ShellCommandWorkdirField].value or "", shell_command.address
+    )
+
+    if interactive and not working_directory:
+        working_directory = "."
 
     command = shell_command[ShellCommandCommandField].value
     if not command:
@@ -240,7 +243,7 @@ async def prepare_process_request_from_target(
 class RunShellCommand(RunFieldSet):
     required_fields = (
         ShellCommandCommandField,
-        ShellCommandRunWorkdirField,
+        ShellCommandWorkdirField,
     )
     run_in_sandbox_behavior = RunInSandboxBehavior.NOT_SUPPORTED
 
@@ -324,7 +327,9 @@ async def run_in_sandbox_request(
     )
     run_field_set: RunFieldSet = field_sets.field_sets[0]
 
-    working_directory = shell_command.address.spec_path
+    working_directory = _parse_working_directory(
+        shell_command[ShellCommandWorkdirField].value or "", shell_command.address
+    )
 
     # Must be run in target environment so that the binaries/envvars match the execution
     # environment when we actually run the process.
@@ -456,8 +461,9 @@ async def prepare_shell_command_process(
     input_digest = await Get(Digest, MergeDigests([shell_command.input_digest, work_dir]))
 
     if interactive:
+        _working_directory = working_directory or "."
         relpath = os.path.relpath(
-            working_directory or ".", start="/" if os.path.isabs(working_directory) else "."
+            _working_directory or ".", start="/" if os.path.isabs(_working_directory) else "."
         )
         boot_script = f"cd {shlex.quote(relpath)}; " if relpath != "." else ""
     else:
@@ -516,6 +522,21 @@ def _output_at_build_root(process: Process, bash: BashBinary) -> Process:
     )
 
 
+def _parse_working_directory(workdir_in: str, address: Address) -> str:
+    """Convert the `workdir` field into something that can be understood by `Process`."""
+
+    reldir = address.spec_path
+
+    if workdir_in == ".":
+        return reldir
+    elif workdir_in.startswith("./"):
+        return os.path.join(reldir, workdir_in[2:])
+    elif workdir_in.startswith("/"):
+        return workdir_in[1:]
+    else:
+        return workdir_in
+
+
 @rule
 async def run_shell_command_request(shell_command: RunShellCommand) -> RunRequest:
     wrapped_tgt = await Get(


Test patch:
diff --git a/src/python/pants/backend/shell/util_rules/shell_command_test.py b/src/python/pants/backend/shell/util_rules/shell_command_test.py
--- a/src/python/pants/backend/shell/util_rules/shell_command_test.py
+++ b/src/python/pants/backend/shell/util_rules/shell_command_test.py
@@ -4,6 +4,7 @@
 from __future__ import annotations
 
 import logging
+import os
 import shlex
 from textwrap import dedent
 
@@ -130,6 +131,7 @@ def test_sources_and_files(rule_runner: RuleRunner) -> None:
                   output_files=["message.txt"],
                   output_directories=["res"],
                   command="./script.sh",
+                  workdir=".",
                 )
 
                 files(
@@ -182,7 +184,7 @@ def test_quotes_command(rule_runner: RuleRunner) -> None:
     assert_shell_command_result(
         rule_runner,
         Address("src", target_name="quotes"),
-        expected_contents={"src/out.log": "foo bar\n"},
+        expected_contents={"out.log": "foo bar\n"},
     )
 
 
@@ -199,6 +201,47 @@ def test_chained_shell_commands(rule_runner: RuleRunner) -> None:
                 )
                 """
             ),
+            "src/b/BUILD": dedent(
+                """\
+                experimental_shell_command(
+                  name="msg",
+                  tools=["cp", "echo"],
+                  output_files=["msg"],
+                  command="echo 'shell_command:b' >> msg",
+                  execution_dependencies=["src/a:msg"],
+                )
+                """
+            ),
+        }
+    )
+
+    assert_shell_command_result(
+        rule_runner,
+        Address("src/a", target_name="msg"),
+        expected_contents={"msg": "shell_command:a\n"},
+    )
+
+    assert_shell_command_result(
+        rule_runner,
+        Address("src/b", target_name="msg"),
+        expected_contents={"msg": "shell_command:a\nshell_command:b\n"},
+    )
+
+
+def test_chained_shell_commands_with_workdir(rule_runner: RuleRunner) -> None:
+    rule_runner.write_files(
+        {
+            "src/a/BUILD": dedent(
+                """\
+                experimental_shell_command(
+                  name="msg",
+                  tools=["echo"],
+                  output_files=["msg"],
+                  command="echo 'shell_command:a' > msg",
+                  workdir=".",
+                )
+                """
+            ),
             "src/b/BUILD": dedent(
                 """\
                 experimental_shell_command(
@@ -207,6 +250,7 @@ def test_chained_shell_commands(rule_runner: RuleRunner) -> None:
                   output_files=["msg"],
                   command="cp ../a/msg . ; echo 'shell_command:b' >> msg",
                   execution_dependencies=["src/a:msg"],
+                  workdir=".",
                 )
                 """
             ),
@@ -310,14 +354,14 @@ def test_shell_command_masquerade_as_a_files_target(rule_runner: RuleRunner) ->
         ],
     )
 
-    assert sources.files == ("src/contents.txt",)
+    assert sources.files == ("contents.txt",)
     assert sources.unrooted_files == sources.files
 
     contents = rule_runner.request(DigestContents, [sources.snapshot.digest])
     assert len(contents) == 1
 
     fc = contents[0]
-    assert fc.path == "src/contents.txt"
+    assert fc.path == "contents.txt"
     assert fc.content == b"contents\n"
 
 
@@ -344,7 +388,7 @@ def test_package_dependencies(caplog, rule_runner: RuleRunner) -> None:
 
                 experimental_shell_command(
                   name="test",
-                  command="ls .",
+                  command="ls src",
                   tools=["ls"],
                   log_output=True,
                   execution_dependencies=[":msg-archive"],
@@ -452,12 +496,12 @@ def test_execution_dependencies(caplog, rule_runner: RuleRunner) -> None:
     assert_shell_command_result(
         rule_runner,
         Address("src", target_name="expect_success_1"),
-        expected_contents={"src/output.txt": "message\nmessage\n"},
+        expected_contents={"output.txt": "message\nmessage\n"},
     )
     assert_shell_command_result(
         rule_runner,
         Address("src", target_name="expect_success_2"),
-        expected_contents={"src/output.txt": "message\nmessage\n"},
+        expected_contents={"output.txt": "message\nmessage\n"},
     )
 
 
@@ -498,7 +542,7 @@ def test_old_style_dependencies(caplog, rule_runner: RuleRunner) -> None:
     assert_shell_command_result(
         rule_runner,
         Address("src", target_name="expect_success"),
-        expected_contents={"src/output.txt": "message\nmessage\n"},
+        expected_contents={"output.txt": "message\nmessage\n"},
     )
 
 
@@ -544,6 +588,7 @@ def test_shell_command_boot_script(rule_runner: RuleRunner) -> None:
                     "python3.8",
                   ],
                   command="./command.script",
+                  workdir=".",
                 )
                 """
             ),
@@ -659,6 +704,41 @@ def test_run_runnable_in_sandbox(rule_runner: RuleRunner) -> None:
         }
     )
 
+    assert_run_in_sandbox_result(
+        rule_runner,
+        Address("src", target_name="run_fruitcake"),
+        expected_contents={"fruitcake.txt": "fruitcake\n"},
+    )
+
+
+def test_run_runnable_in_sandbox_with_workdir(rule_runner: RuleRunner) -> None:
+    rule_runner.write_files(
+        {
+            "src/fruitcake.py": dedent(
+                """\
+                f = open("fruitcake.txt", "w")
+                f.write("fruitcake\\n")
+                f.close()
+                """
+            ),
+            "src/BUILD": dedent(
+                """\
+                python_source(
+                    source="fruitcake.py",
+                    name="fruitcake",
+                )
+
+                experimental_run_in_sandbox(
+                  name="run_fruitcake",
+                  runnable=":fruitcake",
+                  output_files=["fruitcake.txt"],
+                  workdir=".",
+                )
+                """
+            ),
+        }
+    )
+
     assert_run_in_sandbox_result(
         rule_runner,
         Address("src", target_name="run_fruitcake"),
@@ -676,6 +756,7 @@ def test_relative_directories(rule_runner: RuleRunner) -> None:
                   tools=["echo"],
                   command='echo foosh > ../foosh.txt',
                   output_files=["../foosh.txt"],
+                  workdir=".",
                 )
                 """
             ),
@@ -699,6 +780,7 @@ def test_relative_directories_2(rule_runner: RuleRunner) -> None:
                   tools=["echo"],
                   command='echo foosh > ../newdir/foosh.txt',
                   output_files=["../newdir/foosh.txt"],
+                  workdir=".",
                 )
                 """
             ),
@@ -722,6 +804,7 @@ def test_cannot_escape_build_root(rule_runner: RuleRunner) -> None:
                   tools=["echo"],
                   command='echo foosh > ../../invalid.txt',
                   output_files=["../../invalid.txt"],
+                  workdir=".",
                 )
                 """
             ),
@@ -734,3 +817,51 @@ def test_cannot_escape_build_root(rule_runner: RuleRunner) -> None:
             Address("src", target_name="quotes"),
             expected_contents={"../../invalid.txt": "foosh\n"},
         )
+
+
+@pytest.mark.parametrize(
+    ("workdir", "file_location"),
+    (
+        ("src", "src"),
+        (".", "src"),
+        ("./", "src"),
+        ("/", ""),
+        ("", ""),
+        ("/src", "src"),
+    ),
+)
+def test_working_directory_special_values(
+    rule_runner: RuleRunner, workdir: str, file_location: str
+) -> None:
+    rule_runner.write_files(
+        {
+            "src/fruitcake.py": dedent(
+                """\
+                f = open("fruitcake.txt", "w")
+                f.write("fruitcake\\n")
+                f.close()
+                """
+            ),
+            "src/BUILD": dedent(
+                f"""\
+                python_source(
+                    source="fruitcake.py",
+                    name="fruitcake",
+                )
+
+                experimental_run_in_sandbox(
+                  name="run_fruitcake",
+                  runnable=":fruitcake",
+                  output_files=["fruitcake.txt"],
+                  workdir="{workdir}",
+                )
+                """
+            ),
+        }
+    )
+
+    assert_run_in_sandbox_result(
+        rule_runner,
+        Address("src", target_name="run_fruitcake"),
+        expected_contents={os.path.join(file_location, "fruitcake.txt"): "fruitcake\n"},
+    )
diff --git a/src/python/pants/core/util_rules/wrap_source_intergration_test.py b/src/python/pants/core/util_rules/wrap_source_intergration_test.py
--- a/src/python/pants/core/util_rules/wrap_source_intergration_test.py
+++ b/src/python/pants/core/util_rules/wrap_source_intergration_test.py
@@ -17,6 +17,7 @@ def test_synthesized_python_is_included_in_package() -> None:
                 command='echo print\\\\(\\\\"Hello, World!\\\\"\\\\) > hello_world.py',
                 execution_dependencies=(),
                 output_files=["hello_world.py",],
+                workdir=".",
             )
 
             experimental_wrap_as_python_sources(