Repository: pantsbuild/pants

Issue:
`experimental_shell_command`'s sandbox and invalidation includes transitive dependencies of `experimental_shell_command` dependencies, not just outputs
**Describe the bug**

When a `experimental_shell_command` (`:second`) depends on another `experimental_shell_command` (`:first`), the sandbox for `:second` include the dependencies of `:first`, rather than just the output of `:first`.

https://gist.github.com/huonw/c55d0b387ed6030cda611898ee2d0361 provides a reproducer, where `:first` generates `output.txt` by 'consuming' `input.txt` (but does nothing with it), and `:second` includes a `sleep` to demonstrate when it is running. The `archive` package includes all the `.txt` files that ended in the sandbox for `:second`: I'd expect it to only be the direct output of `:first` (`output.txt`), not the `input.txt` dependency of `:first`. 

BUILD file from that gist for convenience:

```python
file(name="input", source="input.txt")
experimental_shell_command(
    name="first",
    command="""
    echo "doesn't affect output"
    echo contents > output.txt
    """,
    tools=["echo"],
    dependencies=[":input"],
    outputs=["output.txt"]
)

experimental_shell_command(
    name="second",
    command="""
    sleep 3 # make 'actually running' obvious
    """,
    dependencies=[":first"],
    tools=["sleep"],
    outputs=["*.txt"],
)
archive(name="archive", files=[":second"], format="zip")
```

```shell
git clone git@gist.github.com:c55d0b387ed6030cda611898ee2d0361.git
cd c55d0b387ed6030cda611898ee2d0361
./pants version # 2.13.0

# initial build (takes about 3 seconds to run :second, due to sleep)
./pants package ::

# check package output:
unzip -l dist/archive.zip # two files: input.txt output.txt

# validating the cache works as expected (~instantaneous)
./pants package ::

# BUG: change to :first dependencies, that doesn't affect output... 
echo 'new contents' > input.txt
# ... reruns :second, and thus takes ~3 seconds
./pants package ::

# EXPECTED: change to :first script that doesn't affect output...
sed -i '' 's/affect output/affect output still/' BUILD
# .... doesn't rerun :second, only :first
./pants package :: 
```

AFAICT, there's no way to break `:second`'s importing of `:input`, e.g. adjusting to `dependencies=[":first", "!:input"]` doesn't do anything: the behaviour is the same (and using `!!:input` isn't supported).

The `:input` being `file` is just for convenience. This appears to apply to all other target types, e.g. a `pex_binary` or yet another `experimental_shell_command`.

I haven't checked how this behaves when depending on explicit codegen targets, only these adhoc `experimental_shell_command` ones.

**Pants version**
2.13.0

**OS**
macOS

**Additional info**

Background: how are we using `experimental_shell_command` to hit this?

We're using `experimental_shell_command` to try to bridge the gap between Pants supported code and unsupported code (JS/TS), as well as for adhoc codegen tasks (to avoid avoid having to write and maintain a plugin). This can result in 'long' chains of `experimental_shell_command`s that depend on each other (and other resources), e.g.:

```python
# some app:
python_sources()
pex_binary(name="app", ...)

# codegen the schema itself:
experimental_shell_command(name="schema", dependencies=[":app"], outputs=["schema.json"], command="./app.pex export-schema > schema.json", tools=["python3.9"])

# set up the NPM dependencies:
experimental_shell_command(name="node_modules", outputs=["node_modules/**"], command="npm ci", ...)

# use the schema to do codegen or generate docs or whatever:
experimental_shell_command(name="codegen-from-schema", dependencies=[":node_modules", ":schema"], outputs=["codegen/**"], command="npm run do-codegen < schema.json", ...)
experimental_shell_command(name="docs-from-schema", dependencies=[":node_modules", ":schema"], outputs=["docs/**"], command="npm run generate-docs < schema.json", ...)

# continues with targets using :codegen-from-schema and :docs-from-schema... (`archive`, `experimental_shell_command` and `experimental_run_shell_command`)
```

The Python code that goes into `pex_binary` changes regularly and thus the PEX itself changes too, but the exported `schema.json` doesn't change so much (i.e. we're often refactoring/adding features/fixing bugs without changing the schema). In theory, if `:app` changes but the schema doesn't, `:schema`'s dependees (`:codegen-from-schema`, `:docs-from-schema`) don't need to run, but those dependees pull in the `app.pex` file from `:app`, and thus do rerun. Rerunning can take significant time.


(See also: https://pantsbuild.slack.com/archives/C046T6T9U/p1666315386420799)


Discussion:
Started a discussion here: https://github.com/pantsbuild/pants/discussions/17368
FWIW This is part of my "re-architect dependencies" proposal I'm hoping to work on late this year or Q1 next year :smile: 
#17680 and follow-up work will have some impact on this 

Resolving patch:
diff --git a/src/python/pants/backend/shell/target_types.py b/src/python/pants/backend/shell/target_types.py
--- a/src/python/pants/backend/shell/target_types.py
+++ b/src/python/pants/backend/shell/target_types.py
@@ -19,6 +19,7 @@
     MultipleSourcesField,
     OverridesField,
     SingleSourceField,
+    SpecialCasedDependencies,
     StringField,
     StringSequenceField,
     Target,
@@ -301,8 +302,41 @@ class ShellCommandOutputDirectoriesField(StringSequenceField):
     )
 
 
-class ShellCommandDependenciesField(ShellDependenciesField):
+class ShellCommandOutputDependenciesField(ShellDependenciesField):
     supports_transitive_excludes = True
+    alias = "output_dependencies"
+    deprecated_alias = "dependencies"
+    deprecated_alias_removal_version = "2.17.0.dev0"
+
+    help = softwrap(
+        """
+        Any dependencies that the output artifacts require in order to be effectively consumed.
+
+        To enable legacy use cases, if `execution_dependencies` is `None`, these dependencies will
+        be materialized in the command execution sandbox. This behavior is deprecated, and will be
+        removed in version 2.17.0.dev0.
+        """
+    )
+
+
+class ShellCommandExecutionDependenciesField(SpecialCasedDependencies):
+    alias = "execution_dependencies"
+    required = False
+    default = None
+
+    help = softwrap(
+        """
+        The execution dependencies for this shell command.
+
+        Dependencies specified here are those required to make the command complete successfully
+        (e.g. file inputs, binaries compiled from other targets, etc), but NOT required to make
+        the output side-effects useful. Dependencies that are required to use the side-effects
+        produced by this command should be specified using the `output_dependencies` field.
+
+        If this field is specified, dependencies from `output_dependencies` will not be added to
+        the execution sandbox.
+        """
+    )
 
 
 class ShellCommandSourcesField(MultipleSourcesField):
@@ -356,7 +390,7 @@ class ShellCommandRunWorkdirField(StringField):
     help = "Sets the current working directory of the command, relative to the project root."
 
 
-class ShellCommandTestDependenciesField(ShellCommandDependenciesField):
+class ShellCommandTestDependenciesField(ShellCommandExecutionDependenciesField):
     pass
 
 
@@ -370,7 +404,8 @@ class ShellCommandTarget(Target):
     alias = "experimental_shell_command"
     core_fields = (
         *COMMON_TARGET_FIELDS,
-        ShellCommandDependenciesField,
+        ShellCommandOutputDependenciesField,
+        ShellCommandExecutionDependenciesField,
         ShellCommandCommandField,
         ShellCommandLogOutputField,
         ShellCommandOutputsField,
@@ -391,7 +426,7 @@ class ShellCommandTarget(Target):
             experimental_shell_command(
                 command="./my-script.sh --flag",
                 tools=["tar", "curl", "cat", "bash", "env"],
-                dependencies=[":scripts"],
+                execution_dependencies=[":scripts"],
                 output_files=["logs/my-script.log"],
                 output_directories=["results"],
             )
@@ -411,7 +446,7 @@ class ShellCommandRunTarget(Target):
     alias = "experimental_run_shell_command"
     core_fields = (
         *COMMON_TARGET_FIELDS,
-        ShellCommandDependenciesField,
+        ShellCommandExecutionDependenciesField,
         ShellCommandCommandField,
         ShellCommandRunWorkdirField,
     )
@@ -423,16 +458,16 @@ class ShellCommandRunTarget(Target):
 
             experimental_run_shell_command(
                 command="./scripts/my-script.sh --data-files-dir={chroot}",
-                dependencies=["src/project/files:data"],
+                execution_dependencies=["src/project/files:data"],
             )
 
         The `command` may use either `{chroot}` on the command line, or the `$CHROOT`
         environment variable to get the root directory for where any dependencies are located.
 
         In contrast to the `experimental_shell_command`, in addition to `workdir` you only have
-        the `command` and `dependencies` fields as the `tools` you are going to use are already
-        on the PATH which is inherited from the Pants environment. Also, the `outputs` does not
-        apply, as any output files produced will end up directly in your project tree.
+        the `command` and `execution_dependencies` fields as the `tools` you are going to use are
+        already on the PATH which is inherited from the Pants environment. Also, the `outputs` does
+        not apply, as any output files produced will end up directly in your project tree.
         """
     )
 
@@ -461,7 +496,7 @@ class ShellCommandTestTarget(Target):
                 name="test",
                 tools=["test"],
                 command="test -r $CHROOT/some-data-file.txt",
-                dependencies=["src/project/files:data"],
+                execution_dependencies=["src/project/files:data"],
             )
 
         The `command` may use either `{chroot}` on the command line, or the `$CHROOT`
diff --git a/src/python/pants/backend/shell/util_rules/shell_command.py b/src/python/pants/backend/shell/util_rules/shell_command.py
--- a/src/python/pants/backend/shell/util_rules/shell_command.py
+++ b/src/python/pants/backend/shell/util_rules/shell_command.py
@@ -3,6 +3,7 @@
 
 from __future__ import annotations
 
+import itertools
 import logging
 import os
 import re
@@ -13,6 +14,7 @@
 from pants.backend.shell.subsystems.shell_setup import ShellSetup
 from pants.backend.shell.target_types import (
     ShellCommandCommandField,
+    ShellCommandExecutionDependenciesField,
     ShellCommandExtraEnvVarsField,
     ShellCommandLogOutputField,
     ShellCommandOutputDirectoriesField,
@@ -24,6 +26,7 @@
     ShellCommandToolsField,
 )
 from pants.backend.shell.util_rules.builtin import BASH_BUILTIN_COMMANDS
+from pants.base.deprecated import warn_or_error
 from pants.core.goals.package import BuiltPackage, PackageFieldSet
 from pants.core.goals.run import RunDebugAdapterRequest, RunFieldSet, RunRequest
 from pants.core.target_types import FileSourceField
@@ -35,6 +38,7 @@
     BinaryPathRequest,
     BinaryPaths,
 )
+from pants.engine.addresses import Addresses, UnparsedAddressInputs
 from pants.engine.env_vars import EnvironmentVars, EnvironmentVarsRequest
 from pants.engine.environment import EnvironmentName
 from pants.engine.fs import (
@@ -102,27 +106,82 @@ async def _prepare_process_request_from_target(shell_command: Target) -> ShellCo
 
     command = shell_command[ShellCommandCommandField].value
     if not command:
-        raise ValueError(f"Missing `command` line in {description}.")
+        raise ValueError(f"Missing `command` line in `{description}.")
 
-    # Prepare `input_digest`: Currently uses transitive targets per old behaviour, but
-    # this will probably change soon, per #17345.
-    transitive_targets = await Get(
+    dependencies_digest = await _execution_environment_from_dependencies(shell_command)
+
+    output_files, output_directories = _parse_outputs_from_command(shell_command, description)
+
+    return ShellCommandProcessRequest(
+        description=description,
+        interactive=interactive,
+        working_directory=working_directory,
+        command=command,
+        timeout=shell_command.get(ShellCommandTimeoutField).value,
+        tools=shell_command.get(ShellCommandToolsField, default_raw_value=()).value or (),
+        input_digest=dependencies_digest,
+        output_files=output_files,
+        output_directories=output_directories,
+        extra_env_vars=shell_command.get(ShellCommandExtraEnvVarsField).value or (),
+    )
+
+
+@rule_helper
+async def _execution_environment_from_dependencies(shell_command: Target) -> Digest:
+
+    runtime_dependencies_defined = (
+        shell_command.get(ShellCommandExecutionDependenciesField).value is not None
+    )
+
+    # If we're specifying the `dependencies` as relevant to the execution environment, then include
+    # this command as a root for the transitive dependency search for execution dependencies.
+    maybe_this_target = (shell_command.address,) if not runtime_dependencies_defined else ()
+
+    # Always include the execution dependencies that were specified
+    if runtime_dependencies_defined:
+        runtime_dependencies = await Get(
+            Addresses,
+            UnparsedAddressInputs,
+            shell_command.get(ShellCommandExecutionDependenciesField).to_unparsed_address_inputs(),
+        )
+    else:
+        runtime_dependencies = Addresses()
+        warn_or_error(
+            "2.17.0.dev0",
+            (
+                "Using `dependencies` to specify execution-time dependencies for "
+                "`experimental_shell_command` "
+            ),
+            (
+                "To clear this warning, use the `output_dependencies` and `execution_dependencies`"
+                "fields. Set `execution_dependencies=()` if you have no execution-time "
+                "dependencies."
+            ),
+            print_warning=True,
+        )
+
+    transitive = await Get(
         TransitiveTargets,
-        TransitiveTargetsRequest([shell_command.address]),
+        TransitiveTargetsRequest(itertools.chain(maybe_this_target, runtime_dependencies)),
+    )
+
+    all_dependencies = (
+        *(i for i in transitive.roots if i is not shell_command),
+        *transitive.dependencies,
     )
 
     sources, pkgs_per_target = await MultiGet(
         Get(
             SourceFiles,
             SourceFilesRequest(
-                sources_fields=[tgt.get(SourcesField) for tgt in transitive_targets.dependencies],
+                sources_fields=[tgt.get(SourcesField) for tgt in all_dependencies],
                 for_sources_types=(SourcesField, FileSourceField),
                 enable_codegen=True,
             ),
         ),
         Get(
             FieldSetsPerTarget,
-            FieldSetsPerTargetRequest(PackageFieldSet, transitive_targets.dependencies),
+            FieldSetsPerTargetRequest(PackageFieldSet, all_dependencies),
         ),
     )
 
@@ -134,20 +193,7 @@ async def _prepare_process_request_from_target(shell_command: Target) -> ShellCo
         Digest, MergeDigests([sources.snapshot.digest, *(pkg.digest for pkg in packages)])
     )
 
-    output_files, output_directories = _parse_outputs_from_command(shell_command, description)
-
-    return ShellCommandProcessRequest(
-        description=description,
-        interactive=interactive,
-        working_directory=working_directory,
-        command=command,
-        timeout=shell_command.get(ShellCommandTimeoutField).value,
-        tools=shell_command.get(ShellCommandToolsField, default_raw_value=()).value or (),
-        input_digest=dependencies_digest,
-        output_files=output_files,
-        output_directories=output_directories,
-        extra_env_vars=shell_command.get(ShellCommandExtraEnvVarsField).value or (),
-    )
+    return dependencies_digest
 
 
 def _parse_outputs_from_command(


Test patch:
diff --git a/src/python/pants/backend/helm/util_rules/post_renderer_test.py b/src/python/pants/backend/helm/util_rules/post_renderer_test.py
--- a/src/python/pants/backend/helm/util_rules/post_renderer_test.py
+++ b/src/python/pants/backend/helm/util_rules/post_renderer_test.py
@@ -277,7 +277,7 @@ def test_use_simple_extra_post_renderer(rule_runner: RuleRunner) -> None:
               experimental_run_shell_command(
                 name="custom_post_renderer",
                 command="src/shell/my-script.sh",
-                dependencies=[":scripts"]
+                execution_dependencies=[":scripts"]
               )
               """
             ),
diff --git a/src/python/pants/backend/shell/goals/test_test.py b/src/python/pants/backend/shell/goals/test_test.py
--- a/src/python/pants/backend/shell/goals/test_test.py
+++ b/src/python/pants/backend/shell/goals/test_test.py
@@ -59,14 +59,14 @@ def test_shell_command_as_test(rule_runner: RuleRunner) -> None:
 
                 experimental_test_shell_command(
                   name="pass",
-                  dependencies=[":msg-gen", ":src"],
+                  execution_dependencies=[":msg-gen", ":src"],
                   tools=["echo"],
                   command="./test.sh msg.txt message",
                 )
 
                 experimental_test_shell_command(
                   name="fail",
-                  dependencies=[":msg-gen", ":src"],
+                  execution_dependencies=[":msg-gen", ":src"],
                   tools=["echo"],
                   command="./test.sh msg.txt xyzzy",
                 )
diff --git a/src/python/pants/backend/shell/util_rules/shell_command_test.py b/src/python/pants/backend/shell/util_rules/shell_command_test.py
--- a/src/python/pants/backend/shell/util_rules/shell_command_test.py
+++ b/src/python/pants/backend/shell/util_rules/shell_command_test.py
@@ -29,14 +29,14 @@
 from pants.engine.addresses import Address
 from pants.engine.environment import EnvironmentName
 from pants.engine.fs import EMPTY_SNAPSHOT, DigestContents
-from pants.engine.process import Process
+from pants.engine.process import Process, ProcessExecutionFailure
 from pants.engine.target import (
     GeneratedSources,
     MultipleSourcesField,
     TransitiveTargets,
     TransitiveTargetsRequest,
 )
-from pants.testutil.rule_runner import QueryRule, RuleRunner
+from pants.testutil.rule_runner import QueryRule, RuleRunner, engine_error
 
 
 @pytest.fixture
@@ -99,7 +99,7 @@ def test_sources_and_files(rule_runner: RuleRunner) -> None:
                 """\
                 experimental_shell_command(
                   name="hello",
-                  dependencies=[":build-utils", ":files"],
+                  execution_dependencies=[":build-utils", ":files"],
                   tools=[
                     "bash",
                     "cat",
@@ -186,7 +186,7 @@ def test_chained_shell_commands(rule_runner: RuleRunner) -> None:
                   tools=["cp", "echo"],
                   output_files=["msg"],
                   command="cp ../a/msg . ; echo 'shell_command:b' >> msg",
-                  dependencies=["src/a:msg"],
+                  execution_dependencies=["src/a:msg"],
                 )
                 """
             ),
@@ -327,7 +327,7 @@ def test_package_dependencies(caplog, rule_runner: RuleRunner) -> None:
                   command="ls .",
                   tools=["ls"],
                   log_output=True,
-                  dependencies=[":msg-archive"],
+                  execution_dependencies=[":msg-archive"],
                 )
                 """
             ),
@@ -345,6 +345,143 @@ def test_package_dependencies(caplog, rule_runner: RuleRunner) -> None:
     )
 
 
+def test_execution_dependencies(caplog, rule_runner: RuleRunner) -> None:
+    caplog.set_level(logging.INFO)
+    caplog.clear()
+
+    rule_runner.write_files(
+        {
+            "src/BUILD": dedent(
+                """\
+                experimental_shell_command(
+                  name="a1",
+                  command="echo message > msg.txt",
+                  outputs=["msg.txt"],
+                )
+
+                experimental_shell_command(
+                    name="a2",
+                    tools=["cat"],
+                    command="cat msg.txt > msg2.txt",
+                    execution_dependencies=[":a1",],
+                    outputs=["msg2.txt",],
+                )
+
+                # Fails because runtime dependencies are not exported
+                # transitively
+                experimental_shell_command(
+                    name="expect_fail_1",
+                    tools=["cat"],
+                    command="cat msg.txt",
+                    execution_dependencies=[":a2",],
+                )
+
+                # Fails because `output_dependencies` are not available at runtime
+                experimental_shell_command(
+                    name="expect_fail_2",
+                    tools=["cat"],
+                    command="cat msg.txt",
+                    execution_dependencies=(),
+                    output_dependencies=[":a1"],
+                )
+
+                # Fails because runtime dependencies are not fetched transitively
+                # even if the root is requested through `output_dependencies`
+                experimental_shell_command(
+                    name="expect_fail_3",
+                    tools=["cat"],
+                    command="cat msg.txt",
+                    output_dependencies=[":a2"],
+                )
+
+                # Succeeds because `a1` and `a2` are requested directly
+                experimental_shell_command(
+                    name="expect_success_1",
+                    tools=["cat"],
+                    command="cat msg.txt msg2.txt > output.txt",
+                    execution_dependencies=[":a1", ":a2",],
+                    outputs=["output.txt"],
+                )
+
+                # Succeeds becuase `a1` and `a2` are requested directly and `output_dependencies`
+                # are made available at runtime
+                experimental_shell_command(
+                    name="expect_success_2",
+                    tools=["cat"],
+                    command="cat msg.txt msg2.txt > output.txt",
+                    output_dependencies=[":a1", ":a2",],
+                    outputs=["output.txt"],
+                )
+                """
+            ),
+        }
+    )
+
+    with engine_error(ProcessExecutionFailure):
+        assert_shell_command_result(
+            rule_runner, Address("src", target_name="expect_fail_1"), expected_contents={}
+        )
+    with engine_error(ProcessExecutionFailure):
+        assert_shell_command_result(
+            rule_runner, Address("src", target_name="expect_fail_2"), expected_contents={}
+        )
+    with engine_error(ProcessExecutionFailure):
+        assert_shell_command_result(
+            rule_runner, Address("src", target_name="expect_fail_3"), expected_contents={}
+        )
+    assert_shell_command_result(
+        rule_runner,
+        Address("src", target_name="expect_success_1"),
+        expected_contents={"src/output.txt": "message\nmessage\n"},
+    )
+    assert_shell_command_result(
+        rule_runner,
+        Address("src", target_name="expect_success_2"),
+        expected_contents={"src/output.txt": "message\nmessage\n"},
+    )
+
+
+def test_old_style_dependencies(caplog, rule_runner: RuleRunner) -> None:
+    caplog.set_level(logging.INFO)
+    caplog.clear()
+
+    rule_runner.write_files(
+        {
+            "src/BUILD": dedent(
+                """\
+                experimental_shell_command(
+                  name="a1",
+                  command="echo message > msg.txt",
+                  outputs=["msg.txt"],
+                )
+
+                experimental_shell_command(
+                    name="a2",
+                    tools=["cat"],
+                    command="cat msg.txt > msg2.txt",
+                    dependencies=[":a1",],
+                    outputs=["msg2.txt",],
+                )
+
+                experimental_shell_command(
+                    name="expect_success",
+                    tools=["cat"],
+                    command="cat msg.txt msg2.txt > output.txt",
+                    dependencies=[":a1", ":a2",],
+                    outputs=["output.txt"],
+                )
+                """
+            ),
+        }
+    )
+
+    assert_shell_command_result(
+        rule_runner,
+        Address("src", target_name="expect_success"),
+        expected_contents={"src/output.txt": "message\nmessage\n"},
+    )
+
+
 def test_run_shell_command_request(rule_runner: RuleRunner) -> None:
     rule_runner.write_files(
         {