Repository: pantsbuild/pants

Issue:
Add output to signal if a test result was cached
**Is your feature request related to a problem? Please describe.**
Coming from Bazelland, it was very nice to see which tests actually ran and which ones didn't due to caching.

Output looks roughly like:
```
//a/b/c/d:e_test        (cached) PASSED in 29.5s
//a/b:c_test            (cached) PASSED in 1.6s
//a/b/c/d:e2_test                PASSED in 2.8s
//a/b/c:d_test                   PASSED in 16.0s
```

This raised the confidence in the build tool's ability to cache entire test runs, and in our understanding of the cache invalidation.

**Describe the solution you'd like**
A similar output for `./pants test` (maybe other rules as well, `lint`/`fmt`/etc...).

**Describe alternatives you've considered**
The stats helps, but isn't quite the same.

**Additional context**
N/A



Discussion:
Additionally, the feature could use different signaling for "memoized" vs. "cached".
We should definitely do this, yea. In fact, I promised @g-cassie I'd open a ticket about it a while back... oops. Thanks!
#13590 exposes the information needed to do this. In a followup change, the [`TestResult`](https://github.com/pantsbuild/pants/blob/a655638b1c22155e5f028fcd40af151ad0a5e525/src/python/pants/core/goals/test.py#L43-L56) type (and any others for which this information might be rendered) should begin carrying along `ProcessResultMetadata`, and the `test` goal should request the `RunId`, and consume the metadata to render cache status and runtime.
I'm going to have to focus on other things for a bit. Maybe someone else would be interested in picking up the rendering changes, as described above?
@stuhood Had a quick play with this, looks straightforward. But not sure how to get the current process run id though in the `test` goal - per "the test goal should request the RunId". 

> @stuhood Had a quick play with this, looks straightforward. But not sure how to get the current process run id though in the `test` goal - per "the test goal should request the RunId".

Ah, yea: sorry that that wasn't clear!

#13590 added an intrinsic that exposes `RunId` as a singleton. To consume a singleton (or any other type which can be computed using what is already in scope), you put it in the argument list of your `@rule`. So:
```python
@goal_rule
def run_tests(
    console: Console,
    ..,
    run_id: RunId,  # NEW
) -> TestGoal:
    ..
```
@riisi do you intend to implement this?
Yes, been a bit busy this week but will come back to this tomorrow. 

> On 11 Dec 2021, at 19:00, Yoav Alon ***@***.***> wrote:
> 
> ﻿
> @riisi do you intend to implement this?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
> Triage notifications on the go with GitHub Mobile for iOS or Android. 

Thoughts on this layout? I'd slightly prefer columns like the Bazel output but this is the least change to get this in, and I don't want to mis-align with the outputs from the other goals - that might require a larger discussion. 

![image](https://user-images.githubusercontent.com/30220524/145707180-784cad18-d15f-4e2c-b7d6-83e36fad9572.png)

The possible cache results are `ran_locally`, `memoized`, `hit_locally`, and `hit_remote`. Suggest these should be translated for console output as 
- `<no print output if ran locally>`
- (memoized)
- (cached locally)
- (cached remote)

@riisi I would appreciate a sort-by option, where you can get the final output sorted by time
BTW you absolutely can add columns (in a followup) if you like. There is no requirement that different goal outputs be consistent, and they aren't really today.
I'd say "cached remotely", so it mirrors "cached locally"?
> BTW you absolutely can add columns (in a followup) if you like.

A concern I'd have is when your terminal isn't wide enough, that wrapping might look even worse. We intentionally use soft wrapping with the output of `test` so that the shell handles wrapping for us.

That's probably not a show stopper and I think that columns would be a net win. Although, doing it in a dedicated followup probably makes sense.

Resolving patch:
diff --git a/src/python/pants/engine/process.py b/src/python/pants/engine/process.py
--- a/src/python/pants/engine/process.py
+++ b/src/python/pants/engine/process.py
@@ -172,24 +172,34 @@ class FallibleProcessResult:
 class ProcessResultMetadata:
     """Metadata for a ProcessResult, which is not included in its definition of equality."""
 
+    class Source(Enum):
+        RAN_LOCALLY = "ran_locally"
+        RAN_REMOTELY = "ran_remotely"
+        HIT_LOCALLY = "hit_locally"
+        HIT_REMOTELY = "hit_remotely"
+        MEMOIZED = "memoized"
+
     # The execution time of the process, in milliseconds, or None if it could not be captured
     # (since remote execution does not guarantee its availability).
     total_elapsed_ms: int | None
     # Whether the ProcessResult (when it was created in the attached run_id) came from the local
     # or remote cache, or ran locally or remotely. See the `self.source` method.
-    # TODO: Consider extracting an enum.
     _source: str
     # The run_id in which a ProcessResult was created. See the `self.source` method.
     source_run_id: int
 
-    def source(self, current_run_id: RunId) -> str:
+    def source(self, current_run_id: RunId) -> Source:
         """Given the current run_id, return the calculated "source" of the ProcessResult.
 
         If a ProcessResult is consumed in any run_id other than the one it was created in, the its
         source implicitly becomes memoization, since the result was re-used in a new run without
         being recreated.
         """
-        return self._source if self.source_run_id == current_run_id else "memoized"
+        return (
+            self.Source(self._source)
+            if self.source_run_id == current_run_id
+            else self.Source.MEMOIZED
+        )
 
 
 class ProcessExecutionFailure(Exception):
@@ -323,9 +333,9 @@ def __post_init__(self):
             )
         if self.append_only_caches and self.run_in_workspace:
             raise ValueError(
-                "InteractiveProcess requested setup of append-only caches and also requested to run in "
-                "the workspace. These options are incompatible since setting up append-only caches would "
-                "modify the workspace."
+                "InteractiveProcess requested setup of append-only caches and also requested to run"
+                " in the workspace. These options are incompatible since setting up append-only"
+                " caches would modify the workspace."
             )
 
     @classmethod


Test patch:
diff --git a/src/python/pants/backend/go/goals/test.py b/src/python/pants/backend/go/goals/test.py
--- a/src/python/pants/backend/go/goals/test.py
+++ b/src/python/pants/backend/go/goals/test.py
@@ -153,6 +153,7 @@ def compilation_failure(exit_code: int, stdout: str | None, stderr: str | None)
             stderr_digest=EMPTY_FILE_DIGEST,
             address=field_set.address,
             output_setting=test_subsystem.output,
+            result_metadata=None,
         )
 
     if maybe_pkg_info.info is None:
diff --git a/src/python/pants/core/goals/test.py b/src/python/pants/core/goals/test.py
--- a/src/python/pants/core/goals/test.py
+++ b/src/python/pants/core/goals/test.py
@@ -21,7 +21,13 @@
 from pants.engine.environment import Environment, EnvironmentRequest
 from pants.engine.fs import EMPTY_FILE_DIGEST, Digest, FileDigest, MergeDigests, Snapshot, Workspace
 from pants.engine.goal import Goal, GoalSubsystem
-from pants.engine.process import FallibleProcessResult, InteractiveProcess, InteractiveProcessResult
+from pants.engine.internals.session import RunId
+from pants.engine.process import (
+    FallibleProcessResult,
+    InteractiveProcess,
+    InteractiveProcessResult,
+    ProcessResultMetadata,
+)
 from pants.engine.rules import Effect, Get, MultiGet, collect_rules, goal_rule, rule
 from pants.engine.target import (
     FieldSet,
@@ -49,6 +55,7 @@ class TestResult(EngineAwareReturnType):
     stderr_digest: FileDigest
     address: Address
     output_setting: ShowOutput
+    result_metadata: ProcessResultMetadata | None
 
     coverage_data: CoverageData | None = None
     xml_results: Snapshot | None = None
@@ -68,6 +75,7 @@ def skip(cls, address: Address, output_setting: ShowOutput) -> TestResult:
             stderr_digest=EMPTY_FILE_DIGEST,
             address=address,
             output_setting=output_setting,
+            result_metadata=None,
         )
 
     @classmethod
@@ -89,6 +97,7 @@ def from_fallible_process_result(
             stderr_digest=process_result.stderr_digest,
             address=address,
             output_setting=output_setting,
+            result_metadata=process_result.metadata,
             coverage_data=coverage_data,
             xml_results=xml_results,
             extra_output=extra_output,
@@ -96,7 +105,7 @@ def from_fallible_process_result(
 
     @property
     def skipped(self) -> bool:
-        return self.exit_code is None and not self.stdout and not self.stderr
+        return self.exit_code is None or self.result_metadata is None
 
     def __lt__(self, other: Any) -> bool:
         """We sort first by status (skipped vs failed vs succeeded), then alphanumerically within
@@ -105,9 +114,9 @@ def __lt__(self, other: Any) -> bool:
             return NotImplemented
         if self.exit_code == other.exit_code:
             return self.address.spec < other.address.spec
-        if self.exit_code is None:
+        if self.skipped or self.exit_code is None:
             return True
-        if other.exit_code is None:
+        if other.skipped or other.exit_code is None:
             return False
         return abs(self.exit_code) < abs(other.exit_code)
 
@@ -384,6 +393,7 @@ async def run_tests(
     workspace: Workspace,
     union_membership: UnionMembership,
     dist_dir: DistDir,
+    run_id: RunId,
 ) -> Test:
     if test_subsystem.debug:
         targets_to_valid_field_sets = await Get(
@@ -428,16 +438,11 @@ async def run_tests(
     if results:
         console.print_stderr("")
     for result in sorted(results):
-        if result.skipped:
-            continue
-        if result.exit_code == 0:
-            sigil = console.sigil_succeeded()
-            status = "succeeded"
-        else:
-            sigil = console.sigil_failed()
-            status = "failed"
+        if result.exit_code != 0:
             exit_code = cast(int, result.exit_code)
-        console.print_stderr(f"{sigil} {result.address} {status}.")
+
+        console.print_stderr(_format_test_summary(result, run_id, console))
+
         if result.extra_output and result.extra_output.files:
             workspace.write_digest(
                 result.extra_output.digest,
@@ -500,6 +505,41 @@ async def run_tests(
     return Test(exit_code)
 
 
+def _format_test_summary(result: TestResult, run_id: RunId, console: Console) -> str:
+    """Format the test summary printed to the console."""
+    source_map = {
+        ProcessResultMetadata.Source.MEMOIZED: "memoized",
+        ProcessResultMetadata.Source.RAN_REMOTELY: "ran remotely",
+        ProcessResultMetadata.Source.HIT_LOCALLY: "cached locally",
+        ProcessResultMetadata.Source.HIT_REMOTELY: "cached remotely",
+    }
+
+    if result.result_metadata:
+        if result.exit_code == 0:
+            sigil = console.sigil_succeeded()
+            status = "succeeded"
+        else:
+            sigil = console.sigil_failed()
+            status = "failed"
+
+        source = source_map.get(result.result_metadata.source(run_id))
+        source_print = f" ({source})" if source else ""
+
+        elapsed_print = ""
+        total_elapsed_ms = result.result_metadata.total_elapsed_ms
+        if total_elapsed_ms is not None:
+            elapsed_secs = total_elapsed_ms / 1000
+            elapsed_print = f"in {elapsed_secs:.2f}s"
+
+        suffix = f" {elapsed_print}{source_print}"
+    else:
+        sigil = console.sigil_skipped()
+        status = "skipped"
+        suffix = ""
+
+    return f"{sigil} {result.address} {status}{suffix}."
+
+
 @dataclass(frozen=True)
 class TestExtraEnv:
     env: Environment
diff --git a/src/python/pants/core/goals/test_test.py b/src/python/pants/core/goals/test_test.py
--- a/src/python/pants/core/goals/test_test.py
+++ b/src/python/pants/core/goals/test_test.py
@@ -29,11 +29,13 @@
     TestFieldSet,
     TestResult,
     TestSubsystem,
+    _format_test_summary,
     build_runtime_package_dependencies,
     run_tests,
 )
 from pants.core.util_rules.distdir import DistDir
 from pants.engine.addresses import Address
+from pants.engine.console import Console
 from pants.engine.desktop import OpenFiles, OpenFilesRequest
 from pants.engine.fs import (
     EMPTY_DIGEST,
@@ -43,7 +45,8 @@
     Snapshot,
     Workspace,
 )
-from pants.engine.process import InteractiveProcess, InteractiveProcessResult
+from pants.engine.internals.session import RunId
+from pants.engine.process import InteractiveProcess, InteractiveProcessResult, ProcessResultMetadata
 from pants.engine.target import (
     MultipleSourcesField,
     Target,
@@ -85,6 +88,11 @@ class MockTestFieldSet(TestFieldSet, metaclass=ABCMeta):
     def exit_code(_: Address) -> int:
         pass
 
+    @staticmethod
+    @abstractmethod
+    def skipped(_: Address) -> bool:
+        pass
+
     @property
     def test_result(self) -> TestResult:
         return TestResult(
@@ -96,6 +104,9 @@ def test_result(self) -> TestResult:
             address=self.address,
             coverage_data=MockCoverageData(self.address),
             output_setting=ShowOutput.ALL,
+            result_metadata=None
+            if self.skipped(self.address)
+            else ProcessResultMetadata(999, "ran_locally", 0),
         )
 
 
@@ -104,12 +115,20 @@ class SuccessfulFieldSet(MockTestFieldSet):
     def exit_code(_: Address) -> int:
         return 0
 
+    @staticmethod
+    def skipped(_: Address) -> bool:
+        return False
+
 
 class ConditionallySucceedsFieldSet(MockTestFieldSet):
     @staticmethod
     def exit_code(address: Address) -> int:
         return 27 if address.target_name == "bad" else 0
 
+    @staticmethod
+    def skipped(address: Address) -> bool:
+        return address.target_name == "skipped"
+
 
 @pytest.fixture
 def rule_runner() -> RuleRunner:
@@ -132,6 +151,7 @@ def run_test_rule(
     xml_dir: str | None = None,
     output: ShowOutput = ShowOutput.ALL,
     valid_targets: bool = True,
+    run_id: RunId = RunId(999),
 ) -> tuple[int, str]:
     test_subsystem = create_goal_subsystem(
         TestSubsystem,
@@ -179,6 +199,7 @@ def mock_coverage_report_generation(
                 workspace,
                 union_membership,
                 DistDir(relpath=Path("dist")),
+                run_id,
             ],
             mock_gets=[
                 MockGet(
@@ -238,22 +259,80 @@ def test_invalid_target_noops(rule_runner: RuleRunner) -> None:
 def test_summary(rule_runner: RuleRunner) -> None:
     good_address = Address("", target_name="good")
     bad_address = Address("", target_name="bad")
+    skipped_address = Address("", target_name="skipped")
 
     exit_code, stderr = run_test_rule(
         rule_runner,
         field_set=ConditionallySucceedsFieldSet,
-        targets=[make_target(good_address), make_target(bad_address)],
+        targets=[make_target(good_address), make_target(bad_address), make_target(skipped_address)],
     )
     assert exit_code == ConditionallySucceedsFieldSet.exit_code(bad_address)
     assert stderr == dedent(
         """\
 
-        ✓ //:good succeeded.
-        𐄂 //:bad failed.
+        ✓ //:good succeeded in 1.00s (memoized).
+        - //:skipped skipped.
+        𐄂 //:bad failed in 1.00s (memoized).
         """
     )
 
 
+def _assert_test_summary(
+    expected: str,
+    *,
+    exit_code: int | None,
+    run_id: int,
+    result_metadata: ProcessResultMetadata | None,
+) -> None:
+    assert expected == _format_test_summary(
+        TestResult(
+            exit_code=exit_code,
+            stdout="",
+            stderr="",
+            stdout_digest=EMPTY_FILE_DIGEST,
+            stderr_digest=EMPTY_FILE_DIGEST,
+            address=Address(spec_path="", target_name="dummy_address"),
+            output_setting=ShowOutput.FAILED,
+            result_metadata=result_metadata,
+        ),
+        RunId(run_id),
+        Console(use_colors=False),
+    )
+
+
+def test_format_summary_remote(rule_runner: RuleRunner) -> None:
+    _assert_test_summary(
+        "✓ //:dummy_address succeeded in 0.05s (ran remotely).",
+        exit_code=0,
+        run_id=0,
+        result_metadata=ProcessResultMetadata(50, "ran_remotely", 0),
+    )
+
+
+def test_format_summary_local(rule_runner: RuleRunner) -> None:
+    _assert_test_summary(
+        "✓ //:dummy_address succeeded in 0.05s.",
+        exit_code=0,
+        run_id=0,
+        result_metadata=ProcessResultMetadata(50, "ran_locally", 0),
+    )
+
+
+def test_format_summary_memoized(rule_runner: RuleRunner) -> None:
+    _assert_test_summary(
+        "✓ //:dummy_address succeeded in 0.05s (memoized).",
+        exit_code=0,
+        run_id=1234,
+        result_metadata=ProcessResultMetadata(50, "ran_locally", 0),
+    )
+
+
+def test_format_summary_skipped(rule_runner: RuleRunner) -> None:
+    _assert_test_summary(
+        "- //:dummy_address skipped.", exit_code=None, run_id=0, result_metadata=None
+    )
+
+
 def test_debug_target(rule_runner: RuleRunner) -> None:
     exit_code, _ = run_test_rule(
         rule_runner,
@@ -324,6 +403,7 @@ def assert_streaming_output(
     output_setting: ShowOutput = ShowOutput.ALL,
     expected_level: LogLevel,
     expected_message: str,
+    result_metadata: ProcessResultMetadata = ProcessResultMetadata(999, "dummy", 0),
 ) -> None:
     result = TestResult(
         exit_code=exit_code,
@@ -333,6 +413,7 @@ def assert_streaming_output(
         stderr_digest=EMPTY_FILE_DIGEST,
         output_setting=output_setting,
         address=Address("demo_test"),
+        result_metadata=result_metadata,
     )
     assert result.level() == expected_level
     assert result.message() == expected_message