Repository: pantsbuild/pants

Issue:
Add support for filtering file vs base targets
As explained in https://www.pantsbuild.org/docs/targets#dependencies-and-dependency-inference, Pants `2.x` automatically creates and uses file-level targets in addition to the targets declared in your BUILD files (aka "base" targets).

Currently there is no way to filter to select one or the other of these with the graph introspection goals. AFAIK, this is the first place where we have needed an option to affect the behavior/rendering of file targets, so where the option lives and how we refer to "file" vs "base" is up in the air.

There are at least two potential ways to implement this:
1. add a ternary option to `filter`: `--granularity={all,file,target}`, to allow for rendering only base targets, only file targets, etc
2. add a global ternary, implemented in the same place as `./pants --tags=..`

This issue is related to #10087.


Discussion:
As @thamenato pointed out in Slack, his requirement is that he is able to get out file names: `src/python/example/file.py`, rather than file target addresses, which would have the name of the owning target following a colon: `src/python/example/file.py:lib`.

Discussion in Slack concluded that using either of the implementations from the description (a flag on `filter`, or a global flag) would allow for something like:
```
# Request file targets, and then strip off the owning target name using unix tools.
./pants --changed-since ... filter --granularity=file | cut -d: -f1 | sort | uniq
# Request base targets, and then ask pants to expand targets to files.
./pants --changed-since ... filter --granularity=target | xargs ./pants filedeps
```

This seems like a sufficient solution: what do you think @thamenato ?

----

Implementation wise: it looks like a flag on `filter` would be straightforward.

But it looks like the global `--tags` and `--exclude-target-regexp` flags currently only work for "address" specs on the commandline, rather than also working for "file" specs. This actually seems like a bug that is worth fixing as well... if we decide that this flag would make more sense as a global filter, we will likely want to fix it via this change. Otherwise, I can open a separate issue. EDIT: Opened #11123 about this.

----

So I think that this comes down to a question of whether we want the "shorthand" that having this be a global option gives you... it's very convenient to be able to use `--tag` for quick filtering of roots. But I don't think that this type of filtering really needs shorthand, and given that, it feels like it is easier to continue to isolate our filtering options to the `filter` goal.
> But it looks like the global --tags and --exclude-target-regexp flags currently only work for "address" specs on the commandline, rather than also working for "file" specs. This actually seems like a bug that is worth fixing as well.

+1 that this is a bug.

Generally, I bias towards not having a global option like `--granularity`. I think its semantics are confusing for something like `test`, which always runs on a file level. The flag only seems relevant to project introspection goals.

Resolving patch:
diff --git a/src/python/pants/backend/project_info/filter_targets.py b/src/python/pants/backend/project_info/filter_targets.py
--- a/src/python/pants/backend/project_info/filter_targets.py
+++ b/src/python/pants/backend/project_info/filter_targets.py
@@ -2,6 +2,7 @@
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
 import re
+from enum import Enum
 from typing import Callable, Pattern
 
 from pants.engine.console import Console
@@ -14,13 +15,20 @@
     UnexpandedTargets,
     UnrecognizedTargetTypeException,
 )
+from pants.util.enums import match
 from pants.util.filtering import and_filters, create_filters
 
 
+class TargetGranularity(Enum):
+    all_targets = "all"
+    file_targets = "file"
+    build_targets = "BUILD"
+
+
 class FilterSubsystem(LineOriented, GoalSubsystem):
     """Filter the input targets based on various criteria.
 
-    Each of the filtering options below is a comma-separated list of filtering criteria, with an
+    Most of the filtering options below are comma-separated lists of filtering criteria, with an
     implied logical OR between them, so that a target passes the filter if it matches any of the
     criteria in the list.  A '-' prefix inverts the sense of the entire comma-separated list, so
     that a target passes the filter only if it matches none of the criteria in the list.
@@ -40,6 +48,15 @@ def register_options(cls, register):
             metavar="[+-]type1,type2,...",
             help="Filter on these target types, e.g. `resources` or `python_library`.",
         )
+        register(
+            "--granularity",
+            type=TargetGranularity,
+            default=TargetGranularity.all_targets,
+            help=(
+                "Filter to rendering only targets declared in BUILD files, only file-level "
+                "targets, or all targets."
+            ),
+        )
         register(
             "--address-regex",
             type=list,
@@ -88,11 +105,22 @@ def filter_tag_regex(tag_regex: str) -> TargetFilter:
         regex = compile_regex(tag_regex)
         return lambda tgt: any(bool(regex.search(tag)) for tag in tgt.get(Tags).value or ())
 
+    def filter_granularity(granularity: TargetGranularity) -> TargetFilter:
+        return match(
+            granularity,
+            {
+                TargetGranularity.all_targets: lambda _: True,
+                TargetGranularity.file_targets: lambda tgt: not tgt.address.is_base_target,
+                TargetGranularity.build_targets: lambda tgt: tgt.address.is_base_target,
+            },
+        )
+
     anded_filter: TargetFilter = and_filters(
         [
             *(create_filters(filter_subsystem.options.target_type, filter_target_type)),
             *(create_filters(filter_subsystem.options.address_regex, filter_address_regex)),
             *(create_filters(filter_subsystem.options.tag_regex, filter_tag_regex)),
+            filter_granularity(filter_subsystem.options.granularity),
         ]
     )
     addresses = sorted(target.address for target in targets if anded_filter(target))


Test patch:
diff --git a/src/python/pants/backend/project_info/filter_targets_test.py b/src/python/pants/backend/project_info/filter_targets_test.py
--- a/src/python/pants/backend/project_info/filter_targets_test.py
+++ b/src/python/pants/backend/project_info/filter_targets_test.py
@@ -7,7 +7,11 @@
 
 import pytest
 
-from pants.backend.project_info.filter_targets import FilterSubsystem, filter_targets
+from pants.backend.project_info.filter_targets import (
+    FilterSubsystem,
+    TargetGranularity,
+    filter_targets,
+)
 from pants.engine.addresses import Address
 from pants.engine.target import (
     RegisteredTargetTypes,
@@ -31,6 +35,7 @@ def run_goal(
     target_type: Optional[List[str]] = None,
     address_regex: Optional[List[str]] = None,
     tag_regex: Optional[List[str]] = None,
+    granularity: Optional[TargetGranularity] = None,
 ) -> str:
     console = MockConsole(use_colors=False)
     run_rule_with_mocks(
@@ -44,6 +49,7 @@ def run_goal(
                 target_type=target_type or [],
                 address_regex=address_regex or [],
                 tag_regex=tag_regex or [],
+                granularity=granularity or TargetGranularity.all_targets,
                 # Deprecated.
                 type=[],
                 target=[],
@@ -145,3 +151,13 @@ def test_filter_by_tag_regex() -> None:
     # Invalid regex.
     with pytest.raises(re.error):
         run_goal(targets, tag_regex=["("])
+
+
+def test_filter_by_granularity() -> None:
+    targets = [
+        MockTarget({}, address=Address("p1")),
+        MockTarget({}, address=Address("p1", relative_file_path="file.txt")),
+    ]
+    assert run_goal(targets, granularity=TargetGranularity.all_targets).strip() == "p1\np1/file.txt"
+    assert run_goal(targets, granularity=TargetGranularity.build_targets).strip() == "p1"
+    assert run_goal(targets, granularity=TargetGranularity.file_targets).strip() == "p1/file.txt"