Repository: pantsbuild/pants

Issue:
Bug in source root stripping in pytest_coverage.py
This is the error message:

<img width="1115" alt="Screenshot 2020-05-30 07 22 29" src="https://user-images.githubusercontent.com/1268088/83330792-802c0380-a246-11ea-9ca0-481fb2c99828.png">

This is the source roots section in pants.toml:
<img width="342" alt="Screenshot 2020-05-30 07 23 00" src="https://user-images.githubusercontent.com/1268088/83330793-815d3080-a246-11ea-8060-201843bf488d.png">


It is not clear (to me...) what the `src` the error is referring to.
I am going to dig thru the pants source (and the PR that triggers this error, I am pretty sure it is https://github.com/pantsbuild/pants/pull/9888)
However, this is not acceptable for a regular pants user.

@benjyw 



Discussion:
Another data point on this, this error happens when I am running tests not when I am running other goals (lint) on the same targets.
i.e. `./pants lint src/python::` works and ./pants test src/python::` fails.
full stack trace:
```

14:54:38:566 [ERROR] 1 Exception encountered:

Engine traceback:
  in Run pytest
Traceback (most recent call last):
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/engine/internals/native.py", line 288, in call
    val = func(*args)
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/source/source_root.py", line 527, in get_source_root_strict
    return source_root_config.get_source_roots().strict_find_by_path(source_root_request.path)
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/source/source_root.py", line 138, in strict_find_by_path
    raise NoSourceRootError(path)
pants.source.source_root.NoSourceRootError: No source root found for `src`. See https://pants.readme.io/docs/source-roots for how to define source roots.
Traceback (most recent call last):
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 356, in run
    engine_result = self._maybe_run_v2(v2)
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 236, in _maybe_run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 253, in _maybe_run_v2_body
    return self.graph_session.run_goal_rules(
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 255, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 614, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 573, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in Run pytest
Traceback (most recent call last):
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/engine/internals/native.py", line 288, in call
    val = func(*args)
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/source/source_root.py", line 527, in get_source_root_strict
    return source_root_config.get_source_roots().strict_find_by_path(source_root_request.path)
  File "/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0rc0_py38/lib/python3.8/site-packages/pants/source/source_root.py", line 138, in strict_find_by_path
    raise NoSourceRootError(path)
pants.source.source_root.NoSourceRootError: No source root found for `src`. See https://pants.readme.io/docs/source-roots for how to define source roots.

```
actually, this is trigged by turning on code coverage in tests:
`./pants test --use-coverage --pytest-coverage-report=xml src/python::`
running w/o code coverage works.
``./pants test  src/python::``
I think the issue is not just the error message. I think this is now broken.
I was able to debug:
here is some more info when running under a debugger.
<img width="1612" alt="Screenshot 2020-05-30 08 32 33" src="https://user-images.githubusercontent.com/1268088/83332467-3c3dfc00-a250-11ea-96c7-76e1e63e8117.png">


Not sure where that `SourceRootRequest` is coming from...

OK, this is because apparently we inject `__init__.py` above the source root... Probably we should inject only after stripping source roots. Working on a fix now.
repo: https://github.com/pantsbuild/pants/pull/9919

Resolving patch:
diff --git a/src/python/pants/backend/python/rules/inject_init.py b/src/python/pants/backend/python/rules/inject_init.py
--- a/src/python/pants/backend/python/rules/inject_init.py
+++ b/src/python/pants/backend/python/rules/inject_init.py
@@ -12,7 +12,8 @@
 # NB: Needed due to graph ambiguity.
 @dataclass(frozen=True)
 class InjectInitRequest:
-    snapshot: Snapshot
+    # NB: This must contain sources already stripped of their source roots.
+    stripped_sources_snapshot: Snapshot
 
 
 @dataclass(frozen=True)
@@ -26,7 +27,7 @@ async def inject_missing_init_files(request: InjectInitRequest) -> InitInjectedS
 
     This will preserve any `__init__.py` files already in the input snapshot.
     """
-    snapshot = request.snapshot
+    snapshot = request.stripped_sources_snapshot
     missing_init_files = sorted(identify_missing_init_files(snapshot.files))
     if not missing_init_files:
         return InitInjectedSnapshot(snapshot)
diff --git a/src/python/pants/backend/python/rules/pytest_coverage.py b/src/python/pants/backend/python/rules/pytest_coverage.py
--- a/src/python/pants/backend/python/rules/pytest_coverage.py
+++ b/src/python/pants/backend/python/rules/pytest_coverage.py
@@ -141,18 +141,25 @@ class CoverageConfig:
 @rule
 async def create_coverage_config(coverage_config_request: CoverageConfigRequest) -> CoverageConfig:
     sources = await Get[SourceFiles](
-        AllSourceFilesRequest((tgt.get(Sources) for tgt in coverage_config_request.targets))
+        AllSourceFilesRequest(
+            (
+                tgt.get(PythonSources)
+                for tgt in coverage_config_request.targets
+                if tgt.has_field(PythonSources)
+            )
+        )
     )
-    init_injected = await Get[InitInjectedSnapshot](InjectInitRequest(sources.snapshot))
 
     source_root_stripped_sources = await Get[SourceRootStrippedSources](
-        StripSnapshotRequest(snapshot=init_injected.snapshot)
+        StripSnapshotRequest(snapshot=sources.snapshot)
     )
 
     # Generate a map from source root stripped source to its source root. eg:
     #  {'pants/testutil/subsystem/util.py': 'src/python'}. This is so that coverage reports
     #  referencing /chroot/path/pants/testutil/subsystem/util.py can be mapped back to the actual
     #  sources they reference when generating coverage reports.
+    # Note that this map doesn't contain injected __init__.py files, but we tell coverage
+    # to ignore those using the -i flag (see below).
     stripped_files_to_source_roots = {}
     for source_root, files in source_root_stripped_sources.root_to_relfiles.items():
         for file in files:
@@ -321,9 +328,8 @@ async def generate_coverage_report(
     """Takes all Python test results and generates a single coverage report."""
     requirements_pex = coverage_setup.requirements_pex
 
-    python_targets = [tgt for tgt in transitive_targets.closure if tgt.has_field(PythonSources)]
     coverage_config = await Get[CoverageConfig](
-        CoverageConfigRequest(Targets(python_targets), is_test_time=False)
+        CoverageConfigRequest(Targets(transitive_targets.closure), is_test_time=False)
     )
 
     sources = await Get[SourceFiles](
@@ -347,7 +353,10 @@ async def generate_coverage_report(
     )
 
     report_type = coverage_subsystem.options.report
-    coverage_args = [report_type.report_name]
+    # The -i flag causes coverage to ignore files it doesn't have sources for.
+    # Specifically, it will ignore the injected __init__.py files, which is what we want since
+    # those are empty and don't correspond to a real source file the user is aware of.
+    coverage_args = [report_type.report_name, "-i"]
 
     process = requirements_pex.create_process(
         pex_path=f"./{coverage_setup.requirements_pex.output_filename}",


Test patch: