Repository: pantsbuild/pants

Issue:
Add support for pytest.ini
We support config files for linters/formatters, but (for no particular reason) do not support it with Pytest. While users can use [conftest.py](https://www.pantsbuild.org/docs/python-test-goal#conftestpy) and [set CLI passthrough flags in `pants.toml`](https://www.pantsbuild.org/docs/python-test-goal#passing-arguments-to-pytest), there's no reason to not support `pytest.ini` too.

You'll be touching the file [`pytest_runner.py`](https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/python/goals/pytest_runner.py).

You'll be using the [Rules API](https://www.pantsbuild.org/docs/rules-api-concepts), although you don't need to master this. 

--

We'll do this very similar to how we do config files for linters and formatters, e.g. Flake8. First, we need an option called `--config`:

 https://github.com/pantsbuild/pants/blob/95ed385f71feec341f02ef5476bf95ade1a0b4dc/src/python/pants/backend/python/lint/flake8/subsystem.py#L39-L45

See https://www.pantsbuild.org/docs/rules-api-subsystems for an explanation of the options system.

Also you'll want to add a `@property` similar to this, for convenience + to make MyPy more useful:

https://github.com/pantsbuild/pants/blob/95ed385f71feec341f02ef5476bf95ade1a0b4dc/src/python/pants/backend/python/lint/flake8/subsystem.py#L55-L57

Confirm the new option shows up by running `./pants help-advanced pytest`.

--

Next, we need to get the config file from the file system. See https://www.pantsbuild.org/docs/rules-api-file-system for how we do this with `PathGlobs`, and what a `Digest` means.

https://github.com/pantsbuild/pants/blob/95ed385f71feec341f02ef5476bf95ade1a0b4dc/src/python/pants/backend/python/lint/flake8/rules.py#L76-L83

This Digest will give us a handle for the config file so that we can include it in the chroot.

We then need to merge it into our final input digest, so that all the files used by the test run get included together:

https://github.com/pantsbuild/pants/blob/95ed385f71feec341f02ef5476bf95ade1a0b4dc/src/python/pants/backend/python/lint/flake8/rules.py#L93-L96

--

We possibly need to update the argv for Pytest to explicitly point to the config file. 

https://github.com/pantsbuild/pants/blob/95ed385f71feec341f02ef5476bf95ade1a0b4dc/src/python/pants/backend/python/lint/flake8/rules.py#L48-L53

I'm not seeing anything about that at https://docs.pytest.org/en/stable/customize.html, but it would be helpful to look at the help message for Pytest to confirm. You can do something like `pex pytest==<version> -m pytest:main -o pytest.pex`, then `./pytest.pex --help`. Try Pytest 6, but also earlier versions.

Why would we want to use this flag if the config is usually auto-detected? For some monorepos, people try to centralize all config files into one place like a `build-support/` folder, and they don't like it being in the root of the repo. Some tools like isort only auto-detect if it's in the root, and you need the flag if in a non-standard place. I'm not sure how Pytest behaves.

--

Finally, add a test, similar to this:

https://github.com/pantsbuild/pants/blob/95ed385f71feec341f02ef5476bf95ade1a0b4dc/src/python/pants/backend/python/lint/flake8/rules_integration_test.py#L156-L158

That is, you want to set up something that would normally fail if the config weren't used, and assert that it did not fail, as it implies the config was picked up. Or vice versa.

See https://www.pantsbuild.org/docs/rules-api-testing#approach-3-rulerunner-integration-tests-for-rules for how testing plugins works.



Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/python/goals/pytest_runner.py b/src/python/pants/backend/python/goals/pytest_runner.py
--- a/src/python/pants/backend/python/goals/pytest_runner.py
+++ b/src/python/pants/backend/python/goals/pytest_runner.py
@@ -42,7 +42,15 @@
 )
 from pants.core.util_rules.source_files import SourceFiles, SourceFilesRequest
 from pants.engine.addresses import UnparsedAddressInputs
-from pants.engine.fs import AddPrefix, Digest, DigestSubset, MergeDigests, PathGlobs, Snapshot
+from pants.engine.fs import (
+    AddPrefix,
+    Digest,
+    DigestSubset,
+    GlobMatchErrorBehavior,
+    MergeDigests,
+    PathGlobs,
+    Snapshot,
+)
 from pants.engine.process import (
     FallibleProcessResult,
     InteractiveProcess,
@@ -133,6 +141,15 @@ async def setup_pytest_for_target(
         ),
     )
 
+    config_digest_request = Get(
+        Digest,
+        PathGlobs(
+            globs=[pytest.config] if pytest.config else [],
+            glob_match_error_behavior=GlobMatchErrorBehavior.error,
+            description_of_origin="the option `--pytest-config`",
+        ),
+    )
+
     prepared_sources_request = Get(
         PythonSourceFiles, PythonSourceFilesRequest(all_targets, include_files=True)
     )
@@ -161,11 +178,18 @@ async def setup_pytest_for_target(
         SourceFiles, SourceFilesRequest([request.field_set.sources])
     )
 
-    pytest_pex, requirements_pex, prepared_sources, field_set_source_files = await MultiGet(
+    (
+        pytest_pex,
+        requirements_pex,
+        prepared_sources,
+        field_set_source_files,
+        config_digest,
+    ) = await MultiGet(
         pytest_pex_request,
         requirements_pex_request,
         prepared_sources_request,
         field_set_source_files_request,
+        config_digest_request,
     )
 
     pytest_runner_pex = await Get(
@@ -187,6 +211,7 @@ async def setup_pytest_for_target(
             (
                 coverage_config.digest,
                 prepared_sources.source_files.snapshot.digest,
+                config_digest,
                 *(binary.digest for binary in assets),
             )
         ),
diff --git a/src/python/pants/backend/python/subsystems/pytest.py b/src/python/pants/backend/python/subsystems/pytest.py
--- a/src/python/pants/backend/python/subsystems/pytest.py
+++ b/src/python/pants/backend/python/subsystems/pytest.py
@@ -3,7 +3,7 @@
 
 from typing import Optional, Tuple, cast
 
-from pants.option.custom_types import shell_str
+from pants.option.custom_types import file_option, shell_str
 from pants.option.subsystem import Subsystem
 
 
@@ -89,6 +89,22 @@ def register_options(cls, register):
                 "to tests under this environment variable name."
             ),
         )
+        register(
+            "--config",
+            type=file_option,
+            default=None,
+            advanced=True,
+            help=(
+                "Path to pytest.ini or alternative Pytest config file.\n\n"
+                "Pytest will attempt to auto-discover the config file,"
+                "meaning that it should typically be an ancestor of your"
+                "tests, such as in the build root.\n\nPants will not automatically"
+                " set --rootdir for you to force Pytest to pick up your config "
+                "file, but you can manually set --rootdir in [pytest].args.\n\n"
+                "Refer to https://docs.pytest.org/en/stable/customize.html#"
+                "initialization-determining-rootdir-and-configfile."
+            ),
+        )
 
     def get_requirement_strings(self) -> Tuple[str, ...]:
         """Returns a tuple of requirements-style strings for Pytest and Pytest plugins."""
@@ -105,3 +121,7 @@ def timeout_default(self) -> Optional[int]:
     @property
     def timeout_maximum(self) -> Optional[int]:
         return cast(Optional[int], self.options.timeout_maximum)
+
+    @property
+    def config(self) -> Optional[str]:
+        return cast(Optional[str], self.options.config)


Test patch:
diff --git a/src/python/pants/backend/python/goals/pytest_runner_integration_test.py b/src/python/pants/backend/python/goals/pytest_runner_integration_test.py
--- a/src/python/pants/backend/python/goals/pytest_runner_integration_test.py
+++ b/src/python/pants/backend/python/goals/pytest_runner_integration_test.py
@@ -52,6 +52,7 @@ def rule_runner() -> RuleRunner:
 SOURCE_ROOT = "tests/python"
 PACKAGE = os.path.join(SOURCE_ROOT, "pants_test")
 GOOD_SOURCE = FileContent(f"{PACKAGE}/test_good.py", b"def test():\n  pass\n")
+GOOD_WITH_PRINT = FileContent(f"{PACKAGE}/test_good.py", b"def test():\n  print('All good!')")
 BAD_SOURCE = FileContent(f"{PACKAGE}/test_bad.py", b"def test():\n  assert False\n")
 PY3_ONLY_SOURCE = FileContent(f"{PACKAGE}/test_py3.py", b"def test() -> None:\n  pass\n")
 LIBRARY_SOURCE = FileContent(f"{PACKAGE}/library.py", b"def add_two(x):\n  return x + 2\n")
@@ -143,6 +144,7 @@ def run_pytest(
     execution_slot_var: Optional[str] = None,
     extra_env_vars: Optional[str] = None,
     env: Optional[Mapping[str, str]] = None,
+    config: Optional[str] = None,
     force: bool = False,
 ) -> TestResult:
     args = [
@@ -162,6 +164,9 @@ def run_pytest(
         args.append("--test-use-coverage")
     if execution_slot_var:
         args.append(f"--pytest-execution-slot-var={execution_slot_var}")
+    if config:
+        rule_runner.create_file(relpath="pytest.ini", contents=config)
+        args.append("--pytest-config=pytest.ini")
     if force:
         args.append("--test-force")
     rule_runner.set_options(args, env=env)
@@ -253,6 +258,13 @@ def test():
     assert f"{PACKAGE}/test_relative_import.py ." in result.stdout
 
 
+def test_respects_config(rule_runner: RuleRunner) -> None:
+    target = create_test_target(rule_runner, [GOOD_WITH_PRINT])
+    result = run_pytest(rule_runner, target, config="[pytest]\naddopts = -s\n")
+    assert result.exit_code == 0
+    assert "All good!" in result.stdout and "Captured" not in result.stdout
+
+
 def test_transitive_dep(rule_runner: RuleRunner) -> None:
     create_python_library(rule_runner, [LIBRARY_SOURCE])
     transitive_dep_fc = FileContent(