Repository: pantsbuild/pants

Issue:
Most built-in tool lockfiles do not work with Python 3.12
**Describe the bug**

The built-in lockfiles for tools like: pytest, mypy, ..., use pip 23.0.1, which doesn't work with Python 3.12:

```
13:28:11.08 [INFO] Completed: Building pytest.pex from resource://pants.backend.python.subsystems/pytest.lock
13:28:11.09 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

ProcessExecutionFailure: Process 'Building pytest.pex from resource://pants.backend.python.subsystems/pytest.lock' failed with exit code 1.
stdout:

stderr:
The Pip requested was pip==23.0.1 but it does not work with the interpreter selected which is CPython 3.12.0 at /Users/huon/.pyenv/versions/3.12.0/bin/python3.12. Pip 23.0.1 requires Python <3.12,>=3.7.
```

Reproducer:

```shell
cd $(mktemp -d)

cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.19.0rc2"

backend_packages = [
  "pants.backend.python",
]

[python]
interpreter_constraints = ["==3.12.*"]
EOF

echo 'python_tests(name="t")' > BUILD
echo 'def test_foo(): pass' > test_example.py

# BUG: fails by default with Python 3.12
pants test ::
#> The Pip requested was pip==23.0.1 but it does not work with the interpreter selected which is CPython 3.12.0 at /Users/huon/.pyenv/versions/3.12.0/bin/python3.12. Pip 23.0.1 requires Python <3.12,>=3.7.

# OKAY: works fine
pants test --python-interpreter-constraints='["==3.11.*"]' ::
```

Workaround: use custom lockfiles, with `python_requirement`s/`install_from_resolve`.

**Pants version**

2.19.0rc2

(NB. 2.18.1 uses a version of pex that doesn't support Python 3.12 by default, but would likely be still affected by this if one specifies a newer pex in `[pex-cli]`.)

**OS**

macOS

**Additional info**
Discussed in https://github.com/pantsbuild/pants/pull/20310#pullrequestreview-1797974655

Details of pip version support in https://github.com/pantsbuild/pex/blob/4eb5c9aa25c6a695bf55263ab239189b720cebaf/pex/pip/version.py#L128-L233

See https://github.com/search?q=repo%3Apantsbuild%2Fpants%20pip_version%2023.0.1&type=code for 27 current lock files using pip version 23.0.1.


Discussion:
The lockfiles are mostly using 23.0.1, and that has a version range of >=3.7,<3.12: https://github.com/pantsbuild/pex/blob/4eb5c9aa25c6a695bf55263ab239189b720cebaf/pex/pip/version.py#L182-L187

Thus, I imagine it's not a problem to regenerate to a newer versions that just _increase_ this range, while still supporting 3.7: https://github.com/pantsbuild/pex/blob/4eb5c9aa25c6a695bf55263ab239189b720cebaf/pex/pip/version.py#L210-L233

That said, pip suggests that pants' default pip version (23.1.2) doesn't work with python 3.12 (this seems potentially incorrect? https://github.com/pantsbuild/pex/pull/2314), so we might need to leap to an even newer pip and/or bump the default pip version too.

Resolving patch:
diff --git a/src/python/pants/backend/python/lint/docformatter/subsystem.py b/src/python/pants/backend/python/lint/docformatter/subsystem.py
--- a/src/python/pants/backend/python/lint/docformatter/subsystem.py
+++ b/src/python/pants/backend/python/lint/docformatter/subsystem.py
@@ -14,7 +14,7 @@ class Docformatter(PythonToolBase):
     help = "The Python docformatter tool (https://github.com/myint/docformatter)."
 
     default_main = ConsoleScript("docformatter")
-    default_requirements = ["docformatter>=1.4,<1.6"]
+    default_requirements = ["docformatter>=1.4,<1.5"]
 
     register_interpreter_constraints = True
 
diff --git a/src/python/pants/backend/python/subsystems/setup.py b/src/python/pants/backend/python/subsystems/setup.py
--- a/src/python/pants/backend/python/subsystems/setup.py
+++ b/src/python/pants/backend/python/subsystems/setup.py
@@ -225,7 +225,7 @@ def interpreter_constraints(self) -> tuple[str, ...]:
         ),
     )
     pip_version = StrOption(
-        default="23.1.2",
+        default="24.0",
         help=softwrap(
             f"""
             Use this version of Pip for resolving requirements and generating lockfiles.


Test patch: