Repository: python/mypy

Issue:
tests failing on master (macOS Monterey / py 3.7)
**Bug Report**

Tests are failing for the master branch on my macOS laptop.

A few tests in `PEP561Suite` are flaky for reasons I don't quite understand. 

**To Reproduce**

```
git pull
git checkout f501cf649d7976077a7196e3548d773d67340a8c
python3.7 -m venv .venv
.venv/bin/pip install -r test-requirements.txt
.venv/bin/pip install -e .
PATH=$(pwd)/.venv/bin:$PATH .venv/bin/pytest -q -k 'PEP561Suite'
```

**Expected Behavior**

Tests pass.

**Actual Behavior**

There's usually exactly one failure per run. On some rare occasions I've seen more than one failure, or no failures.

Here are some sample failures:

```
_____________________________________________________________________________________________________________ testStubPrecedence _____________________________________________________________________________________________________________
[gw3] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:66:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:152: in test_pep561
    testcase.file, testcase.line, iter_count))
E   AssertionError: Invalid output (/System/Volumes/Data/repos/mypy/test-data/unit/pep561.test, line 66)
------------------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------------------
Expected:
  testStubPrecedence.py:5: note: Revealed type is "builtins.list[builtins.str]" (diff)
Actual:
  testStubPrecedence.py:3: error: Module "typedpkg" has no attribute "dne" (diff)
  testStubPrecedence.py:5: note: Revealed type is "builtins.list[builtins.str]" (diff)

Alignment of first line difference:
  E: testStubPrecedence.py:5: note: Revealed type is "builtins.list[builtins....
  A: testStubPrecedence.py:3: error: Module "typedpkg" has no attribute "dne"...
                           ^
```

```
______________________________________________________________________________________________ testNamespacePkgWStubsWithNamespacePackagesFlag _______________________________________________________________________________________________
[gw2] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:204:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:152: in test_pep561
    testcase.file, testcase.line, iter_count))
E   AssertionError: Invalid output (/System/Volumes/Data/repos/mypy/test-data/unit/pep561.test, line 204)
------------------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------------------
Expected:
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:7: error: Argument 1 to "bf" has incompatible type "int"; expected "bool" (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:8: error: Argument 1 to "bf" has incompatible type "int"; expected "bool" (diff)
Actual:
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:3: error: Skipping analyzing "typedpkg_ns.a.bbb": module is installed, but missing library stubs or py.typed marker (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:3: error: Skipping analyzing "typedpkg_ns.a": module is installed, but missing library stubs or py.typed marker (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:8: error: Argument 1 to "bf" has incompatible type "int"; expected "bool" (diff)

Alignment of first line difference:
  E: ...spacePackagesFlag.py:7: error: Argument 1 to "bf" has incompatible ty...
  A: ...spacePackagesFlag.py:3: error: Skipping analyzing "typedpkg_ns.a.bbb"...
                             ^
```

```
_______________________________________________________________________________________________________ testTypedPkgNamespaceRegImport _______________________________________________________________________________________________________
[gw2] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:150:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:152: in test_pep561
    testcase.file, testcase.line, iter_count))
E   AssertionError: Invalid output (/System/Volumes/Data/repos/mypy/test-data/unit/pep561.test, line 150)
------------------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------------------
Expected:
  testTypedPkgNamespaceRegImport.py:4: error: Cannot find implementation or library stub for module named "typedpkg_ns.a.dne" (diff)
  testTypedPkgNamespaceRegImport.py:4: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports (diff)
  testTypedPkgNamespaceRegImport.py:10: error: Argument 1 has incompatible type "bool"; expected "str" (diff)
  testTypedPkgNamespaceRegImport.py:11: error: Argument 1 has incompatible type "int"; expected "bool" (diff)
Actual:
  testTypedPkgNamespaceRegImport.py:3: error: Cannot find implementation or library stub for module named "typedpkg_ns.a.bbb" (diff)
  testTypedPkgNamespaceRegImport.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports (diff)
  testTypedPkgNamespaceRegImport.py:3: error: Cannot find implementation or library stub for module named "typedpkg_ns" (diff)
  testTypedPkgNamespaceRegImport.py:3: error: Cannot find implementation or library stub for module named "typedpkg_ns.a" (diff)
  testTypedPkgNamespaceRegImport.py:4: error: Cannot find implementation or library stub for module named "typedpkg_ns.a.dne" (diff)
  testTypedPkgNamespaceRegImport.py:10: error: Argument 1 has incompatible type "bool"; expected "str" (diff)

Alignment of first line difference:
  E: ...kgNamespaceRegImport.py:4: error: Cannot find implementation or libra...
  A: ...kgNamespaceRegImport.py:3: error: Cannot find implementation or libra...
                                ^
```

```
_______________________________________________________________________________________________________ testTypedPkgNamespaceRegImport _______________________________________________________________________________________________________
[gw0] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:150:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:113: in test_pep561
    install_package(pkg, python_executable, use_pip, editable)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:86: in install_package
    raise Exception(proc.stdout.decode('utf-8') + proc.stderr.decode('utf-8'))
E   Exception: Looking in indexes: https://registry.affirm-stage.com/artifactory/api/pypi/pypi/simple
E   Processing /System/Volumes/Data/repos/mypy/test-data/packages/typedpkg_ns_a
E     Preparing metadata (setup.py): started
E     Preparing metadata (setup.py): finished with status 'done'
E   Building wheels for collected packages: typedpkg-namespace.alpha
E     Building wheel for typedpkg-namespace.alpha (setup.py): started
E     Building wheel for typedpkg-namespace.alpha (setup.py): finished with status 'done'
E     Created wheel for typedpkg-namespace.alpha: filename=typedpkg_namespace.alpha-1.0.0-py3-none-any.whl size=1588 sha256=21e162c192f4c7f108ec1b970a54d677de03e021e7143da28589e81f308a7505
E     Stored in directory: /private/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/pip-ephem-wheel-cache-niwbfkch/wheels/85/1b/b9/19a1923b949574d3f92d0b2fd98b06cfe4eb7aa0b819bbbcf7
E   ERROR: Exception:
E   Traceback (most recent call last):
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2681, in version
E       return self._version
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
E       raise AttributeError(attr)
E   AttributeError: _version
E
E   During handling of the above exception, another exception occurred:
E
E   Traceback (most recent call last):
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
E       status = run_func(*args)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
E       return func(self, options, args)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 366, in run
E       global_options=[],
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/wheel_builder.py", line 354, in build
E       req.editable and req.permit_editable_wheels,
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/wheel_builder.py", line 227, in _build_one
E       _verify_one(req, wheel_path)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/wheel_builder.py", line 175, in _verify_one
E       dist_verstr = str(dist.version)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/metadata/pkg_resources.py", line 147, in version
E       return parse_version(self._dist.version)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2689, in version
E       raise ValueError(msg, self)
E   ValueError: ("Missing 'Version:' header and/or METADATA file at path: [could not detect]", typedpkg-namespace-alpha [unknown version] (/private/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/pip-ephem-wheel-cache-niwbfkch/wheels/85/1b/b9/19a1923b949574d3f92d0b2fd98b06cfe4eb7aa0b819bbbcf7/typedpkg_namespace.alpha-1.0.0-py3-none-any.whl))
```

**Your Environment**

mypy commit `f501cf649d7976077a7196e3548d773d67340a8c`
Python 3.7.13
macOS Monterey 12.3.1
x86_64
tests failing on master (macOS Monterey / py 3.7)
**Bug Report**

Tests are failing for the master branch on my macOS laptop.

A few tests in `PEP561Suite` are flaky for reasons I don't quite understand. 

**To Reproduce**

```
git pull
git checkout f501cf649d7976077a7196e3548d773d67340a8c
python3.7 -m venv .venv
.venv/bin/pip install -r test-requirements.txt
.venv/bin/pip install -e .
PATH=$(pwd)/.venv/bin:$PATH .venv/bin/pytest -q -k 'PEP561Suite'
```

**Expected Behavior**

Tests pass.

**Actual Behavior**

There's usually exactly one failure per run. On some rare occasions I've seen more than one failure, or no failures.

Here are some sample failures:

```
_____________________________________________________________________________________________________________ testStubPrecedence _____________________________________________________________________________________________________________
[gw3] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:66:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:152: in test_pep561
    testcase.file, testcase.line, iter_count))
E   AssertionError: Invalid output (/System/Volumes/Data/repos/mypy/test-data/unit/pep561.test, line 66)
------------------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------------------
Expected:
  testStubPrecedence.py:5: note: Revealed type is "builtins.list[builtins.str]" (diff)
Actual:
  testStubPrecedence.py:3: error: Module "typedpkg" has no attribute "dne" (diff)
  testStubPrecedence.py:5: note: Revealed type is "builtins.list[builtins.str]" (diff)

Alignment of first line difference:
  E: testStubPrecedence.py:5: note: Revealed type is "builtins.list[builtins....
  A: testStubPrecedence.py:3: error: Module "typedpkg" has no attribute "dne"...
                           ^
```

```
______________________________________________________________________________________________ testNamespacePkgWStubsWithNamespacePackagesFlag _______________________________________________________________________________________________
[gw2] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:204:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:152: in test_pep561
    testcase.file, testcase.line, iter_count))
E   AssertionError: Invalid output (/System/Volumes/Data/repos/mypy/test-data/unit/pep561.test, line 204)
------------------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------------------
Expected:
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:7: error: Argument 1 to "bf" has incompatible type "int"; expected "bool" (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:8: error: Argument 1 to "bf" has incompatible type "int"; expected "bool" (diff)
Actual:
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:3: error: Skipping analyzing "typedpkg_ns.a.bbb": module is installed, but missing library stubs or py.typed marker (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:3: error: Skipping analyzing "typedpkg_ns.a": module is installed, but missing library stubs or py.typed marker (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:8: error: Argument 1 to "bf" has incompatible type "int"; expected "bool" (diff)

Alignment of first line difference:
  E: ...spacePackagesFlag.py:7: error: Argument 1 to "bf" has incompatible ty...
  A: ...spacePackagesFlag.py:3: error: Skipping analyzing "typedpkg_ns.a.bbb"...
                             ^
```

```
_______________________________________________________________________________________________________ testTypedPkgNamespaceRegImport _______________________________________________________________________________________________________
[gw2] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:150:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:152: in test_pep561
    testcase.file, testcase.line, iter_count))
E   AssertionError: Invalid output (/System/Volumes/Data/repos/mypy/test-data/unit/pep561.test, line 150)
------------------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------------------
Expected:
  testTypedPkgNamespaceRegImport.py:4: error: Cannot find implementation or library stub for module named "typedpkg_ns.a.dne" (diff)
  testTypedPkgNamespaceRegImport.py:4: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports (diff)
  testTypedPkgNamespaceRegImport.py:10: error: Argument 1 has incompatible type "bool"; expected "str" (diff)
  testTypedPkgNamespaceRegImport.py:11: error: Argument 1 has incompatible type "int"; expected "bool" (diff)
Actual:
  testTypedPkgNamespaceRegImport.py:3: error: Cannot find implementation or library stub for module named "typedpkg_ns.a.bbb" (diff)
  testTypedPkgNamespaceRegImport.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports (diff)
  testTypedPkgNamespaceRegImport.py:3: error: Cannot find implementation or library stub for module named "typedpkg_ns" (diff)
  testTypedPkgNamespaceRegImport.py:3: error: Cannot find implementation or library stub for module named "typedpkg_ns.a" (diff)
  testTypedPkgNamespaceRegImport.py:4: error: Cannot find implementation or library stub for module named "typedpkg_ns.a.dne" (diff)
  testTypedPkgNamespaceRegImport.py:10: error: Argument 1 has incompatible type "bool"; expected "str" (diff)

Alignment of first line difference:
  E: ...kgNamespaceRegImport.py:4: error: Cannot find implementation or libra...
  A: ...kgNamespaceRegImport.py:3: error: Cannot find implementation or libra...
                                ^
```

```
_______________________________________________________________________________________________________ testTypedPkgNamespaceRegImport _______________________________________________________________________________________________________
[gw0] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:150:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:113: in test_pep561
    install_package(pkg, python_executable, use_pip, editable)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:86: in install_package
    raise Exception(proc.stdout.decode('utf-8') + proc.stderr.decode('utf-8'))
E   Exception: Looking in indexes: https://registry.affirm-stage.com/artifactory/api/pypi/pypi/simple
E   Processing /System/Volumes/Data/repos/mypy/test-data/packages/typedpkg_ns_a
E     Preparing metadata (setup.py): started
E     Preparing metadata (setup.py): finished with status 'done'
E   Building wheels for collected packages: typedpkg-namespace.alpha
E     Building wheel for typedpkg-namespace.alpha (setup.py): started
E     Building wheel for typedpkg-namespace.alpha (setup.py): finished with status 'done'
E     Created wheel for typedpkg-namespace.alpha: filename=typedpkg_namespace.alpha-1.0.0-py3-none-any.whl size=1588 sha256=21e162c192f4c7f108ec1b970a54d677de03e021e7143da28589e81f308a7505
E     Stored in directory: /private/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/pip-ephem-wheel-cache-niwbfkch/wheels/85/1b/b9/19a1923b949574d3f92d0b2fd98b06cfe4eb7aa0b819bbbcf7
E   ERROR: Exception:
E   Traceback (most recent call last):
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2681, in version
E       return self._version
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
E       raise AttributeError(attr)
E   AttributeError: _version
E
E   During handling of the above exception, another exception occurred:
E
E   Traceback (most recent call last):
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
E       status = run_func(*args)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
E       return func(self, options, args)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 366, in run
E       global_options=[],
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/wheel_builder.py", line 354, in build
E       req.editable and req.permit_editable_wheels,
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/wheel_builder.py", line 227, in _build_one
E       _verify_one(req, wheel_path)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/wheel_builder.py", line 175, in _verify_one
E       dist_verstr = str(dist.version)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/metadata/pkg_resources.py", line 147, in version
E       return parse_version(self._dist.version)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2689, in version
E       raise ValueError(msg, self)
E   ValueError: ("Missing 'Version:' header and/or METADATA file at path: [could not detect]", typedpkg-namespace-alpha [unknown version] (/private/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/pip-ephem-wheel-cache-niwbfkch/wheels/85/1b/b9/19a1923b949574d3f92d0b2fd98b06cfe4eb7aa0b819bbbcf7/typedpkg_namespace.alpha-1.0.0-py3-none-any.whl))
```

**Your Environment**

mypy commit `f501cf649d7976077a7196e3548d773d67340a8c`
Python 3.7.13
macOS Monterey 12.3.1
x86_64


Discussion:
NB: initial report included a mypyc build failure, which turned out to be caused by using python 3.7.8 built via pyenv prior to the upgrade to Monterey. Using a freshly build 3.7.13 solved that. The flakiness persists however...
For what it's worth, I am seeing the exact same thing on Python 3.10, also on Mac Monterey (12.3.1), not sure if it's relevant but I have the Apple M1 cpu, and so I compile my Pythons for arm64.
I suspect that this is caused by some shared file system state, and only reproduces when running tests in parallel. All the tests passed when I ran them sequentially (on macOS). Also CI has been reliable, and it doesn't use much/any parallelism.
I saw this on Linux too, on Python 3.10.4. Confirm that things work reliably and consistently with `pytest -q  -n0 -k PEP561Suite`. However, as far as I can see from a casual glance, it seems that each case of the PEP 561 suite runs within its own venv (and associated unique tempdir) so I don't understand exactly how parallel mode gives rise to FS race conditions here.
> I saw this on Linux too, on Python 3.10.4. Confirm that things work reliably and consistently with `pytest -q -n0 -k PEP561Suite`. However, as far as I can see from a casual glance, it seems that each case of the PEP 561 suite runs within its own venv (and associated unique tempdir) so I don't understand exactly how parallel mode gives rise to FS race conditions here.

`pip` would be my first guess based on the stack trace I posted in the original bug report... It's notoriously prone to breakage if invoked concurrently, even on separate venvs, because it may still use shared global state, for instance when downloading packages or building wheels.
NB: initial report included a mypyc build failure, which turned out to be caused by using python 3.7.8 built via pyenv prior to the upgrade to Monterey. Using a freshly build 3.7.13 solved that. The flakiness persists however...
For what it's worth, I am seeing the exact same thing on Python 3.10, also on Mac Monterey (12.3.1), not sure if it's relevant but I have the Apple M1 cpu, and so I compile my Pythons for arm64.
I suspect that this is caused by some shared file system state, and only reproduces when running tests in parallel. All the tests passed when I ran them sequentially (on macOS). Also CI has been reliable, and it doesn't use much/any parallelism.
I saw this on Linux too, on Python 3.10.4. Confirm that things work reliably and consistently with `pytest -q  -n0 -k PEP561Suite`. However, as far as I can see from a casual glance, it seems that each case of the PEP 561 suite runs within its own venv (and associated unique tempdir) so I don't understand exactly how parallel mode gives rise to FS race conditions here.
> I saw this on Linux too, on Python 3.10.4. Confirm that things work reliably and consistently with `pytest -q -n0 -k PEP561Suite`. However, as far as I can see from a casual glance, it seems that each case of the PEP 561 suite runs within its own venv (and associated unique tempdir) so I don't understand exactly how parallel mode gives rise to FS race conditions here.

`pip` would be my first guess based on the stack trace I posted in the original bug report... It's notoriously prone to breakage if invoked concurrently, even on separate venvs, because it may still use shared global state, for instance when downloading packages or building wheels.

Resolving patch:
diff --git a/runtests.py b/runtests.py
--- a/runtests.py
+++ b/runtests.py
@@ -58,11 +58,15 @@
     'pytest-slow': ['pytest', '-q', '-k', ' or '.join(
         [SAMPLES,
          TYPESHED,
-         PEP561,
          DAEMON,
          MYPYC_EXTERNAL,
          MYPYC_COMMAND_LINE,
          ERROR_STREAM])],
+
+    # Test cases that might take minutes to run
+    'pytest-slower': ['pytest', '-q', '-k', ' or '.join(
+        [PEP561])],
+
     # Test cases to run in typeshed CI
     'typeshed-ci': ['pytest', '-q', '-k', ' or '.join([CMDLINE,
                                                        EVALUATION,


Test patch:
diff --git a/mypy/test/config.py b/mypy/test/config.py
--- a/mypy/test/config.py
+++ b/mypy/test/config.py
@@ -15,3 +15,12 @@
 # This is *within* the tempfile.TemporaryDirectory that is chroot'ed per testcase.
 # It is also hard-coded in numerous places, so don't change it.
 test_temp_dir = 'tmp'
+
+# The PEP 561 tests do a bunch of pip installs which, even though they operate
+# on distinct temporary virtual environments, run into race conditions on shared
+# file-system state. To make this work reliably in parallel mode, we'll use a
+# FileLock courtesy of the tox-dev/py-filelock package.
+# Ref. https://github.com/python/mypy/issues/12615
+# Ref. mypy/test/testpep561.py
+pip_lock = os.path.join(package_path, '.pip_lock')
+pip_timeout = 60
diff --git a/mypy/test/testpep561.py b/mypy/test/testpep561.py
--- a/mypy/test/testpep561.py
+++ b/mypy/test/testpep561.py
@@ -1,4 +1,5 @@
 from contextlib import contextmanager
+import filelock
 import os
 import pytest
 import re
@@ -9,7 +10,7 @@
 from typing import Tuple, List, Generator
 
 import mypy.api
-from mypy.test.config import package_path
+from mypy.test.config import package_path, pip_lock, pip_timeout
 from mypy.util import try_find_python2_interpreter
 from mypy.test.data import DataDrivenTestCase, DataSuite
 from mypy.test.config import test_temp_dir
@@ -77,11 +78,15 @@ def install_package(pkg: str,
         env = {'PIP_BUILD': dir}
         # Inherit environment for Windows
         env.update(os.environ)
-        proc = subprocess.run(install_cmd,
-                              cwd=working_dir,
-                              stdout=PIPE,
-                              stderr=PIPE,
-                              env=env)
+        try:
+            with filelock.FileLock(pip_lock, timeout=pip_timeout):
+                proc = subprocess.run(install_cmd,
+                                      cwd=working_dir,
+                                      stdout=PIPE,
+                                      stderr=PIPE,
+                                      env=env)
+        except filelock.Timeout as err:
+            raise Exception("Failed to acquire {}".format(pip_lock)) from err
     if proc.returncode != 0:
         raise Exception(proc.stdout.decode('utf-8') + proc.stderr.decode('utf-8'))