Repository: python/mypy
Pull request: Tests autodiscovery replacing hardcoded file list
### Description

This PR hopefully resolves https://github.com/python/mypy/issues/8650.

The main change is the `find_test_files ` helper function that looks for files matching the pattern in the unit tests directory. It has been used in the following test files for autodiscovery:
-  `testcheck.py`
-  `testdeps.py`
- ` testfinegrained.py`
- `testparse.py`
- `testsemanal.py`

I've also updated the readme. 

## Test Plan

This is a question for maintainers - does this helper function needs unit tests and if so, where should they live? I haven't found `testhelpers.py` 🙂 


Problem statement:
Auto-discovery of .test files
Currently all the `.test` files (used for mypy self tests) are enumerated in the test case runners, such as in `mypy/test/testcheck.py`. This is error-prone and can be confusing.

It would be better to do auto-discovery of test files using file name prefixes. For example, all the files with names like `check-*.test` would be executed by `mypy.test.testcheck`.

(Additionally, if there is an unrecognized `.test` file, it would be nice to generate an error and give hints about how to fix the issue. Addressing this is optional for now, however, and may be a bit tricky.)

Hints from issue discussion:
I am interested in looking into this, as I was a bit surprised that the `testpep561.py` file parsed the `pep561.test` file by direct reference (as do other tests I presume?). As I've read they do this by specifying the `files` property in a DataSuite-inheriting class. I've worked with code running in this extract (testpep561.py):
```
class PEP561Suite(DataSuite):
    files = ['pep561.test']

    def run_case(self, test_case: DataDrivenTestCase) -> None:
        test_pep561(test_case)
```

One would expect one (or a couple extended) classes(s) to parse all the tests and generate a recognizable pytest output.

It's going to require a bit more understanding on my side, although I gained some experience through #8524 and @ethanhs his work in #8571. So just for clarity's sake:
- We are interested in intercepting pytest autodiscovery for data-driven tests (any class extending Suite/DataSuite/etc.) . Any pointers to which class to start studying if so? (File: `data.py` or `helpers.py` possibly).
- We are interested in combining unit tests starting with **foo** by regex'ing `foo-*.test` files.
- We need some scanning or mapping state of which tests are picked up only once by a module (f.e. `mypy.test.testcheck` as your example goes). I'd expect this to be more strict: we must enforce a `.test` file to map to one `test*.py` file only. Only then can we make sure the data-driven tests are run by at most one module. A second requirement of course is that the unit test maps to the right test module. A third is that if the test-data can regex to multiple modules, an error must be thrown. Does this sound complete/correct?
- We'd like some way of generating a mapping of the _data unit tests_ in a custom discovery class, with 'missed_cases' and 'multiple_mappings' as output, but it is expected to by tricky because it can generate wrong information/advice or has some edge cases.
@JukkaL it'd be awesome to hear your thoughts so I can get to work.

Just a 'go-ahead' is fine for now.
I will work on this, although any comment would be great. @ethanhs do you have any thoughts? It isn't a small endeavor this time, but pretty similar to data-driven tests like we worked on earlier
I'm afraid I'm not sure the particulars of what @JukkaL has in mind, so it'd be best to ask him.
@JukkaL I'd gladly hear about whether this issue can be acted upon
I'd like to work on this issue.
From what I understand, we need the list of `.test` files to be auto generated. I am thinking of using something like `pathlib.Path().rglob("prefix*.test")` to do so. What do you think?
Up for grabs, @ethanhs @JukkaL could you help this fella out?
Hello Sir ,

I was trying to fix auto-discovery of .test files #8650

Here is the script

```
# List of files that contain test case descriptions.

typecheck_files = [f for f in os.listdir("../../test-data/unit") if re.match(r'check-\w+-?\w+?.test',f)]

# Tests that use Python 3.8-only AST features (like expression-scoped ignores):
if sys.version_info < (3, 8):
    typecheck_files.remove('check-python38.test')
if sys.version_info < (3, 9):
    typecheck_files.remove('check-python39.test')

# Special tests for platforms with case-insensitive filesystems.
if sys.platform not in ('darwin', 'win32'):
    typecheck_files.remove('check-modules-case.test')  
```

But there was path not found error. Can you exactly tell me from where would it be called ?
As seen in [ReadMe](https://github.com/python/mypy/blob/master/test-data/unit/README.md) , you have called many functions as 

```
$ pytest -n0 mypy/test/testcheck.py
```

So is that the default location from where we can call it ? If yes ,then maybe this would work :

```
# List of files that contain test case descriptions.

typecheck_files = [f for f in os.listdir("test-data/unit") if re.match(r'check-\w+-?\w+?.test', f)]


# Tests that use Python 3.8-only AST features (like expression-scoped ignores):
if sys.version_info < (3, 8):
    typecheck_files.remove('check-python38.test')
if sys.version_info < (3, 9):
    typecheck_files.remove('check-python39.test')

# Special tests for platforms with case-insensitive filesystems.
if sys.platform not in ('darwin', 'win32'):
    typecheck_files.remove('check-modules-case.test')
    
```
Please Help !!
Hello, I'd like to solve this issue, although many have already solved this prior to me.
@JukkaL @davidzwa Can anyone reach out, I have a few pointers. Also, it'd be great to start my contribution journey from this issue of mypy.

Thanks!

Commit message: cleanup

Patch to review:
diff --git a/mypy/test/helpers.py b/mypy/test/helpers.py
index af1fefe67ffd..9b2ed6e89183 100644
--- a/mypy/test/helpers.py
+++ b/mypy/test/helpers.py
@@ -1,11 +1,12 @@
 import os
+import pathlib
 import re
 import sys
 import time
 import shutil
 import contextlib
 
-from typing import List, Iterable, Dict, Tuple, Callable, Any, Iterator, Union, Pattern
+from typing import List, Iterable, Dict, Tuple, Callable, Any, Iterator, Union, Pattern, Optional
 
 from mypy import defaults
 import mypy.api as api
@@ -494,3 +495,11 @@ def normalize_file_output(content: List[str], current_abs_path: str) -> List[str
     result = [re.sub(r'\b' + re.escape(base_version) + r'\b', '$VERSION', x) for x in result]
     result = [timestamp_regex.sub('$TIMESTAMP', x) for x in result]
     return result
+
+
+def find_test_files(pattern: str, exclude: Optional[List[str]] = None) -> List[str]:
+    return [
+        str(x.name)
+        for x in (pathlib.Path("./test-data/unit").rglob(pattern))
+        if str(x.name) not in (exclude or [])
+    ]
diff --git a/mypy/test/testcheck.py b/mypy/test/testcheck.py
index ddcb78df8100..11830345507d 100644
--- a/mypy/test/testcheck.py
+++ b/mypy/test/testcheck.py
@@ -17,6 +17,7 @@
     assert_string_arrays_equal, normalize_error_messages, assert_module_equivalence,
     update_testcase_output, parse_options,
     assert_target_equivalence, check_test_output_files, perform_file_operations,
+    find_test_files,
 )
 from mypy.errors import CompileError
 from mypy.semanal_main import core_modules
@@ -29,96 +30,20 @@
 import pytest
 
 # List of files that contain test case descriptions.
-typecheck_files = [
-    'check-basic.test',
-    'check-union-or-syntax.test',
-    'check-callable.test',
-    'check-classes.test',
-    'check-statements.test',
-    'check-generics.test',
-    'check-dynamic-typing.test',
-    'check-inference.test',
-    'check-inference-context.test',
-    'check-kwargs.test',
-    'check-overloading.test',
-    'check-type-checks.test',
-    'check-abstract.test',
-    'check-multiple-inheritance.test',
-    'check-super.test',
-    'check-modules.test',
-    'check-modules-fast.test',
-    'check-typevar-values.test',
-    'check-unsupported.test',
-    'check-unreachable-code.test',
-    'check-unions.test',
-    'check-isinstance.test',
-    'check-lists.test',
-    'check-namedtuple.test',
-    'check-narrowing.test',
-    'check-typeddict.test',
-    'check-type-aliases.test',
-    'check-ignore.test',
-    'check-type-promotion.test',
-    'check-semanal-error.test',
-    'check-flags.test',
-    'check-incremental.test',
-    'check-serialize.test',
-    'check-bound.test',
-    'check-optional.test',
-    'check-fastparse.test',
-    'check-warnings.test',
-    'check-async-await.test',
-    'check-newtype.test',
-    'check-class-namedtuple.test',
-    'check-selftype.test',
-    'check-python2.test',
-    'check-columns.test',
-    'check-functions.test',
-    'check-tuples.test',
-    'check-expressions.test',
-    'check-generic-subtyping.test',
-    'check-varargs.test',
-    'check-newsyntax.test',
-    'check-protocols.test',
-    'check-underscores.test',
-    'check-classvar.test',
-    'check-enum.test',
-    'check-incomplete-fixture.test',
-    'check-custom-plugin.test',
-    'check-default-plugin.test',
-    'check-attr.test',
-    'check-ctypes.test',
-    'check-dataclasses.test',
-    'check-final.test',
-    'check-redefine.test',
-    'check-literal.test',
-    'check-newsemanal.test',
-    'check-inline-config.test',
-    'check-reports.test',
-    'check-errorcodes.test',
-    'check-annotated.test',
-    'check-parameter-specification.test',
-    'check-typevar-tuple.test',
-    'check-generic-alias.test',
-    'check-typeguard.test',
-    'check-functools.test',
-    'check-singledispatch.test',
-    'check-slots.test',
-    'check-formatting.test',
-    'check-native-int.test',
-]
+# Includes all check-* files with the .test extension in the test-data/unit directory
+typecheck_files = find_test_files(pattern="check-*.test")
 
 # Tests that use Python 3.8-only AST features (like expression-scoped ignores):
-if sys.version_info >= (3, 8):
-    typecheck_files.append('check-python38.test')
-if sys.version_info >= (3, 9):
-    typecheck_files.append('check-python39.test')
-if sys.version_info >= (3, 10):
-    typecheck_files.append('check-python310.test')
+if sys.version_info < (3, 8):
+    typecheck_files.remove('check-python38.test')
+if sys.version_info < (3, 9):
+    typecheck_files.remove('check-python39.test')
+if sys.version_info < (3, 10):
+    typecheck_files.remove('check-python310.test')
 
 # Special tests for platforms with case-insensitive filesystems.
-if sys.platform in ('darwin', 'win32'):
-    typecheck_files.extend(['check-modules-case.test'])
+if sys.platform not in ('darwin', 'win32'):
+    typecheck_files.remove('check-modules-case.test')
 
 
 class TypeCheckSuite(DataSuite):
diff --git a/mypy/test/testdeps.py b/mypy/test/testdeps.py
index 7446d44339a0..03869b67316a 100644
--- a/mypy/test/testdeps.py
+++ b/mypy/test/testdeps.py
@@ -14,7 +14,7 @@
 from mypy.server.deps import get_dependencies
 from mypy.test.config import test_temp_dir
 from mypy.test.data import DataDrivenTestCase, DataSuite
-from mypy.test.helpers import assert_string_arrays_equal, parse_options
+from mypy.test.helpers import assert_string_arrays_equal, parse_options, find_test_files
 from mypy.types import Type
 from mypy.typestate import TypeState
 
@@ -23,14 +23,7 @@
 
 
 class GetDependenciesSuite(DataSuite):
-    files = [
-        'deps.test',
-        'deps-types.test',
-        'deps-generics.test',
-        'deps-expressions.test',
-        'deps-statements.test',
-        'deps-classes.test',
-    ]
+    files = find_test_files(pattern="deps*.test")
 
     def run_case(self, testcase: DataDrivenTestCase) -> None:
         src = '\n'.join(testcase.input)
diff --git a/mypy/test/testfinegrained.py b/mypy/test/testfinegrained.py
index 783e21e2869e..97b97f2c078e 100644
--- a/mypy/test/testfinegrained.py
+++ b/mypy/test/testfinegrained.py
@@ -27,7 +27,7 @@
 )
 from mypy.test.helpers import (
     assert_string_arrays_equal, parse_options, assert_module_equivalence,
-    assert_target_equivalence, perform_file_operations,
+    assert_target_equivalence, perform_file_operations, find_test_files,
 )
 from mypy.server.mergecheck import check_consistency
 from mypy.dmypy_util import DEFAULT_STATUS_FILE
@@ -42,15 +42,9 @@
 
 
 class FineGrainedSuite(DataSuite):
-    files = [
-        'fine-grained.test',
-        'fine-grained-cycles.test',
-        'fine-grained-blockers.test',
-        'fine-grained-modules.test',
-        'fine-grained-follow-imports.test',
-        'fine-grained-suggest.test',
-        'fine-grained-attr.test',
-    ]
+    files = find_test_files(
+        pattern="fine-grained*.test", exclude=["fine-grained-cache-incremental.test"]
+    )
 
     # Whether to use the fine-grained cache in the testing. This is overridden
     # by a trivial subclass to produce a suite that uses the cache.
diff --git a/mypy/test/testparse.py b/mypy/test/testparse.py
index 340d6b904476..ec9930bd26b7 100644
--- a/mypy/test/testparse.py
+++ b/mypy/test/testparse.py
@@ -5,7 +5,7 @@
 from pytest import skip
 
 from mypy import defaults
-from mypy.test.helpers import assert_string_arrays_equal, parse_options
+from mypy.test.helpers import assert_string_arrays_equal, parse_options, find_test_files
 from mypy.test.data import DataDrivenTestCase, DataSuite
 from mypy.parse import parse
 from mypy.errors import CompileError
@@ -15,11 +15,10 @@
 class ParserSuite(DataSuite):
     required_out_section = True
     base_path = '.'
-    files = ['parse.test',
-             'parse-python2.test']
+    files = find_test_files(pattern="parse*.test", exclude=["parse-errors.test"])
 
-    if sys.version_info >= (3, 10):
-        files.append('parse-python310.test')
+    if sys.version_info < (3, 10):
+        files.remove('parse-python310.test')
 
     def run_case(self, testcase: DataDrivenTestCase) -> None:
         test_parser(testcase)
diff --git a/mypy/test/testsemanal.py b/mypy/test/testsemanal.py
index c7a623507ac1..d86c6ce2fe4a 100644
--- a/mypy/test/testsemanal.py
+++ b/mypy/test/testsemanal.py
@@ -9,7 +9,8 @@
 from mypy.modulefinder import BuildSource
 from mypy.defaults import PYTHON3_VERSION
 from mypy.test.helpers import (
-    assert_string_arrays_equal, normalize_error_messages, testfile_pyversion, parse_options
+    assert_string_arrays_equal, normalize_error_messages, testfile_pyversion, parse_options,
+    find_test_files,
 )
 from mypy.test.data import DataDrivenTestCase, DataSuite
 from mypy.test.config import test_temp_dir
@@ -21,26 +22,19 @@
 # Semantic analyzer test cases: dump parse tree
 
 # Semantic analysis test case description files.
-semanal_files = [
-    'semanal-basic.test',
-    'semanal-expressions.test',
-    'semanal-classes.test',
-    'semanal-types.test',
-    'semanal-typealiases.test',
-    'semanal-modules.test',
-    'semanal-statements.test',
-    'semanal-abstractclasses.test',
-    'semanal-namedtuple.test',
-    'semanal-typeddict.test',
-    'semenal-literal.test',
-    'semanal-classvar.test',
-    'semanal-python2.test',
-    'semanal-lambda.test',
-]
-
-
-if sys.version_info >= (3, 10):
-    semanal_files.append('semanal-python310.test')
+semanal_files = find_test_files(
+    pattern="semanal-*.test",
+    exclude=[
+        "semanal-errors-python310.test",
+        "semanal-errors.test",
+        "semanal-typeinfo.test",
+        "semanal-symtable.test",
+    ],
+)
+
+
+if sys.version_info < (3, 10):
+    semanal_files.remove('semanal-python310.test')
 
 
 def get_semanal_options(program_text: str, testcase: DataDrivenTestCase) -> Options:
diff --git a/test-data/unit/README.md b/test-data/unit/README.md
index 080e8770d9f7..a414fd646ec6 100644
--- a/test-data/unit/README.md
+++ b/test-data/unit/README.md
@@ -7,8 +7,7 @@ Quick Start
 
 To add a simple unit test for a new feature you developed, open or create a
 `test-data/unit/check-*.test` file with a name that roughly relates to the
-feature you added. If you added a new `check-*.test` file, add it to the list
-of files in `mypy/test/testcheck.py`.
+feature you added. If you added a new `check-*.test` file, it will be autodiscovered during unittests run.
 
 Add the test in this format anywhere in the file:
 


Code review comments:
diff_hunk: @@ -494,3 +495,11 @@ def normalize_file_output(content: List[str], current_abs_path: str) -> List[str
     result = [re.sub(r'\b' + re.escape(base_version) + r'\b', '$VERSION', x) for x in result]
     result = [timestamp_regex.sub('$TIMESTAMP', x) for x in result]
     return result
+
+
+def find_test_files(pattern: str, exclude: Optional[List[str]] = None) -> List[str]:
+    return [
+        str(x.name)
path: mypy/test/helpers.py
text: @user1:
`x.name` is already a string, you can remove the conversion.

@user2:
Also maybe use a bit more descriptive name than `x` (`path` maybe?)

@user2:
@author Could you please implement these two suggestions? ^^ I will then merge this.

@author:
@user2 Done in [7974ae6](https://github.com/python/mypy/pull/13143/commits/7974ae67cf3ca261d585c9aba5032b620274dadc) 🙂