Repository: PrefectHQ/prefect

Issue:
Change how we use mypy in CI
Currently we use `mypy` as part of our CI process. This frequently leads to extra work trying to appease the type checker for code that is definitely correct (see e.g. https://github.com/PrefectHQ/prefect/pull/2406).

We *do* want mypy to check that the type signatures of public methods actually match their implementation, but we *don't* want mypy to error for internal type checking that it thinks is invalid (internal errors can be caught by tests, and frequently the type checker is more of a burden than an aid). Perhaps there's a mypy configuration setting we can change to get this behavior?

We also currently run the mypy tests in two locations:

- As part of the `Check code style and docs` build
- As part of a test in `tests/test_formatting.py`, in the `Run tests` build

These two CI builds have different environments (in particular, prefect (and dependencies) aren't installed in the former but are in the latter). This difference can lead to mypy tests passing in one environment and failing in the other.


Discussion:
I'm starting to look at this. I'd like to not duplicate our mypy (and black) checks in two spots. This means we either:

- Remove the `Check code style and docs` build
- Remove the `py.test` run tests in `tests/test_formatting`

Most projects I've worked on run the code quality checks directly using the tools, rather than scripted as part of pytest, so removing the copies in the `tests` directory would be my vote. Getting in the practice of using these tools directly also makes it easier to check individual files when developing, rather than running the test suite which always checks the whole repo.

Thoughts @joshmeek, @cicdw?
Yup, I like the clarity of having a named CI job for style / static analysis / formatting, and the red X on test failures just because of `black` or `mypy` I find annoying / misleading. 

I'm all for removing them from `py.test` and keeping the "true" CI job.
Yeah I am on board with removing them from `py.test` as well. It's very clear to have them in a separate named job!

Resolving patch:
diff --git a/conftest.py b/conftest.py
deleted file mode 100644
--- a/conftest.py
+++ /dev/null
@@ -1,32 +0,0 @@
-import sys
-
-import pytest
-
-
-def pytest_addoption(parser):
-    parser.addoption(
-        "--skip-formatting",
-        action="store_true",
-        dest="formatting",
-        help="including this flag will skip all formatting tests",
-    )
-
-
-def pytest_configure(config):
-    config.addinivalue_line(
-        "markers",
-        "formatting: mark test as formatting to skip when --skip-formatting flag is provided.",
-    )
-
-
-def pytest_runtest_setup(item):
-    formatting_mark = item.get_closest_marker("formatting")
-
-    # if a test IS marked as "formatting" and the --skip-formatting flag IS set, skip it
-    if (
-        formatting_mark is not None
-        and item.config.getoption("--skip-formatting") is True
-    ):
-        pytest.skip(
-            "Formatting tests skipped because --skip-formatting flag was provided."
-        )


Test patch:
diff --git a/docs/core/development/tests.md b/docs/core/development/tests.md
--- a/docs/core/development/tests.md
+++ b/docs/core/development/tests.md
@@ -21,13 +21,9 @@ cd prefect
 pytest
 ```
 
-Prefect has a few flags for modifying pytest's behavior. Passing `--skip-formatting` will skip the `black` and `mypy` tests. Passing `--airflow` will run a series of Prefect/Airflow integration tests that can take a long time (and are not run unless this flag is invoked).
-
 ## CI
 
-In CI, Prefect's unit tests are run against Python 3.6, 3.7, and 3.8. As much as we love `f-strings`, Prefect code must support all versions. A common pitfall is writing tests that implicitly depend on Python 3.6's implicitly-ordered dictionaries and sets.
-
-Though Prefect's unit tests include tests for black and mypy (see the [style guide](style.md)), a separate "formatting" CI job is also run. Since formatting errors are common in PRs, we have found this to be a useful early-warning during development.
+In CI, Prefect's unit tests are run against Python 3.6, 3.7, and 3.8. A separate "formatting" CI job is also run. Since formatting errors are common in PRs, we have found this to be a useful early-warning during development.
 
 ## Documentation
 
diff --git a/docs/test_generate_docs.py b/docs/test_generate_docs.py
--- a/docs/test_generate_docs.py
+++ b/docs/test_generate_docs.py
@@ -35,7 +35,6 @@
 
 
 pytest.mark.skipif(sys.version_info < (3, 6))
-pytestmark = pytest.mark.formatting
 
 
 def consistency_check(obj, obj_name):
diff --git a/tests/test_formatting.py b/tests/test_formatting.py
deleted file mode 100644
--- a/tests/test_formatting.py
+++ /dev/null
@@ -1,41 +0,0 @@
-import os
-import shlex
-import subprocess
-import sys
-
-import pytest
-
-pytestmark = pytest.mark.formatting
-
-
-FILE_ENDING = os.path.join("tests", "test_formatting.py")
-POSIX = sys.platform != "win32"
-
-
-@pytest.mark.skipif(sys.version_info < (3, 6), reason="Black requires Python 3.6+")
-def test_black_formatting():
-    # make sure we know what working directory we're in
-    assert __file__.endswith(FILE_ENDING)
-
-    prefect_dir = os.path.dirname(os.path.dirname(__file__))
-    result = subprocess.call(
-        shlex.split("black --check {}".format(prefect_dir), posix=POSIX)
-    )
-    assert result == 0, "Prefect repo did not pass Black formatting!"
-
-
-@pytest.mark.skipif(sys.version_info < (3, 6), reason="mypy requires Python 3.6+")
-def test_mypy():
-    # make sure we know what working directory we're in
-    assert __file__.endswith(FILE_ENDING)
-
-    prefect_dir = os.path.dirname(os.path.dirname(__file__))
-    prefect_src_dir = os.path.join(prefect_dir, "src")
-    config_file = os.path.join(prefect_dir, "setup.cfg")
-    result = subprocess.call(
-        shlex.split(
-            "mypy --config-file {0} {1}".format(config_file, prefect_src_dir),
-            posix=POSIX,
-        )
-    )
-    assert result == 0, "Prefect repo did not pass mypy type checks!"