Repository: matplotlib/matplotlib
Pull request: Add option of running stubtest using tox
<!--
Thank you so much for your PR!  To help us review your contribution, please check
out the development guide https://matplotlib.org/devdocs/devel/index.html
-->
This gives developers the option of running stubtests using `tox`. This is useful, because the stubtests can't be run on local editable installs (see https://github.com/matplotlib/matplotlib/issues/27580).

I've added a test commit that breaks stubtest deliberately to show this works - I will leave this here for review purposes, then force push it away once this has one approved review.

~~I can't get this to give me a true negative result locally though 😢 . With the below diff (saved, but unstaged in git), running `tox -e stubtest` still gives me a successful result. I'll investigate a bit further, but I don't suppose anyone has any ideas?~~

Fixes https://github.com/matplotlib/matplotlib/issues/27580.

## PR summary
<!-- Please provide at least 1-2 sentences describing the pull request in detail
(Why is this change required?  What problem does it solve?) and link to relevant
issues and PRs.
Also please summarize the changes in the title, for example "Raise ValueError on
non-numeric input to set_xlim" and avoid non-descriptive titles such as "Addresses
issue #8576".
-->


## PR checklist
<!-- Please mark any checkboxes that do not apply to this PR as [N/A].-->

- [ ] "closes #0000" is in the body of the PR description to [link the related issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)
- [ ] new and changed code is [tested](https://matplotlib.org/devdocs/devel/testing.html)
- [ ] *Plotting related* features are demonstrated in an [example](https://matplotlib.org/devdocs/devel/document.html#write-examples-and-tutorials)
- [ ] *New Features* and *API Changes* are noted with a [directive and release note](https://matplotlib.org/devdocs/devel/coding_guide.html#new-features-and-api-changes)
- [ ] Documentation complies with [general](https://matplotlib.org/devdocs/devel/document.html#write-rest-pages) and [docstring](https://matplotlib.org/devdocs/devel/document.html#write-docstrings) guidelines

<!--We understand that PRs can sometimes be overwhelming, especially as the
reviews start coming in.  Please let us know if the reviews are unclear or
the recommended next step seems overly demanding, if you would like help in
addressing a reviewer's comments, or if you have been waiting too long to hear
back on your PR.-->


Problem statement:
[Bug]: Running stubtest locally fails
### Bug summary

Running `python tools/stubtest.py` fails for me locally with

```
% python tools/stubtest.py              
error: matplotlib failed to find stubs
Stub:
MISSING
Runtime:
<module 'matplotlib' from '/Users/dstansby/software/matplotlib/lib/matplotlib/__init__.py'>

note: unused allowlist entry matplotlib.backends.*
note: unused allowlist entry matplotlib.tests.*
note: unused allowlist entry matplotlib.pylab.*
...
(many more note lines below here)
```

I can't work out what I need to do to fix this, and was hoping someone who knew more about the stubs could help! I followed the contributing guidelines to set up a development environment, which is working fine for tests etc.

### Code for reproduction

```python
n/a
```


### Actual outcome

Stub test fails.

### Expected outcome

Stub test runs.

### Additional information

_No response_

### Operating system

macOS

### Matplotlib Version

3.9.0.dev896+gc1b80c56f7

### Matplotlib Backend

MacOSX

### Python version

3.11.6

### Jupyter version

_No response_

### Installation

git checkout

Hints from issue discussion:
Unfortunately the way meson does editable installs doesn't work with stubtest...

It does work with a normal install.

It also does work if you edit the `.pth` file in site-packages to point to the `lib/matplotlib` folder
> Unfortunately the way meson does editable installs doesn't work with stubtest..

Can you update the docs to say that? I'd do it but not sure I'd get the details right.
I think updating these lines would be good:
https://github.com/matplotlib/matplotlib/blob/e5098e0dc2e5da26f659f44ce758d09c54f6b656/doc/devel/coding_guide.rst?plain=1#L91-L93

If it's not possible to run stubtest on an editable install maybe we should just remove mention of it from this section of the docs?
> If it's not possible to run stubtest on an editable install maybe we should just remove mention of it from this section of the docs?

:/ I added that section so fine w/ removing it if it doesn't make sense, but then my question is - when do you use these tools? Regular mypy should work and is a pre-commit hook.

Commit message: Add instructions on running stubtest

Patch to review:
diff --git a/.github/workflows/mypy-stubtest.yml b/.github/workflows/mypy-stubtest.yml
index f042688feb62..1280926021f1 100644
--- a/.github/workflows/mypy-stubtest.yml
+++ b/.github/workflows/mypy-stubtest.yml
@@ -18,12 +18,6 @@ jobs:
         with:
           python-version: 3.9
 
-      - name: Install mypy
-        run: |
-          pip3 install -r requirements/testing/mypy.txt \
-            -r requirements/testing/all.txt
-          pip3 install .
-
       - name: Set up reviewdog
         run: |
           mkdir -p "$HOME/bin"
@@ -32,13 +26,16 @@ jobs:
               sh -s -- -b "$HOME/bin"
           echo "$HOME/bin" >> $GITHUB_PATH
 
+      - name: Install tox
+        run: python -m pip install tox
+
       - name: Run mypy stubtest
         env:
           REVIEWDOG_GITHUB_API_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           set -o pipefail
-          MPLBACKEND=agg python tools/stubtest.py | \
-              sed -e "s!$pythonLocation/lib/python3.9/site-packages!lib!g" | \
+          tox -e stubtest | \
+              sed -e "s!.tox/stubtest/lib/python3.11/site-packages!lib!g" | \
               reviewdog \
                 -efm '%Eerror: %m' \
                 -efm '%CStub: in file %f:%l' \
diff --git a/.gitignore b/.gitignore
index 41c97b62c51e..b6f9e1ee74f4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -31,6 +31,7 @@
 ################
 # meson-python working directory
 build
+.mesonpy*
 
 # meson-python/build frontend dist directory
 dist
diff --git a/doc/devel/coding_guide.rst b/doc/devel/coding_guide.rst
index 6564de906979..b1c0a810c965 100644
--- a/doc/devel/coding_guide.rst
+++ b/doc/devel/coding_guide.rst
@@ -88,9 +88,8 @@ We generally use `stub files
 the type information for ``colors.py``. A notable exception is ``pyplot.py``,
 which is type hinted inline.
 
-Type hints are checked by the mypy :ref:`pre-commit hook <pre-commit-hooks>`
-and can often be verified using ``tools\stubtest.py`` and occasionally may
-require the use of ``tools\check_typehints.py``.
+Type hints are checked by the mypy :ref:`pre-commit hook <pre-commit-hooks>`,
+can often be verified by running ``tox -e stubtest``.
 
 New modules and files: installation
 ===================================
diff --git a/environment.yml b/environment.yml
index 9f06ab7ff143..d50e544edb0d 100644
--- a/environment.yml
+++ b/environment.yml
@@ -46,6 +46,7 @@ dependencies:
       - sphinxcontrib-svg2pdfconverter>=1.1.0
       - pikepdf
   # testing
+  - black<24
   - coverage
   - flake8>=3.8
   - flake8-docstrings>=1.4.0
@@ -64,4 +65,4 @@ dependencies:
   - pytest-xdist
   - tornado
   - pytz
-  - black<24
+  - tox
diff --git a/tox.ini b/tox.ini
index fc8479269160..cb2fcc979076 100644
--- a/tox.ini
+++ b/tox.ini
@@ -4,7 +4,7 @@
 # and then run "tox" from this directory.
 
 [tox]
-envlist = py38, py39, py310
+envlist = py38, py39, py310, stubtest
 
 [testenv]
 changedir = /tmp
@@ -25,3 +25,12 @@ commands =
 deps =
     pytest
     pytz
+
+[testenv:stubtest]
+changedir = {tox_root}
+commands =
+    python tools/stubtest.py
+usedevelop = False
+deps =
+    -r requirements/testing/mypy.txt
+    -r requirements/testing/all.txt


Code review comments:
diff_hunk: @@ -32,13 +26,16 @@ jobs:
               sh -s -- -b "$HOME/bin"
           echo "$HOME/bin" >> $GITHUB_PATH
 
+      - name: Install tox
+        run: python -m pip install tox
+
       - name: Run mypy stubtest
         env:
           REVIEWDOG_GITHUB_API_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           set -o pipefail
-          MPLBACKEND=agg python tools/stubtest.py | \
-              sed -e "s!$pythonLocation/lib/python3.9/site-packages!lib!g" | \
+          tox -e stubtest | \
+              sed -e "s!.tox/stubtest/lib/python3.11/site-packages!lib!g" | \
path: .github/workflows/mypy-stubtest.yml
text: @user1:
```suggestion
              sed -e "s!.tox/stubtest/lib/python3.9/site-packages!lib!g" | \
```

This is referring to the python version (and removing the verbose string) via sed, but if the python version doesn't match, it won't match anything (and thus the sed line won't _error_, but won't work as intended.