Repository: matplotlib/matplotlib
Pull request: DOC: document the issues with overlaying new mpl on old mpl
closes #26827


Problem statement:
[Bug]: ImportError when using Matplotlib v3.8.0 in Python package tests
### Bug summary

I am encountering an ImportError when attempting to use Matplotlib v3.8.0 in my Python package's tests, specifically when running them with pytest. This issue does not occur with older versions of Matplotlib, such as v3.7.3.

### Code for reproduction

```python
>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/pyplot.py", line 66, in <module>
    from matplotlib.figure import Figure, FigureBase, figaspect
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/figure.py", line 43, in <module>
    from matplotlib import _blocking_input, backend_bases, _docstring, projections
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/projections/__init__.py", line 58, in <module>
    from mpl_toolkits.mplot3d import Axes3D
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in <module>
    from .axes3d import Axes3D
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py", line 23, in <module>
    from matplotlib import _api, cbook, docstring, _preprocess_data
ImportError: cannot import name 'docstring' from 'matplotlib' (/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py)
```


### Actual outcome

When running the tests, I receive the following ImportError:

```python
tests/integration/test_cli.py:263: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.local/lib/python3.10/site-packages/pyKVFinder/main.py:214: in cli
    plot_frequencies(frequencies, output_plot)
../../.local/lib/python3.10/site-packages/pyKVFinder/utils.py:952: in plot_frequencies
    import matplotlib.pyplot as plt
../../.local/lib/python3.10/site-packages/matplotlib/pyplot.py:66: in <module>
    from matplotlib.figure import Figure, FigureBase, figaspect
../../.local/lib/python3.10/site-packages/matplotlib/figure.py:43: in <module>
    from matplotlib import _blocking_input, backend_bases, _docstring, projections
../../.local/lib/python3.10/site-packages/matplotlib/projections/__init__.py:58: in <module>
    from mpl_toolkits.mplot3d import Axes3D
/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py:1: in <module>
    from .axes3d import Axes3D
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    axes3d.py, original mplot3d version by John Porter
    Created: 23 Sep 2005
    
    Parts fixed by Reinier Heeres <reinier@heeres.eu>
    Minor additions by Ben Axelrod <baxelrod@coroware.com>
    Significant updates and revisions by Ben Root <ben.v.root@gmail.com>
    
    Module containing Axes3D, an object which can plot 3D objects on a
    2D matplotlib figure.
    """
    
    from collections import defaultdict
    import functools
    import inspect
    import itertools
    import math
    from numbers import Integral
    import textwrap
    
    import numpy as np
    
>   from matplotlib import _api, cbook, docstring, _preprocess_data
E   ImportError: cannot import name 'docstring' from 'matplotlib' (/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py)

/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py:23: ImportError
```

### Expected outcome

I expected my tests to run successfully with Matplotlib v3.8.0, just as they did with v3.7.3.

### Additional information

- Has this worked in earlier versions?
It is working in previous versions (eg, v3.7.3).

- Do you know why this bug is happening?
When running the following command to get the Matplotlib Backend: 
```python
import matplotlib; print(matplotlib.get_backend())
```
I got the following error:
```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py", line 1275, in get_backend
    return rcParams['backend']
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py", line 759, in __getitem__
    from matplotlib import pyplot as plt
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/pyplot.py", line 66, in <module>
    from matplotlib.figure import Figure, FigureBase, figaspect
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/figure.py", line 43, in <module>
    from matplotlib import _blocking_input, backend_bases, _docstring, projections
  File "/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/projections/__init__.py", line 58, in <module>
    from mpl_toolkits.mplot3d import Axes3D
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in <module>
    from .axes3d import Axes3D
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py", line 23, in <module>
    from matplotlib import _api, cbook, docstring, _preprocess_data
ImportError: cannot import name 'docstring' from 'matplotlib' (/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py)
```

### Operating system

Pop!_OS 22.04 LTS

### Matplotlib Version

3.8.0

### Matplotlib Backend

ImportError: cannot import name 'docstring' from 'matplotlib' (/home/ABTLUS/joao.guerra/.local/lib/python3.10/site-packages/matplotlib/__init__.py)

### Python version

3.10.12

### Jupyter version

6.5.1 (Not using Jupyter notebook)

### Installation

pip

Hints from issue discussion:
breaks my CI too
Your installs are crossed. You are picking up mpl_toolkits from a different distribution.  

On the other hand this happens pretty commonly - I forget the explanation and maybe indicates an issue with how we install the wheels?
This looks like an installation issue?

It is certainly true that `docstring` was removed after a deprecation cycle of 2 releases and changed to be private (`_docstring`)

However, the line that is erroring in `mplot3d/axes3d.py` was updated [#22148](https://github.com/matplotlib/matplotlib/pull/22148), indicating that you have a version of mpl that it is picking up that is <3.6

https://github.com/matplotlib/matplotlib/commit/52df591338e948e05501b0abc9d64fd137544c2d#diff-a67656224bfb26dc6de0a0f89e398cd6999377d1482aca3e867ee550ff05be57


I will also point out that `matplotlib` is coming from `site-packages`, but `mpl_toolkits` is coming from `dist-packages`
@jklymak - "You are ..." sound like we did something actively. I assume we did not. The nightly CI build in jenkins fail. This seems to be just dependency hell as I have seen quite a few times in the last few months. 
Avoiding 3.8.0 fixes the jenkins problem and does not harm the github CI. The github CI also worked with the 3.8.0 version so there might be indeed be a sandboxing problem and some crossover between packages. I donot even know what mpl_toolkits is and why it would have anything to do with the way I am using matplotlib. My code is quite short and it surprises me that it should have such hidden dependencies 
```python
'''
Created on 2020-07-05

@author: wf
'''
import matplotlib.pyplot as plt
from collections import Counter
import numpy as np
import os

class Plot(object):
    '''
    create Plot based on counters
    see https://stackoverflow.com/questions/19198920/using-counter-in-python-to-build-histogram
    '''
    def __init__(self, valueList,title,xlabel=None,ylabel=None,gformat='.png',fontsize=12,plotdir=None,debug=False):
        '''
        Constructor
        '''
        self.counter=Counter(valueList)
        self.valueList=valueList
        self.title=title
        self.xlabel=xlabel
        self.ylabel=ylabel
        self.fontsize=fontsize
        self.gformat=gformat
        self.debug=debug
        path=os.path.dirname(__file__)
        if plotdir is not None:
            self.plotdir=plotdir
        else:
            self.plotdir=path+"/../plots/"
            os.makedirs(self.plotdir,exist_ok=True)
            
    def titleMe(self):   
        ''' set my title and labels '''     
        plt.title(self.title, fontsize=self.fontsize)
        if self.xlabel is not None:
            plt.xlabel(self.xlabel)
        if self.ylabel is not None:    
            plt.ylabel(self.ylabel)
            
    def showMe(self,mode='show',close=True):
        ''' show me in the given mode '''
        if mode=="show":
            plt.show() 
        else:
            plt.savefig(self.plotdir+self.title+self.gformat)
        if close:    
            plt.close()    
            
    def barchart(self,mode='show'):
        ''' barchart based histogram for the given counter '''
        labels, values = zip(*self.counter.items())
        indexes = np.arange(len(labels))
        width = 1
        self.titleMe()
        plt.bar(indexes, values, width)
        plt.xticks(indexes + width * 0.5, labels)
        plt.yticks(np.arange(1,max(values)+1,step=1))
        self.showMe(mode)
        
    def showDebug(self):    
        print("   value  list: ",self.valueList)
        print("counter  items: ",self.counter.items())
        print("counter values: ",self.counter.values())
        print("counter   keys: ",self.counter.keys())
        
    def hist(self,mode="show"):
        ''' create histogram for the given counter '''
        if self.debug:
            self.showDebug()
        self.titleMe()
        # see https://stackoverflow.com/a/2162045/1497139
        plt.hist(self.valueList,bins=len(self.counter.keys()))
        self.showMe(mode)
        pass
            
        
```
I am getting the same problem on Linux Mint 21.2 after I upgraded to 3.8.0.

The import sequence jump from
~/.local/lib/python3.10/site-packages/matplotlib
to
/usr/lib/python3/dist-packages/mpl_toolkits

The latter directory is part of the matplotlib 3.5.1 that is bundled with Mint. I uninstalled the Mint package and the import then started working correctly.

~/.local/lib/python3.10/site-packages comes before /usr/lib/python3/dist-packages in my sys.path, so is something in 3.8.0 changing it at run time?

I have another PC with Mint 21.2 which is still on matplotlib 3.7.2 and it doe not have this problem.

`mpl_toolkits` is distributed with `matplotlib`, in the `matplotlib` pypi (and conda, etc) package. It is a separate python package, where some more experimental features (most notably 3D plotting support), though it is imported (for 3D) by default.

As to why you have multiple and why python is choosing different locations for `matplotlib` and `mpl_toolkits`, I don't honestly know. Is it possible you edit `sys.path` in process (perhaps in test setup)?

Even with Matplotlib 3.7, I'd have expected this to at least be _warning_, as that was deprecated in 3.6.
mpl_toolkits is a namespace package that we ship with Matplotlib (that is where the 3d and axis artist code live).

For 3.8 we did change to using the "new-style" version of doing namespace packages (see https://github.com/matplotlib/matplotlib/issues/25244 https://github.com/matplotlib/matplotlib/pull/25381) because upstream started to actively warn about using the old way.

It looks like all of these issues are in cases where `pip install --local xxx` is being used to shadow an existing Matplotlib installation from the system package manager with a version from pip.   Although this works _most_ of the time, I think that this is one of the corner cases where it does not.

What is the version of setuptools on the cases where it is broken?  I strongly suspect that is going to be the determining factor of when this works or not.
I'm not changing sys.path in my own code. Just executing the following line in the interpreter will trigger it:
```
from mpl_toolkits import mplot3d
```

```
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from mpl_toolkits import mplot3d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in <module>
    from .axes3d import Axes3D
  File "/usr/lib/python3/dist-packages/mpl_toolkits/mplot3d/axes3d.py", line 23, in <module>
    from matplotlib import _api, cbook, docstring, _preprocess_data
ImportError: cannot import name 'docstring' from 'matplotlib' (/home/richardt/.local/lib/python3.10/site-packages/matplotlib/__init__.py)

```

Commit message: Update doc/api/prev_api_changes/api_changes_3.8.0/behaviour.rst

Co-authored-by: Jody Klymak <jklymak@gmail.com>

Patch to review:
diff --git a/doc/api/prev_api_changes/api_changes_3.8.0/behaviour.rst b/doc/api/prev_api_changes/api_changes_3.8.0/behaviour.rst
index 0b598723e26c..b6fbcca5735d 100644
--- a/doc/api/prev_api_changes/api_changes_3.8.0/behaviour.rst
+++ b/doc/api/prev_api_changes/api_changes_3.8.0/behaviour.rst
@@ -171,3 +171,22 @@ saved.
 
 Previously, *mincnt* was inclusive with no *C* provided but exclusive when *C* is provided.
 It is now inclusive of *mincnt* in both cases.
+
+
+``matplotlib.mpl_toolkits`` is now an implicit namespace package
+----------------------------------------------------------------
+
+Following the deprecation of ``pkg_resources.declare_namespace`` in ``setuptools`` 67.3.0,
+``matplotlib.mpl_toolkits`` is now implemented as an implicit namespace, following
+`PEP 420 <https://peps.python.org/pep-0420/>`_.
+
+As a consequence using ``pip`` to install a version of Matplotlib >= 3.8 on top
+of a version of Matplotlib < 3.8 (e.g. via ``pip install --local`` or
+``python -m venv --system-site-packages ...``) will fail because the old
+``matplotlib.mpl_toolkits`` files will be found where as the newer files will be
+found for all other modules.  This will result in errors due to the version
+miss-match.
+
+To avoid this issue you need to avoid having multiple versions of Matplotlib
+in different entries of ``sys.path``.   Either uninstall Matplotlib
+at the system level or use a more isolated virtual environment.


Code review comments:
diff_hunk: @@ -171,3 +171,22 @@ saved.
 
 Previously, *mincnt* was inclusive with no *C* provided but exclusive when *C* is provided.
 It is now inclusive of *mincnt* in both cases.
+
+
+``matplotlib.mpl_toolkits`` is now an implicit namespace package
+----------------------------------------------------------------
+
+Following the deprecation of ``pkg_resources.declare_namespace`` in ``setuptools`` 67.3.0,
+``matplotlib.mpl_toolkits`` is now implemented as an implicit namespace, following
+`PEP 420 <https://peps.python.org/pep-0420/>`_.
+
+As a consequence using ``pip`` to install a version of Matplotlib >= 3.8 on top
+of a version of Matplotlib < 3.8 (e.g. via ``pip install --local`` or
+``python -m venv --system-site-packages ...``) will fail because the old
+``matplotlib.mpl_toolkits`` files will be found where as the newer files will be
+found for all other modules.  This will result in errors due to the version
+miss-match.
path: doc/api/prev_api_changes/api_changes_3.8.0/behaviour.rst
text: @user1:
```suggestion
mismatch.
```

diff_hunk: @@ -171,3 +171,22 @@ saved.
 
 Previously, *mincnt* was inclusive with no *C* provided but exclusive when *C* is provided.
 It is now inclusive of *mincnt* in both cases.
+
+
+``matplotlib.mpl_toolkits`` is now an implicit namespace package
+----------------------------------------------------------------
+
+Following the deprecation of ``pkg_resources.declare_namespace`` in ``setuptools`` 67.3.0,
+``matplotlib.mpl_toolkits`` is now implemented as an implicit namespace, following
+`PEP 420 <https://peps.python.org/pep-0420/>`_.
+
+As a consequence using ``pip`` to install a version of Matplotlib >= 3.8 on top
+of a version of Matplotlib < 3.8 (e.g. via ``pip install --local`` or
+``python -m venv --system-site-packages ...``) will fail because the old
+``matplotlib.mpl_toolkits`` files will be found where as the newer files will be
path: doc/api/prev_api_changes/api_changes_3.8.0/behaviour.rst
text: @user1:
```suggestion
``matplotlib.mpl_toolkits`` files will be found whereas the newer files will be
```