Repository: modin-project/modin

Issue:
BUG: using docstring extensions causes several incorrect behaviors because DataFrame, Series, and BasePandasDataset cannot recognize each other
### Modin version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest released version of Modin.

- [X] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow [this guide](https://modin.readthedocs.io/en/stable/getting_started/installation.html#installing-from-the-github-master-branch).)


### Reproducible Example

```python
import pandas
import numpy as np

import modin.pandas as pd
from modin.config import DocModule

assert pd.DataFrame([True]).all(axis=None) is np.bool_(True)

DocModule.put("modin.config.test.docs_module")

# Test for override
assert (
    pd.DataFrame.apply.__doc__
    == "This is a test of the documentation module for DataFrame."
)
# Test for pandas doc when method is not defined on the plugin module
assert pandas.DataFrame.isna.__doc__ in pd.DataFrame.isna.__doc__
assert pandas.DataFrame.isnull.__doc__ in pd.DataFrame.isnull.__doc__
# Test for override
assert (
    pd.Series.isna.__doc__
    == "This is a test of the documentation module for Series."
)
# Test for pandas doc when method is not defined on the plugin module
assert pandas.Series.isnull.__doc__ in pd.Series.isnull.__doc__
assert pandas.Series.apply.__doc__ in pd.Series.apply.__doc__
# Test for override
assert pd.read_csv.__doc__ == "Test override for functions on the module."
# Test for pandas doc when function is not defined on module.
assert pandas.read_table.__doc__ in pd.read_table.__doc__

assert pd.DataFrame([True]).all(axis=None) is np.bool_(True)
```


### Issue Description

The reproducible example is a modified version of `modin/config/test/test_envvars.py::test_doc_module` at modin commit a616e4cc4944b4ad3c3daf446adf06714e22ad34.

The cause is that we redefine `BasePandasDataset`, `Series`, and `DataFrame` twice each:
1. define `BasePandasDataset`, then `Series`, then `DataFrame` when we first load `modin.pandas`
1. redefine `BasePandasDataset` when we do `importlib.reload(pd.base)` [here](https://github.com/modin-project/modin/blob/553eb5cf5f3bb255999720ab9539ffc4964677f1/modin/config/envvars.py#L880) because we're adding the doc module.
1. redefine `DataFrame` when we do `importlib.reload(pd.dataframe)` [here](https://github.com/modin-project/modin/blob/553eb5cf5f3bb255999720ab9539ffc4964677f1/modin/config/envvars.py#L881) because we're adding the doc module.
1. redefine `Series` when we do `importlib.reload(pd.series)` [here](https://github.com/modin-project/modin/blob/553eb5cf5f3bb255999720ab9539ffc4964677f1/modin/config/envvars.py#L881) because we're adding the doc module.

When we execute `all`, `BasePandasDataset` [here](https://github.com/modin-project/modin/blob/553eb5cf5f3bb255999720ab9539ffc4964677f1/modin/pandas/base.py#L857C35-L857C52), `BasePandasDataset` is the second `BasePandasDataset` class, but `result` is an instance of the first `BasePandasDataset` class because [dataframe imported `Series`](https://github.com/modin-project/modin/blob/553eb5cf5f3bb255999720ab9539ffc4964677f1/modin/pandas/dataframe.py#L66) before the `series` module was reloaded. So `result` should be an instance of `BasePandasDataset`, but it's not, so we skip the extra reduction and end up with a `Series` result instead of a boolean.

There's no good order to do the reloads in because all three classes need to reference each other. We could do `import modin.pandas.base as modin_base` and always do `base.BasePandasDataset` instead of `from modin.pandas.base import BsaePandasDataset`, but that's hadr to read.

Proposed solution: instead of doing `importlib.reload`, we should update docstrings in place when we do `DocModule.put()`.

### Expected Behavior

`all(axis=None)` should always return a boolean. In general, `BasePandasDataset`, `Series`, and `DataFrame` should all be able to reference each other.

### Error Logs

<details>

```python-traceback

___________________________________________________________________ test_doc_module ____________________________________________________________________

    def test_doc_module():
        import pandas
        import numpy as np

        import modin.pandas as pd
        from modin.config import DocModule

        assert pd.DataFrame([True]).all(axis=None) is np.bool_(True)

        DocModule.put("modin.config.test.docs_module")

        # Test for override
        assert (
            pd.DataFrame.apply.__doc__
            == "This is a test of the documentation module for DataFrame."
        )
        # Test for pandas doc when method is not defined on the plugin module
        assert pandas.DataFrame.isna.__doc__ in pd.DataFrame.isna.__doc__
        assert pandas.DataFrame.isnull.__doc__ in pd.DataFrame.isnull.__doc__
        # Test for override
        assert (
            pd.Series.isna.__doc__
            == "This is a test of the documentation module for Series."
        )
        # Test for pandas doc when method is not defined on the plugin module
        assert pandas.Series.isnull.__doc__ in pd.Series.isnull.__doc__
        assert pandas.Series.apply.__doc__ in pd.Series.apply.__doc__
        # Test for override
        assert pd.read_csv.__doc__ == "Test override for functions on the module."
        # Test for pandas doc when function is not defined on module.
        assert pandas.read_table.__doc__ in pd.read_table.__doc__

>       assert pd.DataFrame([True]).all(axis=None) is np.bool_(True)
E       AssertionError: assert 0    True\ndtype: bool is True
E        +  where 0    True\ndtype: bool = <bound method BasePandasDataset.all of       0\n0  True>(axis=None)
E        +    where <bound method BasePandasDataset.all of       0\n0  True> =       0\n0  True.all
E        +      where       0\n0  True = <class 'modin.pandas.dataframe.DataFrame'>([True])
E        +        where <class 'modin.pandas.dataframe.DataFrame'> = <module 'modin.pandas' from '/Users/mvashishtha/sources/modin/modin/pandas/__init__.py'>.DataFrame
E        +  and   True = <class 'numpy.bool_'>(True)
E        +    where <class 'numpy.bool_'> = <module 'numpy' from '/Users/mvashishtha/miniconda3/envs/modin-latest/lib/python3.9/site-packages/numpy/__init__.py'>.bool_

modin/config/test/test_envvars.py:116: AssertionErro

```

</details>


### Installed Versions

<details>

INSTALLED VERSIONS
------------------
commit                : a616e4cc4944b4ad3c3daf446adf06714e22ad34
python                : 3.9.18.final.0
python-bits           : 64
OS                    : Darwin
OS-release            : 23.4.0
Version               : Darwin Kernel Version 23.4.0: Wed Feb 21 21:45:49 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6020
machine               : arm64
processor             : arm
byteorder             : little
LC_ALL                : None
LANG                  : en_US.UTF-8
LOCALE                : en_US.UTF-8

Modin dependencies
------------------
modin                 : 0.28.0+35.ga616e4cc.dirty
ray                   : 2.8.0
dask                  : 2024.3.1
distributed           : 2024.3.1
hdk                   : None

pandas dependencies
-------------------
pandas                : 2.2.1
numpy                 : 1.26.1
pytz                  : 2023.3.post1
dateutil              : 2.8.2
setuptools            : 68.0.0
pip                   : 23.3
Cython                : None
pytest                : 8.1.1
hypothesis            : None
sphinx                : 7.2.6
blosc                 : None
feather               : None
xlsxwriter            : None
lxml.etree            : 5.1.0
html5lib              : None
pymysql               : None
psycopg2              : 2.9.9
jinja2                : 3.1.2
IPython               : 8.17.2
pandas_datareader     : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.12.3
bottleneck            : None
dataframe-api-compat  : None
fastparquet           : 2024.2.0
fsspec                : 2024.3.1
gcsfs                 : None
matplotlib            : 3.8.1
numba                 : None
numexpr               : 2.8.4
odfpy                 : None
openpyxl              : 3.1.2
pandas_gbq            : 0.22.0
pyarrow               : 14.0.1
pyreadstat            : None
python-calamine       : None
pyxlsb                : None
s3fs                  : 2024.3.1
scipy                 : 1.11.3
sqlalchemy            : 2.0.29
tables                : 3.9.2
tabulate              : None
xarray                : 2024.2.0
xlrd                  : 2.0.1
zstandard             : None
tzdata                : 2023.3
qtpy                  : None
pyqt5                 : None

</details>


Discussion:

Patch:
diff --git a/modin/config/envvars.py b/modin/config/envvars.py
index 3f83ec1f..d05a7117 100644
--- a/modin/config/envvars.py
+++ b/modin/config/envvars.py
@@ -13,7 +13,6 @@
 
 """Module houses Modin configs originated from environment variables."""
 
-import importlib
 import os
 import secrets
 import sys
@@ -771,36 +770,6 @@ class DocModule(EnvironmentVariable, type=ExactStr):
     varname = "MODIN_DOC_MODULE"
     default = "pandas"
 
-    @classmethod
-    def put(cls, value: str) -> None:
-        """
-        Assign a value to the DocModule config.
-
-        Parameters
-        ----------
-        value : str
-            Config value to set.
-        """
-        super().put(value)
-        # Reload everything to apply the documentation. This is required since the
-        # docs might already have been created and the implementation will assume
-        # that the new docs are applied when the config is set. This set of operations
-        # does this.
-        import modin.pandas as pd
-
-        importlib.reload(pd.accessor)
-        importlib.reload(pd.base)
-        importlib.reload(pd.dataframe)
-        importlib.reload(pd.general)
-        importlib.reload(pd.groupby)
-        importlib.reload(pd.io)
-        importlib.reload(pd.iterator)
-        importlib.reload(pd.series)
-        importlib.reload(pd.series_utils)
-        importlib.reload(pd.utils)
-        importlib.reload(pd.window)
-        importlib.reload(pd)
-
 
 class DaskThreadsPerWorker(EnvironmentVariable, type=int):
     """Number of threads per Dask worker."""
diff --git a/modin/utils.py b/modin/utils.py
index fdc5260b..2a7df885 100644
--- a/modin/utils.py
+++ b/modin/utils.py
@@ -378,6 +378,129 @@ def _replace_doc(
         target_obj.__doc__ = doc
 
 
+# This is a map from objects whose docstrings we are overriding to functions that
+# take a DocModule string and override the docstring according to the
+# DocModule. When we update DocModule, we can use this map to update all
+# inherited docstrings.
+_docstring_inheritance_calls: list[Callable[[str], None]] = []
+
+
+def _documentable_obj(obj: object) -> bool:
+    """
+    Check whether we can replace the docstring of `obj`.
+
+    Parameters
+    ----------
+    obj : object
+        Object whose docstring we want to replace.
+
+    Returns
+    -------
+    bool
+        Whether we can replace the docstring.
+    """
+    return bool(
+        callable(obj)
+        and not inspect.isclass(obj)
+        or (isinstance(obj, property) and obj.fget)
+        or (isinstance(obj, functools.cached_property))
+        or (isinstance(obj, (staticmethod, classmethod)) and obj.__func__)
+    )
+
+
+def _update_inherited_docstrings(doc_module: DocModule) -> None:
+    """
+    Update all inherited docstrings.
+
+    Parameters
+    ----------
+    doc_module : DocModule
+        The current DocModule.
+    """
+    _doc_module = doc_module.get()
+    for doc_inheritance_call in _docstring_inheritance_calls:
+        doc_inheritance_call(doc_module=_doc_module)  # type: ignore[call-arg]
+
+
+def _inherit_docstrings_in_place(
+    cls_or_func: Fn,
+    doc_module: str,
+    parent: object,
+    excluded: List[object],
+    overwrite_existing: bool = False,
+    apilink: Optional[Union[str, List[str]]] = None,
+) -> None:
+    """
+    Replace `cls_or_func` docstrings with `parent` docstrings in place.
+
+    Parameters
+    ----------
+    cls_or_func : Fn
+        The class or function whose docstrings we need to update.
+    doc_module : str
+        The docs module.
+    parent : object
+        Parent object from which the decorated object inherits __doc__.
+    excluded : list, default: []
+        List of parent objects from which the class does not
+        inherit docstrings.
+    overwrite_existing : bool, default: False
+        Allow overwriting docstrings that already exist in
+        the decorated class.
+    apilink : str | List[str], optional
+        If non-empty, insert the link(s) to pandas API documentation.
+        Should be the prefix part in the URL template, e.g. "pandas.DataFrame".
+    """
+    # Import the docs module and get the class (e.g. `DataFrame`).
+    imported_doc_module = importlib.import_module(doc_module)
+    # Set the default parent so we can use it in case some docs are missing from
+    # parent module.
+    default_parent = parent
+    # Try to get the parent object from the doc module, and if it isn't there,
+    # get it from parent instead. We only do this if we are overriding pandas
+    # documentation. We don't touch other docs.
+    if doc_module != DocModule.default and "pandas" in str(
+        getattr(parent, "__module__", "")
+    ):
+        parent = getattr(imported_doc_module, getattr(parent, "__name__", ""), parent)
+    if parent != default_parent:
+        # Reset API link in case the docs are overridden.
+        apilink = None
+        overwrite_existing = True
+
+    if parent not in excluded:
+        _replace_doc(parent, cls_or_func, overwrite_existing, apilink)
+
+    if not isinstance(cls_or_func, types.FunctionType):
+        seen = set()
+        for base in cls_or_func.__mro__:  # type: ignore[attr-defined]
+            if base is object:
+                continue
+            for attr, obj in base.__dict__.items():
+                if attr in seen:
+                    continue
+                seen.add(attr)
+                # Try to get the attribute from the docs class first, then
+                # from the default parent (pandas), and if it's not in either,
+                # set `parent_obj` to `None`.
+                parent_obj = getattr(parent, attr, getattr(default_parent, attr, None))
+                if (
+                    parent_obj in excluded
+                    or not _documentable_obj(parent_obj)
+                    or not _documentable_obj(obj)
+                ):
+                    continue
+
+                _replace_doc(
+                    parent_obj,
+                    obj,
+                    overwrite_existing,
+                    apilink,
+                    parent_cls=cls_or_func,
+                    attr_name=attr,
+                )
+
+
 def _inherit_docstrings(
     parent: object,
     excluded: List[object] = [],
@@ -416,73 +539,26 @@ def _inherit_docstrings(
     are not defined in target class (but are defined in the ancestor class),
     which means that ancestor class attribute docstrings could also change.
     """
-    # Import the docs module and get the class (e.g. `DataFrame`).
-    imported_doc_module = importlib.import_module(DocModule.get())
-    # Set the default parent so we can use it in case some docs are missing from
-    # parent module.
-    default_parent = parent
-    # Try to get the parent object from the doc module, and if it isn't there,
-    # get it from parent instead. We only do this if we are overriding pandas
-    # documentation. We don't touch other docs.
-    if DocModule.get() != DocModule.default and "pandas" in str(
-        getattr(parent, "__module__", "")
-    ):
-        parent = getattr(imported_doc_module, getattr(parent, "__name__", ""), parent)
-    if parent != default_parent:
-        # Reset API link in case the docs are overridden.
-        apilink = None
-        overwrite_existing = True
-
-    def _documentable_obj(obj: object) -> bool:
-        """Check if `obj` docstring could be patched."""
-        return bool(
-            callable(obj)
-            and not inspect.isclass(obj)
-            or (isinstance(obj, property) and obj.fget)
-            or (isinstance(obj, functools.cached_property))
-            or (isinstance(obj, (staticmethod, classmethod)) and obj.__func__)
-        )
 
     def decorator(cls_or_func: Fn) -> Fn:
-        if parent not in excluded:
-            _replace_doc(parent, cls_or_func, overwrite_existing, apilink)
-
-        if not isinstance(cls_or_func, types.FunctionType):
-            seen = set()
-            for base in cls_or_func.__mro__:  # type: ignore[attr-defined]
-                if base is object:
-                    continue
-                for attr, obj in base.__dict__.items():
-                    if attr in seen:
-                        continue
-                    seen.add(attr)
-                    # Try to get the attribute from the docs class first, then
-                    # from the default parent (pandas), and if it's not in either,
-                    # set `parent_obj` to `None`.
-                    parent_obj = getattr(
-                        parent, attr, getattr(default_parent, attr, None)
-                    )
-                    if (
-                        parent_obj in excluded
-                        or not _documentable_obj(parent_obj)
-                        or not _documentable_obj(obj)
-                    ):
-                        continue
-
-                    _replace_doc(
-                        parent_obj,
-                        obj,
-                        overwrite_existing,
-                        apilink,
-                        parent_cls=cls_or_func,
-                        attr_name=attr,
-                    )
-
+        inherit_docstring_in_place = functools.partial(
+            _inherit_docstrings_in_place,
+            cls_or_func=cls_or_func,
+            parent=parent,
+            excluded=excluded,
+            overwrite_existing=overwrite_existing,
+            apilink=apilink,
+        )
+        inherit_docstring_in_place(doc_module=DocModule.get())
+        _docstring_inheritance_calls.append(inherit_docstring_in_place)
         return cls_or_func
 
     return decorator
 
 
+DocModule.subscribe(_update_inherited_docstrings)
+
+
 def expanduser_path_arg(argname: str) -> Callable[[Fn], Fn]:
     """
     Decorate a function replacing its path argument with "user-expanded" value.


Test patch:
diff --git a/modin/tests/config/test_envvars.py b/modin/tests/config/test_envvars.py
index 4fc1b651..c1fa009d 100644
--- a/modin/tests/config/test_envvars.py
+++ b/modin/tests/config/test_envvars.py
@@ -13,6 +13,7 @@
 
 import os
 
+import pandas
 import pytest
 
 import modin.config as cfg
@@ -79,34 +80,47 @@ def test_custom_help(make_custom_envvar):
     assert "custom var" in make_custom_envvar.get_help()
 
 
-def test_doc_module():
-    import pandas
-
-    import modin.pandas as pd
-    from modin.config import DocModule
-
-    DocModule.put("modin.tests.config.docs_module")
-
-    # Test for override
-    assert (
-        pd.DataFrame.apply.__doc__
-        == "This is a test of the documentation module for DataFrame."
-    )
-    # Test for pandas doc when method is not defined on the plugin module
-    assert pandas.DataFrame.isna.__doc__ in pd.DataFrame.isna.__doc__
-    assert pandas.DataFrame.isnull.__doc__ in pd.DataFrame.isnull.__doc__
-    # Test for override
-    assert (
-        pd.Series.isna.__doc__
-        == "This is a test of the documentation module for Series."
-    )
-    # Test for pandas doc when method is not defined on the plugin module
-    assert pandas.Series.isnull.__doc__ in pd.Series.isnull.__doc__
-    assert pandas.Series.apply.__doc__ in pd.Series.apply.__doc__
-    # Test for override
-    assert pd.read_csv.__doc__ == "Test override for functions on the module."
-    # Test for pandas doc when function is not defined on module.
-    assert pandas.read_table.__doc__ in pd.read_table.__doc__
+class TestDocModule:
+    """
+    Test using a module to replace default docstrings.
+    """
+
+    def test_overrides(self):
+        cfg.DocModule.put("modin.tests.config.docs_module")
+
+        # Test for override
+        assert (
+            pd.DataFrame.apply.__doc__
+            == "This is a test of the documentation module for DataFrame."
+        )
+        # Test for pandas doc when method is not defined on the plugin module
+        assert pandas.DataFrame.isna.__doc__ in pd.DataFrame.isna.__doc__
+        assert pandas.DataFrame.isnull.__doc__ in pd.DataFrame.isnull.__doc__
+        # Test for override
+        assert (
+            pd.Series.isna.__doc__
+            == "This is a test of the documentation module for Series."
+        )
+        # Test for pandas doc when method is not defined on the plugin module
+        assert pandas.Series.isnull.__doc__ in pd.Series.isnull.__doc__
+        assert pandas.Series.apply.__doc__ in pd.Series.apply.__doc__
+        # Test for override
+        assert pd.read_csv.__doc__ == "Test override for functions on the module."
+        # Test for pandas doc when function is not defined on module.
+        assert pandas.read_table.__doc__ in pd.read_table.__doc__
+
+    def test_not_redefining_classes_modin_issue_7138(self):
+        original_dataframe_class = pd.DataFrame
+
+        cfg.DocModule.put("modin.tests.config.docs_module")
+
+        # Test for override
+        assert (
+            pd.DataFrame.apply.__doc__
+            == "This is a test of the documentation module for DataFrame."
+        )
+
+        assert pd.DataFrame is original_dataframe_class
 
 
 @pytest.mark.skipif(cfg.Engine.get() != "Ray", reason="Ray specific test")