Repository: pandas-dev/pandas
Pull request #55171 (2023-09-16 15:46:31)

ENH: set __module__ for objects in pandas pd.DataFrame API
- [x] part of #55178
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR add ```@set_module``` decorator around ```class pd.DataFrame```

Before this PR
```
>>> import pandas as pd
>>> pd.DataFrame
<class 'pandas.core.frame.DataFrame'>
```

After this PR
```
>>> import pandas as pd
>>> pd.DataFrame
<class 'pandas.DataFrame'>
```

This change would discourage users from incorrectly using
```
from pandas.core.frame import DataFrame
```
Inspired by the similar implementation in numpy: 
https://github.com/numpy/numpy/pull/12382




jbrockmendel: > inspired by similar implementation in numpy

Whenever I try to figure out how something in numpy works I have a hard time finding out where something is defined because they use patterns like `from foo import *` at the top level.  I don't know if the pattern in this PR contributes to that pain point, but my spidey sense is tingling that it might.
jorisvandenbossche: 



=================================== FAILURES ===================================
______________ [doctest] pandas.core.indexing.IndexingMixin.iloc _______________
207         c    3
208         d    4
209         Name: 0, dtype: int64
210 
211         With a list of integers.
212 
213         >>> df.iloc[[0]]
214            a  b  c  d
215         0  1  2  3  4
216         >>> type(df.iloc[[0]])
Expected:
    <class 'pandas.core.frame.DataFrame'>
Got:
    <class 'pandas.DataFrame'>

/home/runner/micromamba/envs/test/lib/python3.10/site-packages/pandas/core/indexing.py:216: DocTestFailure
_________________ [doctest] pandas.core.indexing._iLocIndexer __________________
1583 c    3
1584 d    4
1585 Name: 0, dtype: int64
1586 
1587 With a list of integers.
1588 
1589 >>> df.iloc[[0]]
1590    a  b  c  d
1591 0  1  2  3  4
1592 >>> type(df.iloc[[0]])
Expected:
    <class 'pandas.core.frame.DataFrame'>
Got:
    <class 'pandas.DataFrame'>

/home/runner/micromamba/envs/test/lib/python3.10/site-packages/pandas/core/indexing.py:1592: DocTestFailure

> > inspired by similar implementation in numpy
> 
> Whenever I try to figure out how something in numpy works I have a hard time finding out where something is defined because they use patterns like `from foo import *` at the top level. I don't know if the pattern in this PR contributes to that pain point, but my spidey sense is tingling that it might.

@jbrockmendel this has nothing to do with a `*` import though, so I don't fully understand your comment. 
It is true that it _does_ mask where the object actually lives, so it makes it harder to find the location (the file) in the source code. But this masking is actually the purpose, with the idea that users don't need to see this (with the drawback for contributors, but in making the trade-off, there are much more users, so I would prefer prioritizing that use case).
jorisvandenbossche: @deepak-george there are a bunch of tests where it seems we rely on this exact "pandas.core.frame" string that will have to be updated as well.

First, there are some doctest failures (i.e. docstrings that need to be updated):

```



=================================== FAILURES ===================================
______________ [doctest] pandas.core.indexing.IndexingMixin.iloc _______________
207         c    3
208         d    4
209         Name: 0, dtype: int64
210 
211         With a list of integers.
212 
213         >>> df.iloc[[0]]
214            a  b  c  d
215         0  1  2  3  4
216         >>> type(df.iloc[[0]])
Expected:
    <class 'pandas.core.frame.DataFrame'>
Got:
    <class 'pandas.DataFrame'>

/home/runner/micromamba/envs/test/lib/python3.10/site-packages/pandas/core/indexing.py:216: DocTestFailure
_________________ [doctest] pandas.core.indexing._iLocIndexer __________________
1583 c    3
1584 d    4
1585 Name: 0, dtype: int64
1586 
1587 With a list of integers.
1588 
1589 >>> df.iloc[[0]]
1590    a  b  c  d
1591 0  1  2  3  4
1592 >>> type(df.iloc[[0]])
Expected:
    <class 'pandas.core.frame.DataFrame'>
Got:
    <class 'pandas.DataFrame'>

/home/runner/micromamba/envs/test/lib/python3.10/site-packages/pandas/core/indexing.py:1592: DocTestFailure
```

And then there are also normal test failures like:

```
____________________________ test_info_int_columns _____________________________
[gw2] darwin -- Python 3.9.18 /Users/runner/micromamba/envs/test/bin/python3.9

    @pytest.mark.xfail(not IS64, reason="GH 36579: fail on 32-bit system")
    def test_info_int_columns():
        # GH#37245
        df = DataFrame({1: [1, 2], 2: [2, 3]}, index=["A", "B"])
        buf = StringIO()
        df.info(show_counts=True, buf=buf)
        result = buf.getvalue()
        expected = textwrap.dedent(
            """\
            <class 'pandas.core.frame.DataFrame'>
            Index: 2 entries, A to B
            Data columns (total 2 columns):
             #   Column  Non-Null Count  Dtype
            ---  ------  --------------  -----
             0   1       2 non-null      int64
             1   2       2 non-null      int64
            dtypes: int64(2)
            memory usage: 48.0+ bytes
            """
        )
>       assert result == expected
E       AssertionError

pandas/tests/io/formats/test_info.py:501: AssertionError
```

and for all, see the failing CI logs.

It might be possible to fix most of them with a search replace of "pandas.core.frame.DataFrame" -> "pandas.DataFrame"
deepak-george: @jorisvandenbossche Thanks for the feedback! I have made the changes  like pandas.core.frame.DataFrame -> pandas.DataFrame

Now all the tests except "Unit Tests / Linux-32-bit (pull_request)"  passed. I am looking into the logs and trying to fix but not sure yet what is going on.
deepak-george: > @jorisvandenbossche Thanks for the feedback! I have made the changes like pandas.core.frame.DataFrame -> pandas.DataFrame & pylint fix
> 
> Now all the tests except "Unit Tests / Linux-32-bit (pull_request)" passed. I am looking into the logs and trying to fix but not sure yet what is going on.

@jorisvandenbossche 
`../../numpy/meson.build:207:4: ERROR: Problem encountered: No BLAS library detected! Install one, or use the `allow-noblas` build option (note, this may be up to 100x slower for some linear algebra operations).`

I belive BLAS is already installed
jorisvandenbossche: @deepak-george you can ignore that error, that looks unrelated (and probably if you merge with the latest main branch, that might be resolved)
deepak-george: > @deepak-george you can ignore that error, that looks unrelated (and probably if you merge with the latest main branch, that might be resolved)

Ok. Now all the tests including the new test added to check this functionality have passed. Is there something else that need to be done before you could merge? Please let me know the next step. 
deepak-george: @jorisvandenbossche What is the next step for this PR? Could this be merged?
I have pulled the recent changes from main and all checks have passed as well.
jorisvandenbossche: @aimlnerd sorry for the slow follow-up here. But we _might_ want to only merge this after pandas 2.2 is cut, so we merge this in the main branch for 3.0. Reason for this: in theory it could be a breaking change if someone relies on the `__module__` attribute, so for safety it might be better to only do this in the major release.

It is the plan to branch 2.2.x in one week or two, so we can revisit this shortly!
aimlnerd: @jorisvandenbossche Thanks for the context! Please let me know when you want to revisit, happy to make changes and complete it!

Patch:
diff --git a/doc/source/development/contributing_docstring.rst b/doc/source/development/contributing_docstring.rst
index e2881c1087..0b8c1e16dc 100644
--- a/doc/source/development/contributing_docstring.rst
+++ b/doc/source/development/contributing_docstring.rst
@@ -940,7 +940,7 @@ Finally, docstrings can also be appended to with the ``doc`` decorator.
 
 In this example, we'll create a parent docstring normally (this is like
 ``pandas.core.generic.NDFrame``). Then we'll have two children (like
-``pandas.core.series.Series`` and ``pandas.core.frame.DataFrame``). We'll
+``pandas.core.series.Series`` and ``pandas.DataFrame``). We'll
 substitute the class names in this docstring.
 
 .. code-block:: python
diff --git a/doc/source/user_guide/enhancingperf.rst b/doc/source/user_guide/enhancingperf.rst
index 8c51017381..c4721f3a6b 100644
--- a/doc/source/user_guide/enhancingperf.rst
+++ b/doc/source/user_guide/enhancingperf.rst
@@ -453,7 +453,7 @@ by evaluate arithmetic and boolean expression all at once for large :class:`~pan
    :func:`~pandas.eval` is many orders of magnitude slower for
    smaller expressions or objects than plain Python. A good rule of thumb is
    to only use :func:`~pandas.eval` when you have a
-   :class:`.DataFrame` with more than 10,000 rows.
+   :class:`~pandas.core.frame.DataFrame` with more than 10,000 rows.
 
 Supported syntax
 ~~~~~~~~~~~~~~~~
diff --git a/doc/source/user_guide/io.rst b/doc/source/user_guide/io.rst
index 9c48e66daa..db2326d5b9 100644
--- a/doc/source/user_guide/io.rst
+++ b/doc/source/user_guide/io.rst
@@ -6400,7 +6400,7 @@ ignored.
    In [2]: df = pd.DataFrame({'A': np.random.randn(sz), 'B': [1] * sz})
 
    In [3]: df.info()
-   <class 'pandas.core.frame.DataFrame'>
+   <class 'pandas.DataFrame'>
    RangeIndex: 1000000 entries, 0 to 999999
    Data columns (total 2 columns):
    A    1000000 non-null float64
diff --git a/doc/source/whatsnew/v0.24.0.rst b/doc/source/whatsnew/v0.24.0.rst
index cb12962256..c63d047f03 100644
--- a/doc/source/whatsnew/v0.24.0.rst
+++ b/doc/source/whatsnew/v0.24.0.rst
@@ -840,7 +840,7 @@ then all the columns are dummy-encoded, and a :class:`SparseDataFrame` was retur
    In [2]: df = pd.DataFrame({"A": [1, 2], "B": ['a', 'b'], "C": ['a', 'a']})
 
    In [3]: type(pd.get_dummies(df, sparse=True))
-   Out[3]: pandas.core.frame.DataFrame
+   Out[3]: pandas.DataFrame
 
    In [4]: type(pd.get_dummies(df[['B', 'C']], sparse=True))
    Out[4]: pandas.core.sparse.frame.SparseDataFrame
diff --git a/doc/source/whatsnew/v1.0.0.rst b/doc/source/whatsnew/v1.0.0.rst
index 94a8ee7cd1..5dbf6f1c60 100755
--- a/doc/source/whatsnew/v1.0.0.rst
+++ b/doc/source/whatsnew/v1.0.0.rst
@@ -414,7 +414,7 @@ Extended verbose info output for :class:`~pandas.DataFrame`
    ...                    "text_col": ["a", "b", "c"],
    ...                    "float_col": [0.0, 0.1, 0.2]})
    In [2]: df.info(verbose=True)
-   <class 'pandas.core.frame.DataFrame'>
+   <class 'pandas.DataFrame'>
    RangeIndex: 3 entries, 0 to 2
    Data columns (total 3 columns):
    int_col      3 non-null int64
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index 8fd0cd8c66..5d10a5541f 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -65,6 +65,7 @@ from pandas.util._decorators import (
     Appender,
     Substitution,
     doc,
+    set_module,
 )
 from pandas.util._exceptions import (
     find_stack_level,
@@ -498,6 +499,7 @@ ValueError: columns overlap but no suffix specified:
 # DataFrame class
 
 
+@set_module("pandas")
 class DataFrame(NDFrame, OpsMixin):
     """
     Two-dimensional, size-mutable, potentially heterogeneous tabular data.
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
index c8a2e11dce..6b4070ed63 100644
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -227,7 +227,7 @@ class IndexingMixin:
            a  b  c  d
         0  1  2  3  4
         >>> type(df.iloc[[0]])
-        <class 'pandas.core.frame.DataFrame'>
+        <class 'pandas.DataFrame'>
 
         >>> df.iloc[[0, 1]]
              a    b    c    d
diff --git a/pandas/io/formats/format.py b/pandas/io/formats/format.py
index 8566751b9f..c503121328 100644
--- a/pandas/io/formats/format.py
+++ b/pandas/io/formats/format.py
@@ -855,7 +855,7 @@ class DataFrameRenderer:
         - to_csv
         - to_latex
 
-    Called in pandas.core.frame.DataFrame:
+    Called in pandas.DataFrame:
         - to_html
         - to_string
 
diff --git a/pandas/io/formats/info.py b/pandas/io/formats/info.py
index ad595a2be8..bb156f0fbf 100644
--- a/pandas/io/formats/info.py
+++ b/pandas/io/formats/info.py
@@ -72,7 +72,7 @@ frame_examples_sub = dedent(
     Prints information of all columns:
 
     >>> df.info(verbose=True)
-    <class 'pandas.core.frame.DataFrame'>
+    <class 'pandas.DataFrame'>
     RangeIndex: 5 entries, 0 to 4
     Data columns (total 3 columns):
      #   Column     Non-Null Count  Dtype
@@ -87,7 +87,7 @@ frame_examples_sub = dedent(
     information:
 
     >>> df.info(verbose=False)
-    <class 'pandas.core.frame.DataFrame'>
+    <class 'pandas.DataFrame'>
     RangeIndex: 5 entries, 0 to 4
     Columns: 3 entries, int_col to float_col
     dtypes: float64(1), int64(1), object(1)
@@ -115,7 +115,7 @@ frame_examples_sub = dedent(
     ...     'column_3': np.random.choice(['a', 'b', 'c'], 10 ** 6)
     ... })
     >>> df.info()
-    <class 'pandas.core.frame.DataFrame'>
+    <class 'pandas.DataFrame'>
     RangeIndex: 1000000 entries, 0 to 999999
     Data columns (total 3 columns):
      #   Column    Non-Null Count    Dtype
@@ -127,7 +127,7 @@ frame_examples_sub = dedent(
     memory usage: 22.9+ MB
 
     >>> df.info(memory_usage='deep')
-    <class 'pandas.core.frame.DataFrame'>
+    <class 'pandas.DataFrame'>
     RangeIndex: 1000000 entries, 0 to 999999
     Data columns (total 3 columns):
      #   Column    Non-Null Count    Dtype
diff --git a/pandas/util/_decorators.py b/pandas/util/_decorators.py
index 4f6fc0f3d8..d287fa72d5 100644
--- a/pandas/util/_decorators.py
+++ b/pandas/util/_decorators.py
@@ -503,3 +503,24 @@ __all__ = [
     "future_version_msg",
     "Substitution",
 ]
+
+
+def set_module(module):
+    """Private decorator for overriding __module__ on a function or class.
+
+    Example usage::
+
+        @set_module("pandas")
+        def example():
+            pass
+
+
+        assert example.__module__ == "pandas"
+    """
+
+    def decorator(func):
+        if module is not None:
+            func.__module__ = module
+        return func
+
+    return decorator


Test patch:
diff --git a/pandas/conftest.py b/pandas/conftest.py
index 50a94b35c2..65410c3c09 100644
--- a/pandas/conftest.py
+++ b/pandas/conftest.py
@@ -125,7 +125,7 @@ def ignore_doctest_warning(item: pytest.Item, path: str, message: str) -> None:
     item : pytest.Item
         pytest test item.
     path : str
-        Module path to Python object, e.g. "pandas.core.frame.DataFrame.append". A
+        Module path to Python object, e.g. "pandas.DataFrame.append". A
         warning will be filtered when item.name ends with in given path. So it is
         sufficient to specify e.g. "DataFrame.append".
     message : str
diff --git a/pandas/tests/api/test_api.py b/pandas/tests/api/test_api.py
index 15b6c9abae..82c5c305b5 100644
--- a/pandas/tests/api/test_api.py
+++ b/pandas/tests/api/test_api.py
@@ -401,3 +401,7 @@ def test_pandas_array_alias():
         res = pd.arrays.PandasArray
 
     assert res is pd.arrays.NumpyExtensionArray
+
+
+def test_set_module():
+    assert pd.DataFrame.__module__ == "pandas"
diff --git a/pandas/tests/frame/methods/test_info.py b/pandas/tests/frame/methods/test_info.py
index fcb7677f03..4e3726f4dc 100644
--- a/pandas/tests/frame/methods/test_info.py
+++ b/pandas/tests/frame/methods/test_info.py
@@ -40,7 +40,7 @@ def test_info_empty():
     result = buf.getvalue()
     expected = textwrap.dedent(
         """\
-        <class 'pandas.core.frame.DataFrame'>
+        <class 'pandas.DataFrame'>
         RangeIndex: 0 entries
         Empty DataFrame\n"""
     )
@@ -208,7 +208,7 @@ def test_info_memory():
     bytes = float(df.memory_usage().sum())
     expected = textwrap.dedent(
         f"""\
-    <class 'pandas.core.frame.DataFrame'>
+    <class 'pandas.DataFrame'>
     RangeIndex: 2 entries, 0 to 1
     Data columns (total 1 columns):
      #   Column  Non-Null Count  Dtype
@@ -501,7 +501,7 @@ def test_info_int_columns():
     result = buf.getvalue()
     expected = textwrap.dedent(
         """\
-        <class 'pandas.core.frame.DataFrame'>
+        <class 'pandas.DataFrame'>
         Index: 2 entries, A to B
         Data columns (total 2 columns):
          #   Column  Non-Null Count  Dtype
diff --git a/pandas/tests/groupby/test_grouping.py b/pandas/tests/groupby/test_grouping.py
index 699fffe5d0..9ce7a0818a 100644
--- a/pandas/tests/groupby/test_grouping.py
+++ b/pandas/tests/groupby/test_grouping.py
@@ -509,7 +509,7 @@ class TestGrouping:
         assert len(gb.groups.keys()) == 4
 
     def test_grouping_error_on_multidim_input(self, df):
-        msg = "Grouper for '<class 'pandas.core.frame.DataFrame'>' not 1-dimensional"
+        msg = "Grouper for '<class 'pandas.DataFrame'>' not 1-dimensional"
         with pytest.raises(ValueError, match=msg):
             Grouping(df.index, df[["A", "A"]])