DEPR: pandas.core for groupby
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Part of #27522.

Just a PoC for right now to see what the CI says. Various parts of this are stolen from https://github.com/numpy/numpy/pull/24634. 

Not sure if we want to do this all at once or submodule by submodule. I'm thinking doing it one submodule at a time is easier for review and we can try just e.g. groupby first and see how that goes.

All commits prior to "Automated changes" only need to be done the first time. The "Automated changes" commit is the result of the script below. A few manual changes need to be made after.

WARNING: The script below will completely clean out your git repo by doing a `git reset --hard` and `git clean -xfd`

<details>
<summary>Automated changes</summary>

```python
import os
import sys
import subprocess
from pathlib import Path

def run_in_shell(cmd):
    print(cmd)
    response = subprocess.run(cmd, shell=True, capture_output=True)
    stdout_str = response.stdout.decode("utf-8").strip()
    if stdout_str != '':
        print(stdout_str)
    stderr_str = response.stderr.decode("utf-8").strip()
    if stderr_str != '':
        raise ValueError(stderr_str)

base_path = Path("/home/richard/dev/pandas/pandas")
submodule = "groupby"

run_in_shell(f"cd {base_path} && git reset --hard HEAD")
run_in_shell(f"cd {base_path} && git clean -xfd")
os.makedirs(base_path / "_core" / submodule, exist_ok=True)
run_in_shell(f"cd {base_path} && git mv core/{submodule}/* _core/{submodule}/")

old = f'core\.{submodule}'
new = f'_core\.{submodule}'
run_in_shell(f"cd {base_path} && grep -rl '{old}' . --exclude-dir=core/{submodule} | xargs sed -i 's/{old}/{new}/g'")
run_in_shell(f"cd {base_path}/../ci && grep -rl '{old}' . | xargs sed -i 's/{old}/{new}/g'")
run_in_shell(f"cd {base_path}/../doc && grep -rl '{old}' . | xargs sed -i 's/{old}/{new}/g'")
run_in_shell(f"cd {base_path}/../ && sed -i 's/{old}/{new}/g' pyproject.toml")

old = f'from pandas\.core import {submodule}'
new = f'from pandas\._core import {submodule}'
run_in_shell(f"cd {base_path} && grep -rl '{old}' . --exclude-dir=core/{submodule} | xargs sed -i 's/{old}/{new}/g'")

old = f'pandas\/core\/{submodule}'
new = f'pandas\/_core\/{submodule}'
run_in_shell(f"cd {base_path}/../ && sed -i 's/{old}/{new}/g' pyright_reportGeneralTypeIssues.json")

for filename in (base_path / "_core" / submodule).glob("**/*"):
    if not str(filename).endswith(".py"):
        continue
    name = filename.parts[-1][:-len(".py")]
    output_path = str(filename).replace("/_core/", "/core/")

    if name == "__init__":
        contents = f"""from __future__ import annotations
        
from typing import Any

from pandas._core import {submodule} as {submodule}_
from pandas.core.common import _depr_core


def __getattr__(attr_name: str) -> Any:
    attr = getattr({submodule}_, attr_name)
    _depr_core()
    return attr
"""

    else:
        contents = f"""from __future__ import annotations
from pandas._core.{submodule} import {name}
from pandas.core.common import _depr_core

_depr_core()

_globals = globals()

for item in {name}.__dir__():
    _globals[item] = getattr({name}, item)
"""

    with open(output_path, mode="w") as f:
        f.write(contents)

run_in_shell(f"cd {base_path} && git add core/{submodule}/*")
```

</details>

Labels:
name: Groupby
name: API Design
name: Deprecate
description: Functionality to remove in pandas

Comments:
Haven't looked closely, but big picture this seems like something we ought to do eventually and might as well bite the bullet.
> Haven't looked closely, but big picture this seems like something we ought to do eventually and might as well bite the bullet.

I think my latest commits may have messed up the diff, and that we can get it down to < 1000 lines when done properly. I'm hoping I can automate a bit more and will be force-pushing quite a bit here. Will ping when ready.
Looks like my previous comment was wrong - I think because we need to add new content to the old location, GitHub's diff isn't registering this as a move. Bummer.
kind of orthogonal to this but i want to write it down somewhere: the naming in core.groupby is weird to me bc generic.py is for the non-base classes whereas core.generic is for the base class
Just noting that this is how NumPy is deprecating their core usage: https://github.com/numpy/numpy/blob/main/numpy/core/_utils.py 
@mroeschke: happy to make any changes if desired. Because this diff is so large, here is the one I have (in `pandas.core.common._depr_core`):

```
warnings.warn(
        "pandas.core is deprecated and has been renamed to "
        "pandas._core. Accessing `_core` directly is discouraged as "
        "members can change without warning.  You should use a public module "
        "instead that exports the attribute in question. If you still would "
        "like to access an attribute from it, please use pandas._core.",
        DeprecationWarning,
        stacklevel=3,
    )
```
One other open question in my mind: do we do all of `core` at once, or go (roughly) submodule-by-submodule over time (but all within 2.2)?
@jbrockmendel @mroeschke I think this is ready for review. Most of the diff comes from the `Automated changes` commit, produced by the script in the OP.

If we don't do `import pandas.core.groupby` in `pandas/__init__.py`, then we wouldn't be able to do e.g. `from pandas.core.groupby.groupby import GroupBy`. I'm still trying to figure out a better way for this, but maybe it's okay?
Friendly ping @jbrockmendel @mroeschke @phofl 
@mroeschke - friendly ping. Docs build only failed because of the [SciPy URL issue](https://github.com/pandas-dev/pandas/actions/runs/6885153924/job/18728882173?pr=55429#step:6:24).

At this point, I'd think about holding off on this until 2.2 is out. I would like to start on this early in that release cycle though, so good to get any further issues ironed out.

My preference would be to do just one submodule and wait a week or two with it in main before moving on, then do the rest of core all in one go. But I'd be okay with doing the entire thing piece-meal in a series of PRs if others prefer.
> My preference would be to do just one submodule and wait a week or two with it in main before moving on, then do the rest of core all in one go. But I'd be okay with doing the entire thing piece-meal in a series of PRs if others prefer.

No opinion here, strong default to defer to the person doing the actual work: you.

IIRC dask has a place where they do something like

```
import pandas as pd

@inherit_docstring_or_something(pd.core.groupby.DataFrameGroupBy)
class DataFrameGroupBy:
```

that might be affected by this (I think i ran into this when trying to make groupby import lazy)
> IIRC dask has a place where they do something like

Yes that's correct, but that doesn't affect only dask, it affects our Sphinx docs as well (they will show private paths). Richard and I chatted about this a while ago and IIRC having the classes accessible in the typing api module would solve that (we can move dasks imports over)

@rhshadrach is this a correct recollection of our chat?
id want to get @jorisvandenbossche's OK on this before moving forward
During the dev call today, @jorisvandenbossche suggested just doing the move without the deprecation first. It also seems to me it'd be better to do this as two PRs: one moving `core` to `_core` followed by a second that adds the stub files. I will be putting PRs for this approach up in place of this one.