(WIP) CI: Check that private functions are not used across modules
State: closed

- [x] ref https://github.com/pandas-dev/pandas/pull/32932#issuecomment-602704059
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

cc @jbrockmendel 


Labels:
name: CI
description: Continuous Integration

Comments:
neat, thanks for taking the lead on this.  Looks like we've got plenty of violations here.  let's stick a pin in this and try to whittle those down over the coming weeks, then revisit
> neat, thanks for taking the lead on this. Looks like we've got plenty of violations here. let's stick a pin in this and try to whittle those down over the coming weeks, then revisit

 @jbrockmendel Should I open a master issue on this?
> Should I open a master issue on this?

You could open a STY issue, sure.  The other (harder) thing that would go with this would be checking more generally for `foo._bar` (excepting `self._bar` which is OK)

Or could do a round-up for STY issues, since I know there are a bunch.
@MomIsBestFriend how hard would it be to make a special case to allow _shared_docs and similar?
> @MomIsBestFriend how hard would it be to make a special case to allow _shared_docs and similar?

@jbrockmendel Very easy, can you please provide me a list of names you'd like to have an exception for?
> can you please provide me a list of names you'd like to have an exception for?

Subject to change, let's start with ["_merge_doc", "_shared_docs", "_extension_array_shared_docs", "_index_shared_docs"]

Note: ATM I'm only checking ```pandas/core/``` because ```.pyx``` gives me trouble.
I will open a separate PR giving the ability to choose the files extension to check (instead of hard-coding it)
After the changes in #33216 are accepted, the whole ```pandas``` directory will be scanned.
@MomIsBestFriend can you re-push?  re-running the failed GH Actions check isnt working for some reason
rebased @jbrockmendel 
@jbrockmendel As I was going through the code I noticed that for example at 

https://github.com/pandas-dev/pandas/blob/213822a21cb58885af9eca0b9818a2e9fa478b04/pandas/_libs/algos.pyx#L415

```_validate_limit``` is considered private, but it gets used in:

https://github.com/pandas-dev/pandas/blob/24827e5d93f55368803a8e6a599e5b0cf858ac24/pandas/core/missing.py#L223

and in

https://github.com/pandas-dev/pandas/blob/ef1cac3dbddee2684d2f60a3d6a00bd9d405ad65/pandas/core/internals/blocks.py#L379

---

ATM the check is doing "lazy" check, as it only checking for imports of ```from foo import _bar```.

It will __not__ detect things like

```python
import foo as foolib

value = foolib._private_function("baz")
```

Do you want the existing check to validate the case above as well? 
> Do you want the existing check to validate the case above as well?

That's going to be the next step, but its going to have a _lot_ of violations
Let's tentatively add to the ignored list all of the remaining docstring template stuff and pickle-constructor stuff:

_new_Index,
_new_PeriodIndex
_doc_template
_interval_shared_docs
_apply_docs
_arith_doc_FRAME
_flex_comp_doc_FRAME
_make_flex_doc
_op_descriptions
_pipe_template

We should make the _np_under_17 public, will take another look at whats left.
just rebased and pushed
@MomIsBestFriend brainstorming: how hard would it be to check for tests that just silently `pass` or `return`?  e.g. we have a bunch of those in tests.indexing.test_coercion
nice idea. pls comment / open a new PR if able to continue.