BLD: consolidate remaining extensions
State: closed

moves extensions to ``pandas/libs``, which holds the extension code and also the generated builds (as its importable). ``pandas/src`` is now almost an includes dir, holding low-frequency changing code.

This consolidates the import process making it more uniform and consistent throughout the codebase.

Finally this cleans up the remaining top-level namespace (with some deprecations in place for example ``pandas.lib``, ``pandas.tslib``, ``pandas.json``, ``pandas.parser``. I listed all of the changes in the whatsnew, but I don't think worthwhile deprecating anything else.

Labels:
name: Build
description: Library building on various platforms
name: Deprecate
description: Functionality to remove in pandas

Comments:
I'd suggest deprecating at least `tslib`.  I suspect there are some things like `from pandas.tslib import Timestamp` in the wild.
@chris-b1 added
cc @wesm 
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15537?src=pr&el=h1) Report
> Merging [#15537](https://codecov.io/gh/pandas-dev/pandas/pull/15537?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/a347ecb574f4e53f43400ad50b507c481ce12edb?src=pr&el=desc) will **decrease** coverage by `-0.07%`.
> The diff coverage is `95.42%`.


```diff
@@            Coverage Diff             @@
##           master   #15537      +/-   ##
==========================================
- Coverage   91.06%   90.99%   -0.07%     
==========================================
  Files         137      143       +6     
  Lines       49313    49295      -18     
==========================================
- Hits        44906    44858      -48     
- Misses       4407     4437      +30
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15537?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/msgpack/_version.py](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=tree#diff-cGFuZGFzL2lvL21zZ3BhY2svX3ZlcnNpb24ucHk=) | `100% <ø> (ø)` | |
| [pandas/compat/pickle_compat.py](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9waWNrbGVfY29tcGF0LnB5) | `68.29% <ø> (ø)` | :white_check_mark: |
| [pandas/io/msgpack/exceptions.py](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=tree#diff-cGFuZGFzL2lvL21zZ3BhY2svZXhjZXB0aW9ucy5weQ==) | `83.33% <ø> (ø)` | |
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `95.51% <100%> (ø)` | :white_check_mark: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `90.66% <100%> (ø)` | :white_check_mark: |
| [pandas/tslib.py](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=tree#diff-cGFuZGFzL3RzbGliLnB5) | `100% <100%> (ø)` | |
| [pandas/tseries/frequencies.py](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvZnJlcXVlbmNpZXMucHk=) | `96.03% <100%> (-0.01%)` | :x: |
| [pandas/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=tree#diff-cGFuZGFzL2Zvcm1hdHMvZm9ybWF0LnB5) | `95.33% <100%> (ø)` | :white_check_mark: |
| [pandas/core/nanops.py](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbmFub3BzLnB5) | `98.17% <100%> (ø)` | :white_check_mark: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `81.11% <100%> (ø)` | :white_check_mark: |
| ... and [74 more](https://codecov.io/gh/pandas-dev/pandas/pull/15537?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15537?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15537?src=pr&el=footer). Last update [a347ecb...a6d6cfa](https://codecov.io/gh/pandas-dev/pandas/compare/a347ecb574f4e53f43400ad50b507c481ce12edb...a6d6cfa2e6dac1b42f5e2d6e946f2d01a9982950?src=pr&el=footer&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
cc @shoyer 

FYI (I could make this a DeprecationWarning, but those get ignored :<)

```
(pandas) bash-3.2$ python
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:52:12) 
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> import xarray
/Users/jreback/miniconda3/envs/pandas/lib/python3.5/site-packages/xarray/conventions.py:9: FutureWarning: The pandas.tslib module is deprecated and will be removed in a future version. Please import from the pandas.libs.tslib instead
  from pandas.tslib import OutOfBoundsDatetime
```
We used `from pandas.tslib import OutOfBoundsDatetime` because we needed to catch that specific exception and couldn't find it exposed in the public API anywhere.

It would be nice to move that to public API somewhere, maybe `pandas.errors` if the top level namespace is getting too crowded.
This looks fine from superficial glance. Might want to nest src at `pandas/libs/src`. Sort of a "hear lies all ye compiled code" (eventually headed for Davy Jones's locker once the libpandas games get going. I feel like a broken record but it really is going to happen =) )
@wesm sure that sounds good. yeah that's the idea :>
going to merge on pass.
Big +1 on consolidation of where and how the c/cython extensions are organized (and for removing them from the top-level namespace).

Some comments:

- is there a specific reason libwindow is in core and not libs?
- the same as I commented in the other PR, IMO we should think about whether we need to recommend users to import from `pandas.libs` directly (the deprecation warnings now do that).
- Regarding the indication of 'privateness', something like `pandas.tools.libhash` looks more public compared to `pandas._hash` to me (it's not because it is not top-level anymore, that it is not public. We have public non-top-level API). So if we regard that as non-public, I find that not an improvement (that aspect). 

I am personally in favour of using more underscores, making it very clear that something is private stuff.

underscores!
On Fri, Mar 3, 2017 at 8:17 AM Joris Van den Bossche <
notifications@github.com> wrote:

> Big +1 on consolidation of where and how the c/cython extensions are
> organized (and for removing them from the top-level namespace).
>
> Some comments:
>
>    - is there a specific reason libwindow is in core and not libs?
>    - the same as I commented in the other PR, IMO we should think about
>    whether we need to recommend users to import from pandas.libs directly
>    (the deprecation warnings now do that).
>    - Regarding the indication of 'privateness', something like
>    pandas.tools.libhash looks more public compared to pandas._hash to me
>    (it's not because it is not top-level anymore, that it is not public. We
>    have public non-top-level API). So if we regard that as non-public, I find
>    that not an improvement (that aspect).
>
> I am personally in favour of using more underscores, making it very clear
> that something is private stuff.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/15537#issuecomment-283997506>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABKS1ihs1kbBkpLbnU84w2Yh8P8RHiR0ks5riD0ogaJpZM4MP0a7>
> .
>

well, underscores do a lot to make this non-readable.

I *could* make this ``pandas._libs`` (IOW, make all extensions private). I prefer to have imports *inside* pandas code like this

``from pandas._libs import tslib, lib``
rather than
``from pandas.libs import _tslib, _lib``

which looks really ugly to me.


to some of @jorisvandenbossche points

> - is there a specific reason libwindow is in core and not libs?

so from a code organization perspective. directly attributable libs make sense to be right next to there respective libs. general libs like ``lib, algos`` are centrally located. I did consider ``window`` to be with this. But from a dev point of view, editing ``window.py`` and then ``window.pyx`` right next to each is powerful. It moves the code together. 

Similarly for json, sas, and the parser. They are self-contained. Its a tradeoff between dev flexibility and privacy really. I think the ``lib*`` naming really helps to differentiate this though and pretty much solves both problems.

> - the same as I commented in the other PR, IMO we should think about whether we need to recommend users to import from `pandas.libs` directly (the deprecation warnings now do that).

I could do this as ``pandas._libs`` as I indicated above. But in reality having underscores all over the place is simply distracting from a dev point of view (and ugly). So could do that, or put a README.

> - Regarding the indication of 'privateness', something like `pandas.tools.libhash` looks more public compared to `pandas._hash` to me (it's not because it is not top-level anymore, that it is not public. We have public non-top-level API). So if we regard that as non-public, I find that not an improvement (that aspect). 

see comment above. I mean ALL of pandas code is 'public', but that's the point of having ``pandas.api`` in the first place anyhow. Sure *any* code can be used (even ``_`` leading). But I think we need a warning about expectations generally. ``lib*`` should never be regarded as public.
I am personally in favour of using more underscores, making it very clear that something is private stuff.

> I could make this pandas._libs (IOW, make all extensions private). I prefer to have imports inside pandas code like this
> 
> from pandas._libs import tslib, lib
rather than
from pandas.libs import _tslib, _lib
>
> which looks really ugly to me.

Agreed, there's no need to use more than one level of underscores. I also prefer using `_libs.*` to `libs._*`.

> Agreed, there's no need to use more than one level of underscores. I also prefer using _libs.* to libs._*.

ok that seems reasonable. Could also do ``_internals``, though that is a bit longish.
renamed to ``_libs``. This drops if from the main pd namespace as well, so that's a plus.

@jorisvandenbossche w.r.t. the ``lib*``. I can make those private (but still keep the code in those locations) if desired (prob in a followup though)
bombs away.