COMPAT: add back block manager constructors to pandas.core.internals API
State: closed

Those were removed in a for the rest unrelated PR (https://github.com/pandas-dev/pandas/pull/33100, cc @jbrockmendel)

But at least `create_block_manager_from_blocks` is used by dask (in partd: https://github.com/dask/partd/blob/master/partd/pandas.py), so dask's tests are failing with pandas master (cc @TomAugspurger dask isn't testing with pandas master?). But so to be safe just added back both.

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Dask does test against pandas master, but it's allowed to fail. Looking
into it this week.

On Thu, Apr 30, 2020 at 3:44 AM Joris Van den Bossche <
notifications@github.com> wrote:

> Those were removed in a for the rest unrelated PR (#33100
> <https://github.com/pandas-dev/pandas/pull/33100>, cc @jbrockmendel
> <https://github.com/jbrockmendel>)
>
> But at least create_block_manager_from_blocks is used by dask (in partd:
> https://github.com/dask/partd/blob/master/partd/pandas.py), so dask's
> tests are failing with pandas master (cc @TomAugspurger
> <https://github.com/TomAugspurger> dask isn't testing with pandas
> master?). But so to be safe just added back both.
> ------------------------------
> You can view, comment on, or merge this pull request online at:
>
>   https://github.com/pandas-dev/pandas/pull/33892
> Commit Summary
>
>    - COMPAT: add back block manager constructors to pandas.core.internals
>    API
>
> File Changes
>
>    - *M* pandas/core/internals/__init__.py
>    <https://github.com/pandas-dev/pandas/pull/33892/files#diff-cf58b8e0abc60541619ae243fe0ff2ce>
>    (9)
>
> Patch Links:
>
>    - https://github.com/pandas-dev/pandas/pull/33892.patch
>    - https://github.com/pandas-dev/pandas/pull/33892.diff
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/33892>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOITFMAAACTU7FSHIV2LRPE26JANCNFSM4MVJZ63Q>
> .
>

I only catched it in the parquet tests, so the pandas-master build might need to have pyarrow/fastparquet installed to catch it
Is weaning downstream packages off of these a viable option?
> Is weaning downstream packages off of these a viable option?

That is certainly something to look at, but apart from that, we still need to add it back for compat with released versions IMO (and we could also deprecate it first before removing again)
CI fails unrelated; LGTM
It'd be a bit hard for `partd` to move off this. The purpose is to (de)serialize a DataFrame as quickly as possible, which essentially requires working at the block level.
> It'd be a bit hard for partd to move off this. The purpose is to (de)serialize a DataFrame as quickly as possible, which essentially requires working at the block level.

Yes, that's exactly the same reason that pyarrow also interacts with the BlockManager. 

Looking at pyarrow, there we are using the main constructor: `BlockManager(reconstructed_blocks, axes)`, which seems basically what `create_block_manager_from_blocks` does (the function additionally calls `_consolidate_inplace()`). 
But, I am personally not sure it is necessarily "better" to use `BlockManager(..)` over `create_block_manager_from_blocks(..)`. Although in theory it would be nice that all external projects use the same method to create BlockManager.



If we want, we could create a restricted "publich-ish but not necessarily stable" dev API for projects to interact with the block manager. That way we at least know what projects are using.

Dunno if that's worth the effort though. The current setup of downstream projects testing against master seems to work out OK.
For me that "dev API" to interact with blocks is basically what is exposed in `pandas.core.internals` now? (maybe with the exception of `concatenate_block_managers`)
Haven't really looked, but that sounds fine :)
@jorisvandenbossche before i make a PR removing _safe_reshape from `internals.__init__`, are you aware of any downstream packages that need it there?
I am not aware of any `_`-prefixed functions being used
@phofl does dask still need create_block_manager_from_blocks?
yeah that is still in use