REF/INT: concat blocks of same type with preserving block type
State: closed

Related to https://github.com/pandas-dev/pandas/issues/17283

Goal is to get `pd.concat([list of series]` working with Series with external block type. Currently the values are always converted to arrays, concatenated, and converted back to block with block type inference.

This is a proof-of-concept to check whether something like this would be approved. Tests should still break because I didn't specialize the `concat_same_type` for categorical data.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Hello @jorisvandenbossche! Thanks for updating the PR.









Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on October 10, 2017 at 23:27 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=h1) Report
> Merging [#17728](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b8467c00f78eec73efd14f159f1ba935a65b4ee7?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17728/graphs/tree.svg?src=pr&width=650&token=eZ4WkYLtcO&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17728      +/-   ##
==========================================
- Coverage   91.27%   91.26%   -0.01%     
==========================================
  Files         163      163              
  Lines       49765    49793      +28     
==========================================
+ Hits        45421    45442      +21     
- Misses       4344     4351       +7
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.05% <100%> (+0.01%)` | :arrow_up: |
| #single | `40.32% <53.33%> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `94.41% <100%> (+0.03%)` | :arrow_up: |
| [pandas/core/reshape/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9jb25jYXQucHk=) | `97.68% <100%> (+0.07%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.73% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/dtypes/inference.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2luZmVyZW5jZS5weQ==) | `98.36% <0%> (+0.02%)` | :arrow_up: |
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `99.13% <0%> (+0.86%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=footer). Last update [b8467c0...d4ce3df](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=h1) Report
> Merging [#17728](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/727ea20bdb72ff4a5b8380d7f613d4aff058e28a?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17728/graphs/tree.svg?height=150&token=eZ4WkYLtcO&width=650&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17728      +/-   ##
==========================================
+ Coverage   91.22%   91.24%   +0.01%     
==========================================
  Files         163      163              
  Lines       50014    50043      +29     
==========================================
+ Hits        45627    45661      +34     
+ Misses       4387     4382       -5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.05% <100%> (+0.03%)` | :arrow_up: |
| #single | `40.25% <85.36%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `94.45% <100%> (+0.06%)` | :arrow_up: |
| [pandas/core/reshape/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9jb25jYXQucHk=) | `97.57% <100%> (-0.04%)` | :arrow_down: |
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `99.12% <100%> (+0.86%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.77% <0%> (-0.1%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.2% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=footer). Last update [727ea20...bb5a100](https://codecov.io/gh/pandas-dev/pandas/pull/17728?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback this PR introduces some additional code without removing any (due to other complexities like index not being based on blocks, I can't remove any code I think), but I think the code addition is not too excessive for obtaining the goal.

It is still a bit messy where I need to check for the SparseBlock.

It also seems to give a *modest* speed-up on a very quick test, from around 900ms to around 700ms on this test (although performance was not the original motivation):

```
In [1]: s = pd.Series(np.arange(10000))

In [2]: %timeit pd.concat([s]*10)
```
I added a similar logic to `concatenate_block_managers` to also deal with concatting frames (it is reusing the `concat_same_type` methods of the Blocks). 
@jreback could you have a look ?
ping when ready to have a look
So as long we agree on the "external interface", which currently is a `concat_same_type` method on Block that can be overridden, and we keep that stable, then the internal implementation can be iterated on. 
So the question then mainly is, are we OK with this way to override the behaviour?
Just to note that I have no time until friday to do edits (work-related coding event). But, I think the requested changes are more exact internal implementation cosmetics, and is more clean-up and not fundamental code changes. So depending on when you want to release, I would propose to merge this, and I promise to do a follow-up PR. 
Fine by me. I think we can trust you to not flake off :)

On Wed, Oct 11, 2017 at 9:07 PM, Joris Van den Bossche <
notifications@github.com> wrote:

> Just to note that I have no time until friday to do edits (work-related
> coding event). But, I think the requested changes are more exact internal
> implementation cosmetics, and is more clean-up and not fundamental code
> changes. So depending on when you want to release, I would propose to merge
> this, and I promise to do a follow-up PR.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/17728#issuecomment-335999164>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIjQhV0XR9s6m8gMjhn-BRvRzZ8H9ks5srXRNgaJpZM4Ppl3X>
> .
>

thanks @jorisvandenbossche 

yep a clean up would be great.