BUG/PERF: Sparse get_dummies uses concat
State: closed

Working around the DataFrame constructor perf issue in https://github.com/pandas-dev/pandas/issues/24368

Fixes deprecation warnings in the ASV files so there's something to run.

Closes https://github.com/pandas-dev/pandas/issues/24371

Labels:
name: Performance
description: Memory or execution speed performance
name: Sparse
description: Sparse Data Type

Comments:
Hello @TomAugspurger! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`asv_bench/benchmarks/join_merge.py`](https://github.com/TomAugspurger/pandas/blob/b253674bbdf2771221af82b5637fee63efa7f8d0/asv_bench/benchmarks/join_merge.py) !

 - There are no PEP8 issues in the file [`asv_bench/benchmarks/panel_ctor.py`](https://github.com/TomAugspurger/pandas/blob/b253674bbdf2771221af82b5637fee63efa7f8d0/asv_bench/benchmarks/panel_ctor.py) !

 - There are no PEP8 issues in the file [`asv_bench/benchmarks/reindex.py`](https://github.com/TomAugspurger/pandas/blob/b253674bbdf2771221af82b5637fee63efa7f8d0/asv_bench/benchmarks/reindex.py) !

 - There are no PEP8 issues in the file [`asv_bench/benchmarks/timedelta.py`](https://github.com/TomAugspurger/pandas/blob/b253674bbdf2771221af82b5637fee63efa7f8d0/asv_bench/benchmarks/timedelta.py) !

 - There are no PEP8 issues in the file [`asv_bench/benchmarks/timestamp.py`](https://github.com/TomAugspurger/pandas/blob/b253674bbdf2771221af82b5637fee63efa7f8d0/asv_bench/benchmarks/timestamp.py) !

 - There are no PEP8 issues in the file [`pandas/core/dtypes/concat.py`](https://github.com/TomAugspurger/pandas/blob/b253674bbdf2771221af82b5637fee63efa7f8d0/pandas/core/dtypes/concat.py) !

 - There are no PEP8 issues in the file [`pandas/core/reshape/reshape.py`](https://github.com/TomAugspurger/pandas/blob/b253674bbdf2771221af82b5637fee63efa7f8d0/pandas/core/reshape/reshape.py) !


# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=h1) Report
> Merging [#24372](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/f6cf7d9e2606e82fd3934e1dac8119c43f5ee643?src=pr&el=desc) will **decrease** coverage by `49.31%`.
> The diff coverage is `9.09%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24372/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #24372       +/-   ##
===========================================
- Coverage   92.29%   42.97%   -49.32%     
===========================================
  Files         162      162               
  Lines       51832    51836        +4     
===========================================
- Hits        47839    22279    -25560     
- Misses       3993    29557    +25564
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `42.97% <9.09%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/reshape/reshape.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9yZXNoYXBlLnB5) | `13.31% <0%> (-86.24%)` | :arrow_down: |
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `57.35% <50%> (-39.25%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-98.65%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| ... and [122 more](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=footer). Last update [f6cf7d9...6a65cbc](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=h1) Report
> Merging [#24372](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/f6cf7d9e2606e82fd3934e1dac8119c43f5ee643?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24372/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24372      +/-   ##
==========================================
+ Coverage   92.29%   92.29%   +<.01%     
==========================================
  Files         162      162              
  Lines       51832    51836       +4     
==========================================
+ Hits        47839    47843       +4     
  Misses       3993     3993
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.7% <100%> (ø)` | :arrow_up: |
| #single | `42.98% <9.09%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `97.05% <100%> (+0.45%)` | :arrow_up: |
| [pandas/core/reshape/reshape.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9yZXNoYXBlLnB5) | `99.56% <100%> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24372/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.57% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=footer). Last update [f6cf7d9...6a65cbc](https://codecov.io/gh/pandas-dev/pandas/pull/24372?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

n.b. 
https://github.com/pandas-dev/pandas/pull/24372/commits/6a65cbc6c3b640bf1ec489d587d65c772160093b has an API breaking change for SparseSeries.unstack. With this PR that returns a DataFrame of sparse values instead of a SparseDataFrame.
Do you know the method name? I don't see one that does it.

On Thu, Dec 20, 2018 at 7:30 AM Jeff Reback <notifications@github.com>
wrote:

> *@jreback* commented on this pull request.
> ------------------------------
>
> In pandas/core/reshape/reshape.py
> <https://github.com/pandas-dev/pandas/pull/24372#discussion_r243271163>:
>
> > @@ -909,7 +910,15 @@ def _make_col_name(prefix, prefix_sep, level):
>
>          index = None
>
>
>
>      if sparse:
>
> -        sparse_series = {}
>
> +
>
> +        if is_integer_dtype(dtype):
>
>
> we have that too let’s try to not reinvent the wheel
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24372#discussion_r243271163>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIsVh9gjb1l4vfllL5IRhx3YlAv9Mks5u65D5gaJpZM4ZcH5c>
> .
>

I think this should be merged soon if possible. The CI failures are blocking https://github.com/pandas-dev/pandas/pull/20796 and the explode PR.

I haven't made too much progress on fixing https://github.com/pandas-dev/pandas/issues/24368 properly. Too many edge cases in our constructors.