PERF: DataFrame dict constructor with columns
State: closed

When passing a dict and `column=` to DataFrame, we previously
passed the dict of {column: array} to the Series constructor. This
eventually hit `construct_1d_object_array_from_listlike`[1]. For
extension arrays, this ends up calling `ExtensionArray.__iter__`,
iterating over the elements of the ExtensionArray, which is
prohibiatively slow.

---

```python
import pandas as pd
import numpy as np

a = pd.Series(np.arange(1000))
d = {i: a for i in range(30)}

%timeit df = pd.DataFrame(d, columns=list(range(len(d))))
```

before

```
4.06 ms ± 53.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

after

```
2.54 ms ± 113 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

With Series with sparse values instead, the problem is exacerbated (note
the smaller and fewer series).

```python
a = pd.Series(np.arange(1000), dtype="Sparse[int]")
d = {i: a for i in range(50)}

%timeit df = pd.DataFrame(d, columns=list(range(len(d))))
```

Before

```
213 ms ± 7.64 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

after

```
4.41 ms ± 134 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

---

We try to properly handle all the edge cases that we were papering over
earlier by just passing the `data` to Series.

We fix a bug or two along the way, but don't change any *tested*
behavior, even if it looks fishy (e.g. #24385).

[1]: https://github.com/pandas-dev/pandas/issues/24368#issuecomment-449037989

Closes https://github.com/pandas-dev/pandas/issues/24368
Closes https://github.com/pandas-dev/pandas/issues/24386


Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Hello @TomAugspurger! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-03-26 02:13:22 UTC
Note that this passes the tests in `tests/frame/test_constructors.py`, but other tests are failing. I'll be updating to handle additional edge cases that apparently aren't tested in constructors.
I don't really have a preference for whether this goes in 0.24 or 0.24.1.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=h1) Report
> Merging [#24387](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/8c58817bd0eaca46d7c0d0c975715ad37fe43710?src=pr&el=desc) will **decrease** coverage by `49.36%`.
> The diff coverage is `92.1%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24387/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #24387       +/-   ##
===========================================
- Coverage    92.3%   42.93%   -49.37%     
===========================================
  Files         162      162               
  Lines       51875    51892       +17     
===========================================
- Hits        47883    22280    -25603     
- Misses       3992    29612    +25620
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `42.93% <92.1%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `63.7% <92.1%> (-32.95%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-98.65%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.15%)` | :arrow_down: |
| ... and [121 more](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=footer). Last update [8c58817...ed70cef](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=h1) Report
> Merging [#24387](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/8c58817bd0eaca46d7c0d0c975715ad37fe43710?src=pr&el=desc) will **decrease** coverage by `49.35%`.
> The diff coverage is `92.45%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24387/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #24387       +/-   ##
===========================================
- Coverage    92.3%   42.95%   -49.36%     
===========================================
  Files         162      162               
  Lines       51875    51907       +32     
===========================================
- Hits        47883    22296    -25587     
- Misses       3992    29611    +25619
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `42.95% <92.45%> (-0.05%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `66.66% <92.45%> (-29.99%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-98.65%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.15%)` | :arrow_down: |
| ... and [121 more](https://codecov.io/gh/pandas-dev/pandas/pull/24387/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=footer). Last update [8c58817...c1f2a58](https://codecov.io/gh/pandas-dev/pandas/pull/24387?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

ed70cefe012e423d30115de290084046e17c21e6 handles duplicates in `columns`. It wasn't straightforward.
cc @toobaz if you have a chance to look at this. I think you did some work on simplifying the DataFrame constructor recently? I may be undoing some of that work :/
With the latest all of pandas/tests/frame is passing...
@TomAugspurger still active?
Yep.

On Wed, Feb 27, 2019 at 5:56 PM William Ayd <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> still active?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24387#issuecomment-468079100>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIkbvW5XPdn2pRl6IqcqyHtXdLkS-ks5vRxspgaJpZM4Zezaf>
> .
>

pushing the benchmarks here is probably worth it. other parts would need to be revisited. closing.
I'll keep working on this.
closing as stale. pls reopen if rebasing.