PERF: avoid printing object in Dtype.construct_from_string message
State: closed

xref https://github.com/pandas-dev/pandas/pull/26562#issuecomment-500120556

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=h1) Report
> Merging [#26776](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/157a4e35d25cafdca2fbc48ccda2c14eb7f25bcb?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26776/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26776      +/-   ##
==========================================
- Coverage   91.72%   91.72%   -0.01%     
==========================================
  Files         178      178              
  Lines       50779    50781       +2     
==========================================
- Hits        46578    46577       -1     
- Misses       4201     4204       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.31% <100%> (ø)` | :arrow_up: |
| #single | `41.05% <100%> (-0.23%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/26776/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Jhc2UucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26776/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26776/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26776/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.94% <0%> (+0.1%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=footer). Last update [157a4e3...f2017d7](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=h1) Report
> Merging [#26776](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/157a4e35d25cafdca2fbc48ccda2c14eb7f25bcb?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26776/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26776      +/-   ##
==========================================
- Coverage   91.72%   91.72%   -0.01%     
==========================================
  Files         178      178              
  Lines       50779    50781       +2     
==========================================
- Hits        46578    46577       -1     
- Misses       4201     4204       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.31% <100%> (ø)` | :arrow_up: |
| #single | `41.05% <100%> (-0.23%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/26776/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Jhc2UucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26776/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26776/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26776/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.94% <0%> (+0.1%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=footer). Last update [157a4e3...f2017d7](https://codecov.io/gh/pandas-dev/pandas/pull/26776?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Can you post the ASV for one of the affected benchmarks?
Master:

```
In [1]: import scipy.sparse

In [2]: N = 1000 
   ...: sparse = scipy.sparse.rand(N, N, 0.005)

In [3]: import warnings

In [4]: warnings.simplefilter('ignore', FutureWarning)

In [5]: %timeit pd.SparseDataFrame(sparse)
2.7 s ± 512 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

This PR:

```
In [5]: %timeit pd.SparseDataFrame(sparse)
430 ms ± 9.67 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

well that's a difference!

though, why is the construction of the SDF so much? (even now)
> though, why is the construction of the SDF so much? (even now)

It's quite a big matrix / DataFrame (1000 columns), and we need to create a SparseArray for each column of the sparse matrix. So there is some conversion needed. It might be possible to optimize, although the `DataFrame.sparse.from_spmatrix`, which uses a totally different implementation, also takes around 400 ms, so quite similar.
thanks @jorisvandenbossche