PERF: optimize DataFrame.sparse.from_spmatrix performance
State: closed

This optimizes `DataFrame.sparse.from_spmatrix` performance, using an approach proposed by @jorisvandenbossche in https://github.com/pandas-dev/pandas/issues/32196#issuecomment-600824238

Bulds on top of https://github.com/pandas-dev/pandas/pull/32821 and adds another ~4.5x speed up in addition to that PR. Benchmarks for run time (in seconds) are done by running `pd.DataFrame.sparse.from_spmatrix` on a random sparse CSR array of given n_samples, n_features with a density=0.01:
```
label                     PR   master Speed-up master/PR                                                                               
n_samples n_features                                    
100       100000      2.3624  10.1247              4.29x
10000     10000       0.2391   1.1037              4.62x
100000    100         0.0031   0.0134              4.32x
```

with the benchmarking code below,
<details>

```py
import pandas as pd
import numpy as np
import scipy.sparse

from neurtu import timeit, delayed


def bench_cases():
    for n_samples, n_features in [(100, 100000), (10000, 10000), (100000, 100)]:
        X = scipy.sparse.rand(
            n_samples, n_features, random_state=0, density=0.01, format="csr"
        )
        tags = {"n_samples": n_samples, "n_features": n_features, "label": "PR"}

        yield delayed(pd.DataFrame.sparse.from_spmatrix, tags=tags.copy())(X)


res = timeit(bench_cases())

res = (
    res.reset_index()
    .set_index(["n_samples", "n_features", "label"])["wall_time"]
    .unstack(-1)[["PR"]]
    .round(4)
)
# res["master"] = np.array([10.1247, 1.1037, 0.0134])

# res["Speed-up master/PR"] = (res["master"] / res["PR"]).round(2).astype("str") + "x"
print(res)
```

</details>

Closes https://github.com/pandas-dev/pandas/issues/32196 although further optimization might be possible. Around 90% of remaining run time happens in `DataFrame._from_arrays` which goes deeper into pandas internals. Maybe some checks could be disabled there, but that looks less straightforward.

Labels:
name: Performance
description: Memory or execution speed performance
name: Sparse
description: Sparse Data Type

Comments:
> pls add an asv with a fixed random seed

There is actually already an asv benchmark for this,
```
$ asv continuous --bench SparseDataFrameConstructor upstream/master HEAD
[...]
[100.00%] ··· sparse.SparseDataFrameConstructor.time_from_scipy                                                                 117±1ms
       before           after         ratio
     [dbd7a5d3]       [508fda51]
     <master>         <opt-sparse-init>
-         117±1ms      26.8±0.07ms     0.23  sparse.SparseDataFrameConstructor.time_from_scipy

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
that shows comparable improvement to the benchmarks I have done earlier in the issue description.
> Around 90% of remaining run time happens in DataFrame._from_arrays which goes deeper into pandas internals. Maybe some checks could be disabled there, but that looks less straightforward.

Yeah, it's not really optimized for the case you know you have all well-behaved arrays you just want to put in a DataFrame. 
Investigating this a bit, one factor seems to be the unnecessary consolidation -> https://github.com/pandas-dev/pandas/pull/32826


Thanks @WillAyd , I addressed your comments.

"Web and docs" CI job failed with the following, but I don't think it's due to this PR,
```
 Check ipython directive errors
0s
##[error]Process completed with exit code 1.
Run ! grep -B1 "^<<<-------------------------------------------------------------------------$" sphinx.log
  FutureWarning,
<<<-------------------------------------------------------------------------
##[error]Process completed with exit code 1.
```
It's not very visible, but the actual error is :

```
 >>>-------------------------------------------------------------------------
Warning in /home/runner/work/pandas/pandas/doc/source/user_guide/scale.rst at block ending on line 254
Specify :okwarning: as an option in the ipython:: block to suppress this message
----------------------------------------------------------------------------
/home/runner/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/fsspec/implementations/local.py:33: FutureWarning: The default value of auto_mkdir=True has been deprecated and will be changed to auto_mkdir=False by default in a future release.
  FutureWarning,
<<<-------------------------------------------------------------------------
```
so indeed not from this PR.
See https://github.com/pandas-dev/pandas/issues/32832
@jreback Do you have any other comments on this? I addressed your review comment.
@rth I merged https://github.com/pandas-dev/pandas/pull/32858, so you can update this now to use the `verify_integrity=False`
Thanks @jorisvandenbossche . After merging in https://github.com/pandas-dev/pandas/pull/32858 the benchmarks results are
```
$ asv continuous --bench SparseDataFrameConstructor upstream/master HEAD
...
[100.00%] ··· sparse.SparseDataFrameConstructor.time_from_scipy                                                               104±0.3ms
       before           after         ratio
     [3b406a32]       [e063c8a5]
     <opt-sparse-init~2^2>       <opt-sparse-init>
-       104±0.3ms       12.0±0.3ms     0.12  sparse.SparseDataFrameConstructor.time_from_scipy
```

(previous results in https://github.com/pandas-dev/pandas/pull/32825#issuecomment-601182632)
With https://github.com/pandas-dev/pandas/pull/32856 merged, there is a new failure for non unique column names. In particular, `TestFrameAccessor::test_from_spmatrix_columns[columns2]` calling,
```py
pd.DataFrame.sparse.from_spmatrix(mat, columns=['a', 'a'])
```
fails with,
```py
pandas/core/arrays/sparse/accessor.py:251: in from_spmatrix
    return DataFrame._from_arrays(
pandas/core/frame.py:1919: in _from_arrays
    mgr = arrays_to_mgr(
pandas/core/internals/construction.py:77: in arrays_to_mgr
    return create_block_manager_from_arrays(arrays, arr_names, axes)
pandas/core/internals/managers.py:1689: in create_block_manager_from_arrays
    blocks = form_blocks(arrays, names, axes)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

arrays = [[0, 0.7151164908889105, 0, 0, 0.7790091888007832, 0.8726851819885113, 0.8634012065028459, 0, 0, 0.6582977581693065]
F...13, 0, 0, 0.6245750826925724, 0, 0, 0.6870139075931206]
Fill: 0
IntIndex
Indices: array([1, 2, 3, 6, 9], dtype=int32)
]
names = ['a', 'a'], axes = [['a', 'a'], RangeIndex(start=0, stop=10, step=1)]

    def form_blocks(arrays, names, axes):
        # put "leftover" items in float bucket, where else?
        # generalize?
        items_dict = defaultdict(list)
        extra_locs = []
    
        names_idx = ensure_index(names)
        if names_idx.equals(axes[0]):
            names_indexer = np.arange(len(names_idx))
        else:
>           assert names_idx.intersection(axes[0]).is_unique
E           AssertionError
```

any suggestions @jorisvandenbossche ?
Hmm, it's strange that it would be due to #32856, because it's failing inside `form_blocks` before the location that was changed in that PR. Maybe the `verify_integrity=False` causes it? 
> Hmm, it's strange that it would be due to #32856, because it's failing inside form_blocks before the location that was changed in that PR. Maybe the verify_integrity=False causes it?

Yes, sorry, you are right. This failure was already present 2 days ago before I merged master today.
Thanks @jorisvandenbossche and @jreback ! CI is green now.
thanks @rth very nice