BUG: GH10536 in concat for SparseSeries
State: closed

To address #10536, but it's clearly not enough. What should be done for `SparseSeries` of different `kind`s and different `fill` values?


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Sparse
description: Sparse Data Type

Comments:
you can only promote if the things u r concatting have the same fill 

the kind doesn't matter 

Not sure if this is accurate, but would calling `values` on the all the concatenated objects convert them to dense first?  It would be nice if sparse concat didn't do this.

`.values` returns a `SparseArray` and there is a `_concat_compat` specialized for sparse types, but beyond that I haven't really checked. 

Since `SparseSeries` is for `float64` only according to [this doc string](https://github.com/pydata/pandas/blob/master/pandas/sparse/series.py#L2-3), are the tests [here](https://github.com/pydata/pandas/blob/master/pandas/tests/test_series.py#L7052-7065) valid?

@jreback Out of curiosity --- is there any particular reason for the `SparseDataFrame` vs `DataFrame` class distinction (as opposed to just having sparse vs dense underlying block managers?)

@artemyk I made this comment in another issue I think, but can't seem to find it.

The short answer is sort-of. I think the original intent of `SparseDataFrame` was to have a matrix like object that understood indexes. Kind of like a csr/coo (or pick your favorite storage format). That would then for example be all a single dtype, and thus could handle auto-filling.

However after the refactor to move Series out of sub-classing ndarray (0.13). This was no longer necessary (e.g. the SparseBlock became a real thing). So at this point `SparseDataFrame` is not really necessary at all. `SparseSeries` OTOH, does have utiliity as it carries around the filling property for example.

So bottom line, I _think_ we could entirely drop `SparseDataFrame` and everything would work (and of course drop `SparsePanel` which has an original non-useful internal implementation).

@jreback Thanks, that makes sense.  I"m actually feeling like the underlying sparse implementation could use an overhaul.  For example,  if I understand correctly, sparse blocks cannot be consolidated right now.  So, if get_dummies is called on a column with 100,000 values, we get a dataframe with 100,000 blocks.  I imagine this can strongly hurt performance in some areas.  An underlying sparse block manager that can truly handle sparse matrices (not just series) of various data types would be nice.

well, we have been thinking about what to do with sparse. I am not 100% sure why @wesm originally wrote this. But then again I don't know the state of scipy sparse back in 2010.

So we _could_ have an alternative (or maybe THE) sparse repr that actually is something from scipy, e.g. coo/csr. And just use the current index stuff. It would be something of a project, but yes, could be very nice.

Unfortunately it seems that scipy.sparse only supports numeric datatypes.  Perhaps a wrapper around something like the Boost sparse matrix library (http://www.boost.org/doc/libs/1_45_0/libs/numeric/ublas/doc/matrix_sparse.htm#2CompressedMatrix) could be used instead.

@jreback AFAIK isn't `pandas.sparse` the only solution out there for "mostly NA" sparse tables (`scipy.sparse` is "mostly 0", which is not the same thing at all)? That was the original use case: panel regressions involving future contract data over a long history. So you might have a time series that only appears for 3-6 months out of a 30 year period, and want to convert that to a long (dense) panel form without insane memory use

yeah, have been thinking about sparse and what to do. a suggestion was made to split this off into `pandas-sparse`, to allow a more freedom to experiment (but still keep some integration). The actual sparse impl is pretty ok. I think we need to blow away `SparsePanel` (as it has an older impl and not really much supported). Maybe `SparseDataFrame` (as the blocks can simply be included in a `DataFrame`), though the usecase for a 2-d sparse repr is there for it. Suggestions welcome, esp someone to lead this effort.

@kawochen you want to rebase / update and can review

@kawochen yeh, the sparse stuff needs an overhaul....

closing, but if you'd like to update, pls reopen