Remove SparseSeries and SparseDataFrame
State: closed

As discussed on the dev call.

Closes https://github.com/pandas-dev/pandas/issues/28407
Closes https://github.com/pandas-dev/pandas/issues/28025
Closes https://github.com/pandas-dev/pandas/issues/26777
Closes https://github.com/pandas-dev/pandas/issues/26227
~Not https://github.com/pandas-dev/pandas/issues/26123~
Closes https://github.com/pandas-dev/pandas/issues/25694
Closes https://github.com/pandas-dev/pandas/issues/25378
~Not https://github.com/pandas-dev/pandas/issues/25270~
~Not https://github.com/pandas-dev/pandas/issues/24817~
Closes https://github.com/pandas-dev/pandas/issues/24270
Closes https://github.com/pandas-dev/pandas/issues/22630
Closes https://github.com/pandas-dev/pandas/issues/21993
Closes https://github.com/pandas-dev/pandas/issues/21909
~Not https://github.com/pandas-dev/pandas/issues/21818~
Closes https://github.com/pandas-dev/pandas/issues/21231
Closes https://github.com/pandas-dev/pandas/issues/20766
Closes https://github.com/pandas-dev/pandas/issues/20172
~Not https://github.com/pandas-dev/pandas/issues/19278~
Closes https://github.com/pandas-dev/pandas/issues/18969
~Not https://github.com/pandas-dev/pandas/issues/18414~
Closes https://github.com/pandas-dev/pandas/issues/17993
Closes https://github.com/pandas-dev/pandas/issues/15737
Closes https://github.com/pandas-dev/pandas/issues/15634
Closes https://github.com/pandas-dev/pandas/issues/14427
Closes https://github.com/pandas-dev/pandas/issues/14310
Closes https://github.com/pandas-dev/pandas/issues/13665
~Not https://github.com/pandas-dev/pandas/issues/11726~
Closes https://github.com/pandas-dev/pandas/issues/10627
Closes https://github.com/pandas-dev/pandas/issues/9275
~Not https://github.com/pandas-dev/pandas/issues/9265~
Closes https://github.com/pandas-dev/pandas/issues/6427
Closes https://github.com/pandas-dev/pandas/issues/5470
Closes https://github.com/pandas-dev/pandas/issues/5078


Labels:
name: Sparse
description: Sparse Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
@WillAyd it looks like you fixed up the pickle panel stuff in https://github.com/pandas-dev/pandas/pull/27082/files.

Would you recommend removing SparseSeries / SparseDataFrame from the pickle files as a precursor to this PR?
3cc47659dc16f57370bc8e4a82463ec69fae7dc0 has the changes to the pickle stuff. I can split that out into a precursor PR if people want.

I regenerated the pickle files with python 3.5.6 and pandas 0.20.3.

Do we still have pickle files with sparse data in them? 
What error do you get if you read such a file with this branch?
> Do we still have pickle files with sparse data in them?
> What error do you get if you read such a file with this branch?

As of https://github.com/pandas-dev/pandas/commit/3cc47659dc16f57370bc8e4a82463ec69fae7dc0 we have a pickle file with Series[Sparse] and DataFrame[Sparse], but no SparseSeries and SparseDataFrame. That converted the use of SparseSeries/Frame to Series/Frame with sparse values.

With https://github.com/pandas-dev/pandas/commit/3cc47659dc16f57370bc8e4a82463ec69fae7dc0, the pickle tests on this branch pass. It would also pass on master, so I could split those changes out.
I don't have a strong opinion on splitting or not, either way.

For older pickle files with SparseDataFrame in it, you currently get:

```
AttributeError: Can't get attribute 'SparseDataFrame' on <module 'pandas.core.sparse.frame' from '/home/joris/scipy/pandas/pandas/core/sparse/frame.py'>
```

With our pickle_compat, I suppose we could actually map this to a function that creates a DataFrame[sparse] of it. But not sure that's worth the effort. 
But a better error message indicating they need to read it in 0.25, convert and save again to read it in 1.0 would be welcome I think.


Huh, 766a2f2b1cd6ee53b0a8124ca04aac3460ad66fd has a bit of compat code to allow reading a file saved with SparseSeries / SparseFrame. I don't really know how it works, but it seems to. I've restored the old pickle file with SparseSeries / SparseFrame and it passes locally.

This will hopefully pass. To summarize a few things

- I'm skipping the majority of the Feather reader / writer tests because of users would have to add another dependency (e.g conda install pandas-stata) and possibly an extra import in their code
- We're still able to read old pickle files with SparseSeries or SparseDataFrame because of the loaders I added in pickle_compat
- We're semi-blocked by https://github.com/pandas-dev/pandas/pull/28425#discussion_r324212982 (HDF5 io). IMO, we can merge this first and restore sparse IO for HDF5 later (but before 1.0), but either works for me.
@jorisvandenbossche did you say you wanted / could look into HDF5 serialization? If not I can take care of it.

I looked at things briefly, and it seems pretty doable. We should be able to read existing HDF5 files w/ sparse data, and write news using most of the same code.
I am working on the doc theme / geopandas stuff today, so if you want to do it today, go ahead!
> For the issues you are closing with this, we actually need to see if some of them are not also valid for Series[sparse] / SparseArray

I already went through the most recent ones, and crossed a few in the top post.
> added new legacy pickle files 

Maybe rather add files for a different (later) version, since we already have them for 0.21? 
Just FYI, since the last round of reviews I've added

1. A dummy `SparseDataFrame` and `SparseSeries` class, mirroring what we did for Panel. This prevents the immediate error with feather (I've removed the test xfails and reverted the doc changes working around it)
2. Added a warning when loading a pickle file with a SparseDataFrame / SparseSeries saying they should re-save it
Going through the rest of the issues to validate that they are specific to the sparse subclasses.

Edit: done. The list at the top should be accurate. All the remaining sparse issues still open should be relevant to SparseArray.
Let's merge this?
lgtm. I am -0 on providing the pickle compat. but I guess ok.
merge when ready.
Gave another glance through, and I think we're good.

Mostly agreed about the pickle compat. But since it turned out to be so easy (same internal representation) I think we're OK.
Woohoo! :)

>  I am -0 on providing the pickle compat. but I guess ok.

I think a reason to have it (apart from that it was easy to add) is because this is a very recent deprecation, and that it also maps nicely to a supported data structure.