BUG: fix issue with concat creating SparseFrame if not all series are sparse.
State: closed

Ok so after trying a few things out. This seems to be the problem.

When concatting multiple data frames together if any of the containing series are sparse then the entire data frame becomes sparse (or SparseDataFrame). 

This is in fact not what we want. We want a DataFrame that contains a SparseSeries and a Series.

closes #18914, 
closes #18686 
closes #16874
closes #18551
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

  

Labels:
name: Bug
name: Sparse
description: Sparse Data Type

Comments:
this should not to be necessary. rather this is a bug in how reindexing is implemented 
K I'll look at reindex then. I was a bit hesitant to push this fix because it felt a bit off.
I think it might still be get_dummies because if I create a SparseDataFrame from scratch it works fine

```python
import pandas.util.testing as tm

df2 = pd.SparseDataFrame()
df2['GDP'] = pd.Series([1,2])
df2['Nation_AB'] = pd.SparseSeries([1,0])
df2['Nation_CD'] = pd.SparseSeries([0,1])

df = pd.DataFrame.from_items([('GDP', [1, 2]),('Nation', ['AB', 'CD'])])
df = pd.get_dummies(df, columns=['Nation'], sparse=True)  # SparseDataFrame

tm.assert_frame_equal(df, df2) # => Works

df2.reindex(columns=['GDP']) #=> doesn't fail
df.reindex(columns=['GDP']) #=> fails

df.GDP #=> fails since it's cast as a SparseSeries but has an IntBlock underneath
df[['GDP']] #=> works fine

df2.GDP.block #=> gets cast to SparseBlock somewhere in transmission
```

Somehow... GDP is being stored as a IntBlock but marked as SparseBlock. It doesn't feel like this is reindex's fault but I can definitely be wrong.

I might have to come back to this in the morning. Not sure where to take it.

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18924?src=pr&el=h1) Report
> Merging [#18924](https://codecov.io/gh/pandas-dev/pandas/pull/18924?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/238499ab0a48a0ad4a2011e2ce1c6a02c86124eb?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18924/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/18924?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18924      +/-   ##
==========================================
- Coverage   91.62%    91.6%   -0.03%     
==========================================
  Files         150      150              
  Lines       48724    48726       +2     
==========================================
- Hits        44642    44634       -8     
- Misses       4082     4092      +10
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.97% <100%> (-0.03%)` | :arrow_down: |
| #single | `41.75% <100%> (+0.01%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18924?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/18924/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2dlbmVyaWMucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/18924/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `99.14% <100%> (ø)` | :arrow_up: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18924/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.22% <0%> (-1.74%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/18924/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `95.91% <0%> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/18924/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `83.85% <0%> (+0.2%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18924?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18924?src=pr&el=footer). Last update [238499a...0768990](https://codecov.io/gh/pandas-dev/pandas/pull/18924?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

See https://github.com/pandas-dev/pandas/issues/18686 as well.

Agreed with @jreback that the issue is in `reindex`, not `get_dummies`. I don't think that `sparse=True` implies that numeric, non-sparse blocks should become sparse. I think those numeric columns should go through untouched.
Alright since it's after the holiday I'll take a look today :smile:.
I think I need some guidance on this one... It really doesn't appear to be reindex to me. Doing some debugging I couldn't re-create it without utilizing concat. 

The problem is that this column "GDP" is being cast as a SparseSeries when the underlying block is an IntBlock. But reindex thinks that since it's a SparseSeries it should re-index using SparseArray only which is why it's throwing errors.

IF I create the sparse data frame from scratch I can't re-create the issue but if I use concat I can.

```python
from pandas.core.reshape.reshape import _get_dummies_1d
from pandas.core.reshape.concat import concat

df = pd.DataFrame.from_items([('GDP', [1, 2]),('Nation', ['AB', 'CD'])])
dummy = _get_dummies_1d(df['Nation'], sparse=True, prefix='Nation')

isinstance(dummy, pd.SparseDataFrame)
isinstance(df, pd.DataFrame)

# I think this is the real problem.
concatted = concat([df.drop('Nation', axis=1), dummy], axis=1)
concatted.reindex(columns=['GDP']) # => Fails because reindex thinks it's reindexing a SparseArray but it's an IntBlock
concatted.GDP # => Fails since the underlying block is an IntBlock not a SparseBlock
```

I also looked at concat but have run out of my open source time for the day :(.

Let me know if I'm on the right track or barking up a tree I shouldn't be barking up. Thanks @jreback and @TomAugspurger.
Alright I looked at this again today...

Here is the problem as I see it:

SparseFrame is instantiating only SparseSeries even though 'GDP' isn't a SparseSeries.

On top of that SparseSeries when wrapping a SingleBlockManager doesn't coerce it into a Sparse data structure.

I think SingleBlockManager should be coerced into a sparse structure since everything else is on instantiation. So I will try that tomorrow.




yeah this should actually be returning a ``DataFrame`` which would contain SparseSeries (and a regular Series). SparseDataFrame is kind of a special DataFrame that has filling behavior.
Hello @hexgnu! Thanks for updating the PR.











Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on January 31, 2018 at 11:26 Hours UTC
@jreback omg thank you! found the bug. I was hitting my head trying to get SparseDataFrame to have both dense and sparse series in it but in fact needed to use DataFrame.

Found the issue and honestly it's a simple enough fix. This should close both #18914 and #18686
Alright so I just pushed some test fixes.

One thing that kind of confuses me is why is there both:

```python
pandas.core.dtypes.concat._get_series_result_type
pandas.core.dtypes.concat._get_frame_result_type
```

They are almost identical minus some things. I didn't want to get stuck in a refactor loop but it seems like these functions should be one instead of two?



have a look at https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3ASparse+label%3AReshaping, see if this closes any of those (aside from the already marked ones)
Also I found one more issue that was directly tied to this. I opened up a new issue though asking about DataFrame.density and whether that should be defined or not. 
looks like we can add some tests from the issues. (make the new tests and add the gh issue on them).
Yea I wondered if you wanted me to do that. I’ll get on it soon thanks!

On Thu, Jan 4, 2018 at 8:44 AM Jeff Reback <notifications@github.com> wrote:

> looks like we can add some tests from the issues. (make the new tests and
> add the gh issue on them).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/18924#issuecomment-355168914>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAIdDu0VpdVzT6fxj5n5M_0wg32MRz1mks5tHB7ugaJpZM4RL0Op>
> .
>

So on #18686 it will close _that_ problem but to_coo won't be defined on DataFrame. Do you want me to tack onto that existing issue I made?

I added a test for the other one plus a fix for what you pointed out in whatsnew.
Ok this should be good to go after tests run @jreback 
@hexgnu close now! Just some linting failures: https://travis-ci.org/pandas-dev/pandas/jobs/334545951#L3209 Can you fix those and repush?
@TomAugspurger done! thanks :) been waiting for that fix in master to show up.
lgtm. ping on green.
alrighty all green thanks! @jreback 
thanks @hexgnu nice patch! keep em coming!