Repository: pandas-dev/pandas

Issue:
stack() method of SparseDataFrame should return a SparseSeries and optimize memory usage
#### Code Sample, a copy-pastable example if possible

```python
import numpy as np
import pandas as pd

df = pd.DataFrame({'a': [np.nan, np.nan, np.nan, np.nan, np.nan, 1., np.nan],
                   'b': [1., np.nan, np.nan, 1., np.nan, np.nan, np.nan]}).to_sparse()
print(type(df))
print(type(df.stack()))
```

> <class 'pandas.sparse.frame.SparseDataFrame'>
> <class 'pandas.core.series.Series'>

#### Problem description

I'm trying to convert a SparseDataFrame (obtained it from pd.get_dummies()) into a scipy sparse matrix, by using the experimental .to_coo(). As this method accepts a MultiIndex Series, instead of a DataFrame, i call the .stack() method of this SparseDataFrame.

The problem is that it looks like the .stack() method doesn't process the SparseDataFrame as sparse, and instead stacks it as dense, consuming too much memory, and returning a (dense) Series.

Returning a dense Series could be all right, as np.nan values are drop by default with the dropna parameters, but the memory consumption is a problem.

I'm aware the whole sparse functionality is not yet mature. And I saw the function pd.sparse.frame.stack_sparse_frame which I guess it's a step to fix this problem (which doesn't work for me). But as I couldn't find a specific issue for this problem, I thought it was worth opening it.

#### Expected Output

> <class 'pandas.sparse.frame.SparseDataFrame'>
> <class 'pandas.sparse.series.SparseSeries'>

#### Output of ``pd.show_versions()``

<details>

> commit: None
> python: 3.5.2.final.0
> python-bits: 64
> OS: Linux
> OS-release: 4.7.5-100.fc23.x86_64
> machine: x86_64
> processor: x86_64
> byteorder: little
> LC_ALL: None
> LANG: en_US.utf8
> LOCALE: en_US.UTF-8
> 
> pandas: 0.19.2+0.g825876c.dirty
> nose: 1.3.7
> pip: 9.0.1
> setuptools: 23.0.0
> Cython: 0.24
> numpy: 1.11.1
> scipy: 0.17.1
> statsmodels: 0.6.1
> xarray: None
> IPython: 4.2.0
> sphinx: 1.4.1
> patsy: 0.4.1
> dateutil: 2.5.3
> pytz: 2016.4
> blosc: None
> bottleneck: 1.1.0
> tables: 3.2.3.1
> numexpr: 2.6.0
> matplotlib: 1.5.1
> openpyxl: 2.3.2
> xlrd: 1.0.0
> xlwt: 1.1.2
> xlsxwriter: 0.9.2
> lxml: 3.6.0
> bs4: 4.4.1
> html5lib: None
> httplib2: None
> apiclient: None
> sqlalchemy: 1.0.13
> pymysql: None
> psycopg2: 2.6.1 (dt dec pq3 ext)
> jinja2: 2.8
> boto: 2.40.0
> pandas_datareader: None
> 

</details>

SparseSeries.unstack() raises error
#### Code Sample, a copy-pastable example if possible

```python
>>> s = pd.Series([1, 2], index=pd.MultiIndex.from_tuples([(0, 0), (1, 2)]))
>>> s
0  0    1
1  2    2
dtype: int64

>>> s.unstack()
     0    2
0  1.0  NaN
1  NaN  2.0

>>> s.to_sparse()
0  0    1
1  2    2
dtype: int64
BlockIndex
Block locations: array([0], dtype=int32)
Block lengths: array([2], dtype=int32

>>> s.to_sparse().unstack()
---------------------------------------------------------------------------
TypeError: expected dimension <= 1 data
```
#### Problem description

Unstacking a SparseSeries — i.e. one with a two-level index that should neatly expand into a SparseDataFrame — doesn't work, errors.

#### Expected Output

```py
>>> s.to_sparse().unstack()
     0    2
0  1.0  NaN
1  NaN  2.0

>>> s.to_sparse().unstack().__class__
pandas.core.sparse.frame.SparseDataFrame
```

#### Output of ``pd.show_versions()``

<details>
0.21.0.dev+91.g2aafce4

</details>



Discussion:
I think your use-case sounds reasonable. API-wise, I guess we would add a `sparse=True/False` keyword to `SparseDataFrame.stack`. Would the default be True or False? I would think True.

Would you be interested in submitting a PR?

I'll try to work on a PR.

But do you think it will be a case when a user calling stack in a SparseDataFrame wants it to be stacked as dense? The sparse keyword argument doesn't seem necessary to me. But if you think there is a reason for that, I'm happy to have it in the implementation.
> But do you think it will be a case when a user calling stack in a SparseDataFrame wants it to be stacked as dense? 

I'm having trouble coming up with a case where `.stack` densifying would be desired. @sinhrks thoughts?
this is a dupe of #14493. though those are essentially for unstack, so I guess we can leave this one.
note this is actually non-trivial. We are not simply doing ``.to_sparse()`` on the data, rather constructing it directly (which is way more efficient).
One concern is a case when stacking columns have different ``fill_value`` (a value omitted in sparse repr). In this case, the result is not efficient in sparse repr. Should raise?


Resolving patch:
diff --git a/pandas/core/categorical.py b/pandas/core/categorical.py
--- a/pandas/core/categorical.py
+++ b/pandas/core/categorical.py
@@ -124,10 +124,16 @@ def f(self, other):
     return f
 
 
-def maybe_to_categorical(array):
-    """ coerce to a categorical if a series is given """
+def _maybe_to_categorical(array):
+    """
+    Coerce to a categorical if a series is given.
+
+    Internal use ONLY.
+    """
     if isinstance(array, (ABCSeries, ABCCategoricalIndex)):
         return array._values
+    elif isinstance(array, np.ndarray):
+        return Categorical(array)
     return array
 
 
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -56,7 +56,7 @@
 
 from pandas.core.index import Index, MultiIndex, _ensure_index
 from pandas.core.indexing import maybe_convert_indices, length_of_indexer
-from pandas.core.categorical import Categorical, maybe_to_categorical
+from pandas.core.categorical import Categorical, _maybe_to_categorical
 from pandas.core.indexes.datetimes import DatetimeIndex
 from pandas.io.formats.printing import pprint_thing
 
@@ -1484,6 +1484,35 @@ def equals(self, other):
             return False
         return array_equivalent(self.values, other.values)
 
+    def _unstack(self, unstacker_func, new_columns):
+        """Return a list of unstacked blocks of self
+
+        Parameters
+        ----------
+        unstacker_func : callable
+            Partially applied unstacker.
+        new_columns : Index
+            All columns of the unstacked BlockManager.
+
+        Returns
+        -------
+        blocks : list of Block
+            New blocks of unstacked values.
+        mask : array_like of bool
+            The mask of columns of `blocks` we should keep.
+        """
+        unstacker = unstacker_func(self.values.T)
+        new_items = unstacker.get_new_columns()
+        new_placement = new_columns.get_indexer(new_items)
+        new_values, mask = unstacker.get_new_values()
+
+        mask = mask.any(0)
+        new_values = new_values.T[mask]
+        new_placement = new_placement[mask]
+
+        blocks = [make_block(new_values, placement=new_placement)]
+        return blocks, mask
+
     def quantile(self, qs, interpolation='linear', axis=0, mgr=None):
         """
         compute the quantiles of the
@@ -1712,6 +1741,38 @@ def _slice(self, slicer):
     def _try_cast_result(self, result, dtype=None):
         return result
 
+    def _unstack(self, unstacker_func, new_columns):
+        """Return a list of unstacked blocks of self
+
+        Parameters
+        ----------
+        unstacker_func : callable
+            Partially applied unstacker.
+        new_columns : Index
+            All columns of the unstacked BlockManager.
+
+        Returns
+        -------
+        blocks : list of Block
+            New blocks of unstacked values.
+        mask : array_like of bool
+            The mask of columns of `blocks` we should keep.
+        """
+        # NonConsolidatable blocks can have a single item only, so we return
+        # one block per item
+        unstacker = unstacker_func(self.values.T)
+        new_items = unstacker.get_new_columns()
+        new_placement = new_columns.get_indexer(new_items)
+        new_values, mask = unstacker.get_new_values()
+
+        mask = mask.any(0)
+        new_values = new_values.T[mask]
+        new_placement = new_placement[mask]
+
+        blocks = [self.make_block_same_class(vals, [place])
+                  for vals, place in zip(new_values, new_placement)]
+        return blocks, mask
+
 
 class NumericBlock(Block):
     __slots__ = ()
@@ -2227,7 +2288,7 @@ class CategoricalBlock(NonConsolidatableMixIn, ObjectBlock):
     def __init__(self, values, placement, fastpath=False, **kwargs):
 
         # coerce to categorical if we can
-        super(CategoricalBlock, self).__init__(maybe_to_categorical(values),
+        super(CategoricalBlock, self).__init__(_maybe_to_categorical(values),
                                                fastpath=True,
                                                placement=placement, **kwargs)
 
@@ -4167,6 +4228,38 @@ def canonicalize(block):
         return all(block.equals(oblock)
                    for block, oblock in zip(self_blocks, other_blocks))
 
+    def unstack(self, unstacker_func):
+        """Return a blockmanager with all blocks unstacked.
+
+        Parameters
+        ----------
+        unstacker_func : callable
+            A (partially-applied) ``pd.core.reshape._Unstacker`` class.
+
+        Returns
+        -------
+        unstacked : BlockManager
+        """
+        dummy = unstacker_func(np.empty((0, 0)), value_columns=self.items)
+        new_columns = dummy.get_new_columns()
+        new_index = dummy.get_new_index()
+        new_blocks = []
+        columns_mask = []
+
+        for blk in self.blocks:
+            blocks, mask = blk._unstack(
+                partial(unstacker_func,
+                        value_columns=self.items[blk.mgr_locs.indexer]),
+                new_columns)
+
+            new_blocks.extend(blocks)
+            columns_mask.extend(mask)
+
+        new_columns = new_columns[columns_mask]
+
+        bm = BlockManager(new_blocks, [new_columns, new_index])
+        return bm
+
 
 class SingleBlockManager(BlockManager):
     """ manage a single block with """
diff --git a/pandas/core/reshape/reshape.py b/pandas/core/reshape/reshape.py
--- a/pandas/core/reshape/reshape.py
+++ b/pandas/core/reshape/reshape.py
@@ -2,6 +2,7 @@
 # pylint: disable=W0703,W0622,W0613,W0201
 from pandas.compat import range, text_type, zip
 from pandas import compat
+from functools import partial
 import itertools
 import re
 
@@ -10,7 +11,7 @@
 from pandas.core.dtypes.common import (
     _ensure_platform_int,
     is_list_like, is_bool_dtype,
-    needs_i8_conversion)
+    needs_i8_conversion, is_sparse)
 from pandas.core.dtypes.cast import maybe_promote
 from pandas.core.dtypes.missing import notna
 import pandas.core.dtypes.concat as _concat
@@ -75,10 +76,15 @@ def __init__(self, values, index, level=-1, value_columns=None,
                  fill_value=None):
 
         self.is_categorical = None
+        self.is_sparse = is_sparse(values)
         if values.ndim == 1:
             if isinstance(values, Categorical):
                 self.is_categorical = values
                 values = np.array(values)
+            elif self.is_sparse:
+                # XXX: Makes SparseArray *dense*, but it's supposedly
+                # a single column at a time, so it's "doable"
+                values = values.values
             values = values[:, np.newaxis]
         self.values = values
         self.value_columns = value_columns
@@ -177,7 +183,8 @@ def get_result(self):
                                   ordered=ordered)
                       for i in range(values.shape[-1])]
 
-        return DataFrame(values, index=index, columns=columns)
+        klass = SparseDataFrame if self.is_sparse else DataFrame
+        return klass(values, index=index, columns=columns)
 
     def get_new_values(self):
         values = self.values
@@ -469,36 +476,12 @@ def unstack(obj, level, fill_value=None):
 
 
 def _unstack_frame(obj, level, fill_value=None):
-    from pandas.core.internals import BlockManager, make_block
-
     if obj._is_mixed_type:
-        unstacker = _Unstacker(np.empty(obj.shape, dtype=bool),  # dummy
-                               obj.index, level=level,
-                               value_columns=obj.columns)
-        new_columns = unstacker.get_new_columns()
-        new_index = unstacker.get_new_index()
-        new_axes = [new_columns, new_index]
-
-        new_blocks = []
-        mask_blocks = []
-        for blk in obj._data.blocks:
-            blk_items = obj._data.items[blk.mgr_locs.indexer]
-            bunstacker = _Unstacker(blk.values.T, obj.index, level=level,
-                                    value_columns=blk_items,
-                                    fill_value=fill_value)
-            new_items = bunstacker.get_new_columns()
-            new_placement = new_columns.get_indexer(new_items)
-            new_values, mask = bunstacker.get_new_values()
-
-            mblk = make_block(mask.T, placement=new_placement)
-            mask_blocks.append(mblk)
-
-            newb = make_block(new_values.T, placement=new_placement)
-            new_blocks.append(newb)
-
-        result = DataFrame(BlockManager(new_blocks, new_axes))
-        mask_frame = DataFrame(BlockManager(mask_blocks, new_axes))
-        return result.loc[:, mask_frame.sum(0) > 0]
+        unstacker = partial(_Unstacker, index=obj.index,
+                            level=level, fill_value=fill_value)
+        blocks = obj._data.unstack(unstacker)
+        klass = type(obj)
+        return klass(blocks)
     else:
         unstacker = _Unstacker(obj.values, obj.index, level=level,
                                value_columns=obj.columns,
@@ -559,7 +542,9 @@ def factorize(index):
         mask = notna(new_values)
         new_values = new_values[mask]
         new_index = new_index[mask]
-    return Series(new_values, index=new_index)
+
+    klass = type(frame)._constructor_sliced
+    return klass(new_values, index=new_index)
 
 
 def stack_multiple(frame, level, dropna=True):


Test patch:
diff --git a/pandas/tests/sparse/test_reshape.py b/pandas/tests/sparse/test_reshape.py
new file mode 100644
--- /dev/null
+++ b/pandas/tests/sparse/test_reshape.py
@@ -0,0 +1,38 @@
+import pytest
+import numpy as np
+
+import pandas as pd
+import pandas.util.testing as tm
+
+
+@pytest.fixture
+def sparse_df():
+    return pd.SparseDataFrame({0: {0: 1}, 1: {1: 1}, 2: {2: 1}})  # eye
+
+
+@pytest.fixture
+def multi_index3():
+    return pd.MultiIndex.from_tuples([(0, 0), (1, 1), (2, 2)])
+
+
+def test_sparse_frame_stack(sparse_df, multi_index3):
+    ss = sparse_df.stack()
+    expected = pd.SparseSeries(np.ones(3), index=multi_index3)
+    tm.assert_sp_series_equal(ss, expected)
+
+
+def test_sparse_frame_unstack(sparse_df):
+    mi = pd.MultiIndex.from_tuples([(0, 0), (1, 0), (1, 2)])
+    sparse_df.index = mi
+    arr = np.array([[1, np.nan, np.nan],
+                    [np.nan, 1, np.nan],
+                    [np.nan, np.nan, 1]])
+    unstacked_df = pd.DataFrame(arr, index=mi).unstack()
+    unstacked_sdf = sparse_df.unstack()
+
+    tm.assert_numpy_array_equal(unstacked_df.values, unstacked_sdf.values)
+
+
+def test_sparse_series_unstack(sparse_df, multi_index3):
+    frame = pd.SparseSeries(np.ones(3), index=multi_index3).unstack()
+    tm.assert_sp_frame_equal(frame, sparse_df)