Repository: pandas-dev/pandas

Issue:
BUG: setting a sparse column in a frame buggy
from [SO](http://stackoverflow.com/questions/25517342/adding-a-sparse-series-to-a-dense-dataframe-in-pandas)

thought this was well tested.....

```
df = pd.DataFrame({'c_1':['a', 'b', 'c'], 'n_1': [1., 2., 3.]})
df['new_column'] = pd.Series([0, 0, 1]).to_sparse(fill_value=0)
# AssertionError: Shape of new values must be compatible with manager shape
```



Discussion:
cc @immerrr

Let me check that...

The issue is in `df._sanitize_column` that returns a 2d dense array with non-fill elements:

``` python
In [9]: df = pd.DataFrame({'c_1': list('abc')})

In [10]: sp_col = pd.Series([0,0,1]).to_sparse(fill_value=0)

In [11]: df._sanitize_column('n', sp_col)
Out[11]: array([[1]])
```

Hacking sanitize column is easy, but it uncovers yet another issue with ndarray subclassing:

``` python
In [54]: sp_arr = pd.SparseArray([0,0,1], fill_value=0)

In [55]: sp_arr
Out[55]: 
[0, 0, 1.0]
Fill: 0
IntIndex
Indices: array([2], dtype=int32)


In [56]: np.asarray(sp_arr)
Out[56]: array([ 1.])
```

This happens because `np.asarray` checks on C level that `sp_arr` provides PEP3118 buffer interface (which ndarray does) and uses that representation which contains only non-fill elements. Which is unfortunate because it can not be overridden by inheriting class on Python level (see [python issue](http://bugs.python.org/issue13797)).

yep, prob `SpareseArray`  just needs to be tested for (similar to what just did with `Categorical`),
needs to be passed thru directly.


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -43,6 +43,7 @@
 from pandas.compat import(range, zip, lrange, lmap, lzip, StringIO, u,
                           OrderedDict, raise_with_traceback)
 from pandas import compat
+from pandas.sparse.array import SparseArray
 from pandas.util.decorators import deprecate, Appender, Substitution, \
     deprecate_kwarg
 
@@ -2164,8 +2165,8 @@ def reindexer(value):
             value = np.repeat(value, len(self.index)).astype(dtype)
             value = com._possibly_cast_to_datetime(value, dtype)
 
-        # return categoricals directly
-        if isinstance(value, Categorical):
+        # return unconsolidatables directly
+        if isinstance(value, (Categorical, SparseArray)):
             return value
 
         # broadcast across multiple columns if necessary


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -1791,6 +1791,20 @@ def test_getitem_ix_float_duplicates(self):
         expect = df.iloc[[1, -1], 0]
         tm.assert_series_equal(df.loc[0.2, 'a'], expect)
 
+    def test_setitem_with_sparse_value(self):
+        # GH8131
+        df = pd.DataFrame({'c_1':['a', 'b', 'c'], 'n_1': [1., 2., 3.]})
+        sp_series = pd.Series([0, 0, 1]).to_sparse(fill_value=0)
+        df['new_column'] = sp_series
+        tm.assert_series_equal(df['new_column'], sp_series)
+
+    def test_setitem_with_unaligned_sparse_value(self):
+        df = pd.DataFrame({'c_1':['a', 'b', 'c'], 'n_1': [1., 2., 3.]})
+        sp_series = (pd.Series([0, 0, 1], index=[2, 1, 0])
+                     .to_sparse(fill_value=0))
+        df['new_column'] = sp_series
+        tm.assert_series_equal(df['new_column'], pd.Series([1, 0, 0]))
+
 
 _seriesd = tm.getSeriesData()
 _tsd = tm.getTimeSeriesData()