Repository: pandas-dev/pandas
Pull request: BUG GH23744 ufuncs on DataFrame keeps dtype sparseness
- [X] closes #23744 
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
BUG: DataFrame.apply looses sparse dtype
```
In [151]: df = pd.SparseDataFrame(np.array([[0, 1, 0], [0, 0, 0], [0, 0, 1]]), 
                                  columns=['a', 'b', 'c'], default_fill_value=0)
In [152]: df2 = pd.DataFrame(df)

In [153]: df.apply(np.exp)['a']
Out[153]: 
0    1.0
1    1.0
2    1.0
Name: a, dtype: Sparse[float64, 1.0]
BlockIndex
Block locations: array([], dtype=int32)
Block lengths: array([], dtype=int32)

In [155]: df2.apply(np.exp)['a']
Out[155]: 
0    1.0
1    1.0
2    1.0
Name: a, dtype: float64
```

Hints from issue discussion:
I observe the sparse dtype being lost after the copy, even before the apply

```
>>> df = pd.SparseDataFrame(np.array([[0, 1, 0], [0, 0, 0], [0, 0, 1]]), columns=list('abc'), default_fill_value=0)
>>> df2 = pd.DataFrame(df)
>>> type(df)
<class 'pandas.core.sparse.frame.SparseDataFrame'>
>>> type(df2)
<class 'pandas.core.frame.DataFrame'>
```

`df2 = pd.SparseDataFrame(df)` results in expected behaviour.
@JustinZhengBC The `pd.DataFrame(df)` instead of `pd.SparseDataFrame(df)` is on purpose, as I was testing the sparse support in a *normal* DataFrame. That was not fully clear from the issue.

Normal DataFrames also support storing sparse columns in it:

```
In [14]: type(df2)                                                                                                                                                              
Out[14]: pandas.core.frame.DataFrame

In [15]: df2.dtypes                                                                                                                                                             
Out[15]: 
a    Sparse[int64, 0]
b    Sparse[int64, 0]
c    Sparse[int64, 0]
dtype: object
```

Commit message: BUG-23744 Fix memory usage

Patch to review:
diff --git a/doc/source/whatsnew/v0.24.0.rst b/doc/source/whatsnew/v0.24.0.rst
index b5d0532c6dfa3..e33aab0f6be45 100644
--- a/doc/source/whatsnew/v0.24.0.rst
+++ b/doc/source/whatsnew/v0.24.0.rst
@@ -1259,6 +1259,7 @@ Numeric
 - Bug in :meth:`Series.rpow` with object dtype ``NaN`` for ``1 ** NA`` instead of ``1`` (:issue:`22922`).
 - :meth:`Series.agg` can now handle numpy NaN-aware methods like :func:`numpy.nansum` (:issue:`19629`)
 - Bug in :meth:`Series.rank` and :meth:`DataFrame.rank` when ``pct=True`` and more than 2:sup:`24` rows are present resulted in percentages greater than 1.0 (:issue:`18271`)
+- Bug in :meth:`DataFrame.apply` where dtypes would lose sparseness (:issue:`23744`)
 
 Strings
 ^^^^^^^
diff --git a/pandas/core/apply.py b/pandas/core/apply.py
index 40cd952a62138..cfe65f248c6c4 100644
--- a/pandas/core/apply.py
+++ b/pandas/core/apply.py
@@ -7,7 +7,8 @@
     is_extension_type,
     is_dict_like,
     is_list_like,
-    is_sequence)
+    is_sequence,
+    is_sparse)
 from pandas.util._decorators import cache_readonly
 
 from pandas.io.formats.printing import pprint_thing
@@ -131,10 +132,13 @@ def get_result(self):
 
         # ufunc
         elif isinstance(self.f, np.ufunc):
-            with np.errstate(all='ignore'):
-                results = self.f(self.values)
-            return self.obj._constructor(data=results, index=self.index,
-                                         columns=self.columns, copy=False)
+            result = self.obj._constructor(index=self.index, copy=False)
+            for col in self.columns:
+                if is_sparse(self.obj.dtypes[col]):
+                    result[col] = self.f(self.obj[col].values)
+                else:
+                    result[col] = self.f(self.obj[col])
+            return result
 
         # broadcasting
         if self.result_type == 'broadcast':
diff --git a/pandas/tests/frame/test_apply.py b/pandas/tests/frame/test_apply.py
index c43872bfc3ddb..799694f54aad6 100644
--- a/pandas/tests/frame/test_apply.py
+++ b/pandas/tests/frame/test_apply.py
@@ -570,6 +570,16 @@ def test_apply_dup_names_multi_agg(self):
 
         tm.assert_frame_equal(result, expected)
 
+    def test_apply_keep_sparse_dtype(self):
+        # GH 23744
+        df = pd.SparseDataFrame(np.array([[0, 1, 0], [0, 0, 0], [0, 0, 1]]),
+                                columns=['a', 'b', 'c'], default_fill_value=1)
+        df2 = pd.DataFrame(df)
+
+        df = df.apply(np.exp)
+        df2 = df2.apply(np.exp)
+        tm.assert_frame_equal(df, df2)
+
 
 class TestInferOutputShape(object):
     # the user has supplied an opaque UDF where


Code review comments:
diff_hunk: @@ -131,10 +132,13 @@ def get_result(self):
 
         # ufunc
         elif isinstance(self.f, np.ufunc):
-            with np.errstate(all='ignore'):
-                results = self.f(self.values)
-            return self.obj._constructor(data=results, index=self.index,
-                                         columns=self.columns, copy=False)
+            result = self.obj._constructor(index=self.index, copy=False)
+            for col in self.columns:
+                if is_sparse(self.obj.dtypes[col]):
path: pandas/core/apply.py
text: @user1:
This is closer, but I don't think it should be special casing sparse. The bug applies to any extension array.

https://github.com/pandas-dev/pandas/pull/23293 is fixing ufuncs on series. So extracting `.values` won't be necessary. I'd recommend waiting for https://github.com/pandas-dev/pandas/pull/23293 and seeing if you can just do `result[col] = self.f(self.obj[col])`.

@author:
I removed the sparse check in the last commit because it turns out calling `values` on normal columns is also acceptable (should generalize to other array types as well). I do agree that it would look neater if calling `values` wasn't required at all though.