BUG GH23744 ufuncs on DataFrame keeps dtype sparseness
State: closed

- [X] closes #23744 
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Sparse
description: Sparse Data Type
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @JustinZhengBC! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`pandas/core/apply.py`](https://github.com/JustinZhengBC/pandas/blob/c7ea063fee1adff6d6cdcb0d5932fb476ebd4dcd/pandas/core/apply.py) !

 - There are no PEP8 issues in the file [`pandas/tests/frame/test_apply.py`](https://github.com/JustinZhengBC/pandas/blob/c7ea063fee1adff6d6cdcb0d5932fb476ebd4dcd/pandas/tests/frame/test_apply.py) !


# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23755?src=pr&el=h1) Report
> Merging [#23755](https://codecov.io/gh/pandas-dev/pandas/pull/23755?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/dc0674d66c1e8cd1917a6ec28f0b5a6d3e23c5e6?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23755/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23755?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #23755      +/-   ##
==========================================
+ Coverage   92.29%   92.31%   +0.01%     
==========================================
  Files         161      161              
  Lines       51498    51483      -15     
==========================================
- Hits        47530    47525       -5     
+ Misses       3968     3958      -10
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.7% <100%> (+0.01%)` | :arrow_up: |
| #single | `42.43% <0%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23755?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/apply.py](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXBwbHkucHk=) | `98.61% <100%> (ø)` | :arrow_up: |
| [pandas/plotting/\_misc.py](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19taXNjLnB5) | `38.68% <0%> (-0.31%)` | :arrow_down: |
| [pandas/core/arrays/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lcy5weQ==) | `98.37% <0%> (-0.14%)` | :arrow_down: |
| [pandas/core/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvb3BzLnB5) | `94.14% <0%> (-0.14%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `97.63% <0%> (-0.05%)` | :arrow_down: |
| [pandas/core/dtypes/dtypes.py](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2R0eXBlcy5weQ==) | `95.59% <0%> (-0.03%)` | :arrow_down: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.34% <0%> (-0.02%)` | :arrow_down: |
| [pandas/core/arrays/sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS5weQ==) | `91.93% <0%> (-0.02%)` | :arrow_down: |
| [pandas/core/reshape/tile.py](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS90aWxlLnB5) | `94.73% <0%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.02% <0%> (ø)` | :arrow_up: |
| ... and [10 more](https://codecov.io/gh/pandas-dev/pandas/pull/23755/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23755?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23755?src=pr&el=footer). Last update [dc0674d...551ced8](https://codecov.io/gh/pandas-dev/pandas/pull/23755?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Can't reproduce the one failing test, and I don't think it's related
edit: never mind
does this cover all of the examples in both issues?
@jreback sorry, does not fix the second issue, edited
can you merge master
Considering #23875, I think we will need a for loop in `get_result` to check for all the different dtypes. This PR only covers the sparse case but the categorical case could be covered as well by adding:
```
elif is_categorical(self.obj.dtypes[col]):
    before = self.obj.dtypes[col].categories
    after = self.f(before)
    map = dict(zip(before, after))
    result = self.obj[col].replace(map).astype('category')
```
@jreback, general question: how firm should we be on preserving blockwise application of things? I don't think we should give that up yet (as the current PR does).
@TomAugspurger I agree generally with your point about block-wise application. Sparse however I think is already a single block, so there should be no difference in this approach.
My concern is about dataframes with non-sparse columns. Previously, for an all-float DataFrame, `DataFrame.apply(np.abs)` would (I think) call `np.abs` once. Now we'll be calling it once per column, even if there are no sparse columns.
> My concern is about dataframes with non-sparse columns. Previously, for an all-float DataFrame, DataFrame.apply(np.abs) would (I think) call np.abs once. Now we'll be calling it once per column, even if there are no sparse columns.

this could actually dispatch to the block impl then (prob better).

@JustinZhengBC  basically you can call

``self._data.apply('apply', .....)`` I think
Ah, I forgot we had `Block.apply`. Yes, this should hopefully work

```python
In [20]: df = pd.DataFrame({"A": [1, 2, 3], "B": pd.SparseArray([0, 1, 2])})

In [21]: df._data.apply('apply', func=np.sin)
Out[21]:
BlockManager
Items: Index(['A', 'B'], dtype='object')
Axis 1: RangeIndex(start=0, stop=3, step=1)
FloatBlock: slice(0, 1, 1), 1 x 3, dtype: float64
ExtensionBlock: slice(1, 2, 1), 1 x 3, dtype: Sparse[float64, 0.0]
```
@jreback you're right, it works. I also changed switched up the columns in the test so order preservation is tested
@jreback green
thanks @JustinZhengBC