ENH: SparseDataFrame/SparseSeries value assignment
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/21818
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Works by as much as possible using `SparseBlock.setitem()` which calls into `SparseArray.set_values()`, which returns a new (replacement) `SparseArray` object.

Labels:
name: Sparse
description: Sparse Data Type

Comments:
Hello @kernc! Thanks for updating the PR.





















Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on August 14, 2018 at 01:24 Hours UTC
@kernc : A couple of points:

* Travis and company are complaining pretty loudly about some failing tests.  Could you address those?
* Could you point to the issue number that you're fixing?  It looks like you pointed to your own PR.
Handling the warnings now. What do I do when there are two warnings to catch with `assert_produces_warning` (FutureWarning and PerformanceWarning)? Should I even emit PerformanceWarning as I do?
the FutureWarning is prob enough
The FutureWarning is about the recent `.set_value()` deprecation, PerformanceWarning is about setting values on a sparse structure (makes a copy of data, sometimes densifies data).

The problem with `assert_produces_warning` is that if any other warnings are encountered, the whole thing fails.

Can I amend `assert_produces_warning` to also accept a tuple of expected warning types? Alternatively, I can add a flag `error_unexpected=True` which, when set to False, won't error on _additional_ unexpected warnings?
can you rebase / update
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=h1) Report
> Merging [#17785](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/3493abaa9c47e39b410752833c901fd27f5b3a76?src=pr&el=desc) will **decrease** coverage by `0.21%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17785/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17785      +/-   ##
==========================================
- Coverage   91.42%   91.21%   -0.22%     
==========================================
  Files         163      163              
  Lines       50064    50036      -28     
==========================================
- Hits        45773    45640     -133     
- Misses       4291     4396     +105
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.02% <100%> (-0.2%)` | :arrow_down: |
| #single | `40.22% <21.27%> (-0.2%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.98% <ø> (-0.03%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.76% <100%> (-0.14%)` | :arrow_down: |
| [pandas/core/sparse/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL3Nlcmllcy5weQ==) | `95.84% <100%> (+0.56%)` | :arrow_up: |
| [pandas/core/sparse/array.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2FycmF5LnB5) | `91.96% <100%> (-0.33%)` | :arrow_down: |
| [pandas/core/sparse/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2ZyYW1lLnB5) | `94.92% <100%> (+0.14%)` | :arrow_up: |
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `94.26% <100%> (-0.28%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `92.09% <0%> (-3.64%)` | :arrow_down: |
| [pandas/core/indexes/range.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9yYW5nZS5weQ==) | `92.83% <0%> (-2.83%)` | :arrow_down: |
| [pandas/util/\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2RlY29yYXRvcnMucHk=) | `78% <0%> (-2.71%)` | :arrow_down: |
| ... and [58 more](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=footer). Last update [3493aba...137304e](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=h1) Report
> Merging [#17785](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/ffae158749c9b72720a7fe3295d323082a5fe420?src=pr&el=desc) will **decrease** coverage by `0.16%`.
> The diff coverage is `93.33%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17785/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17785      +/-   ##
==========================================
- Coverage   92.08%   91.91%   -0.17%     
==========================================
  Files         169      164       -5     
  Lines       50706    50018     -688     
==========================================
- Hits        46691    45975     -716     
- Misses       4015     4043      +28
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.3% <93.33%> (-0.19%)` | :arrow_down: |
| #single | `42.14% <21.66%> (-0.19%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.11% <ø> (+0.39%)` | :arrow_up: |
| [pandas/core/sparse/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL3Nlcmllcy5weQ==) | `95.77% <100%> (+0.55%)` | :arrow_up: |
| [pandas/core/sparse/array.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2FycmF5LnB5) | `91.81% <100%> (+0.43%)` | :arrow_up: |
| [pandas/core/sparse/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2ZyYW1lLnB5) | `94.91% <100%> (+0.12%)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `85.44% <100%> (-0.62%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.19% <100%> (-0.05%)` | :arrow_down: |
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `95.2% <77.77%> (ø)` | |
| [pandas/util/\_doctools.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2RvY3Rvb2xzLnB5) | `0% <0%> (-12.88%)` | :arrow_down: |
| [pandas/core/reshape/util.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS91dGlsLnB5) | `90.32% <0%> (-9.68%)` | :arrow_down: |
| [pandas/util/\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2RlY29yYXRvcnMucHk=) | `82.25% <0%> (-9.09%)` | :arrow_down: |
| ... and [74 more](https://codecov.io/gh/pandas-dev/pandas/pull/17785/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=footer). Last update [ffae158...4779c36](https://codecov.io/gh/pandas-dev/pandas/pull/17785?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Of course; thanks for coming around to it!

There's still the issue of two distinct Warnings being emitted but `assert_produces_warning` only tests for one, raising on the other. Leaning towards extending `assert_produces_warning` for such a case.

whatsnew is also missing as this is all highly preliminary. Thanks.
The tests in fact pass (only the linter complains about unused imports I don't actually intend to keep). @jreback I think this PR while working is full of incorrect solutions so I'd really appreciate if you could have a look-over.
closing as stale. if you want to continue working, pls ping.

@kernc happy to have sparse fixes. pls ping if you want to fixup.
This was a planned enhancement. I guess I should wait for https://github.com/pandas-dev/pandas/issues/19239 (https://github.com/pandas-dev/pandas/pull/19174). If `SparseArray` survives, the bits are there.
@kernc actually no, #19174 is a long ways off. I would certainly like to do this *before* if at all possible. that way the tests (and code) are there. I am happy to have this. let's try to get over the finish line.
Ok. Then please give it a first pass as it is! 😉 
FWIW, this has been tidied up.
definitly want to fix this up, but closing this PR as stale
@jreback : Besides patching failing tests (and rebasing I imagine), was there anything else that needed to be addressed before merging?
its looking very complicated and adding lots of code. happy for you to take a look though.
How can this become un-stale? **Nobody ever took a first look at it**, despite it being feature complete and working+passing for quite a while.

> was there anything else that needed to be addressed 

My query-comments needed to be addressed!  :confused:

> its looking very complicated and adding lots of code. 

It's in fact _very_ simple: uses `SparseBlock.setitem()` which calls into `SparseArray.set_values()`, which always returns a new (replacement) SparseArray object. The only code added, really, is tests!

And it's all very neatly sorted in distinct commits. I don't get the standards here.

If you think this can be done in significantly fewer and/or less complicated lines ... well, I didn't.
@kernc : We can certainly re-open this PR.  Could you correct the test failures and rebase?
So the travis tests pass. Now if anyone could have a first look over.

I'm particularly interested in `_try_coerce_result`, `_can_hold_element`, extending of `assert_produces_warning` ... i.e. anything an insider finds obviously wrong or lacking.
@jreback please have a look. It's all green. :pray: The first four commits are where the magic is, the rest was mostly figuring out warnings in CI tests.
@jreback Please have another look. The docstrings for the two new override methods have been copied nearly verbatim from the superclass, without clean-up. Also the `tm.assert_produces_warning()` addition was done simpler in another way.
This PR is likely obsoleted by https://github.com/pandas-dev/pandas/issues/21978 and its PR https://github.com/pandas-dev/pandas/pull/22325. Understood and approved.
I’m hoping to steal pieces from this once #22325 is in. 
@TomAugspurger can this PR build upon the SparseArray work or has that superseded this PR? 
SparseArray doesn't support setting values yet. I haven't had a chance to
into this. I think the core of the approach in
https://github.com/pandas-dev/pandas/pull/17785/files#diff-71caf9627e9687e837e4b1f86ecc6271R373
should
still be valid on top of SparseArray.

On Mon, Nov 12, 2018 at 11:56 PM Matthew Roeschke <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> can this PR build upon
> the SparseArray work or has that superseded this PR?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/17785#issuecomment-438146776>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIg9fUZocG2EOwoSJyj3PmuI0Btuqks5uul8WgaJpZM4Pt5Os>
> .
>

Closing based off previous comments - seems like this PR is dead (though may inspire others).

@kernc if you want to reopen please ping