ENH: ExtensionArray.setitem
State: closed

xref https://github.com/pandas-dev/pandas/issues/19696

Adds `ExtensionBlock.__setitem__`.

This is only tested for with `DecimalArray` and `Categorical`. Supporting something like `JSONArray` where the "scalar" elements are actual sequences runs into issues elsewhere in internals. Maybe someday we can support that.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19907?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@0bd8a5a`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/19907/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/19907?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master   #19907   +/-   ##
=========================================
  Coverage          ?   91.84%           
=========================================
  Files             ?      153           
  Lines             ?    49289           
  Branches          ?        0           
=========================================
  Hits              ?    45269           
  Misses            ?     4020           
  Partials          ?        0
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.23% <100%> (?)` | |
| #single | `41.9% <42.1%> (?)` | |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/19907?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/19907/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `95.53% <100%> (ø)` | |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/19907/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.16% <100%> (ø)` | |
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/19907/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `93.08% <100%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19907?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19907?src=pr&el=footer). Last update [0bd8a5a...3cbe078](https://codecov.io/gh/pandas-dev/pandas/pull/19907?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

This currently fails when the repr is triggered, not in the setitem as it should

```python
df = pd.DataFrame({"B": [1, 2, 3]})
df['A'] = IPArray([1, 2])
df._repr_html_()
```

What would we expect the type of this to be

```python
df = pd.DataFrame({"A": IPArray([1, 2, 3])})
df.loc[:, 'A'] = 1
```

This results in an integer dtype column. I think that's correct, right? The previous block is being entirely replaced? Then we have symmetry with `df['A'] = 1`.

One unhandled case

```python
In [3]: s = pd.Series(IPArray([1, 2]), index=[(0, 1), (1, 2)])

In [4]: s[(0, 1)] = 2
```

```pytb
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/sandbox/pandas-ip/pandas/pandas/core/series.py in setitem(key, value)
    871             try:
--> 872                 self._set_with_engine(key, value)
    873                 return

~/sandbox/pandas-ip/pandas/pandas/core/series.py in _set_with_engine(self, key, value)
    930         try:
--> 931             self.index._engine.set_value(values, key, value)
    932             return

TypeError: Argument 'arr' has incorrect type (expected numpy.ndarray, got IPArray)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-4-6c3a0d7e7408> in <module>()
----> 1 s[(0, 1)] = 2

~/sandbox/pandas-ip/pandas/pandas/core/series.py in __setitem__(self, key, value)
    922         # do the setitem
    923         cacher_needs_updating = self._check_is_chained_assignment_possible()
--> 924         setitem(key, value)
    925         if cacher_needs_updating:
    926             self._maybe_update_cacher()

~/sandbox/pandas-ip/pandas/pandas/core/series.py in setitem(key, value)
    904                 if (isinstance(key, tuple) and
    905                         not isinstance(self.index, MultiIndex)):
--> 906                     raise ValueError("Can only tuple-index with a MultiIndex")
    907
    908                 # python 3 type errors should be raised

ValueError: Can only tuple-index with a MultiIndex
```

The first exception is the problem. That cython method is expecting an ndarray, but we're giving it an ndarray or extensionarray.
Running asv with  43dfd7d   now.
The last commit allowing EAs in the `index.engine.set*` had trouble with us using DatetimeIndex as a storage container for Datetime with TZ. We tried to directly modify the DatetimeIndex with `values[key] = value`. Previously this wasn't hit because we raised a TypeError earlier on, sending us down a different code path. I think that someday we'll want something like 43dfd7d, but not right now.
Hmm, perhaps it's worth just punting on `__setitem__` on EAs with tuples for now. It's already failing on master for non-ndarrays

```python
In [1]: import pandas as pd

In [2]: arr = pd.date_range('2017', periods=4, tz='US/Eastern')

In [3]: s = pd.Series(arr, index=[(0, 1), (0, 2), (0, 3), (0, 4)])

In [4]: s[(0, 1)] = float('NaN')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/sandbox/pandas-ip/pandas/pandas/core/series.py in setitem(key, value)
    871             try:
--> 872                 self._set_with_engine(key, value)
    873                 return

~/sandbox/pandas-ip/pandas/pandas/core/series.py in _set_with_engine(self, key, value)
    930         try:
--> 931             self.index._engine.set_value(values, key, value)
    932             return

TypeError: Argument 'arr' has incorrect type (expected numpy.ndarray, got DatetimeIndex)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-4-0a0fdcf8dcd6> in <module>()
----> 1 s[(0, 1)] = float('NaN')

~/sandbox/pandas-ip/pandas/pandas/core/series.py in __setitem__(self, key, value)
    922         # do the setitem
    923         cacher_needs_updating = self._check_is_chained_assignment_possible()
--> 924         setitem(key, value)
    925         if cacher_needs_updating:
    926             self._maybe_update_cacher()

~/sandbox/pandas-ip/pandas/pandas/core/series.py in setitem(key, value)
    904                 if (isinstance(key, tuple) and
    905                         not isinstance(self.index, MultiIndex)):
--> 906                     raise ValueError("Can only tuple-index with a MultiIndex")
    907
    908                 # python 3 type errors should be raised

ValueError: Can only tuple-index with a MultiIndex
```

With this hack, I can get it working

```python
diff --git a/pandas/core/series.py b/pandas/core/series.py
index e48012420..cd412844e 100644
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -869,6 +869,18 @@ class Series(base.IndexOpsMixin, generic.NDFrame):
 
         def setitem(key, value):
             try:
+                if self._data.blocks[0].is_extension:
+                    raise TypeError
                 self._set_with_engine(key, value)
                 return
             except com.SettingWithCopyError:
```

I don't think that's worthwhile including to support tuples in the index. I'd prefer to see a proper fix for whatever is going wrong there.
> perhaps it's worth just punting on __setitem__ on EAs with tuples for now. It's already failing on master for non-ndarrays

No problem with that for now.

> [on df.loc[:, 'A'] = 1] This results in an integer dtype column. I think that's correct, right? The previous block is being entirely replaced? Then we have symmetry with df['A'] = 1.

I thought we somehow handled `df.loc[:, 'A'] = 1` and `df['A'] = 1` differently right now, but it seems that also in case of `df.loc[:, 'A'] = 1`  with 'normal' dtypes like int and float, the full block gets replaced. In that case, yes, I think we want to do the same here.


I had to remove the `_values` alias for now, as factorize uses the `getattr(values, '_values', values)` pattern to unbox a Series / Index.
can you rebase
I think all the tests have passed an there's no merge conflict.

On Sun, Apr 1, 2018 at 9:26 AM, Jeff Reback <notifications@github.com>
wrote:

> can you rebase
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/19907#issuecomment-377790457>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIrvrkqmLuCjxyufLE-LJRyKtIlhVks5tkOOMgaJpZM4STfE9>
> .
>

@TomAugspurger 

- let's rebase one more time.
- doc-strings looks good, but pls give a once over
- I would collect all the issues that are EA and add onto the sub-section in whatsnew (just list them)

merge when ready- 
thanks @TomAugspurger