BUG: Redefine IndexOpsMixin.size, fix #25580.
State: closed



- [x] closes #25580
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=h1) Report
> Merging [#25584](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/74a9ae309d057d5da50717ffb18fd3dd9212fad7?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25584/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25584      +/-   ##
==========================================
+ Coverage   91.26%   91.26%   +<.01%     
==========================================
  Files         173      173              
  Lines       52966    52968       +2     
==========================================
+ Hits        48337    48340       +3     
+ Misses       4629     4628       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.83% <100%> (ø)` | :arrow_up: |
| #single | `41.7% <100%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `97.76% <100%> (ø)` | :arrow_up: |
| [pandas/plotting/\_core.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb3JlLnB5) | `83.53% <0%> (-0.05%)` | :arrow_down: |
| [pandas/core/config.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnLnB5) | `87% <0%> (-0.05%)` | :arrow_down: |
| [pandas/core/arrays/sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS5weQ==) | `92.17% <0%> (ø)` | :arrow_up: |
| [pandas/core/groupby/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9iYXNlLnB5) | `91.83% <0%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.79% <0%> (ø)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `93.65% <0%> (ø)` | :arrow_up: |
| [pandas/io/feather\_format.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2ZlYXRoZXJfZm9ybWF0LnB5) | `89.74% <0%> (ø)` | :arrow_up: |
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `97.2% <0%> (ø)` | :arrow_up: |
| ... and [5 more](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=footer). Last update [74a9ae3...8b6a6f4](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=h1) Report
> Merging [#25584](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/74a9ae309d057d5da50717ffb18fd3dd9212fad7?src=pr&el=desc) will **decrease** coverage by `49.54%`.
> The diff coverage is `50%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25584/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #25584       +/-   ##
===========================================
- Coverage   91.26%   41.71%   -49.55%     
===========================================
  Files         173      173               
  Lines       52966    52968        +2     
===========================================
- Hits        48337    22094    -26243     
- Misses       4629    30874    +26245
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `41.71% <50%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `43.85% <50%> (-52.47%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.36%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.15%)` | :arrow_down: |
| ... and [131 more](https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=footer). Last update [74a9ae3...3f28f1e](https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Do we just add a size property to the EA interface?

```
@property
def size(self):
    return (len(self),)
```

It seems like Series.size expects it.

On Thu, Mar 7, 2019 at 2:31 AM codecov[bot] <notifications@github.com>
wrote:

> Codecov <https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=h1>
> Report
>
> Merging #25584
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=desc> into
> master
> <https://codecov.io/gh/pandas-dev/pandas/commit/74a9ae309d057d5da50717ffb18fd3dd9212fad7?src=pr&el=desc>
> will *decrease* coverage by 49.54%.
> The diff coverage is 50%.
>
> [image: Impacted file tree graph]
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=tree>
>
> @@             Coverage Diff             @@
> ##           master   #25584       +/-   ##
>
> ===========================================
> - Coverage   91.26%   41.71%   -49.55%
>
> ===========================================
>
>   Files         173      173
>
>   Lines       52966    52968        +2
>
> ===========================================
> - Hits        48337    22094    -26243
> - Misses       4629    30874    +26245
>
> Flag Coverage Δ
> #multiple ?
> #single 41.71% <50%> (-0.01%) ⬇️
> Impacted Files
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=tree> Coverage
> Δ
> pandas/core/arrays/integer.py
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=> 43.85%
> <50%> (-52.47%) ⬇️
> pandas/io/formats/latex.py
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=> 0%
> <0%> (-100%) ⬇️
> pandas/core/categorical.py
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=> 0%
> <0%> (-100%) ⬇️
> pandas/io/sas/sas_constants.py
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5> 0%
> <0%> (-100%) ⬇️
> pandas/tseries/plotting.py
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=> 0%
> <0%> (-100%) ⬇️
> pandas/tseries/converter.py
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5> 0%
> <0%> (-100%) ⬇️
> pandas/io/formats/html.py
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==> 0%
> <0%> (-99.36%) ⬇️
> pandas/core/groupby/categorical.py
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==> 0%
> <0%> (-95.46%) ⬇️
> pandas/io/sas/sas7bdat.py
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==> 0%
> <0%> (-91.17%) ⬇️
> pandas/io/sas/sas_xport.py
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=> 0%
> <0%> (-90.15%) ⬇️
> ... and 131 more
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584/diff?src=pr&el=tree-more>
> ------------------------------
>
> Continue to review full report at Codecov
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=continue>.
>
> *Legend* - Click here to learn more
> <https://docs.codecov.io/docs/codecov-delta>
> Δ = absolute <relative> (impact), ø = not affected, ? = missing data
> Powered by Codecov
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=footer>.
> Last update 74a9ae3...3f28f1e
> <https://codecov.io/gh/pandas-dev/pandas/pull/25584?src=pr&el=lastupdated>.
> Read the comment docs <https://docs.codecov.io/docs/pull-request-comments>
> .
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/25584#issuecomment-470433108>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIveYINIfkMJEfHehzSUm4zEwPrfNks5vUM5qgaJpZM4bikg6>
> .
>

@sighingnow can you avoid force pushing? Makes it a bit easier to review incrementally. It'd be good to include a release note for the bug this is fixing too.

@jorisvandenbossche do you remember if we discussed a `size` property on EA? It's not standard on other arrays in the ecosystem (none of array.array, memoryview, pyarrow define it). Maybe just re-define `IndexOpsMixin.size` to be `(len(self._values),)`?

Noted, thanks @TomAugspurger.

The `size` method should return an integer, rather than the shape `(len(self._values),)`.
Whoops, yes it should return an integer (I was mixing it up with shape).

That said, in the name of keeping a smaller API footprint, I'm leaning towards redefining IndexOpsMixin.size to use `len(self._values)`. We'll need to update MultiIndex.size for this change, since that's the only 2-D index.

I believe that this will work for MultiIndex (have to be careful about unused levels)

```python
def size(self):
    return np.prod(self.remove_unused_levels().levshape)
```
I agree to move redefine `size` in `IndexOpsMixin` to avoid adding a new API, but I think the `size` for `MultiIndex` above is not correct.

~~~python
>>> a = pd.MultiIndex.from_arrays(np.array([[1,2,3], [4,5,6], [7,8,9]]))
>>> a.size
3
>>> np.prod(a.remove_unused_levels().levshape)
27
~~~

I just want to confirm whether current behavior of `size` for `MultiIndex` is correct? (I'm a newcomer for pandas.) Thanks!

Edit: if redefine `size` in `IndexOpsMixin` as `len(self._values)`, the behavior for `MultiIndex` is consistent with current implementation.
Oh, hmm that's surprising. But I don't think we can easily change it at this point... I guess that means that MultiIndex.size can continue to use IndexOpsMixin.size.

+1 on for now not adding `size` to ExtensionArray. It is always easy to add it later if there are good use cases for it.

The `size` of a MultiIndex is consistent with its `shape` and `ndim`, ànd with how it is represented as an array:

```
In [22]: a = pd.MultiIndex.from_arrays(np.array([[1,2,3], [4,5,6], [7,8,9]]))                                                                                                   

In [23]: a.shape                                                                                                                                                                
Out[23]: (3,)

In [24]: a.ndim                                                                                                                                                                 
Out[24]: 1

In [25]: a.size                                                                                                                                                                 
Out[25]: 3

In [26]: np.asarray(a)                                                                                                                                                          
Out[26]: array([(1, 4, 7), (2, 5, 8), (3, 6, 9)], dtype=object)
```

so a MultiIndex is regarded as a 1D object, not 2D (for sure can be confusing :-))
I think we should add .size to ea generally

this has come a number of times (and to be honest it’s a trivial addition to the base class )


lgtm. though something failing.
It's the same sparse failure that should be fixed on master.
@sighingnow thanks!
MeeseeksDev doesn't seem to be picking this one up (and I also assume there will be a conflict in the whatsnew), so will do a manual backport
@jorisvandenbossche many thanks for helping me get this PR merged into master! Sorry for the late response.

BTW, I'm wondering if there were conflicts between pull request and master, how should I resolve that? Should I merge the master into the pull request, or rebase the pull request on top of master?
@sighingnow you should merge master into the branch
Noted, thanks! @WillAyd