BUG: Cast ExtensionArray to numpy ndarray before plot
State: closed

Cast `ExtensionArray`(eg. `IntegerArray`) to `np.ndarray` before send to matplotlib to plot, fixes #25587.

- [x] closes #25587
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Visualization
description: plotting
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=h1) Report
> Merging [#25590](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/74a9ae309d057d5da50717ffb18fd3dd9212fad7?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `80%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25590/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25590      +/-   ##
==========================================
+ Coverage   91.26%   91.26%   +<.01%     
==========================================
  Files         173      173              
  Lines       52966    52971       +5     
==========================================
+ Hits        48337    48343       +6     
+ Misses       4629     4628       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.83% <80%> (ø)` | :arrow_up: |
| #single | `41.7% <20%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/plotting/\_core.py](https://codecov.io/gh/pandas-dev/pandas/pull/25590/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb3JlLnB5) | `83.64% <80%> (+0.05%)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25590/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.66% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=footer). Last update [74a9ae3...b3445fc](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=h1) Report
> Merging [#25590](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/998e1de593d6bbf11c985eae9243b83f4710c1c6?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25590/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25590      +/-   ##
==========================================
+ Coverage   91.25%   91.25%   +<.01%     
==========================================
  Files         172      172              
  Lines       52973    52976       +3     
==========================================
+ Hits        48338    48341       +3     
  Misses       4635     4635
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.82% <100%> (ø)` | :arrow_up: |
| #single | `41.74% <0%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/plotting/\_core.py](https://codecov.io/gh/pandas-dev/pandas/pull/25590/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb3JlLnB5) | `83.64% <100%> (+0.1%)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25590/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88.98% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=footer). Last update [998e1de...3afc9eb](https://codecov.io/gh/pandas-dev/pandas/pull/25590?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Yeah, ignoring those plots for now seems fine.

On Fri, Mar 8, 2019 at 8:43 PM Tao He <notifications@github.com> wrote:

> *@sighingnow* commented on this pull request.
> ------------------------------
>
> In pandas/plotting/_core.py
> <https://github.com/pandas-dev/pandas/pull/25590#discussion_r263982950>:
>
> > @@ -574,6 +576,13 @@ def _get_xticks(self, convert_period=False):
>
>      @classmethod
>      def _plot(cls, ax, x, y, style=None, is_errorbar=False, **kwds):
> +        # GH25587: cast ExtensionArray of pandas (IntegerArray, etc.) to
>
> After move the conversion to _compute_plot_data, kde and area plots still
> don't work because we cast IntegerArray to nd.array(dtype=object), but kde
> and area needs dtype that can be used to ufunc add of numpy. Could we
> ignore the tests for kde and area and open a new issue to track that
> trouble?
> The traceback of `kde`
>
>   File "D:\Open\pandas\pandas\plotting\_core.py", line 1474, in _plot
>     gkde = gaussian_kde(y, bw_method=bw_method)
>   File "D:\Program\Conda\lib\site-packages\scipy\stats\kde.py", line 208, in __init__
>     self.set_bandwidth(bw_method=bw_method)
>   File "D:\Program\Conda\lib\site-packages\scipy\stats\kde.py", line 540, in set_bandwidth
>     self._compute_covariance()
>   File "D:\Program\Conda\lib\site-packages\scipy\stats\kde.py", line 551, in _compute_covariance
>     aweights=self.weights))
>   File "D:\Program\Conda\lib\site-packages\numpy\lib\function_base.py", line 2427, in cov
>     avg, w_sum = average(X, axis=1, weights=w, returned=True)
>   File "D:\Program\Conda\lib\site-packages\numpy\lib\function_base.py", line 419, in average
>     scl = wgt.sum(axis=axis, dtype=result_dtype)
>   File "D:\Program\Conda\lib\site-packages\numpy\core\_methods.py", line 36, in _sum
>     return umr_sum(a, axis, dtype, out, keepdims, initial)
> TypeError: No loop matching the specified signature and casting was found for ufunc add
>
> The traceback of `area`
>
>   File "D:\Open\pandas\pandas\plotting\_core.py", line 252, in generate
>     self._make_plot()
>   File "D:\Open\pandas\pandas\plotting\_core.py", line 1000, in _make_plot
>     **kwds)
>   File "D:\Open\pandas\pandas\plotting\_core.py", line 1151, in _plot
>     cls._update_stacker(ax, stacking_id, y)
>   File "D:\Open\pandas\pandas\plotting\_core.py", line 1083, in _update_stacker
>     ax._stacker_pos_prior[stacking_id] += values
> TypeError: ufunc 'add' output (typecode 'O') could not be coerced to provided output parameter (typecode 'd') according to the casting rule ''same_kind''
>
> —
> You are receiving this because your review was requested.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/25590#discussion_r263982950>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIvIcdLkUY6mBqeRC7WEZe7yGwcLMks5vUx_RgaJpZM4bjP6l>
> .
>

Fix the lint error and add more tests for different kinds of plotting.

@TomAugspurger I'm wondering how should I resolve the conflicts between this PR and master? Merge or rebase and push with `--force`?

Thanks in advance!
I don't use kde and area plots so I am fine with the current PR. Hopefully it can make it before the next release.
Something like `df = df.assign(**dict(zip(df.columns, values)))`  should
work.

On Wed, Mar 13, 2019 at 10:06 PM Tao He <notifications@github.com> wrote:

> *@sighingnow* commented on this pull request.
> ------------------------------
>
> In pandas/plotting/_core.py
> <https://github.com/pandas-dev/pandas/pull/25590#discussion_r265405834>:
>
> > @@ -365,6 +366,12 @@ def _compute_plot_data(self):
>
>          if is_empty:
>
>              raise TypeError('no numeric data to plot')
>
>
>
> +        # GH25587: cast ExtensionArray of pandas (IntegerArray, etc.) to
>
>
> .assign still doesn't work (if I don‘’t miss something). .assign requires
> kwargs, which is a dict of {str. callable or Series}, however in many
> dataframes the column label may not be a string (like
> pd.DataFrame([1,2,3])).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/25590#discussion_r265405834>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIlT3NBvXCkZ6r-N0DjPezf1VXDvgks5vWby4gaJpZM4bjP6l>
> .
>

@TomAugspurger thanks for the suggestion, but we would still get `TypeError: assign() keywords must be strings`.
Huh, sorry thought that worked.

You should be able to `.copy()` before doing the updates. That would avoid
any settingwithcopy warnings.

On Thu, Mar 14, 2019 at 6:45 AM Tao He <notifications@github.com> wrote:

> @TomAugspurger <https://github.com/TomAugspurger> thanks for the
> suggestion, but we would still get TypeError: assign() keywords must be
> strings.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/25590#issuecomment-472818085>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIs6KU6djbRJgW9hN1Yy8N3zFK1taks5vWjZVgaJpZM4bjP6l>
> .
>

Thanks @sighingnow!