Repository: pandas-dev/pandas
Pull request: BUG: Fixes plotting with nullable integers (#32073)
- [x] closes #32073
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry




Problem statement:
Plotting Int64 columns with nulled integers (NAType) fails
#### Code Sample, a copy-pastable example if possible

```
import pandas as pd
import numpy as np

df = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [7, 5, np.nan, 3, 2]})
df.plot(x='A', y='B')
df = df.astype('Int64')
df.plot(x='A', y='B')
```
#### Problem description

The first plotting command works, the second throws the error message

> TypeError: float() argument must be a string or a number, not 'NAType'

#### Expected Output

NAType should be treated the same way as numpy nan in plotting. Maybe transformed on the fly?

(I'm unsure if this is a pandas, a numpy, or a matplotlib issue, I'm starting here)

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.6.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 79 Stepping 1, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en
LOCALE           : None.None

pandas           : 1.0.1
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 45.2.0.post20200209
Cython           : 0.29.15
pytest           : None
hypothesis       : None
sphinx           : 2.4.1
blosc            : None
feather          : None
xlsxwriter       : 1.2.7
lxml.etree       : 4.5.0
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : 7.12.0
pandas_datareader: None
bs4              : 4.8.2
bottleneck       : None
fastparquet      : 0.3.3
gcsfs            : None
lxml.etree       : 4.5.0
matplotlib       : 3.1.3
numexpr          : None
odfpy            : None
openpyxl         : 3.0.3
pandas_gbq       : None
pyarrow          : 0.16.0
pytables         : None
pytest           : None
pyxlsb           : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : 1.3.13
tables           : None
tabulate         : None
xarray           : None
xlrd             : 1.2.0
xlwt             : None
xlsxwriter       : 1.2.7
numba            : 0.48.0

</details>

Hints from issue discussion:
Can you post the full traceback?
Gladly:

```
Traceback (most recent call last):

  File "C:\Users\My.Name\Documents\Python_Projects\GIT\miscellaneous\temp.py", line 14, in <module>
    df.plot(x='A', y='B')

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_core.py", line 847, in __call__
    return plot_backend.plot(data, kind=kind, **kwargs)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\__init__.py", line 61, in plot
    plot_obj.generate()

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\core.py", line 263, in generate
    self._make_plot()

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\core.py", line 1085, in _make_plot
    **kwds,

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\core.py", line 1104, in _plot
    lines = MPLPlot._plot(ax, x, y_values, style=style, **kwds)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\converter.py", line 66, in wrapper
    return func(*args, **kwargs)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\plotting\_matplotlib\core.py", line 656, in _plot
    return ax.plot(*args, **kwds)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\axes\_axes.py", line 1667, in plot
    self.add_line(line)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\axes\_base.py", line 1902, in add_line
    self._update_line_limits(line)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\axes\_base.py", line 1924, in _update_line_limits
    path = line.get_path()

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\lines.py", line 1027, in get_path
    self.recache()

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\lines.py", line 675, in recache
    y = _to_unmasked_float_array(yconv).ravel()

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\matplotlib\cbook\__init__.py", line 1388, in _to_unmasked_float_array
    return np.ma.asarray(x, float).filled(np.nan)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\numpy\ma\core.py", line 7849, in asarray
    subok=False, order=order)

  File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\numpy\ma\core.py", line 2795, in __new__
    order=order, subok=True, ndmin=ndmin)

TypeError: float() argument must be a string or a number, not 'NAType'
```
We need to convert to floats with NaNs before passing the data to matplotlib, I suppose.
Hi! Can I make an attempt on this one? I'm looking for issues that can be useful to the community and help me to understand better how pandas works in depth :) 
@jeandersonbc yes, that's welcome! 
If you have any questions, don't hesitate to ask
take
@AnnaDaglis someone else (@jeandersonbc) already just commented he would like to look at it, so I would first give him some time before taking this issue
Thanks @jorisvandenbossche, I didn't manage time to work on the issue since last time that I posted so that's why I didn't reply early.
As far as I can see, it is my understanding from your suggestion that there should be verification for NA values before calling `plot_backend.plot(data, kind=kind, **kwargs)`, right? I'll be submitting a PR later today with some tests to see how it goes.
> there should be verification for NA values before calling plot_backend.plot(data, kind=kind, **kwargs)

Yes, I think so. But the check might need to be done lower in the stack, as when calling `plot_backend.plot()`, `data` is still a dataframe. I think we can focus here on the matplotlib backend included in pandas, and thus we need to ensure converting to float dtype (with `to_numpy(float, na_value=np.nan)` if it is nullable integer dtype) before passing it to matplotlib's plot().
take

Commit message: Added checking for NA values

Patch to review:
diff --git a/doc/source/whatsnew/v1.2.0.rst b/doc/source/whatsnew/v1.2.0.rst
index 996fb828010ba..bec66ac22b428 100644
--- a/doc/source/whatsnew/v1.2.0.rst
+++ b/doc/source/whatsnew/v1.2.0.rst
@@ -681,6 +681,7 @@ Plotting
   indexed by a :class:`TimedeltaIndex` with a fixed frequency when x-axis lower limit was greater than upper limit (:issue:`37454`)
 - Bug in :meth:`DataFrameGroupBy.boxplot` when ``subplots=False``, a KeyError would raise (:issue:`16748`)
 - Bug in :meth:`DataFrame.plot` and :meth:`Series.plot` was overwriting matplotlib's shared y axes behaviour when no sharey parameter was passed (:issue:`37942`)
+- Bug in :meth:`DataFrame.plot` to handle nullable integers (:issue:`32073`)
 
 Groupby/resample/rolling
 ^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/pandas/plotting/_matplotlib/core.py b/pandas/plotting/_matplotlib/core.py
index c01cfb9a8b487..93ed636032831 100644
--- a/pandas/plotting/_matplotlib/core.py
+++ b/pandas/plotting/_matplotlib/core.py
@@ -9,9 +9,11 @@
 from pandas.util._decorators import cache_readonly
 
 from pandas.core.dtypes.common import (
+    is_extension_array_dtype,
     is_float,
     is_hashable,
     is_integer,
+    is_integer_dtype,
     is_iterator,
     is_list_like,
     is_number,
@@ -423,13 +425,19 @@ def _compute_plot_data(self):
         if is_empty:
             raise TypeError("no numeric data to plot")
 
-        # GH25587: cast ExtensionArray of pandas (IntegerArray, etc.) to
-        # np.ndarray before plot.
-        numeric_data = numeric_data.copy()
-        for col in numeric_data:
-            numeric_data[col] = np.asarray(numeric_data[col])
+        def convert_to_ndarray(data):
+            # GH32073: cast to float if values contain nulled integers
+            if is_integer_dtype(data.dtype) and is_extension_array_dtype(data.dtype):
+                return data.to_numpy(dtype="float", na_value=np.nan)
 
-        self.data = numeric_data
+            # GH25587: cast ExtensionArray of pandas (IntegerArray, etc.) to
+            # np.ndarray before plot.
+            if len(data) > 0:
+                return np.asarray(data)
+
+            return data
+
+        self.data = numeric_data.apply(convert_to_ndarray)
 
     def _make_plot(self):
         raise AbstractMethodError(self)
diff --git a/pandas/tests/plotting/frame/test_frame.py b/pandas/tests/plotting/frame/test_frame.py
index 77a4c4a8faf5e..ccd77ea822762 100644
--- a/pandas/tests/plotting/frame/test_frame.py
+++ b/pandas/tests/plotting/frame/test_frame.py
@@ -148,6 +148,23 @@ def test_plot(self):
         result = ax.axes
         assert result is axes[0]
 
+    def test_nullable_int_plot(self):
+        # GH 32073
+        dates = ["2008", "2009", None, "2011", "2012"]
+        df = pd.DataFrame(
+            {
+                "A": [1, 2, 3, 4, 5],
+                "B": [7, 5, np.nan, 3, 2],
+                "C": pd.to_datetime(dates, format="%Y"),
+                "D": pd.to_datetime(dates, format="%Y", utc=True),
+            }
+        )
+
+        _check_plot_works(df.plot, x="A", y="B")
+        _check_plot_works(df[["A", "B"]].astype("Int64").plot, x="A", y="B")
+        _check_plot_works(df[["A", "C"]].plot, x="A", y="C")
+        _check_plot_works(df[["A", "D"]].plot, x="A", y="D")
+
     def test_integer_array_plot(self):
         # GH 25587
         arr = integer_array([1, 2, 3, 4], dtype="UInt32")


Code review comments:
diff_hunk: @@ -423,13 +425,19 @@ def _compute_plot_data(self):
         if is_empty:
             raise TypeError("no numeric data to plot")
 
-        # GH25587: cast ExtensionArray of pandas (IntegerArray, etc.) to
-        # np.ndarray before plot.
-        numeric_data = numeric_data.copy()
-        for col in numeric_data:
-            numeric_data[col] = np.asarray(numeric_data[col])
+        def convert_to_ndarray(data):
path: pandas/plotting/_matplotlib/core.py
text: @user1:
Consider moving ``convert_to_ndarray`` out of the scope of the function ``_compute_plot_data``.

@user2:
yes pls do this

@author:
@user2 I added the changes to the latest commit. Can you please review it again ?

diff_hunk: @@ -681,6 +681,7 @@ Plotting
   indexed by a :class:`TimedeltaIndex` with a fixed frequency when x-axis lower limit was greater than upper limit (:issue:`37454`)
 - Bug in :meth:`DataFrameGroupBy.boxplot` when ``subplots=False``, a KeyError would raise (:issue:`16748`)
 - Bug in :meth:`DataFrame.plot` and :meth:`Series.plot` was overwriting matplotlib's shared y axes behaviour when no sharey parameter was passed (:issue:`37942`)
+- Bug in :meth:`DataFrame.plot` to handle nullable integers (:issue:`32073`)
path: doc/source/whatsnew/v1.2.0.rst
text: @user1:
We want to be more specific:
```
Bug in :meth:`DataFrame.plot` was raising a ``TypeError`` with ``ExtensionDtype`` columns (:issue:`32073`)
```

@author:
Thank you @user1 for your comments. I will add these changes to the next commit. 

@author:
@user1 I added the changes to the latest commit. Can you please review it again ?

diff_hunk: @@ -423,13 +425,19 @@ def _compute_plot_data(self):
         if is_empty:
             raise TypeError("no numeric data to plot")
 
-        # GH25587: cast ExtensionArray of pandas (IntegerArray, etc.) to
-        # np.ndarray before plot.
-        numeric_data = numeric_data.copy()
-        for col in numeric_data:
-            numeric_data[col] = np.asarray(numeric_data[col])
+        def convert_to_ndarray(data):
path: pandas/plotting/_matplotlib/core.py
text: @user1:
you could use `pd.to_numeric` here I think. Illustrative example:
``` python
In [8]: s = pd.Series([1, 2, 3, pd.NA], dtype="Int64")

In [9]: pd.to_numeric(s.to_numpy(), downcast="integer")
Out[9]: array([ 1.,  2.,  3., nan])
```

@author:
Thank you @user1 for the example. I think I didn't get your point here. Can you please explain it a bit more?

@user2:
no don't use `pd.to_numeric` we already know the type and its an internal function

diff_hunk: @@ -148,6 +148,23 @@ def test_plot(self):
         result = ax.axes
         assert result is axes[0]
 
+    def test_nullable_int_plot(self):
+        # GH 32073
+        dates = ["2008", "2009", None, "2011", "2012"]
+        df = pd.DataFrame(
+            {
+                "A": [1, 2, 3, 4, 5],
+                "B": [7, 5, np.nan, 3, 2],
+                "C": pd.to_datetime(dates, format="%Y"),
+                "D": pd.to_datetime(dates, format="%Y", utc=True),
+            }
+        )
+
+        _check_plot_works(df.plot, x="A", y="B")
+        _check_plot_works(df[["A", "B"]].astype("Int64").plot, x="A", y="B")
path: pandas/tests/plotting/frame/test_frame.py
text: @user1:
I think we'd rather to specify the dtype in the dataframe construction

@author:
Sure. Thank you for the suggestion. I will rectify this in the next commit. 

@author:
@user1 I added the changes to the latest commit. Can you please review it again ?

diff_hunk: @@ -423,13 +425,19 @@ def _compute_plot_data(self):
         if is_empty:
             raise TypeError("no numeric data to plot")
 
-        # GH25587: cast ExtensionArray of pandas (IntegerArray, etc.) to
-        # np.ndarray before plot.
-        numeric_data = numeric_data.copy()
-        for col in numeric_data:
-            numeric_data[col] = np.asarray(numeric_data[col])
+        def convert_to_ndarray(data):
+            # GH32073: cast to float if values contain nulled integers
+            if is_integer_dtype(data.dtype) and is_extension_array_dtype(data.dtype):
path: pandas/plotting/_matplotlib/core.py
text: @user1:
actually we want to also catch / test nullable floats as well

@author:
@user1 I fixed this in the latest commit.