Repository: spyder-ide/spyder

Issue:
DataFrame Editor background coloring does not support `pandas.Int*Dtype` dtypes
<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->

### Issue Report Checklist

* [x] Searched the [issues page](https://github.com/spyder-ide/spyder/issues?q=is%3Aissue) for similar reports
* [x] Read the relevant sections of the [Spyder Troubleshooting Guide](https://github.com/spyder-ide/spyder/wiki/Troubleshooting-Guide-and-FAQ) and followed its advice
* [x] Reproduced the issue after updating with ``conda update spyder`` (or ``pip``, if not using Anaconda)
* [ ] Could not reproduce inside ``jupyter qtconsole`` (if console-related)
* [x] Tried basic troubleshooting (if a bug/error)
    * [x] Restarted Spyder
    * [x] Reset preferences with ``spyder --reset``
    * [ ] Reinstalled the latest version of [Anaconda](https://www.anaconda.com/download/)
    * [x] Tried the other applicable steps from the Troubleshooting Guide
* [x] Completed the **Problem Description**, **Steps to Reproduce** and **Version** sections below


## Problem Description
### Introduction
Pandas nowadays has nullable types for integers etc. so that missing values can be represented without conversion to floats. An example is `pandas.Int64Dtype` described here: https://pandas.pydata.org/docs/user_guide/integer_na.html Although the concept of nullable types is relatively new and mostly non-default, it can be very useful. Moreover, some methods of `pandas.StringDtype` return nullable integers, and the documentation specifically [recommends](https://pandas.pydata.org/docs/user_guide/text.html) using `pandas.StringDtype` for text data.

### Problem
The DataFrame Editor in Spyder appears to lack background coloring support for pandas nullable integer data types.


### What steps reproduce the problem?

In the IPython console, type the following:
```python
import pandas as pd

vals = [1, 2, 3, 4, 5]
vals_na = [1, 2, 3, None, 5]

df = pd.DataFrame({
    'old': pd.Series(vals),
    'old_na': pd.Series(vals_na),
    'new': pd.Series(vals, dtype='Int64'),
    'new_na': pd.Series(vals_na, dtype='Int64')
})
```
This produces a DataFrame with the following dtypes (as printed by `df.dtypes`; here `Int64` means `pandas.Int64Dtype`):
```
old         int64
old_na    float64
new         Int64
new_na      Int64
```
Next, double-click `df` in the Variable Explorer.

### What is the expected output? What do you see instead?
In the DataFrame Editor, the following is shown. In the first two columns, the data which uses traditional dtypes is correctly colored, whereas in the last two columns, the data which uses the new nullable integer type is not colored.
![Screenshot_20230806_161952](https://github.com/spyder-ide/spyder/assets/96844552/2f832e7f-cfef-4368-a7b6-0ab257e92f00)



## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 5.4.4  (conda)
* Python version: 3.11.4 64-bit
* Qt version: 5.15.8
* PyQt5 version: 5.15.9
* Operating System: Linux-5.4.0-153-generic-x86_64-with-glibc2.31

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```
# Mandatory:
atomicwrites >=1.2.0                                                        :  1.4.1 (OK)
chardet >=2.0.0                                                             :  5.1.0 (OK)
cloudpickle >=0.5.0                                                         :  2.2.1 (OK)
cookiecutter >=1.6.0                                                        :  2.3.0 (OK)
diff_match_patch >=20181111                                                 :  20230430 (OK)
intervaltree >=3.0.2                                                        :  3.1.0 (OK)
IPython >=7.31.1,<9.0.0,!=8.8.0,!=8.9.0,!=8.10.0,!=8.11.0,!=8.12.0,!=8.12.1 :  8.14.0 (OK)
jedi >=0.17.2,<0.19.0                                                       :  0.18.2 (OK)
jellyfish >=0.7                                                             :  1.0.0 (OK)
jsonschema >=3.2.0                                                          :  4.18.6 (OK)
keyring >=17.0.0                                                            :  24.2.0 (OK)
nbconvert >=4.0                                                             :  7.7.3 (OK)
numpydoc >=0.6.0                                                            :  1.5.0 (OK)
parso >=0.7.0,<0.9.0                                                        :  0.8.3 (OK)
pexpect >=4.4.0                                                             :  4.8.0 (OK)
pickleshare >=0.4                                                           :  0.7.5 (OK)
psutil >=5.3                                                                :  5.9.5 (OK)
pygments >=2.0                                                              :  2.15.1 (OK)
pylint >=2.5.0,<3.0                                                         :  2.17.5 (OK)
pylint_venv >=3.0.2                                                         :  3.0.2 (OK)
pyls_spyder >=0.4.0                                                         :  0.4.0 (OK)
pylsp >=1.7.4,<1.8.0                                                        :  1.7.4 (OK)
pylsp_black >=1.2.0,<3.0.0                                                  :  1.3.0 (OK)
qdarkstyle >=3.0.2,<3.2.0                                                   :  3.1 (OK)
qstylizer >=0.2.2                                                           :  0.2.2 (OK)
qtawesome >=1.2.1                                                           :  1.2.3 (OK)
qtconsole >=5.4.2,<5.5.0                                                    :  5.4.3 (OK)
qtpy >=2.1.0                                                                :  2.3.1 (OK)
rtree >=0.9.7                                                               :  1.0.1 (OK)
setuptools >=49.6.0                                                         :  68.0.0 (OK)
sphinx >=0.6.6                                                              :  7.1.2 (OK)
spyder_kernels >=2.4.4,<2.5.0                                               :  2.4.4 (OK)
textdistance >=4.2.0                                                        :  4.5.0 (OK)
three_merge >=0.1.1                                                         :  0.1.1 (OK)
watchdog >=0.10.3                                                           :  3.0.0 (OK)
xdg >=0.26                                                                  :  0.28 (OK)
zmq >=22.1.0                                                                :  25.1.0 (OK)

# Optional:
cython >=0.21                                                               :  3.0.0 (OK)
matplotlib >=3.0.0                                                          :  3.7.2 (OK)
numpy >=1.7                                                                 :  1.25.1 (OK)
pandas >=1.1.1                                                              :  2.0.3 (OK)
scipy >=0.17.0                                                              :  1.11.1 (OK)
sympy >=0.7.3                                                               :  1.12 (OK)
```



Discussion:
This makes sense to me and should be quite easy to incorporate. The only potential issue is that pandas describes these nullable integer types as experimental, so some details may change.

Resolving patch:
diff --git a/spyder/plugins/variableexplorer/widgets/dataframeeditor.py b/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
--- a/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
+++ b/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
@@ -63,7 +63,7 @@
 from spyder.utils.palette import QStylePalette
 
 
-# Supported Numbers and complex numbers
+# Supported real and complex number types
 REAL_NUMBER_TYPES = (float, int, np.int64, np.int32)
 COMPLEX_NUMBER_TYPES = (complex, np.complex64, np.complex128)
 
@@ -91,6 +91,18 @@
 BACKGROUND_MISC_ALPHA = 0.3
 
 
+def is_any_real_numeric_dtype(dtype) -> bool:
+    """
+    Test whether a Pandas dtype is a real numeric type.
+    """
+    try:
+        import pandas.api.types
+        return pandas.api.types.is_any_real_numeric_dtype(dtype)
+    except Exception:
+        # Pandas version 1
+        return dtype in REAL_NUMBER_TYPES
+
+
 def bool_false_check(value):
     """
     Used to convert bool entrance to false.
@@ -250,8 +262,11 @@ def max_min_col_update(self):
             # the maximum of a column.
             # Fixes spyder-ide/spyder#17145
             try:
-                if col.dtype in REAL_NUMBER_TYPES + COMPLEX_NUMBER_TYPES:
-                    if col.dtype in REAL_NUMBER_TYPES:
+                if (
+                    is_any_real_numeric_dtype(col.dtype)
+                    or col.dtype in COMPLEX_NUMBER_TYPES
+                ):
+                    if is_any_real_numeric_dtype(col.dtype):
                         vmax = col.max(skipna=True)
                         vmin = col.min(skipna=True)
                     else:


Test patch:
diff --git a/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py b/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py
--- a/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py
+++ b/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py
@@ -358,6 +358,41 @@ def test_dataframemodel_get_bgcolor_with_missings():
             'Wrong bg color for missing of type ' + column
 
 
+def test_dataframemodel_get_bgcolor_with_nullable_numbers():
+    """
+    Test background colors for nullable integer data types
+
+    Regression test for spyder-ide/spyder#21222.
+    """
+    vals = [1, 2, 3, 4, 5]
+    vals_na = [1, 2, 3, None, 5]
+    df = DataFrame({
+        'old': Series(vals),
+        'old_na': Series(vals_na),
+        'new': Series(vals, dtype='Int64'),
+        'new_na': Series(vals_na, dtype='Int64')
+    })
+    dfm = DataFrameModel(df)
+    dfm.colum_avg(0)
+
+    # Test numbers
+    h0 = dataframeeditor.BACKGROUND_NUMBER_MINHUE
+    dh = dataframeeditor.BACKGROUND_NUMBER_HUERANGE
+    s = dataframeeditor.BACKGROUND_NUMBER_SATURATION
+    v = dataframeeditor.BACKGROUND_NUMBER_VALUE
+    a = dataframeeditor.BACKGROUND_NUMBER_ALPHA
+    for col_index in range(4):
+        assert colorclose(bgcolor(dfm, 0, col_index), (h0 + dh, s, v, a))
+    assert colorclose(bgcolor(dfm, 3, 0), (h0 + 1 / 4 * dh, s, v, a))
+    assert colorclose(bgcolor(dfm, 3, 2), (h0 + 1 / 4 * dh, s, v, a))
+
+    # Test null values
+    h, s, v, __ = QColor(dataframeeditor.BACKGROUND_NONNUMBER_COLOR).getHsvF()
+    alpha = dataframeeditor.BACKGROUND_MISC_ALPHA
+    assert colorclose(bgcolor(dfm, 3, 1), (h, s, v, alpha))
+    assert colorclose(bgcolor(dfm, 3, 3), (h, s, v, alpha))
+
+
 def test_dataframemodel_with_format_percent_d_and_nan():
     """
     Test DataFrameModel with format `d` and dataframe containing NaN