Repository: pandas-dev/pandas
Pull request: ENH: Styler.background_gradient to accept vmin vmax and dtype Int64
For `vmin` `vmax` use the same implementation as `Styler.bar`
For dtype `Int64` issue, deprecated `.values` and use `.to_numpy` instead
Here explicitly assign the dtype to float since we are doing normalize

- [x] closes #12145 and #28869
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
background_gradient vmin and vmax
Working with the new style API, I would like to turn several columns of a dataframe into something like a heatmap. I would like to use the same range for all columns in the subset, i.e., the colour map should take a minimum and maximum value I specify, regardless of what actual range each column has. Currently using the "min" and "max" this is not intuitive. I would much prefer to be able to specify explicit vmin and vmax values, analogous to the seaborn.heatmap function.

Thanks for the style API, extremely useful.

Hints from issue discussion:
That's what I wanted initially as well, but didn't have time to do it properly before the last release. I wonder if we should just require seaborn for those parameters, and use its heuristics for determining a good colormap and handling vmin/vmax.

Hi, 

I have some feedback which IIUC is related to what @alimanfoo is saying. The `background_gradient` as well as the `bar` style methods should have the same axis options as the `highlight_max` method:

`axis: int, str, or None; default None`

It can be very misleading if the color scale is only applied to each column separately - particulary when I specify a subset of columns (which represent the same measure) I expected them to be represented with the same color scale by default. I tried to set `axis=None` for the above mentioned methods, but unluckily an error is being raised from that. 
I get that the feature is new and hopefully this will be possible soon. Apart from my negative criticism: Kudo's for the good work.

Couldn't agree more with @mpschr --- `axis = None` should even be the default. Anybody know a workaround for now?

@alimanfoo I'm not sure it applies directly to your problem, but to get the equivalent of vmin/vmax for the colormap, I do the following:
```python
# given df, cmap, vmin, vmax:
df_data = df.copy()
df = df.clip(vmin, vmax)
df = df.style.background_gradient(cmap=cmap)
df.data = df_data
```
FYI: `axis = None` support is being added under PR #21259

Commit message: fix typo

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.0.rst b/doc/source/whatsnew/v1.0.0.rst
index 4007ecd5a9412..70121e70fc5cb 100644
--- a/doc/source/whatsnew/v1.0.0.rst
+++ b/doc/source/whatsnew/v1.0.0.rst
@@ -112,6 +112,7 @@ Other enhancements
 - :meth:`read_stata` can read Stata 119 dta files. (:issue:`28250`)
 - Added ``encoding`` argument to :meth:`DataFrame.to_string` for non-ascii text (:issue:`28766`)
 - Added ``encoding`` argument to :func:`DataFrame.to_html` for non-ascii text (:issue:`28663`)
+- :meth:`Styler.background_gradient` now accepts ``vmin`` and ``vmax`` arguments (:issue:`12145`)
 
 Build Changes
 ^^^^^^^^^^^^^
@@ -385,6 +386,7 @@ I/O
 - Bug in :meth:`DataFrame.read_excel` with ``engine='ods'`` when ``sheet_name`` argument references a non-existent sheet (:issue:`27676`)
 - Bug in :meth:`pandas.io.formats.style.Styler` formatting for floating values not displaying decimals correctly (:issue:`13257`)
 - Bug in :meth:`DataFrame.to_html` when using ``formatters=<list>`` and ``max_cols`` together. (:issue:`25955`)
+- Bug in :meth:`Styler.background_gradient` not able to work with dtype ``Int64`` (:issue:`28869`)
 
 Plotting
 ^^^^^^^^
diff --git a/pandas/io/formats/style.py b/pandas/io/formats/style.py
index abf2caf3914e0..0bd19eb16c362 100644
--- a/pandas/io/formats/style.py
+++ b/pandas/io/formats/style.py
@@ -958,6 +958,8 @@ def background_gradient(
         axis=0,
         subset=None,
         text_color_threshold=0.408,
+        vmin=None,
+        vmax=None,
     ):
         """
         Color the background in a gradient according to
@@ -986,6 +988,18 @@ def background_gradient(
 
             .. versionadded:: 0.24.0
 
+        vmin : float, optional
+            Minimum data value that corresponds to colormap minimum value.
+            When None (default): the minimum value of the data will be used.
+
+            .. versionadded:: 1.0.0
+
+        vmax : float, optional
+            Maximum data value that corresponds to colormap maximum value.
+            When None (default): the maximum value of the data will be used.
+
+            .. versionadded:: 1.0.0
+
         Returns
         -------
         self : Styler
@@ -1012,11 +1026,15 @@ def background_gradient(
             low=low,
             high=high,
             text_color_threshold=text_color_threshold,
+            vmin=vmin,
+            vmax=vmax,
         )
         return self
 
     @staticmethod
-    def _background_gradient(s, cmap="PuBu", low=0, high=0, text_color_threshold=0.408):
+    def _background_gradient(
+        s, cmap="PuBu", low=0, high=0, text_color_threshold=0.408, vmin=None, vmax=None
+    ):
         """
         Color background in a range according to the data.
         """
@@ -1028,14 +1046,18 @@ def _background_gradient(s, cmap="PuBu", low=0, high=0, text_color_threshold=0.4
             raise ValueError(msg)
 
         with _mpl(Styler.background_gradient) as (plt, colors):
-            smin = s.values.min()
-            smax = s.values.max()
+            smin = s.min() if vmin is None else vmin
+            if isinstance(smin, ABCSeries):
+                smin = smin.min()
+            smax = s.max() if vmax is None else vmax
+            if isinstance(smax, ABCSeries):
+                smax = smax.max()
             rng = smax - smin
             # extend lower / upper bounds, compresses color range
             norm = colors.Normalize(smin - (rng * low), smax + (rng * high))
             # matplotlib colors.Normalize modifies inplace?
             # https://github.com/matplotlib/matplotlib/issues/5427
-            rgbas = plt.cm.get_cmap(cmap)(norm(s.values))
+            rgbas = plt.cm.get_cmap(cmap)(norm(s.to_numpy(dtype=float)))
 
             def relative_luminance(rgba):
                 """
@@ -1121,7 +1143,7 @@ def _bar(s, align, colors, width=100, vmin=None, vmax=None):
             smax = max(abs(smin), abs(smax))
             smin = -smax
         # Transform to percent-range of linear-gradient
-        normed = width * (s.values - smin) / (smax - smin + 1e-12)
+        normed = width * (s.to_numpy(dtype=float) - smin) / (smax - smin + 1e-12)
         zero = -width * smin / (smax - smin + 1e-12)
 
         def css_bar(start, end, color):
diff --git a/pandas/tests/io/formats/test_style.py b/pandas/tests/io/formats/test_style.py
index 0f1402d7da389..fa725ccae66f9 100644
--- a/pandas/tests/io/formats/test_style.py
+++ b/pandas/tests/io/formats/test_style.py
@@ -1648,6 +1648,23 @@ def test_background_gradient_axis(self):
         assert result[(1, 0)] == mid
         assert result[(1, 1)] == high
 
+    def test_background_gradient_vmin_vmax(self):
+        # GH 12145
+        df = pd.DataFrame(range(5))
+        ctx = df.style.background_gradient(vmin=1, vmax=3)._compute().ctx
+        assert ctx[(0, 0)] == ctx[(1, 0)]
+        assert ctx[(4, 0)] == ctx[(3, 0)]
+
+    def test_background_gradient_int64(self):
+        # GH 28869
+        df1 = pd.Series(range(3)).to_frame()
+        df2 = pd.Series(range(3), dtype="Int64").to_frame()
+        ctx1 = df1.style.background_gradient()._compute().ctx
+        ctx2 = df2.style.background_gradient()._compute().ctx
+        assert ctx2[(0, 0)] == ctx1[(0, 0)]
+        assert ctx2[(1, 0)] == ctx1[(1, 0)]
+        assert ctx2[(2, 0)] == ctx1[(2, 0)]
+
 
 def test_block_names():
     # catch accidental removal of a block


Code review comments:
diff_hunk: @@ -1028,14 +1046,18 @@ def _background_gradient(s, cmap="PuBu", low=0, high=0, text_color_threshold=0.4
             raise ValueError(msg)
 
         with _mpl(Styler.background_gradient) as (plt, colors):
-            smin = s.values.min()
-            smax = s.values.max()
+            smin = s.min() if vmin is None else vmin
+            if isinstance(smin, ABCSeries):
+                smin = smin.min()
+            smax = s.max() if vmax is None else vmax
+            if isinstance(smax, ABCSeries):
+                smax = smax.max()
             rng = smax - smin
             # extend lower / upper bounds, compresses color range
             norm = colors.Normalize(smin - (rng * low), smax + (rng * high))
             # matplotlib colors.Normalize modifies inplace?
             # https://github.com/matplotlib/matplotlib/issues/5427
-            rgbas = plt.cm.get_cmap(cmap)(norm(s.values))
+            rgbas = plt.cm.get_cmap(cmap)(norm(s.to_numpy(dtype=float)))
path: pandas/io/formats/style.py
text: @user1:
OK with this but can just use default constructor without `dtype` argument I think? As it it seems like this is trying to coerce, which I'm not sure why that would be needed

@author:
unfortunately, `.to_numpy()` alone will produce `dtype=object` and rejected by matplotlib.
OTOH, it will immediately convert to float anyway since `cmap` require normalizing to [0,1], nothing to loose for coercing in the first place.

diff_hunk: @@ -1028,14 +1046,18 @@ def _background_gradient(s, cmap="PuBu", low=0, high=0, text_color_threshold=0.4
             raise ValueError(msg)
 
         with _mpl(Styler.background_gradient) as (plt, colors):
-            smin = s.values.min()
-            smax = s.values.max()
+            smin = s.min() if vmin is None else vmin
+            if isinstance(smin, ABCSeries):
path: pandas/io/formats/style.py
text: @user1:
Is this not already covered by the line directly preceding it?

@author:
Here depends on the `axis` argument provided to `Styler.background_gradient`, `s` can be a DataFrame or a Series. if `s` is a DataFrame and `vmin` not provided, `smin` will first become a Series then become the min value of the full data, the code here is consistent with (aka. direct copy from) `Styler.bar` method, do we want to change it?

@author:
side note: try to use `s.values.min()` to get overall minima will produce `AttributeError: 'IntegerArray' object has no attribute 'min'` as described in #28869. OTOH, use `np.min(s.values)` will also produce `NotImplementedError: The 'reduce' method is not supported.` I thought IntegerArray might be a WIP and `.min().min()` is the workaround.

@user1:
Can you move this into a helper function? I think this affects `highlight_extrema` as well but seems to tackle slightly differently; should keep unified

`squeeze` might also be more appropriate to use here

@author:
Thanks! didn't notice the `highlight_extrema`, they really should be consistent. turned-out a fairly simple approach can be used: `np.nanmin(s.to_numpy())`, s can be any shape

diff_hunk: @@ -958,6 +958,8 @@ def background_gradient(
         axis=0,
         subset=None,
         text_color_threshold=0.408,
+        vmin=None,
path: pandas/io/formats/style.py
text: @user1:
Can you annotate the new parameters?

@author:
Thanks for your review @user1, but I don't quite get it what do you mean by "annotate", can you elaborate a little more? (I have docstring already so I suppose you don't mean docstring right?)

@author:
I got it! you mean the type-hints annotation right?

@user1:
Yes that is correct

@author:
done! :smile: