Repository: matplotlib/matplotlib

Issue:
[Bug]: `imshow` allows RGB(A) images with `np.nan` values to pass
### Bug summary

Using `imshow` to visualize a 2x2x1 array with nans scattered throughout works as expected. With the default "bad" color of black with alpha = 0, you'll see an axis' `facecolor` where the `nan`s are.

With a 2x2x3 array, the `nan`s are always black.

With a 2x2x4 array, and the alpha manually set to 0 where the `nan`s are, you get the correct behavior (i.e. you can see the axis where the nans are).

### Code for reproduction

```python
import numpy
from matplotlib import pyplot

img = numpy.ones((2, 2))  # single channel 2x2
img_nan = img.copy()
img_nan[0, 0] = float('nan')

img3 = numpy.ones((2, 2, 3))  # RGB 2x2
img3_nan = img3.copy()
img3_nan[0, 0, :] = float('nan')

img4 = numpy.ones((2, 2, 4))  # RGBA 2x2
img4_nan = img4.copy()
img4_nan[0, 0, :] = float('nan')
img4_nan[0, 0, 3] = 0  # alpha


def imshow(a):
    pyplot.figure()
    im = pyplot.imshow(a)
    # im.get_cmap().set_bad(color='y', alpha=1) # only works for single channel images
    im.get_cmap().set_bad(alpha=0)
    pyplot.gca().set_facecolor('y')


imshow(img_nan)   # nan shows up as 'y'
imshow(img3_nan)  # nan shows up as black
imshow(img4_nan)  # nan shows up as 'y'


pyplot.show()
```


### Actual outcome

![image](https://github.com/matplotlib/matplotlib/assets/26465923/bc067e95-da1a-4ddf-b55b-929b8a18054b)
![image](https://github.com/matplotlib/matplotlib/assets/26465923/7afe79b4-64cc-4b41-a909-d2d6fe966af0)
![image](https://github.com/matplotlib/matplotlib/assets/26465923/ef71c33e-991d-4fd5-95a2-80ce3a66584c)


### Expected outcome

Figures 2 and 3 above should look identical.

### Additional information

I traced through `imshow` and cannot see any difference between the single- and 3-channel image; the mask is there and it's correct in both cases. So I wonder if this is a backend issue?

### Operating system

Windows

### Matplotlib Version

3.7.3

### Matplotlib Backend

QtAgg

### Python version

3.8.10  

### Jupyter version

_No response_

### Installation

pip


Discussion:
colormaps aren't used for 3 or 4 channel imshow()'s, so the "bad" is never
applicable there. With the 4 channel one, having an alpha of zero just
reveals the color underneath.

On Thu, Nov 9, 2023 at 4:37 PM Emilio Graff ***@***.***>
wrote:

> Bug summary
>
> Using imshow to visualize a 2x2x1 array with nans scattered throughout
> works as expected. With the default "bad" color of black with alpha = 0,
> you'll see an axis' facecolor where the nans are.
>
> With a 2x2x3 array, the nans are always black.
>
> With a 2x2x4 array, and the alpha manually set to 0 where the nans are,
> you get the correct behavior (i.e. you can see the axis where the nans are.
> Code for reproduction
>
> import numpyfrom matplotlib import pyplot
> img = numpy.ones((2, 2))  # single channel 2x2img_nan = img.copy()img_nan[0, 0] = float('nan')
> img3 = numpy.ones((2, 2, 3))  # RGB 2x2img3_nan = img3.copy()img3_nan[0, 0, :] = float('nan')
> img4 = numpy.ones((2, 2, 4))  # RGBA 2x2img4_nan = img4.copy()img4_nan[0, 0, :] = float('nan')img4_nan[0, 0, 3] = 0  # alpha
>
> def imshow(a):
>     pyplot.figure()
>     im = pyplot.imshow(a)
>     # im.get_cmap().set_bad(color='y', alpha=1) # only works for single channel images
>     im.get_cmap().set_bad(alpha=0)
>     pyplot.gca().set_facecolor('y')
>
> imshow(img_nan)   # nan shows up as 'y'imshow(img3_nan)  # nan shows up as blackimshow(img4_nan)  # nan shows up as 'y'
>
> pyplot.show()
>
> Actual outcome
>
> [image: image]
> <https://user-images.githubusercontent.com/26465923/281868384-bc067e95-da1a-4ddf-b55b-929b8a18054b.png>
> [image: image]
> <https://user-images.githubusercontent.com/26465923/281868405-7afe79b4-64cc-4b41-a909-d2d6fe966af0.png>
> [image: image]
> <https://user-images.githubusercontent.com/26465923/281868433-ef71c33e-991d-4fd5-95a2-80ce3a66584c.png>
> Expected outcome
>
> Figures 2 and 3 above should look identical.
> Additional information
>
> I traced through imshow and cannot see any difference between the single-
> and 3-channel image; the mask is there and it's correct in both cases. So I
> wonder if this is a backend issue?
> Operating system
>
> Windows
> Matplotlib Version
>
> 3.7.3
> Matplotlib Backend
>
> QtAgg
> Python version
>
> 3.8.10
> Jupyter version
>
> *No response*
> Installation
>
> pip
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/matplotlib/matplotlib/issues/27301>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACHF6FMWP2UDJGCCPC3SILYDVEJRAVCNFSM6AAAAAA7FHKK6KVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE4DMNBUGA4DGOA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

There's an additional thing here.

Moving the mouse pointer over the image:
1. In the 1- and 3-channel case, the area where the `nan` is shows `[]`
2. in the 4-channel case, it shows `[0]`
> colormaps aren't used for 3 or 4 channel imshow()'s, so the "bad" is never applicable there. With the 4 channel one, having an alpha of zero just reveals the color underneath.

OK fine, colormaps are ignored for 3- and 4-channel images. I guess I'm arguing that masking out `nan` should still happen. It actually is, and then getting thrown out / ignored somewhere down the line.
For float RGB(A) images I am not sure what I would expect `np.nan` to do.  Empirically, it looks like it is getting converted to `0` by Agg (as `ScalarMappable.to_rgba` appears to pass it through just fine and does not trigger any of the warnings as the `np.nan` poisons the min/max computations and reports false for all comparisons).

My inclination is that the second two cases should be raising on invalid input (as they would if you passed `2` for one of the channels.
and the reason the second RGB(A) case looks like it works is that the alpha channel is 0 so it does not matter what the RGB channels do.
@tacaswell, if I understand correctly, you're now going to raise an exception if an RGB image has `nan`s in it?

This seems quite harsh, since single-channel images are allowed to have `nan`s and even have a mechanism to control their display with `set_bad`.

My RGBA example is a workaround; in our application, I went that route because I want 1- and 3- channel images to behave the same with regards to `nan`.
I think I would expect points with nans to be treated the same as masked points.  We have explicit handling that sets the alpha to zero if any of the channels are masked at the given point:

https://github.com/matplotlib/matplotlib/blob/c0e9ebc10d28daa337fba4884fd52caddc4b732c/lib/matplotlib/cm.py#L501-L504



Patch:
diff --git a/lib/matplotlib/cm.py b/lib/matplotlib/cm.py
index 7e46cf567f70..e9a7aad6c456 100644
--- a/lib/matplotlib/cm.py
+++ b/lib/matplotlib/cm.py
@@ -328,7 +328,7 @@ def to_rgba(self, x, alpha=None, bytes=False, norm=True):
         treated as an RGB or RGBA array, and no mapping will be done.
         The array can be `~numpy.uint8`, or it can be floats with
         values in the 0-1 range; otherwise a ValueError will be raised.
-        If it is a masked array, any masked elements will be set to 0 alpha.
+        Any NaNs or masked elements will be set to 0 alpha.
         If the last dimension is 3, the *alpha* kwarg (defaulting to 1)
         will be used to fill in the transparency.  If the last dimension
         is 4, the *alpha* kwarg is ignored; it does not
@@ -360,6 +360,12 @@ def to_rgba(self, x, alpha=None, bytes=False, norm=True):
                 else:
                     raise ValueError("Third dimension must be 3 or 4")
                 if xx.dtype.kind == 'f':
+                    # If any of R, G, B, or A is nan, set to 0
+                    if np.any(nans := np.isnan(x)):
+                        if xx.shape[2] == 4:
+                            xx = xx.copy()
+                        xx[np.any(nans, axis=2), :] = 0
+
                     if norm and (xx.max() > 1 or xx.min() < 0):
                         raise ValueError("Floating point image RGB values "
                                          "must be in the 0..1 range.")


Test patch:
diff --git a/lib/matplotlib/tests/test_colors.py b/lib/matplotlib/tests/test_colors.py
index 2d71fea83472..5b5332100c2a 100644
--- a/lib/matplotlib/tests/test_colors.py
+++ b/lib/matplotlib/tests/test_colors.py
@@ -1352,6 +1352,32 @@ def test_scalarmappable_to_rgba(bytes):
     np.testing.assert_almost_equal(sm.to_rgba(xm[..., :3], bytes=bytes), expected)
 
 
+@pytest.mark.parametrize("bytes", (True, False))
+def test_scalarmappable_nan_to_rgba(bytes):
+    sm = cm.ScalarMappable()
+
+    # RGBA
+    x = np.ones((2, 3, 4), dtype=float) * 0.5
+    x[0, 0, 0] = np.nan
+    expected = x.copy()
+    expected[0, 0, :] = 0
+    if bytes:
+        expected = (expected * 255).astype(np.uint8)
+    np.testing.assert_almost_equal(sm.to_rgba(x, bytes=bytes), expected)
+    assert np.any(np.isnan(x))  # Input array should not be changed
+
+    # RGB
+    expected[..., 3] = 255 if bytes else 1
+    expected[0, 0, 3] = 0
+    np.testing.assert_almost_equal(sm.to_rgba(x[..., :3], bytes=bytes), expected)
+    assert np.any(np.isnan(x))  # Input array should not be changed
+
+    # Out-of-range fail
+    x[1, 0, 0] = 42
+    with pytest.raises(ValueError, match='0..1 range'):
+        sm.to_rgba(x[..., :3], bytes=bytes)
+
+
 def test_failed_conversions():
     with pytest.raises(ValueError):
         mcolors.to_rgba('5')