ConfusionMatrixDisplay does not correctly change text color when confusion matrix contains NaN

### Describe the bug

In our specific usecase we generate a Confusion Matrix using our own software to be passed on to ConfusionMatrixDisplay. Due to the nature of our needs this confusion matrix could contain one or more NaN values (as we want to make it explicit that this combination does not exist instead of replacing NaN by 0).

When passing a confusion matrix to ConfusionMatrixDisplay and `include_values` is set to True the `.plot()` method is unable to properly colour the text of the included values. This is due to the fact that the `thresh` variable ends up being NaN:
https://github.com/scikit-learn/scikit-learn/blob/52b692511450f7439eb929ad16933871c9a6ba68/sklearn/metrics/_plot/confusion_matrix.py#L156 

Both `.max()` and `.min()` propagate NaN as per the [Numpy docs](https://numpy.org/doc/stable/reference/generated/numpy.min.html). Their suggestion is to use `nanmin()` and `nanmax()`.

I am now wondering if this could be a sensible adaptation for this method (I'm willing to implement it myself and open a pull request). 



### Steps/Code to Reproduce

```
import numpy as np
from sklearn.metrics import ConfusionMatrixDisplay

conf_mat = np.array([[1, 2], [3, np.nan]])

disp = ConfusionMatrixDisplay(confusion_matrix = conf_mat)

disp.plot()
```

### Expected Results

Expected result would be a confusion matrix plot with correctly coloured text in the cells, similar to this example without a NaN value:
![Unknown-1](https://github.com/scikit-learn/scikit-learn/assets/4438111/2ca5b20e-a51f-4f8d-a758-c54668f50e26)


### Actual Results

As you can see the label in the top left cell is unreadable:
![Unknown-2](https://github.com/scikit-learn/scikit-learn/assets/4438111/e03de808-079f-4475-881d-db974e7c36f0)


### Versions

```shell
System:
    python: 3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:23:33) [GCC 12.3.0]
executable: /opt/conda/bin/python
   machine: Linux-6.1.21-v8+-aarch64-with-glibc2.35

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 68.1.2
        numpy: 1.24.4
        scipy: 1.11.2
       Cython: 3.0.2
       pandas: 2.1.0
   matplotlib: 3.7.2
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 4
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: cortexa72

       user_api: openmp
   internal_api: openmp
    num_threads: 4
         prefix: libomp
       filepath: /opt/conda/lib/libomp.so
        version: None
```

COMMENT:
@ogrisel I don't understand your question, as the display does not have to guess the expected value for a `np.NaN`? Our usecase is specifically leaving some cells in the confusion matrix blank (to explicitly express the combination did not exist), however passing a `np.NaN` led to undefined behaviour in the text rendering due nan-propagation by two Numpy methods.

@glemaitre I think there are several ways to look at this issue: at the basis the input of a `np.NaN` leads to an issue with the text contrast rendering of the annotations. The ways to handle this that I see:

- Changing out the `min()` and `max()` functions for `nanmin()` and `nanmax()`, this should fix the text contrast issue. This does however not warn the user that a `np.NaN` was passed.
- The abovementioned fix, but a warning is raised that a `np.NaN` was passed. This both fixes the text contrast issue and informs the user that their input was "malformed"
- Your suggestion, `_assert_all_finite` to raise an error if a `np.NaN` is passed in the confusion matrix. 

I think the second option might be the best option as this fixes the unexpected behaviour when presented with a `np.NaN` but also warns the user.

Labels: Bug