ConfusionMatrixDisplay does not correctly change text color when confusion matrix contains NaN

### Describe the bug

In our specific usecase we generate a Confusion Matrix using our own software to be passed on to ConfusionMatrixDisplay. Due to the nature of our needs this confusion matrix could contain one or more NaN values (as we want to make it explicit that this combination does not exist instead of replacing NaN by 0).

When passing a confusion matrix to ConfusionMatrixDisplay and `include_values` is set to True the `.plot()` method is unable to properly colour the text of the included values. This is due to the fact that the `thresh` variable ends up being NaN:
https://github.com/scikit-learn/scikit-learn/blob/52b692511450f7439eb929ad16933871c9a6ba68/sklearn/metrics/_plot/confusion_matrix.py#L156 

Both `.max()` and `.min()` propagate NaN as per the [Numpy docs](https://numpy.org/doc/stable/reference/generated/numpy.min.html). Their suggestion is to use `nanmin()` and `nanmax()`.

I am now wondering if this could be a sensible adaptation for this method (I'm willing to implement it myself and open a pull request). 



### Steps/Code to Reproduce

```
import numpy as np
from sklearn.metrics import ConfusionMatrixDisplay

conf_mat = np.array([[1, 2], [3, np.nan]])

disp = ConfusionMatrixDisplay(confusion_matrix = conf_mat)

disp.plot()
```

### Expected Results

Expected result would be a confusion matrix plot with correctly coloured text in the cells, similar to this example without a NaN value:
![Unknown-1](https://github.com/scikit-learn/scikit-learn/assets/4438111/2ca5b20e-a51f-4f8d-a758-c54668f50e26)


### Actual Results

As you can see the label in the top left cell is unreadable:
![Unknown-2](https://github.com/scikit-learn/scikit-learn/assets/4438111/e03de808-079f-4475-881d-db974e7c36f0)


### Versions

```shell
System:
    python: 3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:23:33) [GCC 12.3.0]
executable: /opt/conda/bin/python
   machine: Linux-6.1.21-v8+-aarch64-with-glibc2.35

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 68.1.2
        numpy: 1.24.4
        scipy: 1.11.2
       Cython: 3.0.2
       pandas: 2.1.0
   matplotlib: 3.7.2
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 4
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: cortexa72

       user_api: openmp
   internal_api: openmp
    num_threads: 4
         prefix: libomp
       filepath: /opt/conda/lib/libomp.so
        version: None
```

COMMENT:
> Correct, but the advantage of the NaN route is that NaNs are represented as white squares as they do not fit on the colourscale. This makes them visually distinct from values that lie very close to the lower bound of the colourscale.

But the value `np.nan` is semantically wrong. I don't think this the right things to do. Maybe a better solution is to use a non-linear colormap to highlight these specific values.

Labels: Bug