partial dependence display generates empty plot with all grid values being nan

### Describe the bug

I trained a binary classifier using XGBoost, and I was trying to generate partial dependence plot for each feature in my dataset. The partial_dependence() and PartialDependenceDisplay.from_estimator() function worked fine for all my features except one - an empty plot was generated, and all the grid values were nan. I suspect that it might be due to the unusual distribution of this feature since 50% of the values are zero. I tried replacing all zeros with np.nan to correct the distribution, but it didn't solve the issue. Could you please help troubleshoot this problem? Thank you.
<img width="161" alt="image" src="https://github.com/scikit-learn/scikit-learn/assets/75292024/961da072-6646-4016-97d2-f37bc1cd4932">


### Steps/Code to Reproduce

```python
fig, ax = plt.subplots(1, 1, figsize=(12, 10))                          
PartialDependenceDisplay.from_estimator(estimator=xgb_best, 
                                        X=X_oot, 
                                        features=[feat],
                                        categorical_features=cat_features,
                                        grid_resolution=3,
                                        ax=ax)
                           
raw_values = partial_dependence(xgb_best, 
                                temp, 
                                feat, 
                                kind="average",
                                categorical_features=cat_features
                                )
```

### Expected Results

Expected a partial dependence plot with a line.

### Actual Results

```python
partial_dependence.py:972: UserWarning: Attempting to set identical low and high ylims makes transformation singular; automatically expanding.
  ax.set_ylim([min_val, max_val])
```

<img width="590" alt="image" src="https://github.com/scikit-learn/scikit-learn/assets/75292024/b1a6ecd0-9f26-47c4-ba51-ba145f2590dc">
<img width="611" alt="image" src="https://github.com/scikit-learn/scikit-learn/assets/75292024/e86f4fe9-f558-4463-82b5-8a128e43fd92">

### Versions

```shell
System:
    python: 3.8.18 (default, Sep 11 2023, 08:17:16)  [Clang 14.0.6 ]
   machine: macOS-13.6-arm64-arm-64bit

Python dependencies:
      sklearn: 1.3.1
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.24.4
        scipy: 1.10.1
       Cython: None
       pandas: 2.0.3
   matplotlib: 3.7.3
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libopenblas
        version: 0.3.21
threading_layer: pthreads
   architecture: armv8

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libomp
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libopenblas
        version: 0.3.18
threading_layer: pthreads
   architecture: armv8

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libomp
        version: None
```

COMMENT:
@wennaq Can you provide a minimal reproducer:
https://scikit-learn.org/stable/developers/minimal_reproducer.html

Labels: Needs Reproducible Code