Partial Dependence Plot orients differently compared to Partial Dependence values

### Describe the bug

The issue is that the 2D partial dependence plot from scikit-learn orients in a different way that what you would get using raw pdp values from sklearn as well.



### Steps/Code to Reproduce

```python
import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingRegressor


DATA_COUNT = 10000

RANDOM_SEED = 137

TRAIN_FRACTION = 0.9

np.random.seed(RANDOM_SEED)

xs = pd.DataFrame(np.random.uniform(size = (DATA_COUNT, 3)))
xs.columns = ['x0', 'x1', 'x2']

y = pd.DataFrame(xs.x0*xs.x1 + xs.x2/xs.x1 + pd.Series(0.1*np.random.randn(DATA_COUNT)))
y.columns = ['y']

gbr_1 = GradientBoostingRegressor(random_state = RANDOM_SEED)
gbr_1.fit(xs, y)


from sklearn.inspection import partial_dependence, PartialDependenceDisplay

PartialDependenceDisplay.from_estimator(gbr_1, xs, [(0, 1)],grid_resolution=15,percentiles=(0.01,0.99),
                                            contour_kw=dict(cmap='bwr'))

# Versus 

pdp1 = partial_dependence(gbr_1, xs, features=[(0, 1)], kind='average', percentiles=(0.01,0.99),grid_resolution=15)
y_pred1 = pdp1['average']
grid1 = pdp1['values']

x,y = np.meshgrid(grid1[0],grid1[1])
plt.contourf(x,y,y_pred1[0],cmap='bwr',levels=7)


```

### Expected Results

```python
from sklearn.inspection import plot_partial_dependence,partial_dependence, PartialDependenceDisplay

PartialDependenceDisplay.from_estimator(gbr_1, xs, [(0, 2)],grid_resolution=15,percentiles=(0.01,0.99),
                                            contour_kw=dict(cmap='bwr'))
```


![image](https://user-images.githubusercontent.com/49596459/214959496-168287b8-6be1-447f-ae39-3b536c9ace4e.png)


### Actual Results

```python
pdp1 = partial_dependence(gbr_1, xs, features=[(0, 2)], kind='average', percentiles=(0.01,0.99),grid_resolution=15)
y_pred1 = pdp1['average']
grid1 = pdp1['values']

x,y = np.meshgrid(grid1[0],grid1[1])
plt.contourf(x,y,y_pred1[0],cmap='bwr',levels=7)
plt.xlabel('x0')
plt.ylabel('x2')
```

![image](https://user-images.githubusercontent.com/49596459/214959548-c40d8324-c204-40d4-ad7f-0d2cee1e1f70.png)


### Versions

```shell
System:
    python: 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:04:59) [GCC 10.3.0]
executable: /glade/work/chayan/conda-envs/pangeo_chayan/bin/python
   machine: Linux-3.10.0-1127.18.2.el7.x86_64-x86_64-with-glibc2.17

Python dependencies:
      sklearn: 1.1.1
          pip: 22.1.2
   setuptools: 63.2.0
        numpy: 1.23.5
        scipy: 1.8.1
       Cython: 0.29.30
       pandas: 1.4.3
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /glade/work/chayan/conda-envs/pangeo_chayan/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 1

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /glade/work/chayan/conda-envs/pangeo_chayan/lib/libgomp.so.1.0.0
        version: None
    num_threads: 1

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /glade/work/chayan/conda-envs/pangeo_chayan/lib/python3.10/site-packages/xgboost.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 1
```

COMMENT:
Your example is programmatically working because we are using a square grid.
Another way to see that your way of plotting is wrong is to compute the average prediction by hand for a specific case:

```python
In [18]: x0 = 0.3
In [20]: x2 = 0.8
In [24]: X_for_pred = np.array([[x0, data.x1, x2] for _, data in xs.iterrows()])
In [27]: gbr_1.predict(X_for_pred).mean()
/Users/glemaitre/Documents/packages/scikit-learn/sklearn/base.py:432: UserWarning: X does not have valid feature names, but GradientBoostingRegressor was fitted with feature names
  warnings.warn(
Out[27]: 12.16840549200049
```

It corresponds to the red area on the partial dependence of `PartialDependenceDisplay`. For your plot, you see that you have a low prediction in comparison to the average prediction.

Labels: Bug, Needs Triage