Partial dependence broken in sklearn 1.6.1 when grid has only two values

### Describe the bug

When our input feature has two possible values (and that the grid built in that function hence has two values), partial_dependence will raise an error `ValueError: cannot reshape array of size 1 into shape (2)`

What I suspect is happening is that inside `_partial_dependence_brute` function, there is a (wrongful) check to see if there are only two predicted values. This check should not be here because there `_get_response_values` seems to do the job of only getting the positive class already.


### Steps/Code to Reproduce

```python
from sklearn.tree import DecisionTreeClassifier
import numpy as np
from sklearn.inspection._partial_dependence import _partial_dependence_brute
from sklearn.inspection import partial_dependence

X_test = np.array([[1., 0], [0., 1], [0., 1], [0., 0], [1., 0], [0., 0], [0., 0]])
clf = DecisionTreeClassifier()
clf.fit(X_test, np.array([0, 1, 1, 0, 0, 0, 0]))

partial_dependence(clf, X=X_test, features=[0], grid_resolution=10, response_method="predict_proba")
```

### Expected Results

.

### Actual Results

```python-traceback
File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:216, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    [210](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:210) try:
    [211](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:211)     with config_context(
    [212](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:212)         skip_parameter_validation=(
    [213](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:213)             prefer_skip_nested_validation or global_skip_validation
    [214](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:214)         )
    [215](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:215)     ):
--> [216](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:216)         return func(*args, **kwargs)
    [217](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:217) except InvalidParameterError as e:
    [218](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:218)     # When the function is just a wrapper around an estimator, we allow
    [219](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:219)     # the function to delegate validation to the estimator, but we replace
    [220](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:220)     # the name of the estimator by the name of the function in the error
    [221](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:221)     # message to avoid confusion.
    [222](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:222)     msg = re.sub(
    [223](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:223)         r"parameter of \w+ must be",
    [224](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:224)         f"parameter of {func.__qualname__} must be",
    [225](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:225)         str(e),
    [226](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:226)     )

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/inspection/_partial_dependence.py:688, in partial_dependence(estimator, X, features, sample_weight, categorical_features, feature_names, response_method, percentiles, grid_resolution, method, kind)
    [684](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/inspection/_partial_dependence.py:684) print("value shapes", [val.shape[0] for val in values])
    [686](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/inspection/_partial_dependence.py:686) # reshape averaged_predictions to
    [687](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/inspection/_partial_dependence.py:687) # (n_outputs, n_values_feature_0, n_values_feature_1, ...)
--> [688](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/inspection/_partial_dependence.py:688) averaged_predictions = averaged_predictions.reshape(
    [689](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/inspection/_partial_dependence.py:689)     -1, *[val.shape[0] for val in values]
    [690](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/inspection/_partial_dependence.py:690) )
    [691](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/inspection/_partial_dependence.py:691) pdp_results = Bunch(grid_values=values)
    [693](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/inspection/_partial_dependence.py:693) if kind == "average":
```

ValueError: cannot reshape array of size 1 into shape (2)

### Versions

```shell
System:
    python: 3.10.11 (v3.10.11:7d4cc5aa85, Apr  4 2023, 19:05:19) [Clang 13.0.0 (clang-1300.0.29.30)]
executable: /usr/local/bin/python3.10
   machine: macOS-14.4.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.6.1
          pip: 24.2
   setuptools: 74.0.0
        numpy: 1.26.4
        scipy: 1.13.1
       Cython: 3.0.10
       pandas: 1.5.3
   matplotlib: 3.8.4
       joblib: 1.2.0
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: armv8

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.27
threading_layer: pthreads
   architecture: neoversen1

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sklearn/.dylibs/libomp.dylib
        version: None
```

COMMENT:
Hello, do you have an opinion on how things should be handled?

Labels: Bug, Regression