GridSearchCV cannot obtain the optimal parameter results.

### Describe the bug

When I used GridSearchCV to obtain the optimal parameters, I found that different cross-validation folds produced inconsistent results.

### Steps/Code to Reproduce

```python3
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.7, random_state=0)
param_grid_rf = {'n_estimators': [5, 10, 15, 35, 50, 70]}
grid_search = GridSearchCV(RandomForestClassifier(random_state=42), param_grid_rf, cv=3, scoring='roc_auc')
grid_search.fit(X_train, y_train)
best_params = grid_search.best_params_
print(f'{roc_auc_score(y_test, y_proba):.4f}')
```
> auc: 0.8859
best_params is 50

```python3
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.7, random_state=0)
param_grid_rf = {'n_estimators': [50, 70, 80, 90]}
grid_search = GridSearchCV(RandomForestClassifier(random_state=42), param_grid_rf, cv=3, scoring='roc_auc')
grid_search.fit(X_train, y_train)
best_params = grid_search.best_params_
print(f'{roc_auc_score(y_test, y_proba):.4f}')
```
> auc: 0.8725
best_params is 80

### Expected Results

Both results above should be identical.
>auc: 0.8859
best_params is 50

### Actual Results

Between the two parameter lists, we should obtain the parameters with the best AUC, not inconsistent ones.

### Versions

```shell
System:
    python: 3.10.12 (main, May 27 2025, 17:12:29) [GCC 11.4.0]
executable: /work/users/suny/mm/.venv/bin/python
   machine: Linux-5.15.0-143-generic-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.7.2
          pip: 25.2
   setuptools: 80.9.0
        numpy: 2.2.6
        scipy: 1.15.3
       Cython: None
       pandas: 2.3.2
   matplotlib: 3.10.6
       joblib: 1.5.2
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 64
         prefix: libscipy_openblas
       filepath: /work/users/suny/mm/.venv/lib/python3.10/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so
        version: 0.3.29
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 64
         prefix: libscipy_openblas
       filepath: /work/users/suny/mm/.venv/lib/python3.10/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 144
         prefix: libgomp
       filepath: /work/users/suny/mm/.venv/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: openmp
   internal_api: openmp
    num_threads: 144
         prefix: libgomp
       filepath: /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
        version: None
```

COMMENT:
Hi @SunYong0821, please provide a fully reproducible snippet for us to be able to reproduce the problem (for instance here we don't have `x` and `y`).

Let me try to rephrase the issue:
"Since 50 is the best param in the first case and since 50 is in the param grid of the second case, then the second case should have a better auc on the test set."

If this is what you expect, there's no guarantee for that. The best param is the one that provides the best score on the left-out fold in the internal cross-validation, but it doesn't mean that it will provide the best score on the test set.

What should be guaranteed is that in the second case the `best_score_` attribute is higher that in the first case.

Let me know if I understood your issue correctly.

Labels: Bug, Needs Info, Needs Reproducible Code