cross_validate error? when processing regression estiamtor.

### Describe the bug

The test_score return  by cross_validate is not equal to the estimator.score() function.
![image](https://github.com/scikit-learn/scikit-learn/assets/35194180/865a696b-04ee-4576-b1e5-e3aa358b4f31)
I know test_score is not exactly equal to svr[0].score(X,y), but the results are too different. 
And I use KFold to split dataset, and get test scores, they are very different to the test_score return  by cross_validate.
![image](https://github.com/scikit-learn/scikit-learn/assets/35194180/dfb5bc0f-dead-4bad-bbca-167505aec4a8)


### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.model_selection import train_test_split,cross_val_score,KFold,cross_validate
from sklearn import svm
X = np.sort(5 * np.random.rand(40, 1), axis=0)
y = np.sin(X).ravel()

# add noise to targets
y[::5] += 3 * (0.5 - np.random.rand(8))
svr_rbf = svm.SVR(kernel="rbf", C=100, gamma=0.1, epsilon=0.1)

scores = cross_validate(svr_rbf, X, y, cv=5,return_estimator=True)
scores

svr=scores["estimator"]
svr[0].score(X,y)

#add KF results
kf=KFold(n_splits=5,random_state=40,shuffle=True)
s=[]
for train_index, test_index in kf.split(X):
    X_train=X[train_index]
    y_train=y[train_index]
    X_test=X[test_index]
    y_test=y[test_index]
    s.append(svr[0].score(X_test,y_test))
print(s)


```


### Expected Results

like this. should be the r2 scores.
[0.1900817162767654, 0.9454646872235107, 0.5383648752174341, 0.7844985877839026, 0.9717599124665652]

### Actual Results

[-0.17922386, -0.25855697,  0.14471102,  0.28956888, -0.20603142]
too many negative numbers and the number is too small.

### Versions

```shell
System:
    python: 3.11.5 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:26:23) [MSC v.1916 64 bit (AMD64)]
executable: D:\ProgramData\Anaconda3\envs\sklearn-env\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.2.2
          pip: 23.0.1
   setuptools: 67.3.2
        numpy: 1.26.0
        scipy: 1.11.3
       Cython: None
       pandas: 2.1.1
   matplotlib: 3.7.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: mkl
         prefix: mkl_rt
       filepath: D:\ProgramData\Anaconda3\envs\sklearn-env\Library\bin\mkl_rt.2.dll
        version: 2023.1-Product
threading_layer: intel
    num_threads: 4

       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: D:\ProgramData\Anaconda3\envs\sklearn-env\vcomp140.dll
        version: None
    num_threads: 8
```

COMMENT:
The `cv` is not the same in both cross-validation (manual and thought the functions). Also you are not retraining the model. In short, the equivalence will be:

```python
import numpy as np
from sklearn.model_selection import cross_validate, KFold
from sklearn import svm

rng = np.random.RandomState(42)
X = np.sort(5 * rng.rand(40, 1), axis=0)
y = np.sin(X).ravel()

# add noise to targets
y[::5] += 3 * (0.5 - rng.rand(8))
svr_rbf = svm.SVR(kernel="rbf", C=100, gamma=0.1, epsilon=0.1)

cv = KFold(n_splits=5)
scores = cross_validate(svr_rbf, X, y, cv=cv,return_estimator=True)
scores["test_score"]
```

```
array([-0.3530594 , -0.17250208, -0.32750824, -0.27194168, -3.10684952])
```

```python
scores = []
for train_index, test_index in cv.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    scores.append(
        svr_rbf.fit(X_train, y_train).score(X_test, y_test)
    )
np.array(scores)
```

```
array([-0.3530594 , -0.17250208, -0.32750824, -0.27194168, -3.10684952])
```

You can add `shuffle=True, random_state=0` in the `KFold` but it will still be deterministic.

Labels: Bug, Needs Triage