decomposition.PCA(svd_solver='covariance_eigh') is less stable with numpy==2.0

### Describe the bug

`decomposition.PCA(svd_solver='covariance_eigh')` is less stable with numpy==2.0

I noticed this issue as some tests started failing at the downstream [dask-ml/#997](https://github.com/dask/dask-ml/pull/997)

For a certain data input, `pca.transform` gives incredibly large values, which are not seen with numpy==1.24.3

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn import datasets, decomposition
X = datasets.make_low_rank_matrix(1000, 10, effective_rank=2, random_state=0, tail_strength=0)
pca = decomposition.PCA(n_components=None, whiten=True, svd_solver='auto').fit(X)
np.max(np.abs(pca.transform(X)))
```


### Expected Results

```python
>>> 3.9065841446726326  # with numpy==1.24.3
```


### Actual Results

```python
>>> np.float64(874957.7078303652)  # with numpy==2.0
```

Both uses sklearn==1.5.1, so probably an upstream issue.

Indeed, the singular values from numpy==2.0 contains zero
```python
pca.singular_values_
# with numpy==1.24.3
>>> array([9.99821683e-01, 7.78610978e-01, 3.67623087e-01, 1.05238902e-01,
       1.83047062e-02, 1.92838400e-03, 1.23400336e-04, 4.77881459e-06,
       1.12514225e-07, 2.93106569e-09])
# with numpy==2.0
>>> array([9.99821683e-01, 7.78610978e-01, 3.67623087e-01, 1.05238902e-01,
       1.83047062e-02, 1.92838400e-03, 1.23400336e-04, 4.77880939e-06,
       1.12582366e-07, 0.00000000e+00])
```
Probably this zero makes something wrong?

### Versions

```shell
### numpy==2.0 configuration

In [10]: sklearn.show_versions()

System:
    python: 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 15:03:56) [MSC v.1929 64 bit (AMD64)]
executable: C:\Users\oqf\AppData\Local\anaconda3\envs\numpy2\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.5.1
          pip: 24.0
   setuptools: 69.5.1
        numpy: 2.0.0
        scipy: 1.14.0
       Cython: None
       pandas: 2.2.2
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: vcomp
       filepath: C:\Users\oqf\AppData\Local\anaconda3\envs\numpy2\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libscipy_openblas
       filepath: C:\Users\oqf\AppData\Local\anaconda3\envs\numpy2\Lib\site-packages\numpy.libs\libscipy_openblas64_-fb1711452d4d8cee9f276fd1449ee5c7.dll    
        version: 0.3.27
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libscipy_openblas
       filepath: C:\Users\oqf\AppData\Local\anaconda3\envs\numpy2\Lib\site-packages\scipy.libs\libscipy_openblas-5b1ec8b915dfb81d11cebc0788069d2d.dll       
        version: 0.3.27.dev
threading_layer: pthreads
   architecture: Haswell


### numpy==1.24.3 configuration
```python
System:
    python: 3.11.5 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:26:23) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\oqf\AppData\Local\anaconda3\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.5.1
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.24.3
        scipy: 1.11.1
       Cython: 3.0.10
       pandas: 2.0.3
   matplotlib: 3.7.2
       joblib: 1.2.0
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: mkl
    num_threads: 10
         prefix: mkl_rt
       filepath: C:\Users\oqf\AppData\Local\anaconda3\Library\bin\mkl_rt.2.dll
        version: 2023.1-Product
threading_layer: intel

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: vcomp
       filepath: C:\Users\oqf\AppData\Local\anaconda3\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libiomp
       filepath: C:\Users\oqf\AppData\Local\anaconda3\Library\bin\libiomp5md.dll
        version: None
```
```

COMMENT:
@jeremiedbb was your solution here to replace the eigenvalues<0 with finf.eps?

Labels: Bug