Importing sklearn takes too much time compared to other imports except spaCy

### Describe the bug

              Can you open a separate issue please with more details about your problem?

In particular, please include the following information in your new issue:
- is this a regression in scikit-learn 1.5, i.e. did your code work scikit-learn 1.4?
+ I tested other versions still same but it's slower than compared to older versions probably due to new features etc. But 1.5.1 less problematic than 1.5
- ideally a stand-alone snippet that reproduces the behaviour on your machine and that others can try to run. Without a stand-alone snippet to reproduce, there is very little we can do to help ...
`import sklearn` Takes 155 seconds to load. 
`from sklearn.metrics.pairwise import cosine_similarity` 384 seconds in my machine at python 3.12 and scikit-learn version 1.5.1 Windows 11 Latest version of Windows SDK and VM with 8gb ram. Other modules loaded less than 1 seconds. Full code of script: https://github.com/HydraDragonAntivirus/HydraDragonAntivirus/blob/main/antivirus.py
_Originally posted by @lesteve in https://github.com/scikit-learn/scikit-learn/issues/29145#issuecomment-2334105586_


### Steps/Code to Reproduce

`from sklearn.metrics.pairwise import cosine_similarity` 

### Expected Results

Excepted: Loaded in 2 minutes

### Actual Results

Results: `import sklearn` takes 155 seconds, `from sklearn.metrics.pairwise import cosine_similarity` takes 384 seconds

### Versions

```shell
Main machine: 
 python: 3.12.5 (tags/v3.12.5:ff3bc82, Aug  6 2024, 20:45:27) [MSC v.1940 64 bit (AMD64)]
executable: c:\Users\victim\AppData\Local\Programs\Python\Python312\python.exe
   machine: Windows-11-10.0.22631-SP0

Python dependencies:
      sklearn: 1.5.1
          pip: 24.2
   setuptools: 73.0.1
        numpy: 1.26.4
        scipy: 1.14.1
       Cython: 3.0.11
       pandas: None
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\victim\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\victim\AppData\Local\Programs\Python\Python312\Lib\site-packages\numpy.libs\libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Zen

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libscipy_openblas
       filepath: C:\Users\victim\AppData\Local\Programs\Python\Python312\Lib\site-packages\scipy.libs\libscipy_openblas-5b1ec8b915dfb81d11cebc0788069d2d.dll
        version: 0.3.27.dev
threading_layer: pthreads
   architecture: Zen
```
Notice: spaCy takes 1469 seconds to load.
Possible solution: Use multithreading or something similar at loading of import.

COMMENT:
I also checked the sizes of the files in that module:

```
tmp ❯ python -c "import sklearn.metrics.pairwise; print(sklearn.metrics.__path__[0])"
/Users/ogrisel/miniforge3/envs/tmp/lib/python3.12/site-packages/sklearn/metrics


tmp ❯ ls -lh /Users/ogrisel/miniforge3/envs/tmp/lib/python3.12/site-packages/sklearn/metrics
total 2664
-rw-rw-r--   5 ogrisel  staff   4.4K Jul  3 11:22 __init__.py
drwxr-xr-x   9 ogrisel  staff   288B Sep  6 17:27 __pycache__
-rw-rw-r--   5 ogrisel  staff   7.1K Jul  3 11:22 _base.py
-rw-rw-r--   5 ogrisel  staff   123K Jul  3 11:22 _classification.py
-rwxrwxr-x   5 ogrisel  staff   584K Jul  3 11:22 _dist_metrics.cpython-312-darwin.so
-rw-rw-r--   5 ogrisel  staff   7.2K Jul  3 11:22 _dist_metrics.pxd
-rw-rw-r--   5 ogrisel  staff   4.3K Jul  3 11:22 _dist_metrics.pxd.tp
-rw-rw-r--   5 ogrisel  staff    90K Jul  3 11:22 _dist_metrics.pyx.tp
drwxr-xr-x  26 ogrisel  staff   832B Sep  6 17:27 _pairwise_distances_reduction
-rwxrwxr-x   5 ogrisel  staff   226K Jul  3 11:22 _pairwise_fast.cpython-312-darwin.so
-rw-rw-r--   5 ogrisel  staff   3.4K Jul  3 11:22 _pairwise_fast.pyx
drwxr-xr-x  10 ogrisel  staff   320B Sep  6 17:27 _plot
-rw-rw-r--   5 ogrisel  staff    75K Jul  3 11:22 _ranking.py
-rw-rw-r--   5 ogrisel  staff    61K Jul  3 11:22 _regression.py
-rw-rw-r--   5 ogrisel  staff    35K Jul  3 11:22 _scorer.py
drwxr-xr-x  11 ogrisel  staff   352B Sep  6 17:27 cluster
-rw-rw-r--   5 ogrisel  staff   1.5K Jul  3 11:22 meson.build
-rw-rw-r--   5 ogrisel  staff    85K Jul  3 11:22 pairwise.py
drwxr-xr-x  12 ogrisel  staff   384B Sep  6 17:27 tests
```

The .so files (.dll under windows) are quite big (a few hundred kB) but still reasonable.

Labels: Bug, Needs Info