Importing sklearn takes too much time compared to other imports except spaCy

### Describe the bug

              Can you open a separate issue please with more details about your problem?

In particular, please include the following information in your new issue:
- is this a regression in scikit-learn 1.5, i.e. did your code work scikit-learn 1.4?
+ I tested other versions still same but it's slower than compared to older versions probably due to new features etc. But 1.5.1 less problematic than 1.5
- ideally a stand-alone snippet that reproduces the behaviour on your machine and that others can try to run. Without a stand-alone snippet to reproduce, there is very little we can do to help ...
`import sklearn` Takes 155 seconds to load. 
`from sklearn.metrics.pairwise import cosine_similarity` 384 seconds in my machine at python 3.12 and scikit-learn version 1.5.1 Windows 11 Latest version of Windows SDK and VM with 8gb ram. Other modules loaded less than 1 seconds. Full code of script: https://github.com/HydraDragonAntivirus/HydraDragonAntivirus/blob/main/antivirus.py
_Originally posted by @lesteve in https://github.com/scikit-learn/scikit-learn/issues/29145#issuecomment-2334105586_


### Steps/Code to Reproduce

`from sklearn.metrics.pairwise import cosine_similarity` 

### Expected Results

Excepted: Loaded in 2 minutes

### Actual Results

Results: `import sklearn` takes 155 seconds, `from sklearn.metrics.pairwise import cosine_similarity` takes 384 seconds

### Versions

```shell
Main machine: 
 python: 3.12.5 (tags/v3.12.5:ff3bc82, Aug  6 2024, 20:45:27) [MSC v.1940 64 bit (AMD64)]
executable: c:\Users\victim\AppData\Local\Programs\Python\Python312\python.exe
   machine: Windows-11-10.0.22631-SP0

Python dependencies:
      sklearn: 1.5.1
          pip: 24.2
   setuptools: 73.0.1
        numpy: 1.26.4
        scipy: 1.14.1
       Cython: 3.0.11
       pandas: None
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\victim\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\victim\AppData\Local\Programs\Python\Python312\Lib\site-packages\numpy.libs\libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Zen

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libscipy_openblas
       filepath: C:\Users\victim\AppData\Local\Programs\Python\Python312\Lib\site-packages\scipy.libs\libscipy_openblas-5b1ec8b915dfb81d11cebc0788069d2d.dll
        version: 0.3.27.dev
threading_layer: pthreads
   architecture: Zen
```
Notice: spaCy takes 1469 seconds to load.
Possible solution: Use multithreading or something similar at loading of import.

COMMENT:
Here are the numbers on my system with a new conda env with a freshly install scikit-learn 1.5.1:

```
tmp ❯ time python -c "from sklearn.metrics.pairwise import cosine_similarity"
python -c "from sklearn.metrics.pairwise import cosine_similarity"  0.67s user 0.22s system 9% cpu 9.838 total

tmp ❯ time python -c "from sklearn.metrics.pairwise import cosine_similarity"
python -c "from sklearn.metrics.pairwise import cosine_similarity"  0.34s user 0.07s system 81% cpu 0.503 total
```

The first import is significantly slower because I think macOS is running some code signature checks in the background for newly created executable files upon first time access. But still it's only 9s. And after that it's very fast the next times.

Labels: Bug, Needs Info