"The Python kernel is unresponsive" when fitting a reasonable sized sparse matrix into NearestNeighbors

### Describe the bug

Hi all,

I have a python code that has been running every day for the past years, which uses NearestNeighbors to find best matches.
All of a sudden, in both our TEST and PRD environments, our code has been crashing on the NearestNeighbors function with the following message: "The Python kernel is unresponsive". This started last Friday 21st of March 2025.

What puzzles me is that we haven't made any modifications to our code, the data hasn't changed (at least in our TEST environment) and we didn't change the version of scikit-learn.
The exact command that throws the error is:

```python
nbrs = NearestNeighbors(n_neighbors = 1, metric = 'cosine').fit(X)
```

where X is a sparse matrix compressed to sparse rows that contains 38506x53709 elements.

We run the code on Databricks (runtime 15.4LTS, where scikit-learn is on 1.3.0).
I also tried with scikit-learn 1.4.2 (preinstalled in Databricks runtime 16.2) but had the same issue.

The error suggests a memory issue, but I'm struggling to understand why this would happen now while the context is exactly the same as what it was before. Furthermore, we use the same code with the same Databricks cluster for another data set which is at least 6x bigger and that one runs successfully in just a few seconds.

I'm not a data scientist and therefore quite confused as to why this would no longer run. Since our environment didn't change, I was wondering if anything would have changed in respect to scikit-learn v1.3.0 for any odd reason, or if you heard anything similar recently from some other user(s)?

### Steps/Code to Reproduce

```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.neighbors import NearestNeighbors

df_table = df_table.toPandas().add_prefix("b.")
vectorizer = TfidfVectorizer(analyzer = 'char', ngram_range = (1, 4))
X = vectorizer.fit_transform(df_table['b.concat_match_col'].values.astype('U'))
nbrs = NearestNeighbors(n_neighbors = 1, metric = 'cosine').fit(X)

# df_table['b.concat_match_col'] is a pandas dataframe that contains 7 columns
# and 38506 rows. I can't place the whole code to build the dataframe here
# because it's quite long
```

### Expected Results

No error should be thrown

### Actual Results

The NearestNeighbor function now returns this:

![Image](https://github.com/user-attachments/assets/d499e136-d4ad-4d00-bc6f-8477f5132232)

### Versions

```shell
System:
    python: 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0]
executable: /local_disk0/.ephemeral_nfs/envs/pythonEnv-de91d56e-aaea-4084-9eca-24ddcb3a19d4/bin/python
   machine: Linux-5.15.0-1078-azure-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.3.0
          pip: 23.2.1
   setuptools: 68.0.0
        numpy: 1.23.5
        scipy: 1.11.1
       Cython: 0.29.32
       pandas: 1.5.3
   matplotlib: 3.7.2
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True
Exception ignored on calling ctypes callback function: <function _ThreadpoolInfo._find_modules_with_dl_iterate_phdr.<locals>.match_module_callback at 0x7f407fbc1b20>
Traceback (most recent call last):
  File "/databricks/python/lib/python3.11/site-packages/threadpoolctl.py", line 400, in match_module_callback
    self._make_module_from_path(filepath)
  File "/databricks/python/lib/python3.11/site-packages/threadpoolctl.py", line 515, in _make_module_from_path
    module = module_class(filepath, prefix, user_api, internal_api)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/databricks/python/lib/python3.11/site-packages/threadpoolctl.py", line 606, in __init__
    self.version = self.get_version()
                   ^^^^^^^^^^^^^^^^^^
  File "/databricks/python/lib/python3.11/site-packages/threadpoolctl.py", line 646, in get_version
    config = get_config().split()
             ^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'split'

threadpoolctl info:
       filepath: /databricks/python3/lib/python3.11/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
         prefix: libopenblas
       user_api: blas
   internal_api: openblas
        version: 0.3.21.dev
    num_threads: 8
threading_layer: pthreads
   architecture: Zen

       filepath: /databricks/python3/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
         prefix: libgomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 8
```

COMMENT:
I know that we introduced some improvement in pairwise distance but it was in 1.2.
It would indeed but useful to get the traceback to be able to have an actionable here. I'm not closing this issue but I'm adding the tag "Needs info". @fabienarnaud Feel free to report any additional information that could be useful.

Labels: Bug, Needs Info