KNeighborsClassifier reports different nearest neighbors and decision boundary depending on sys.platform

### Describe the bug

Training a `KNeighborsClassifier` on the iris dataset produces output that seems to depend on the system architecture (Linux, Mac, Windows tested). The order of neighboring points returned by `KNeighborsClassifier.kneighbors()` is slightly different for specific points near the decision boundary, and in some cases there are differences in the actual neighbors returned (presumably when the order is different causing a difference near `n_neighbors`).

I can theorise reasons why there might be these small differences, but I cannot find this documented anywhere, and therefore wondered if it were a bug or (if I hadn't missed something) whether something could be added to the documentation. There is no difference in the data produced by `load_iris()` and `train_test_split()`, so I know it's not that.

### Steps/Code to Reproduce

```python
# /// script
# requires-python = "==3.12.8"
# dependencies = [
#     "matplotlib",
#     "numpy==2.2.1",
#     "pandas==2.2.3",
#     "scikit-learn==1.6.1",
#     "scipy==1.15.1",
# ]
# ///

import hashlib
import sys

import numpy as np
from sklearn.datasets import load_iris
from sklearn.inspection import DecisionBoundaryDisplay
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(as_frame=True, return_X_y=True)
X = X[["sepal length (cm)", "sepal width (cm)"]]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)

breakpoint()

model = KNeighborsClassifier(n_neighbors=20).fit(X_train, y_train)

# Pick a point that is classified differently on different platforms
point_of_interest = [6.35, 2.80]
print("classification", model.predict([point_of_interest]))
print("neighbors", model.kneighbors([point_of_interest], return_distance=False))
print("hash of training data", hashlib.md5((str(X_train) + str(y_train)).encode()).hexdigest())

display = DecisionBoundaryDisplay.from_estimator(model, X)
display.ax_.plot(*point_of_interest, 'rx')
display.ax_.set_title(sys.platform)
display.figure_.savefig(f"decision_boundary_{sys.platform}.png")
```

### Expected Results

Results as consistent across platforms.

### Actual Results

On Linux (WSL):

```
classification [1]
neighbors [[26 86 10 52 25 79 72 49  6 59 70 71 73 29 57 76 16 82 68 87]]
hash of training data 40c745f2bcd69857949d2229ac5bcdd4
```

![Decision boundary on Linux](https://github.com/user-attachments/assets/f758a181-d2fe-4da3-9e73-071358affcaf)

---

On Windows:

```
classification [2]
neighbors [[26 86 10 52 25 79 72  6 49 59 70 73 71 29 76 57 16 82 77 68]]
hash of training data 40c745f2bcd69857949d2229ac5bcdd4
```

![Decision boundary on Windows](https://github.com/user-attachments/assets/0eeb56c8-c98b-4d5a-b999-d4ed3ba0ade6)

---

On Mac:

```
classification [2]
neighbors [[26 86 10 52 25 79 72  6 49 59 70 73 71 29 76 57 16 82 77  2]]
hash of training data 40c745f2bcd69857949d2229ac5bcdd4
```

![Decision boundary on Mac](https://github.com/user-attachments/assets/e186ccff-e084-46fc-927b-71e68a07549a)

### Versions

```shell
On Linux:

System:
    python: 3.12.8 (main, Jan  5 2025, 05:33:15) [Clang 19.1.6 ]
executable: /home/user/.cache/uv/archive-v0/WfveLKj0nlDrbuzgwXgFU/bin/python3
   machine: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.6.1
          pip: None
   setuptools: None
        numpy: 2.2.1
        scipy: 1.15.1
       Cython: None
       pandas: 2.2.3
   matplotlib: 3.10.0
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: /home/user/.cache/uv/archive-v0/WfveLKj0nlDrbuzgwXgFU/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-6bb31eeb.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: /home/user/.cache/uv/archive-v0/WfveLKj0nlDrbuzgwXgFU/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 20
         prefix: libgomp
       filepath: /home/user/.cache/uv/archive-v0/WfveLKj0nlDrbuzgwXgFU/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None


On Windows:

System:
    python: 3.12.8 (main, Dec 19 2024, 14:41:01) [MSC v.1942 64 bit (AMD64)]
executable: C:\Users\user\AppData\Local\uv\cache\archive-v0\h7HWruVa4wULKAO9-PD5v\Scripts\python.exe
   machine: Windows-11-10.0.22631-SP0

Python dependencies:
      sklearn: 1.6.1
          pip: None
   setuptools: None
        numpy: 2.2.1
        scipy: 1.15.1
       Cython: None
       pandas: 2.2.3
   matplotlib: 3.10.0
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: C:\Users\user\AppData\Local\uv\cache\archive-v0\h7HWruVa4wULKAO9-PD5v\Lib\site-packages\numpy.libs\libscipy_openblas64_-43e11ff0749b8cbe0a615c9cf6737e0e.dll
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 20
         prefix: vcomp
       filepath: C:\Users\user\AppData\Local\uv\cache\archive-v0\h7HWruVa4wULKAO9-PD5v\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: C:\Users\user\AppData\Local\uv\cache\archive-v0\h7HWruVa4wULKAO9-PD5v\Lib\site-packages\scipy.libs\libscipy_openblas-f07f5a5d207a3a47104dca54d6d0c86a.dll
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell


On Mac:

System:
    python: 3.12.8 (main, Dec  6 2024, 19:42:06) [Clang 18.1.8 ]
executable: /Users/user/.cache/uv/archive-v0/AKRvDW3X4a5KTGwOrcQJq/bin/python3
   machine: macOS-15.2-arm64-arm-64bit

Python dependencies:
      sklearn: 1.6.1
          pip: None
   setuptools: None
        numpy: 2.2.1
        scipy: 1.15.1
       Cython: None
       pandas: 2.2.3
   matplotlib: 3.10.0
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /Users/user/.cache/uv/archive-v0/AKRvDW3X4a5KTGwOrcQJq/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib
        version: None
```

COMMENT:
Hi @jatonline, This kind of behavior is not that surprising for an estimator that relies on ordered distances. Scikit-learn uses a BLAS library (MKL, OpenBLAS, ...) to compute distances and it's possible that they produce slightly different results on different architectures due to different rounding error accumulations for instance. Then selecting the n smallest distances can make 1 neighbor switch if 2 points are at almost the same distance (with a difference close to machine precision).

I don't think it's relevant to document this for `KNN` especially since it's a general behavior. But adding an item in the FAQ would make sense imo. Would you be interested in making a PR ?

Labels: Documentation