Repository: scipy/scipy

Issue:
Does kmeans "drop" clusters?
<!-- 
Thank you for taking the time to file a bug report. 
Please fill in the fields below, deleting the sections that 
don't apply to your issue. You can view the final output
by clicking the preview button above.
Note: This is a comment, and won't appear in the output.
-->

Hi, I am using [`scipy.cluster.vq.kmeans`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.vq.kmeans.html), and I noticed that on occasion, the number of clusters returned is not the same as the number of clusters passed in via the `k_or_guess` parameter. Specifically, I am passing in 10,000 points, and asking for 2000 clusters (I am approximating a Centroidal Voronoi Tesselation), and on occasion, I am getting back 1,999 or 1,998 clusters.

Is this behavior intended? If so:

1. Is it the result of clusters not having points assigned to them?
1. It does not seem to be mentioned in the documentation (though perhaps I missed it).
1. What can I do to mitigate it? Would increasing number of iterations (i.e. number of times kmeans is run) or number of samples help?

Thanks!

#### Reproducing code example:
<!-- 
If you place your code between the triple backticks below, 
it will be rendered as a code block. 
-->

This code should roughly replicate the error.

```python
import numpy as np
from scipy.cluster.vq import kmeans

samples = np.random.uniform(
    np.array([-1, -1]),
    np.array([1, 1]),
    size=(10_000, 2),
)

for i in range(10):  # Or however many itrs.
    centroids = kmeans(samples, 2000, iter=1)[0]
    print(len(centroids))
```

#### Error message:
<!-- If any, paste the *full* error message inside a code block
as above (starting from line Traceback)
-->

The output from the above code is shown below. I would expect to only see 2000, no 1999.

```text
2000
2000
2000
1999
1999
1999
2000
2000
2000
2000
```

#### Scipy/Numpy/Python version information:
<!-- You can simply run the following and paste the result in a code block
```
import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
```
-->

```
1.5.4 1.19.4 sys.version_info(major=3, minor=7, micro=8, releaselevel='final', serial=0)
```


Discussion:
Hi. Thanks for reporting. I checked this behavior, same thing happened in my environment.

> Is this behavior intended? If so:

I think this is intended. The reason why the centroid number was changed is that some controids was assigned to no points, and this code removed these non-assigned centroids. 
https://github.com/scipy/scipy/blob/5f4c4d802e5a56708d86909af6e5685cd95e6e66/scipy/cluster/vq.py#L311

> 1. Is it the result of clusters not having points assigned to them?

Yes. I think your understanding is correct.

> It does not seem to be mentioned in the documentation (though perhaps I missed it).

Yes. I think the document is missing it. I think the document should be improved.

> What can I do to mitigate it? Would increasing number of iterations (i.e. number of times kmeans is run) or number of samples help?

You can use [scipy\.cluster\.vq\.kmeans2](http://scipy.github.io/devdocs/generated/scipy.cluster.vq.kmeans2.html#scipy.cluster.vq.kmeans2)

This function can keep the number of centroids.
https://github.com/scipy/scipy/blob/21d2b6ff75d0f82dc48f70f5d9e0fde07ee2d813/scipy/cluster/vq.py#L750-L755

Resolving patch:
diff --git a/scipy/cluster/vq.py b/scipy/cluster/vq.py
--- a/scipy/cluster/vq.py
+++ b/scipy/cluster/vq.py
@@ -368,6 +368,9 @@ def kmeans(obs, k_or_guess, iter=20, thresh=1e-5, check_finite=True):
        codebook[i] is represented with the code i. The centroids
        and codes generated represent the lowest distortion seen,
        not necessarily the globally minimal distortion.
+       Note that the number of centroids is not necessarily the same as the
+       ``k_or_guess`` parameter, because centroids assigned to no observations
+       are removed during iterations.
 
     distortion : float
        The mean (non-squared) Euclidean distance between the observations
@@ -384,6 +387,13 @@ def kmeans(obs, k_or_guess, iter=20, thresh=1e-5, check_finite=True):
     whiten : must be called prior to passing an observation matrix
        to kmeans.
 
+    Notes
+    -----
+    For more functionalities or optimal performance, you can use
+    `sklearn.cluster.KMeans <https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html>`_.
+    `This <https://hdbscan.readthedocs.io/en/latest/performance_and_scalability.html#comparison-of-high-performance-implementations>`_
+    is a benchmark result of several implementations.
+
     Examples
     --------
     >>> from numpy import array


Test patch: