KMeans `predict` method fails when run in ray remote function due to in-place modification of `cluster_centers_` attribute

### Describe the bug

When KMeans object is passed to a ray remote function, calls to method `predict` fail with the following error:

```
---------------------------------------------------------------------------
RayTaskError(ValueError)                  Traceback (most recent call last)
Input In [2], in <cell line: 14>()
     10     y = model.predict(iris.data)
     12 initialize_ray("auto")
---> 14 ray.get(predict.remote(trained_model))

File /opt/miniconda/envs/playground/lib/python3.8/site-packages/ray/_private/client_mode_hook.py:105, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    103     if func.__name__ != "init" or is_client_mode_enabled_by_default:
    104         return getattr(ray, func.__name__)(*args, **kwargs)
--> 105 return func(*args, **kwargs)

File /opt/miniconda/envs/playground/lib/python3.8/site-packages/ray/worker.py:1831, in get(object_refs, timeout)
   1829     worker.core_worker.dump_object_store_memory_usage()
   1830 if isinstance(value, RayTaskError):
-> 1831     raise value.as_instanceof_cause()
   1832 else:
   1833     raise value

RayTaskError(ValueError): ray::predict() (pid=143, ip=10.134.69.151)
  File "/tmp/ipykernel_1163/365058326.py", line 10, in predict
  File "/opt/miniconda/envs/playground/lib/python3.8/site-packages/sklearn/cluster/_kmeans.py", line 1334, in predict
    return _labels_inertia_threadpool_limit(
  File "/opt/miniconda/envs/playground/lib/python3.8/site-packages/sklearn/cluster/_kmeans.py", line 756, in _labels_inertia_threadpool_limit
    labels, inertia = _labels_inertia(
  File "/opt/miniconda/envs/playground/lib/python3.8/site-packages/sklearn/cluster/_kmeans.py", line 733, in _labels_inertia
    _labels(
  File "sklearn/cluster/_k_means_lloyd.pyx", line 31, in sklearn.cluster._k_means_lloyd.lloyd_iter_chunked_dense
  File "stringsource", line 658, in View.MemoryView.memoryview_cwrapper
  File "stringsource", line 349, in View.MemoryView.memoryview.__cinit__
ValueError: buffer source array is read-only
```

The root cause of the issue is the fact that the `predict` function is trying to do an in-place modification of `self.cluster_centers_`.

The `predict` function passes self.cluster_centers_ to `_labels_inertia_threadpool_limit` that passes it to `_labels_inertia`. This latter function calls [sklearn.cluster._k_means_lloyd.lloyd_iter_chunked_dense](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/cluster/_k_means_lloyd.pyx#L24) passing the reference to `self.cluster_centers_` as 4th and 5th parameter (see [this](https://github.com/scikit-learn/scikit-learn/blob/36958fb240fbe435673a9e3c52e769f01f36bec0/sklearn/cluster/_kmeans.py#L760-L771)).

[This function](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/cluster/_k_means_lloyd.pyx#L24) tries to modify this object [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/cluster/_k_means_lloyd.pyx#L112) causing the job to fail since the numpy array is stored as a read-only object (see [ray docs](https://docs.ray.io/en/master/ray-core/objects/serialization.html#numpy-arrays)).





### Steps/Code to Reproduce

Train script (works)
```python
import pickle

from sklearn.cluster import KMeans
from sklearn import datasets

iris = datasets.load_iris()
X = iris.data
y = iris.target
est = KMeans(n_clusters=8)
est.fit(X)
with open('model.pk', 'wb') as file:
    pickle.dump(est, file)
```

Predict script (fails)
```python
import ray
import pickle

with open('model.pk', 'rb') as file:
    trained_model = pickle.load(file)

@ray.remote
def predict(model):
    iris = datasets.load_iris()
    y = model.predict(iris.data)

ray.init()

ray.get(predict.remote(trained_model))
```

### Expected Results

No error is thrown

### Actual Results

```
---------------------------------------------------------------------------
RayTaskError(ValueError)                  Traceback (most recent call last)
Input In [2], in <cell line: 14>()
     10     y = model.predict(iris.data)
     12 initialize_ray("auto")
---> 14 ray.get(predict.remote(trained_model))

File /opt/miniconda/envs/playground/lib/python3.8/site-packages/ray/_private/client_mode_hook.py:105, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    103     if func.__name__ != "init" or is_client_mode_enabled_by_default:
    104         return getattr(ray, func.__name__)(*args, **kwargs)
--> 105 return func(*args, **kwargs)

File /opt/miniconda/envs/playground/lib/python3.8/site-packages/ray/worker.py:1831, in get(object_refs, timeout)
   1829     worker.core_worker.dump_object_store_memory_usage()
   1830 if isinstance(value, RayTaskError):
-> 1831     raise value.as_instanceof_cause()
   1832 else:
   1833     raise value

RayTaskError(ValueError): ray::predict() (pid=143, ip=10.134.69.151)
  File "/tmp/ipykernel_1163/365058326.py", line 10, in predict
  File "/opt/miniconda/envs/playground/lib/python3.8/site-packages/sklearn/cluster/_kmeans.py", line 1334, in predict
    return _labels_inertia_threadpool_limit(
  File "/opt/miniconda/envs/playground/lib/python3.8/site-packages/sklearn/cluster/_kmeans.py", line 756, in _labels_inertia_threadpool_limit
    labels, inertia = _labels_inertia(
  File "/opt/miniconda/envs/playground/lib/python3.8/site-packages/sklearn/cluster/_kmeans.py", line 733, in _labels_inertia
    _labels(
  File "sklearn/cluster/_k_means_lloyd.pyx", line 31, in sklearn.cluster._k_means_lloyd.lloyd_iter_chunked_dense
  File "stringsource", line 658, in View.MemoryView.memoryview_cwrapper
  File "stringsource", line 349, in View.MemoryView.memoryview.__cinit__
ValueError: buffer source array is read-only
```

### Versions

```shell
System:
    python: 3.8.12 | packaged by conda-forge | (default, Jan 30 2022, 23:42:07)  [GCC 9.4.0]
executable: /opt/miniconda/envs/playground/bin/python
   machine: Linux-5.10.90+-x86_64-with-glibc2.10

Python dependencies:
          pip: 21.3.1
   setuptools: 59.8.0
      sklearn: 1.0.2
        numpy: 1.21.6
        scipy: 1.9.0
       Cython: 0.29.32
       pandas: 1.3.5
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
My knowledge of the KMeans algorithm might be limited but I find it strange that the `predict` method tries to modify an object attribute.

Labels: Bug, module:cluster