Repository: ray-project/ray

Issue:
[tune] Ray Cluster deployed in Kubernetes unable to sync checkpoint directories to the driver node
### What is the problem?

Getting the following error (Stacktrace attached):
```
2020-07-17 21:53:48,101	ERROR trial_runner.py:550 -- Trial TrainExample_fd24b_00001: Error handling checkpoint /root/ray_results/TrainExample/TrainExample_1_randomforestclassifier__n_estimators=5_2020-07-17_21-53-462l3hkjfs/checkpoint_1/
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 546, in _process_trial_save
    trial.on_checkpoint(trial.saving_to)
  File "/opt/conda/lib/python3.6/site-packages/ray/tune/trial.py", line 448, in on_checkpoint
    self, checkpoint.value))
ray.tune.error.TuneError: Trial TrainExample_fd24b_00001: Checkpoint path /root/ray_results/TrainExample/TrainExample_1_randomforestclassifier__n_estimators=5_2020-07-17_21-53-462l3hkjfs/checkpoint_1/ not found after successful sync down.
```

This is output of `ray_random_forest.py`. From investigation it looks like the checkpoint directories ie. **checkpoint_1, checkpoint_2** that were created in other nodes (workers) of the cluster do not get synced back to the driver node (from where I am running the python script).
Rest of the files in the Trial directories seem to be in sync. 

This problem is not reproducible by running it on a single-machine with a cluster started up using bare `init()`. My guess is because all the workers point to the same file-system, and thus, the `os.path.exist()` returns True (This is reference to the line of code which checks and throws the above error)

Taking a look at `validate_save_restore()` to validate a Trainable, I have written a bare-bone script which proves that `checkpoint` directories are not synced across workers.


The stacktrace from `checkpoint_validation_failed.py`
```
Traceback (most recent call last):
  File "code/checkpoint_validation_failed.py", line 115, in <module>
    _main()
  File "code/checkpoint_validation_failed.py", line 94, in _main
    print(f"VALIDATE TRAINABLE CLASS: {validate_save_restore(TrainExample)}")
  File "code/checkpoint_validation_failed.py", line 67, in validate_save_restore
    restore_check = ray.get(trainable_2.restore.remote(old_trainable))
  File "/opt/conda/lib/python3.6/site-packages/ray/worker.py", line 1526, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(FileNotFoundError): ray::TrainExample.restore() (pid=2704, ip=172.17.0.8)
  File "python/ray/_raylet.pyx", line 471, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 424, in ray._raylet.execute_task.function_executor
  File "/opt/conda/lib/python3.6/site-packages/ray/tune/trainable.py", line 444, in restore
    with open(checkpoint_path + ".tune_metadata", "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/root/ray_results/2020-07-17_21-43-17dbvo01tp/checkpoint_3/.tune_metadata'
command terminated with exit code 1
```

Here, Worker X is trying to restore checkpoint created by Worker Y.


*Ray version and other system information (Python version, TensorFlow version, OS):*

Setup:
- Ray cluster running in K8s.
- Ray version
`pip list | grep ray
ray 0.9.0.dev0`
- Docker image: 
`
REPOSITORY                                                       TAG                 IMAGE ID            CREATED             SIZE
rayproject/autoscaler                                            latest              3f8d747cb8af        7 days ago          5.25GB
`

Playing with `sync_to_driver`, `sync_on_checkpoint` parameters did not help.

Thanks!

### Reproduction (REQUIRED)
Step 1. Spin-up a Ray cluster using `ray-cluster.yaml` in a namespace called `ray`. ie 
1.  `kubectl create -f ray-namespace.yaml`
2.  `kubectl apply -f ray-cluster.yaml`

Step 2. Copy the python script to any worker node.

1. `kubectl -n ray get pods`.
2. Copy WORKER_1_POD_NAME
3. Exec into pod `kubectl -n ray exec -it <WORKER_1_POD_NAME> -- bin/bash`
4. `mkdir code`
5. Copy the python script `kubectl -n ray cp checkpoint_validation_failed.py  <WORKER_1_POD_NAME>:/code/checkpoint_validation_failed.py`
6. Exit pod
7. Execute script `kubectl -n ray exec <WORKER_1_POD_NAME> -- bin/bash -c "python code/checkpoint_validation_failed.py`

You can do the same steps to run `ray_random_forest.py`.
[ray_issue_related_files.zip](https://github.com/ray-project/ray/files/4940566/ray_issue_related_files.zip)

The above zip contains:
1. ray-cluster.yaml
2. ray-namespace.yaml
3. checkpoint_validation_failed.py
4. ray_random_forest.py


If we cannot run your script, we cannot fix your issue.

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [ ] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:
Do you have an NFS for your kubernetes cluster? That might save you a lot of trouble.
Hello @richardliaw,

Thank you for a quick reply.

No, I do not have NFS for the cluster. Does it have to be a NFS? How does ray sync files like
1. events.out.tfevents.1595022826.ray-worker-6d9f6b9b8d-g4zvk
2. params.json
3. params.pkl
4. progress.csv
5. result.json

But not `checkpoint_1`, `checkpoint_2` etc.?
The files are actually generated on the driver, not on the node. Do you have access to S3? S3 could work too.
Okay. So what gets synced on `sync_on_checkpoint`?

sync_on_checkpoint (bool) – Force sync-down of trial **checkpoint** to driver. If set to
False, checkpoint syncing from worker to driver is asynchronous and best-effort. This does
not affect persistent storage syncing. Defaults to True. 

In the above documentation snippet I am interpreting **checkpoint** as the `checkpoint_N` directories described above. 

Would the suggested S3 solution involve implementing DurableTrainable rather than Trainable?
Also I wanted to clarify, is the above issue a bug in Ray Tune and a network/cloud storage a work-around for it?

Checkpoints (directories) are synced on sync_on_checkpoint, yep.

The above is just a limitation of our file-transfer mechanism; we use rsync to move files between VMs, but I think that mechanism breaks down on kubernetes. 

The two approaches to workaround this limitation are to use 1. a networked storage or 2. to implement your own file-transfer protocol (check out `sync_to_driver`). 

Does that make sense? Let me know if you have any further questions!
Thank you Richard for this clarification.

Upon searching more, I found this: https://serverfault.com/questions/741670/rsync-files-to-a-kubernetes-pod
I see similar solution applied to `KubernetesCommandRunner` in Ray. Maybe we can do the same for sync-checkpoint mechanism to resolve this issue. Or, we just implement it as a custom file-transfer protocol and make it part of the pallet. 

On a side note, I have taken your suggestion to use a NFS. It has unblocked me from aggregating all my checkpoints. 

Thanks once again.

Great! yeah, KubernetesCommandRunner is what we will want to use. More than happy to accept a PR if you want to take a quick pass! 

Otherwise, we'll try to get to this in the next couple of weeks.
I spent sometime last-week to understand if I can fix it, but for me it was not a going to be a quick pass. So I will defer it to the team. Thanks once again Richard. :)

Resolving patch:
diff --git a/python/ray/autoscaler/command_runner.py b/python/ray/autoscaler/command_runner.py
--- a/python/ray/autoscaler/command_runner.py
+++ b/python/ray/autoscaler/command_runner.py
@@ -189,10 +189,16 @@ def run_rsync_up(self, source, target):
             logger.warning(self.log_prefix +
                            "rsync failed: '{}'. Falling back to 'kubectl cp'"
                            .format(e))
-            self.process_runner.check_call(self.kubectl + [
-                "cp", source, "{}/{}:{}".format(self.namespace, self.node_id,
-                                                target)
-            ])
+            self.run_cp_up(source, target)
+
+    def run_cp_up(self, source, target):
+        if target.startswith("~"):
+            target = "/root" + target[1:]
+
+        self.process_runner.check_call(self.kubectl + [
+            "cp", source, "{}/{}:{}".format(self.namespace, self.node_id,
+                                            target)
+        ])
 
     def run_rsync_down(self, source, target):
         if target.startswith("~"):
@@ -209,10 +215,16 @@ def run_rsync_down(self, source, target):
             logger.warning(self.log_prefix +
                            "rsync failed: '{}'. Falling back to 'kubectl cp'"
                            .format(e))
-            self.process_runner.check_call(self.kubectl + [
-                "cp", "{}/{}:{}".format(self.namespace, self.node_id, source),
-                target
-            ])
+            self.run_cp_down(source, target)
+
+    def run_cp_down(self, source, target):
+        if target.startswith("~"):
+            target = "/root" + target[1:]
+
+        self.process_runner.check_call(self.kubectl + [
+            "cp", "{}/{}:{}".format(self.namespace, self.node_id, source),
+            target
+        ])
 
     def remote_shell_command_str(self):
         return "{} exec -it {} bash".format(" ".join(self.kubectl),
diff --git a/python/ray/tune/integration/kubernetes.py b/python/ray/tune/integration/kubernetes.py
new file mode 100644
--- /dev/null
+++ b/python/ray/tune/integration/kubernetes.py
@@ -0,0 +1,169 @@
+import kubernetes
+import subprocess
+
+from ray import services, logger
+from ray.autoscaler.command_runner import KubernetesCommandRunner
+from ray.tune.syncer import NodeSyncer
+from ray.tune.sync_client import SyncClient
+
+
+def NamespacedKubernetesSyncer(namespace, use_rsync=False):
+    """Wrapper to return a ``KubernetesSyncer`` for a Kubernetes namespace.
+
+    Args:
+        namespace (str): Kubernetes namespace.
+        use_rsync (bool):  Use ``rsync`` if True or ``kubectl cp``
+            if False. If True, ``rsync`` will need to be
+            installed in the Kubernetes pods for this to work.
+            If False, ``tar`` will need to be installed instead.
+
+    Returns: A ``KubernetesSyncer`` class to be passed to
+        ``tune.run(sync_to_driver)``.
+
+    Example:
+
+    .. code-block:: python
+
+        from ray.tune.integration.kubernetes import NamespacedKubernetesSyncer
+        tune.run(train,
+                 sync_to_driver=NamespacedKubernetesSyncer("ray"))
+
+    """
+
+    class _NamespacedKubernetesSyncer(KubernetesSyncer):
+        _namespace = namespace
+        _use_rsync = use_rsync
+
+    return _NamespacedKubernetesSyncer
+
+
+class KubernetesSyncer(NodeSyncer):
+    """KubernetesSyncer used for synchronization between Kubernetes pods.
+
+    This syncer extends the node syncer, but is usually instantiated
+    without a custom sync client. The sync client defaults to
+    ``KubernetesSyncClient`` instead.
+
+    KubernetesSyncer uses the default namespace ``ray``. You should
+    probably use ``NamespacedKubernetesSyncer`` to return a class
+    with a custom namespace instead.
+    """
+
+    _namespace = "ray"
+    _use_rsync = False
+
+    def __init__(self, local_dir, remote_dir, sync_client=None):
+        self.local_ip = services.get_node_ip_address()
+        self.local_node = self._get_kubernetes_node_by_ip(self.local_ip)
+        self.worker_ip = None
+        self.worker_node = None
+
+        sync_client = sync_client or KubernetesSyncClient(
+            namespace=self.__class__._namespace,
+            use_rsync=self.__class__._use_rsync)
+
+        super(NodeSyncer, self).__init__(local_dir, remote_dir, sync_client)
+
+    def set_worker_ip(self, worker_ip):
+        self.worker_ip = worker_ip
+        self.worker_node = self._get_kubernetes_node_by_ip(worker_ip)
+
+    def _get_kubernetes_node_by_ip(self, node_ip):
+        """Return node name by internal or external IP"""
+        kubernetes.config.load_incluster_config()
+        api = kubernetes.client.CoreV1Api()
+        pods = api.list_namespaced_pod(self._namespace)
+        for pod in pods.items:
+            if pod.status.host_ip == node_ip or \
+               pod.status.pod_ip == node_ip:
+                return pod.metadata.name
+
+        logger.error(
+            "Could not find Kubernetes pod name for IP {}".format(node_ip))
+        return None
+
+    @property
+    def _remote_path(self):
+        return (self.worker_node, self._remote_dir)
+
+
+class KubernetesSyncClient(SyncClient):
+    """KubernetesSyncClient to be used by KubernetesSyncer.
+
+    This client takes care of executing the synchronization
+    commands for Kubernetes clients. In its ``sync_down`` and
+    ``sync_up`` commands, it expects tuples for the source
+    and target, respectively, for compatibility with the
+    KubernetesCommandRunner.
+
+    Args:
+        namespace (str): Namespace in which the pods live.
+        use_rsync (bool): Use ``rsync`` if True or ``kubectl cp``
+            if False. If True, ``rsync`` will need to be
+            installed in the Kubernetes pods for this to work.
+            If False, ``tar`` will need to be installed instead.
+        process_runner: How commands should be called.
+            Defaults to ``subprocess``.
+
+    """
+
+    def __init__(self, namespace, use_rsync=False, process_runner=subprocess):
+        self.namespace = namespace
+        self.use_rsync = use_rsync
+        self._process_runner = process_runner
+        self._command_runners = {}
+
+    def _create_command_runner(self, node_id):
+        """Create a command runner for one Kubernetes node"""
+        return KubernetesCommandRunner(
+            log_prefix="KubernetesSyncClient: {}:".format(node_id),
+            namespace=self.namespace,
+            node_id=node_id,
+            auth_config=None,
+            process_runner=self._process_runner)
+
+    def _get_command_runner(self, node_id):
+        """Create command runner if it doesn't exist"""
+        # Todo(krfricke): These cached runners are currently
+        # never cleaned up. They are cheap so this shouldn't
+        # cause much problems, but should be addressed if
+        # the SyncClient is used more extensively in the future.
+        if node_id not in self._command_runners:
+            command_runner = self._create_command_runner(node_id)
+            self._command_runners[node_id] = command_runner
+        return self._command_runners[node_id]
+
+    def sync_up(self, source, target):
+        """Here target is a tuple (target_node, target_dir)"""
+        target_node, target_dir = target
+
+        # Add trailing slashes for rsync
+        source += "/" if not source.endswith("/") else ""
+        target_dir += "/" if not target_dir.endswith("/") else ""
+
+        command_runner = self._get_command_runner(target_node)
+        if self.use_rsync:
+            command_runner.run_rsync_up(source, target_dir)
+        else:
+            command_runner.run_cp_up(source, target_dir)
+        return True
+
+    def sync_down(self, source, target):
+        """Here source is a tuple (source_node, source_dir)"""
+        source_node, source_dir = source
+
+        # Add trailing slashes for rsync
+        source_dir += "/" if not source_dir.endswith("/") else ""
+        target += "/" if not target.endswith("/") else ""
+
+        command_runner = self._get_command_runner(source_node)
+        if self.use_rsync:
+            command_runner.run_rsync_down(source_dir, target)
+        else:
+            command_runner.run_cp_down(source_dir, target)
+        return True
+
+    def delete(self, target):
+        """No delete function because it is only used by
+        the KubernetesSyncer, which doesn't call delete."""
+        return True
diff --git a/python/ray/tune/syncer.py b/python/ray/tune/syncer.py
--- a/python/ray/tune/syncer.py
+++ b/python/ray/tune/syncer.py
@@ -3,6 +3,7 @@
 import os
 import time
 
+from inspect import isclass
 from shlex import quote
 
 from ray import ray_constants
@@ -285,6 +286,9 @@ def get_node_syncer(local_dir, remote_dir=None, sync_function=None):
     key = (local_dir, remote_dir)
     if key in _syncers:
         return _syncers[key]
+    elif isclass(sync_function) and issubclass(sync_function, Syncer):
+        _syncers[key] = sync_function(local_dir, remote_dir, None)
+        return _syncers[key]
     elif not remote_dir or sync_function is False:
         sync_client = NOOP
     elif sync_function and sync_function is not True:


Test patch:
diff --git a/python/ray/tune/tests/test_integration_kubernetes.py b/python/ray/tune/tests/test_integration_kubernetes.py
new file mode 100644
--- /dev/null
+++ b/python/ray/tune/tests/test_integration_kubernetes.py
@@ -0,0 +1,147 @@
+import unittest
+
+from ray.autoscaler.command_runner import KUBECTL_RSYNC
+from ray.tune.integration.kubernetes import KubernetesSyncer, \
+    KubernetesSyncClient
+from ray.tune.syncer import NodeSyncer
+
+
+class _MockProcessRunner:
+    def __init__(self):
+        self.history = []
+
+    def check_call(self, command):
+        self.history.append(command)
+        return True
+
+
+class _MockLookup:
+    def __init__(self, node_ips):
+        self.node_to_ip = {}
+        self.ip_to_node = {}
+        for node, ip in node_ips.items():
+            self.node_to_ip[node] = ip
+            self.ip_to_node[ip] = node
+
+    def get_ip(self, node):
+        return self.node_to_ip[node]
+
+    def get_node(self, ip):
+        return self.ip_to_node[ip]
+
+    def __call__(self, ip):
+        return self.ip_to_node[ip]
+
+
+def _create_mock_syncer(namespace, lookup, use_rsync, process_runner, local_ip,
+                        local_dir, remote_dir):
+    class _MockSyncer(KubernetesSyncer):
+        _namespace = namespace
+        _use_rsync = use_rsync
+        _get_kubernetes_node_by_ip = lookup
+
+        def __init__(self, local_dir, remote_dir, sync_client):
+            self.local_ip = local_ip
+            self.local_node = self._get_kubernetes_node_by_ip(self.local_ip)
+            self.worker_ip = None
+            self.worker_node = None
+
+            sync_client = sync_client
+            super(NodeSyncer, self).__init__(local_dir, remote_dir,
+                                             sync_client)
+
+    return _MockSyncer(
+        local_dir,
+        remote_dir,
+        sync_client=KubernetesSyncClient(
+            namespace=namespace,
+            use_rsync=use_rsync,
+            process_runner=process_runner))
+
+
+class KubernetesIntegrationTest(unittest.TestCase):
+    def setUp(self):
+        self.namespace = "test_ray"
+        self.lookup = _MockLookup({
+            "head": "1.0.0.0",
+            "w1": "1.0.0.1",
+            "w2": "1.0.0.2"
+        })
+        self.process_runner = _MockProcessRunner()
+        self.local_dir = "/tmp/local"
+        self.remote_dir = "/tmp/remote"
+
+    def tearDown(self):
+        pass
+
+    def testKubernetesCpUpDown(self):
+        syncer = _create_mock_syncer(
+            self.namespace, self.lookup, False, self.process_runner,
+            self.lookup.get_ip("head"), self.local_dir, self.remote_dir)
+
+        syncer.set_worker_ip(self.lookup.get_ip("w1"))
+
+        # Test sync up. Should add / to the dirs and call kubectl cp
+        syncer.sync_up()
+
+        self.assertEqual(self.process_runner.history[-1], [
+            "kubectl", "-n", self.namespace, "cp", self.local_dir + "/",
+            "{}/{}:{}".format(self.namespace, "w1", self.remote_dir + "/")
+        ])
+
+        # Test sync down.
+        syncer.sync_down()
+        self.assertEqual(self.process_runner.history[-1], [
+            "kubectl", "-n", self.namespace, "cp", "{}/{}:{}".format(
+                self.namespace,
+                "w1",
+                self.remote_dir + "/",
+            ), self.local_dir + "/"
+        ])
+
+        # Sync to same node should be ignored
+        syncer.set_worker_ip(self.lookup.get_ip("head"))
+        syncer.sync_up()
+        self.assertTrue(len(self.process_runner.history) == 2)
+
+        syncer.sync_down()
+        self.assertTrue(len(self.process_runner.history) == 2)
+
+    def testKubernetesRsyncUpDown(self):
+        syncer = _create_mock_syncer(
+            self.namespace, self.lookup, True, self.process_runner,
+            self.lookup.get_ip("head"), self.local_dir, self.remote_dir)
+
+        syncer.set_worker_ip(self.lookup.get_ip("w1"))
+
+        # Test sync up. Should add / to the dirs and call rsync
+        syncer.sync_up()
+        self.assertEqual(self.process_runner.history[-1][0], KUBECTL_RSYNC)
+        self.assertEqual(self.process_runner.history[-1][-2],
+                         self.local_dir + "/")
+        self.assertEqual(
+            self.process_runner.history[-1][-1], "{}@{}:{}".format(
+                "w1", self.namespace, self.remote_dir + "/"))
+
+        # Test sync down.
+        syncer.sync_down()
+        self.assertEqual(self.process_runner.history[-1][0], KUBECTL_RSYNC)
+        self.assertEqual(
+            self.process_runner.history[-1][-2], "{}@{}:{}".format(
+                "w1", self.namespace, self.remote_dir + "/"))
+        self.assertEqual(self.process_runner.history[-1][-1],
+                         self.local_dir + "/")
+
+        # Sync to same node should be ignored
+        syncer.set_worker_ip(self.lookup.get_ip("head"))
+        syncer.sync_up()
+        self.assertTrue(len(self.process_runner.history) == 2)
+
+        syncer.sync_down()
+        self.assertTrue(len(self.process_runner.history) == 2)
+
+
+if __name__ == "__main__":
+    import pytest
+    import sys
+    sys.exit(pytest.main(["-v", __file__]))