Repository: sql-machine-learning/elasticdl

Issue:
PodManager Observer Pattern
The main responsibility of `PodManager` is to maintain the cluster for ElasticDL distributed job. It interacts with Kubernetes directly.  It calls K8S API to create pods/services for the ElasticDL job cluster, relaunches the pod if required and also watches the event published from Kubernetes.
As the job goes on, there may be some changes happening in the members of Job cluster such as worker / ps. The other components may be interested in this event and do some action according to various events. So we propose to apply the observer design pattern to PodManager.

```Python
import collections

PodInfo = collections.namedtuple(
    "PodInfo",
    (
        "type",
        "id",
        "name",
        "address"
    )
)

ClusterContext = collections.namedtuple(
    "ClusterContext",
    (
        "pod_manager"
    )
)

class PodManager(object):
    def __init__(self):
        self._listeners = []

    def add_listener(self, job_cluster_listener):
        self._listeners.append(job_cluster_listener)
```

Option 1:
```Python
class JobClusterListener(object):
    def on_pod_started(self, job_member, job_cluster_context):
        pass

    def on_pod_completed(self, job_member, job_cluster_context):
        pass

    def on_pod_failed(self, job_member, job_cluster_context):
        pass

    def on_pod_deleted(self, job_member, job_cluster_context):
        pass
```

Option 2:
```Python
class JobClusterListener(object):
    def on_worker_started(self, job_member, job_cluster_context):
        pass

    def on_worker_completed(self, job_member, job_cluster_context):
        pass

    def on_worker_failed(self, job_member, job_cluster_context):
        pass

    def on_worker_deleted(self, job_member, job_cluster_context):
        pass

    def on_ps_started(self, job_member, job_cluster_context):
        pass

    def on_ps_completed(self, job_member, job_cluster_context):
        pass

    def on_ps_failed(self, job_member, job_cluster_context):
        pass

    def on_ps_deleted(self, job_member, job_cluster_context):
        pass
```

Currently, there will be two listeners, no listener observes the event of PS.
- TaskManager: focus on `worker failed event`, reassign the tasks belongs to the failed worker to others
- HorovodRendezvousServer: focus on `worker started/completed/failed/deleted event`, will rebuild the configuration for Horovod AllReduce.


Q&A
1. Why not include alive_worker_addresses in JobClusterContext?
    Not all the listeners cares about this information. And it's a little heavy to build alive_worker_addresses. The listener can get the reference of `pod_manager` in JobClusterContext and call its method to get this information on demand.


Discussion:
How to get `alive_worker_addresses`?
1. Option 1 is better.

2. Make the name simple, as there is no job definition in the pod manager. 
JobMember ->PodInfo
JobClusterContext -> just use the reference of pod_manager itself.

> alive_worker_addresses

pod_manager will have a method `get_alive_worker_addresses`, the listener can call it on demand.
> 1. Option 1 is better.
> 2. Make the name simple, as there is no job definition in the pod manager.
>    JobMember ->PodInfo
>    JobClusterContext -> just use the reference of pod_manager itself.

Rename JobMember to PodInfo, JobClusterContext to ClusterContext.
After discussion, we have made agreement on the proposal. Close this issue.

Resolving patch:
diff --git a/elasticdl/python/master/pod_event_callbacks.py b/elasticdl/python/master/pod_event_callbacks.py
new file mode 100644
--- /dev/null
+++ b/elasticdl/python/master/pod_event_callbacks.py
@@ -0,0 +1,92 @@
+# Copyright 2020 The ElasticDL Authors. All rights reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import abc
+import collections
+
+PodInfo = collections.namedtuple("PodInfo", ("type", "id", "name"))
+
+ClusterContext = collections.namedtuple("ClusterContext", ("pod_manager"))
+
+
+class PodEventCallback(metaclass=abc.ABCMeta):
+    """
+    The interface for the observers that are interested in the pod event.
+    The subclass observers can override the following methods to handle
+    various events.
+    """
+
+    @abc.abstractmethod
+    def on_pod_started(self, pod_info, cluster_context):
+        """
+        The handler for the pod started event.
+        Args:
+            pod_info: A PodInfo object. It's the pod that just becomes running.
+            cluster_context: A ClusterContext object. It contains all the
+                context information about the cluster for the job.
+        """
+        pass
+
+    @abc.abstractmethod
+    def on_pod_succeeded(self, pod_info, cluster_context):
+        """
+        The handler for the pod succeeded event.
+        Args:
+            pod_info: A PodInfo object. It's the pod that just terminates
+                in success.
+            cluster_context: A ClusterContext object. It contains all the
+                context information about the cluster for the job.
+        """
+        pass
+
+    @abc.abstractmethod
+    def on_pod_failed(self, pod_info, cluster_context):
+        """
+        The handler for the pod failed event.
+        Args:
+            pod_info: A PodInfo object. It's the pod that just terminates
+                in failure.
+            cluster_context: A ClusterContext object. It contains all the
+                context information about the cluster for the job.
+        """
+        pass
+
+    @abc.abstractmethod
+    def on_pod_deleted(self, pod_info, cluster_context):
+        """
+        The handler for the pod deleted event.
+        Args:
+            pod_info: A PodInfo object. It's the pod which is just deleted.
+            cluster_context: A ClusterContext object. It contains all the
+                context information about the cluster for the job.
+        """
+        pass
+
+
+class TaskRescheduleCallback(PodEventCallback):
+    def __init__(self, task_manager):
+        self._task_manager = task_manager
+
+    def on_pod_started(self, pod_info, cluster_context):
+        pass
+
+    def on_pod_succeeded(self, pod_info, cluster_context):
+        pass
+
+    def on_pod_failed(self, pod_info, cluster_context):
+        # TODO: Call task_manager to reschedule the task
+        # of the failed worker to another worker
+        pass
+
+    def on_pod_deleted(self, pod_info, cluster_context):
+        pass
diff --git a/elasticdl/python/master/pod_manager.py b/elasticdl/python/master/pod_manager.py
--- a/elasticdl/python/master/pod_manager.py
+++ b/elasticdl/python/master/pod_manager.py
@@ -14,7 +14,10 @@
 
 class PodManager(object):
     def __init__(self, args):
-        pass
+        self._pod_event_callbacks = []
 
     def start(self):
         pass
+
+    def add_pod_event_callback(self, pod_event_callback):
+        self._pod_event_callbacks.append(pod_event_callback)


Test patch: