Repository: sql-machine-learning/elasticdl

Issue:
Master occasionally does not clear and relaunch the OOM pod.
Now，master will clear the OOM worker pod and relaunch a new worker pod. However, we can see that the master occasionally does not clear and relaunch the OOM pod.

![image](https://user-images.githubusercontent.com/18071380/84876630-01462000-b0ba-11ea-8f75-c7eea7e2934d.png)

There are two problems:
1. Should master relaunch the OOM worker pod? 
Master will relaunch the OOM worker pod frequently if the configured memory for the worker is not enough. Then, the training is very slow and the job cannot exit. So, the user does not know why it is very slow. The proposed solution is that master does not relaunch the OOM worker pod and the job exits with an OOM error if all workers exit with OOM. Then, the user can restart the job with more memory for workers.

1. We may need a robust solution to relaunch pods.
Except for watching events, master can check the worker status periodically and relaunch the pod.


Discussion:

Resolving patch:
diff --git a/elasticdl/python/master/k8s_instance_manager.py b/elasticdl/python/master/k8s_instance_manager.py
--- a/elasticdl/python/master/k8s_instance_manager.py
+++ b/elasticdl/python/master/k8s_instance_manager.py
@@ -305,32 +305,36 @@ def _event_cb(self, event):
             if pod_name in self._failed_pods:
                 return
 
-            # When a pod fails with exit_code == 137, it may be deleted,
-            # preempted, or OOMkilled. Master will try to relaunch it.
-            # For OOMkilled, the relaunch is a workaround for memory leak
-            # issues in tf eager mode.
             relaunch_failed_pod = False
-            if (
-                evt_type == "MODIFIED"
-                and phase == "Failed"
-                and evt_obj.status.container_statuses
-                and evt_obj.status.container_statuses[0].state.terminated
-                and evt_obj.status.container_statuses[
-                    0
-                ].state.terminated.exit_code
-                == 137
-            ):
+            if evt_type == "MODIFIED" and phase == "Failed":
                 self._failed_pods.append(pod_name)
-                relaunch_failed_pod = True
-                logger.info(
-                    "Pod %s is killed with reason %s."
-                    % (
-                        pod_name,
-                        evt_obj.status.container_statuses[
-                            0
-                        ].state.terminated.reason,
+                worker_id = self._worker_pod_name_to_id.get(pod_name, None)
+                if worker_id is not None:
+                    # Recover tasks when the worker failed
+                    self._task_d.recover_tasks(worker_id)
+
+                if (
+                    evt_obj.status.container_statuses
+                    and evt_obj.status.container_statuses[0].state.terminated
+                    and evt_obj.status.container_statuses[
+                        0
+                    ].state.terminated.exit_code
+                    == 137
+                    and evt_obj.status.container_statuses[
+                        0
+                    ].state.terminated.reason
+                    != "OOMKilled"
+                ):
+                    relaunch_failed_pod = True
+                    logger.info(
+                        "Pod %s is killed with reason %s."
+                        % (
+                            pod_name,
+                            evt_obj.status.container_statuses[
+                                0
+                            ].state.terminated.reason,
+                        )
                     )
-                )
 
             if pod_name in self._worker_pod_name_to_id:
                 worker_id = self._worker_pod_name_to_id.get(pod_name)
@@ -338,7 +342,6 @@ def _event_cb(self, event):
                 if evt_type == "DELETED" or relaunch_failed_pod:
                     del self._worker_pods_phase[worker_id]
                     del self._worker_pod_name_to_id[pod_name]
-                    self._task_d.recover_tasks(worker_id)
 
                     # If a deleted pod was not "Succeeded", relaunch a worker.
                     relaunch_worker = (


Test patch: