Repository: DataBiosphere/toil

Issue:
Kubernetes Toil jobs can get apparently stuck with all or almost all of their memory used
Here's what k9s says about a particular pod (one of many):
```
NAME ↑                                                        READY      STATUS         RS      CPU       MEM          IP                NODE         QOS      AGE 
root-toil-a3413b4c-cb9c-4f21-90b9-dc39a75409a6-68-7qzng       1/1        Running        0       948m      32763Mi      10.244.5.126      k1.kube      GA       17h
```

And here is the pod described:

```
Name:               root-toil-a3413b4c-cb9c-4f21-90b9-dc39a75409a6-330-szf2t
Namespace:          vg
Priority:           0
PriorityClassName:  <none>
Node:               k1.kube/172.31.53.234
Start Time:         Wed, 13 Nov 2019 16:58:50 -0800
Labels:             controller-uid=277bd9e1-9ca5-4c35-9449-9467dfee39de
                    job-name=root-toil-a3413b4c-cb9c-4f21-90b9-dc39a75409a6-330
Annotations:        <none>
Status:             Running
IP:                 10.244.5.141
Controlled By:      Job/root-toil-a3413b4c-cb9c-4f21-90b9-dc39a75409a6-330
Containers:
  runner-container:
    Container ID:  docker://5b70030a0e720e491cbce0d41d5cfdaaffdff0b544e60385bfd30b68f02aa0f7
    Image:         quay.io/adamnovak/toil:3.21.0a1-e98a7bde0788a2d6f4e55da71ad6e072f8f6a821
    Image ID:      docker-pullable://quay.io/adamnovak/toil@sha256:734c456697a7059f6945b2c7948692937b5e18ddf3b8caf84313aa733e7fed17
    Port:          <none>
    Host Port:     <none>
    Command:
      _toil_kubernetes_executor
      gAJ9cQAoVQtlbnZpcm9ubWVudHEBfXECKFUXVE9JTF9SVF9MT0dHSU5HX0FERFJFU1NxA1USMTAuMjQ0LjEuMTU2OjM4MTkxcQRVFVRPSUxfUlRfTE9HR0lOR19MRVZFTHEFVQRJTkZPcQZ1VQdjb21tYW5kcQdYdwAAAF90b2lsX3dvcmtlciBydW5fY29uc3RydWN0X3JlZ2lvbl9ncmFwaCBhd3M6dXMtd2VzdC0yOmFkYW1ub3Zhay1idWlsZGdpcmFmZmVncmFwaHMgM2M0ZjExNmItMmM2NC00NzdiLWE3ZjktMDk2NTBhOTQ5YjcwcQhVCnVzZXJTY3JpcHRxCWNkaWxsLmRpbGwKX2NyZWF0ZV9uYW1lZHR1cGxlCnEKVRJWaXJ0dWFsRW52UmVzb3VyY2VxCyhVBG5hbWVxDFUIcGF0aEhhc2hxDVUDdXJscQ5VC2NvbnRlbnRIYXNocQ90cRBVDXRvaWwucmVzb3VyY2VxEYdxElJxEyhVDXNpdGUtcGFja2FnZXNxFFUgNTcyMTJmOTAzYTM4ZDJlZTY5ZWMxYTA3MmVhOWVjNjdxFWNmdXR1cmUudHlwZXMubmV3c3RyCm5ld3N0cgpxFliWAAAAaHR0cHM6Ly9hZGFtbm92YWstYnVpbGRnaXJhZmZlZ3JhcGhzLS1maWxlcy5zMy11cy13ZXN0LTIuYW1hem9uYXdzLmNvbS8xOWU4OGM3YS1kNjkyLTVjNTktOTA0NS1iNjM0OTE3M2JmNjA/dmVyc2lvbklkPVJjRTFiZkY4Y1NYbXlGN2JWam1wNHVKTVFSV1pQUk1KcReFcRiBcRl9cRpiVSA5NzhiZTJhZWU2YzVmMDRjNzFkMzU2MDAwYTdjNmRhY3EbdHEcgXEddS4=
    State:          Running
      Started:      Wed, 13 Nov 2019 16:58:51 -0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                1
      ephemeral-storage:  34359738368
      memory:             34359738368
    Requests:
      cpu:                1
      ephemeral-storage:  34359738368
      memory:             34359738368
    Environment:          <none>
    Mounts:
      /root/.aws from s3-credentials (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dq8zg (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  s3-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  shared-s3-credentials
    Optional:    false
  default-token-dq8zg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dq8zg
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
Vyoma:vg anovak$ kubectl describe pod root-toil-a3413b4c-cb9c-4f21-90b9-dc39a75409a6-68-7qzng
Name:               root-toil-a3413b4c-cb9c-4f21-90b9-dc39a75409a6-68-7qzng
Namespace:          vg
Priority:           0
PriorityClassName:  <none>
Node:               k1.kube/172.31.53.234
Start Time:         Wed, 13 Nov 2019 16:55:58 -0800
Labels:             controller-uid=06c54793-c85d-4444-84f3-9edb08e02601
                    job-name=root-toil-a3413b4c-cb9c-4f21-90b9-dc39a75409a6-68
Annotations:        <none>
Status:             Running
IP:                 10.244.5.126
Controlled By:      Job/root-toil-a3413b4c-cb9c-4f21-90b9-dc39a75409a6-68
Containers:
  runner-container:
    Container ID:  docker://07ba5e432267a254fa35ee146ca403aa1faa32b8cd566e10be9d9c70cabdde38
    Image:         quay.io/adamnovak/toil:3.21.0a1-e98a7bde0788a2d6f4e55da71ad6e072f8f6a821
    Image ID:      docker-pullable://quay.io/adamnovak/toil@sha256:734c456697a7059f6945b2c7948692937b5e18ddf3b8caf84313aa733e7fed17
    Port:          <none>
    Host Port:     <none>
    Command:
      _toil_kubernetes_executor
      gAJ9cQAoVQtlbnZpcm9ubWVudHEBfXECKFUXVE9JTF9SVF9MT0dHSU5HX0FERFJFU1NxA1USMTAuMjQ0LjEuMTU2OjM4MTkxcQRVFVRPSUxfUlRfTE9HR0lOR19MRVZFTHEFVQRJTkZPcQZ1VQdjb21tYW5kcQdYdwAAAF90b2lsX3dvcmtlciBydW5fY29uc3RydWN0X3JlZ2lvbl9ncmFwaCBhd3M6dXMtd2VzdC0yOmFkYW1ub3Zhay1idWlsZGdpcmFmZmVncmFwaHMgNWM1NmVmZjctMmQ3YS00N2Q4LWFkOTMtZDU3M2IyNDJiMmNmcQhVCnVzZXJTY3JpcHRxCWNkaWxsLmRpbGwKX2NyZWF0ZV9uYW1lZHR1cGxlCnEKVRJWaXJ0dWFsRW52UmVzb3VyY2VxCyhVBG5hbWVxDFUIcGF0aEhhc2hxDVUDdXJscQ5VC2NvbnRlbnRIYXNocQ90cRBVDXRvaWwucmVzb3VyY2VxEYdxElJxEyhVDXNpdGUtcGFja2FnZXNxFFUgNTcyMTJmOTAzYTM4ZDJlZTY5ZWMxYTA3MmVhOWVjNjdxFWNmdXR1cmUudHlwZXMubmV3c3RyCm5ld3N0cgpxFliWAAAAaHR0cHM6Ly9hZGFtbm92YWstYnVpbGRnaXJhZmZlZ3JhcGhzLS1maWxlcy5zMy11cy13ZXN0LTIuYW1hem9uYXdzLmNvbS8xOWU4OGM3YS1kNjkyLTVjNTktOTA0NS1iNjM0OTE3M2JmNjA/dmVyc2lvbklkPVJjRTFiZkY4Y1NYbXlGN2JWam1wNHVKTVFSV1pQUk1KcReFcRiBcRl9cRpiVSA5NzhiZTJhZWU2YzVmMDRjNzFkMzU2MDAwYTdjNmRhY3EbdHEcgXEddS4=
    State:          Running
      Started:      Wed, 13 Nov 2019 16:55:59 -0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                1
      ephemeral-storage:  34359738368
      memory:             34359738368
    Requests:
      cpu:                1
      ephemeral-storage:  34359738368
      memory:             34359738368
    Environment:          <none>
    Mounts:
      /root/.aws from s3-credentials (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dq8zg (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  s3-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  shared-s3-credentials
    Optional:    false
  default-token-dq8zg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dq8zg
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
```

As far as I can tell the pod is stuck and I suspect not doing any useful work. I can't exec anything in the pod:

```
$ kubectl exec -it root-toil-a3413b4c-cb9c-4f21-90b9-dc39a75409a6-330-szf2t -- /bin/bash
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "read init-p: connection reset by peer": unknown
command terminated with exit code 126
```

(I can also get `broken pipe`).

I think the inability to shell into the container is caused by https://github.com/opencontainers/runc/issues/1914 and happens I think because the container memory limit is too full to start the shell process. Apparently it is not full enough to get the pod OOM-killed, but I can't imagine it is doing its work properly if the memory limit is pegged.

Toil should either rig the pods to fail if the memory limit is hit like this (somehow; maybe with an `exec`-based liveness check?), or it should catch these pods as stuck just like it does for pods that get stuck in ImagePullBackoff and clean them up itself.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-454)
┆Issue Number: TOIL-454



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/kubernetes.py b/src/toil/batchSystems/kubernetes.py
--- a/src/toil/batchSystems/kubernetes.py
+++ b/src/toil/batchSystems/kubernetes.py
@@ -47,6 +47,7 @@
 from toil import applianceSelf, customDockerInitCmd
 from toil.batchSystems.abstractBatchSystem import (AbstractBatchSystem,
                                                    BatchSystemLocalSupport)
+from toil.lib.humanize import human2bytes
 from toil.resource import Resource
 
 logger = logging.getLogger(__name__)
@@ -127,8 +128,8 @@ def _api(self, kind, max_age_seconds = 5 * 60):
         goes on longer than our credentials last, though.
         
         This method is the Right Way to get any Kubernetes API. You call it
-        with the API you want ('batch' or 'core') and it returns an API object
-        with guaranteed fresh credentials.
+        with the API you want ('batch', 'core', or 'customObjects') and it
+        returns an API object with guaranteed fresh credentials.
         
         It also recognizes 'namespace' and returns our namespace as a string.
         
@@ -157,6 +158,7 @@ def _api(self, kind, max_age_seconds = 5 * 60):
         # Now fill in the API objects with these credentials
         self._apis['batch'] = kubernetes.client.BatchV1Api()
         self._apis['core'] = kubernetes.client.CoreV1Api()
+        self._apis['customObjects'] = kubernetes.client.CustomObjectsApi()
         
         # And save the time
         self.credential_time = now
@@ -238,7 +240,9 @@ def issueBatchJob(self, jobNode):
             # Add on a bit for Kubernetes overhead (Toil worker's memory, hot deployed
             # user scripts).
             # Kubernetes needs some lower limit of memory to run the pod at all without
-            # OOMing.
+            # OOMing. We also want to provision some extra space so that when
+            # we test _isPodStuckOOM we never get True unless the job has
+            # exceeded jobNode.memory.
             requirements_dict = {'cpu': jobNode.cores,
                                  'memory': jobNode.memory + 1024 * 1024 * 512,
                                  'ephemeral-storage': jobNode.disk + 1024 * 1024 * 512}
@@ -434,7 +438,71 @@ def _getLogForPod(self, podObject):
 
         return self._api('core').read_namespaced_pod_log(podObject.metadata.name,
                                                          namespace=self.namespace)
-   
+
+    def _isPodStuckOOM(self, podObject, minFreeBytes=1024 * 1024 * 2):
+        """
+        Poll the current memory usage for the pod from the cluster.
+
+        Return True if the pod looks to be in a soft/stuck out of memory (OOM)
+        state, where it is using too much memory to actually make progress, but
+        not enough to actually trigger the OOM killer to kill it. For some
+        large memory limits, on some Kubernetes clusters, pods can get stuck in
+        this state when their memory limits are high (approx. 200 Gi).
+
+        We operationalize "OOM" as having fewer than minFreeBytes bytes free.
+
+        We assume the pod has only one container, as Toil's pods do.
+
+        :param kubernetes.client.V1Pod podObject: a Kubernetes pod with one
+                                       container to check up on.
+        :param int minFreeBytes: Minimum free bytes to not be OOM.
+
+        :return: True if the pod is OOM, false otherwise.
+        :rtype: bool
+        """
+
+        # Compose a query to get just the pod we care about
+        query = 'metadata.name=' + podObject.metadata.name
+
+        # Look for it
+        # TODO: When the Kubernetes Python API actually wraps the metrics API, switch to that
+        response = self._api('customObjects').list_namespaced_custom_object('metrics.k8s.io', 'v1beta1',
+                                                                            self.namespace, 'pods',
+                                                                            field_selector=query)
+
+        # Pull out the items
+        items = response.get('items', [])
+
+        if len(items) == 0:
+            # If there's no statistics we can't say we're stuck OOM
+            return False
+
+        # Assume the first result is the right one, because of the selector
+        # Assume it has exactly one pod, because we made it
+        containers = items[0].get('containers', [{}])
+        
+        if len(containers) == 0:
+            # If there are no containers (because none have started yet?), we can't say we're stuck OOM
+            return False
+        
+        # Otherwise, assume it just has one container.
+        # Grab the memory usage string, like 123Ki, and convert to bytes.
+        # If anything is missing, assume 0 bytes used.
+        bytesUsed = human2bytes(containers[0].get('usage', {}).get('memory', '0'))
+
+        # Also get the limit out of the pod object's spec
+        bytesAllowed = human2bytes(podObject.spec.containers[0].resources.limits['memory'])
+
+        if bytesAllowed - bytesUsed < minFreeBytes:
+            # This is too much!
+            logger.warning('Pod %s has used %d of %d bytes of memory; reporting as stuck due to OOM.',
+                           podObject.metadata.name, bytesUsed, bytesAllowed)
+
+            return True
+
+
+
+
     def _getIDForOurJob(self, jobObject):
         """
         Get the JobID number that belongs to the given job that we own.
@@ -551,8 +619,7 @@ def _getUpdatedBatchJobImmediately(self):
                     break
 
         if jobObject is None:
-            # If no jobs are failed, look for jobs with pods with
-            # containers stuck in Waiting with reason ImagePullBackOff
+            # If no jobs are failed, look for jobs with pods that are stuck for various reasons.
             for j in self._ourJobObjects():
                 pod = self._getPodForJob(j)
 
@@ -560,6 +627,8 @@ def _getUpdatedBatchJobImmediately(self):
                     # Skip jobs with no pod
                     continue
                     
+                # Containers can get stuck in Waiting with reason ImagePullBackOff
+
                 # Get the statuses of the pod's containers
                 containerStatuses = pod.status.container_statuses
                 if containerStatuses is None or len(containerStatuses) == 0:
@@ -580,8 +649,17 @@ def _getUpdatedBatchJobImmediately(self):
                     logger.warning('Failing stuck job; did you try to run a non-existent Docker image?'
                                    ' Check TOIL_APPLIANCE_SELF.')
                     break
-       
-            
+
+                # Pods can also get stuck nearly but not quite out of memory,
+                # if their memory limits are high and they try to exhaust them.
+
+                if self._isPodStuckOOM(pod):
+                    # We found a job that probably should be OOM! Report it as stuck.
+                    # Polling function takes care of the logging.
+                    jobObject = j
+                    chosenFor = 'stuck'
+                    break
+
         if jobObject is None:
             # Say we couldn't find anything
             return None


Test patch: