Repository: DataBiosphere/toil

Issue:
Latest master doesn't work with cactus/kubernetes
@adamnovak created this appliance to resolve some Cactus issues described in #2854.
```
quay.io/adamnovak/toil:3.22.0a1-72750c175eb84fde0e6d3302283f032edc39038b
```
It got cactus to run (however slowly, but that's another story) on kubernetes on small inputs.   But even though the changes from this branch are in master (I think), the latest master (https://github.com/DataBiosphere/toil/commit/91eab2b3a8c29e10db3a35a2ad5053fb462849c3) fails with this new error:

```
File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 606, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/usr/local/lib/python2.7/dist-packages/toil/batchSystems/kubernetes.py", line 243, in issueBatchJob
    launched = self.batchApi.create_namespaced_job(self.namespace, job)
  File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/batch_v1_api.py", line 60, in create_namespaced_job
    (data) = self.create_namespaced_job_with_http_info(namespace, body, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/batch_v1_api.py", line 151, in create_namespaced_job_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 377, in request
    body=body)
  File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/rest.py", line 266, in POST
    body=body)
  File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 04 Dec 2019 19:22:09 GMT', 'Content-Length': '588', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Job.batch \"root-toil-44f30e94-357d-4167-80f7-49f1b50f2f53-195\" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: \"100m\": must be less than or equal to cpu limit","reason":"Invalid","details":{"name":"root-toil-44f30e94-357d-4167-80f7-49f1b50f2f53-195","group":"batch","kind":"Job","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"100m\": must be less than or equal to cpu limit","field":"spec.template.spec.containers[0].resources.requests"}]},"code":422}
```

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-463)
┆Issue Number: TOIL-463



Discussion:
This is my fault here:

https://github.com/DataBiosphere/toil/blob/91eab2b3a8c29e10db3a35a2ad5053fb462849c3/src/toil/batchSystems/kubernetes.py#L191

I'm trying to set the limits to 1.5x the requests, but also an integer number of bytes. But for CPUs fractional requests make sense, and this gives a limit of 0 when you ask for 0.1 CPUs.

Since Erich wants us to use limits equal to the requests on our cluster, I could just rip this out and pass the same value twice. But that means the Toil job requirements might have to be raised in some workflows to account for Kubernetes overhead. We also will have cases where the limits were too low all along, but only Kubernetes actually kills jobs to enforce them so nobody noticed.

Resolving patch:
diff --git a/src/toil/batchSystems/kubernetes.py b/src/toil/batchSystems/kubernetes.py
--- a/src/toil/batchSystems/kubernetes.py
+++ b/src/toil/batchSystems/kubernetes.py
@@ -181,14 +181,16 @@ def issueBatchJob(self, jobNode):
             # https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Job.md
 
             # Make a definition for the container's resource requirements.
-            # Don't let people request too-small amounts of memory or disk.
-            # Kubernetes needs some lower limit to run the pod at all without OOMing.
+            # Add on a bit for Kubernetes overhead (Toil worker's memory, hot deployed
+            # user scripts).
+            # Kubernetes needs some lower limit of memory to run the pod at all without
+            # OOMing.
             requirements_dict = {'cpu': jobNode.cores,
-                                 'memory': max(jobNode.memory, 1024 * 1024 * 512),
-                                 'ephemeral-storage': max(jobNode.disk, 1024 * 1024 * 512)}
-            # Set a higher limit to give jobs some room to go over what they
-            # think they need, as is the Kubernetes way.
-            limits_dict = {k: int(v * 1.5) for k, v in requirements_dict.items()}
+                                 'memory': jobNode.memory + 1024 * 1024 * 512,
+                                 'ephemeral-storage': jobNode.disk + 1024 * 1024 * 512}
+            # Use the requirements as the limits, for predictable behavior, and because
+            # the UCSC Kubernetes admins want it that way.
+            limits_dict = requirements_dict
             resources = kubernetes.client.V1ResourceRequirements(limits=limits_dict,
                                                                  requests=requirements_dict)
             


Test patch: