Repository: DataBiosphere/toil

Issue:
Flaky Kubernetes failure?
```
[2022-09-14T12:48:03+0000] [MainThread] [D] [toil.provisioners.node] Running ['rsync', '-e', 'ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no', '--rsync-path', 'docker exec -i toil_leader rsync -v', 'core@52.10.65.120:/etc/kubernetes/worker.ini', '/tmp/tmppi1r0bl3/worker.ini'].
Warning: Permanently added '52.10.65.120' (ECDSA) to the list of known hosts.
rsync: link_stat "/etc/kubernetes/worker.ini" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1677) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)
[2022-09-14T12:48:03+0000] [MainThread] [D] [toil.provisioners.node] Rsync from new node failed, trying again. Error message: Command '['rsync', '-e', 'ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no', '--rsync-path', 'docker exec -i toil_leader rsync -v', 'core@52.10.65.120:/etc/kubernetes/worker.ini', '/tmp/tmppi1r0bl3/worker.ini']' returned non-zero exit status 23.
Traceback (most recent call last):
  File "/builds/databiosphere/toil/venv/bin/toil", line 33, in <module>
    sys.exit(load_entry_point('toil', 'console_scripts', 'toil')())
  File "/builds/databiosphere/toil/src/toil/utils/toilMain.py", line 33, in main
    get_or_die(module, 'main')()
  File "/builds/databiosphere/toil/src/toil/utils/toilLaunchCluster.py", line 163, in main
    cluster.launchCluster(leaderNodeType=options.leaderNodeType,
  File "/builds/databiosphere/toil/src/toil/provisioners/aws/awsProvisioner.py", line 398, in launchCluster
    self._setLeaderWorkerAuthentication(leaderNode)
  File "/builds/databiosphere/toil/src/toil/provisioners/abstractProvisioner.py", line 250, in _setLeaderWorkerAuthentication
    self._leaderWorkerAuthentication = self._getKubernetesJoiningInfo(leader=leader)
  File "/builds/databiosphere/toil/src/toil/provisioners/abstractProvisioner.py", line 333, in _getKubernetesJoiningInfo
    leader.extractFile('/etc/kubernetes/worker.ini', localFile, 'toil_leader')
  File "/builds/databiosphere/toil/src/toil/provisioners/node.py", line 118, in extractFile
    raise RuntimeError(f"Failed to extract file {fromFile} from {role} with ip {self.effectiveIP}")
RuntimeError: Failed to extract file /etc/kubernetes/worker.ini from toil_leader with ip 52.10.65.120
```

See: https://ucsc-ci.com/databiosphere/toil/-/jobs/22647

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1218)
┆friendlyId: TOIL-1218

Flaky Kubernetes failure?
```
[2022-09-14T12:48:03+0000] [MainThread] [D] [toil.provisioners.node] Running ['rsync', '-e', 'ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no', '--rsync-path', 'docker exec -i toil_leader rsync -v', 'core@52.10.65.120:/etc/kubernetes/worker.ini', '/tmp/tmppi1r0bl3/worker.ini'].
Warning: Permanently added '52.10.65.120' (ECDSA) to the list of known hosts.
rsync: link_stat "/etc/kubernetes/worker.ini" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1677) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)
[2022-09-14T12:48:03+0000] [MainThread] [D] [toil.provisioners.node] Rsync from new node failed, trying again. Error message: Command '['rsync', '-e', 'ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no', '--rsync-path', 'docker exec -i toil_leader rsync -v', 'core@52.10.65.120:/etc/kubernetes/worker.ini', '/tmp/tmppi1r0bl3/worker.ini']' returned non-zero exit status 23.
Traceback (most recent call last):
  File "/builds/databiosphere/toil/venv/bin/toil", line 33, in <module>
    sys.exit(load_entry_point('toil', 'console_scripts', 'toil')())
  File "/builds/databiosphere/toil/src/toil/utils/toilMain.py", line 33, in main
    get_or_die(module, 'main')()
  File "/builds/databiosphere/toil/src/toil/utils/toilLaunchCluster.py", line 163, in main
    cluster.launchCluster(leaderNodeType=options.leaderNodeType,
  File "/builds/databiosphere/toil/src/toil/provisioners/aws/awsProvisioner.py", line 398, in launchCluster
    self._setLeaderWorkerAuthentication(leaderNode)
  File "/builds/databiosphere/toil/src/toil/provisioners/abstractProvisioner.py", line 250, in _setLeaderWorkerAuthentication
    self._leaderWorkerAuthentication = self._getKubernetesJoiningInfo(leader=leader)
  File "/builds/databiosphere/toil/src/toil/provisioners/abstractProvisioner.py", line 333, in _getKubernetesJoiningInfo
    leader.extractFile('/etc/kubernetes/worker.ini', localFile, 'toil_leader')
  File "/builds/databiosphere/toil/src/toil/provisioners/node.py", line 118, in extractFile
    raise RuntimeError(f"Failed to extract file {fromFile} from {role} with ip {self.effectiveIP}")
RuntimeError: Failed to extract file /etc/kubernetes/worker.ini from toil_leader with ip 52.10.65.120
```

See: https://ucsc-ci.com/databiosphere/toil/-/jobs/22647

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1218)
┆friendlyId: TOIL-1218



Discussion:
➤ Melaina Legaspi commented:

Adam Novak :"Needs further investigation: is something actually going wrong with kubernetes set up or is the file just not available by the time we look for it?”
➤ Melaina Legaspi commented:

Adam Novak :"Needs further investigation: is something actually going wrong with kubernetes set up or is the file just not available by the time we look for it?”

Resolving patch:
diff --git a/src/toil/batchSystems/kubernetes.py b/src/toil/batchSystems/kubernetes.py
--- a/src/toil/batchSystems/kubernetes.py
+++ b/src/toil/batchSystems/kubernetes.py
@@ -600,7 +600,7 @@ def issueBatchJob(self, job_desc, job_environment: Optional[Dict[str, str]] = No
     def _ourJobObject(self, onlySucceeded=False):
         """
         Yield Kubernetes V1Job objects that we are responsible for that the
-        cluster knows about.
+        cluster knows about. Ignores jobs in the process of being deleted.
 
         Doesn't support a free-form selector, because there's only about 3
         things jobs can be selected on: https://stackoverflow.com/a/55808444
@@ -636,7 +636,7 @@ def _ourJobObject(self, onlySucceeded=False):
             else:
                 results = self._try_kubernetes(self._api('batch').list_namespaced_job, self.namespace,
                                                 label_selector=f"toil_run={self.run_id}", **kwargs)
-            yield from results.items  # These jobs belong to us
+            yield from (j for j in results.items if not self._is_deleted(j))
 
             # Remember the continuation token, if any
             token = getattr(results.metadata, 'continue', None)
@@ -665,7 +665,7 @@ def _ourPodObject(self):
 
             results = self._try_kubernetes(self._api('core').list_namespaced_pod, self.namespace, label_selector=f"toil_run={self.run_id}", **kwargs)
 
-            yield from results.items
+            yield from (j for j in results.items if not self._is_deleted(j))
             # Remember the continuation token, if any
             token = getattr(results.metadata, 'continue', None)
 
@@ -849,6 +849,20 @@ def _isPodStuckWaiting(self, pod_object: kubernetes.client.V1Pod, reason: Option
 
         return True
 
+    def _is_deleted(self, kube_thing: Union['V1Job', 'V1Pod']) -> bool:
+        """
+        Determine if a job or pod is in the process od being deleted, and
+        shouldn't count anymore.
+        """
+
+        # Kubernetes "Terminating" is the same as having the deletion_timestamp
+        # set in the metadata of the object.
+
+        deletion_timestamp: Optional[datetime.datetime] = getattr(getattr(kube_thing, 'metadata', None), 'deletion_timestamp', None)
+        # If the deletion timestamp is set to anything, it is in the process of
+        # being deleted. We will treat that as as good as gone.
+        return deletion_timestamp is not None
+
     def _getIDForOurJob(self, jobObject):
         """
         Get the JobID number that belongs to the given job that we own.
@@ -879,6 +893,10 @@ def getUpdatedBatchJob(self, maxWait):
                                                         timeout_seconds=maxWait):
                 # Grab the metadata data, ID, the list of conditions of the current job, and the total pods
                 jobObject = event['object']
+                if self._is_deleted(jobObject):
+                    # Job is already deleted, so ignore it.
+                    logger.warning('Kubernetes job %s is deleted; ignore its update', getattr(getattr(jobObject, 'metadata', None), 'name', None))
+                    continue
                 jobID = int(jobObject.metadata.name[len(self.job_prefix):])
                 jobObjectListConditions =jobObject.status.conditions
                 totalPods = jobObject.status.active + jobObject.status.finished + jobObject.status.failed
@@ -917,6 +935,7 @@ def getUpdatedBatchJob(self, maxWait):
                                             jobObject.metadata.name,
                                             self.namespace,
                                             propagation_policy='Foreground')
+                        # Make sure the job is deleted so we won't see it again.
                         self._waitForJobDeath(jobObject.metadata.name)
                         return result
                     continue
@@ -1127,11 +1146,8 @@ def _getUpdatedBatchJobImmediately(self):
             # That just kicks off the deletion process. Foreground doesn't
             # actually block. See
             # https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/#foreground-cascading-deletion
-            # We have to either wait until the deletion is done and we can't
-            # see the job anymore, or ban the job from being "updated" again if
-            # we see it. If we don't block on deletion, we can't use limit=1
-            # on our query for succeeded jobs. So we poll for the job's
-            # non-existence.
+            # We wait for the job to vanish, or at least pass _is_deleted so
+            # we ignore it later.
             self._waitForJobDeath(jobObject.metadata.name)
 
         except ApiException as e:
@@ -1155,8 +1171,12 @@ def _waitForJobDeath(self, jobName):
         while True:
             try:
                 # Look for the job
-                self._try_kubernetes_expecting_gone(self._api('batch').read_namespaced_job, jobName, self.namespace)
-                # If we didn't 404, wait a bit with exponential backoff
+                job_object = self._try_kubernetes_expecting_gone(self._api('batch').read_namespaced_job, jobName, self.namespace)
+                if self._is_deleted(job_object):
+                    # The job looks deleted, so we can treat it as not being there.
+                    return
+                # If we didn't 404, and the job doesn't look deleted, wait a
+                # bit with exponential backoff
                 time.sleep(backoffTime)
                 if backoffTime < maxBackoffTime:
                     backoffTime *= 2
@@ -1282,7 +1302,12 @@ def killBatchJobs(self, jobIDs):
             # Work out what the job would be named
             jobName = self.job_prefix + str(jobID)
 
-            # Block until it doesn't exist
+            # Block until the delete takes.
+            # The user code technically might stay running forever, but we push
+            # the potential deadlock (if the user code needs exclusive access to
+            # a resource) onto the user code, instead of always hanging
+            # whenever we can't certify that a faulty node is no longer running
+            # the user code. 
             self._waitForJobDeath(jobName)
 
     @classmethod
diff --git a/src/toil/provisioners/abstractProvisioner.py b/src/toil/provisioners/abstractProvisioner.py
--- a/src/toil/provisioners/abstractProvisioner.py
+++ b/src/toil/provisioners/abstractProvisioner.py
@@ -684,6 +684,7 @@ def addVolumesService(self, config: InstanceConfiguration):
                 sudo mount --bind /mnt/ephemeral/var/$directory /var/$directory
             done
             """))
+        # TODO: Make this retry?
         config.addUnit("volume-mounting.service", contents=textwrap.dedent("""\
             [Unit]
             Description=mounts ephemeral volumes & bind mounts toil directories
@@ -832,6 +833,7 @@ def getKubernetesValues(self, architecture: str = 'amd64'):
             CRICTL_VERSION="v1.17.0",
             CNI_DIR="/opt/cni/bin",
             DOWNLOAD_DIR="/opt/bin",
+            SETUP_STATE_DIR="/etc/toil/kubernetes",
             # This is the version of Kubernetes to use
             # Get current from: curl -sSL https://dl.k8s.io/release/stable.txt
             # Make sure it is compatible with the kubelet.service unit we ship, or update that too.
@@ -901,9 +903,18 @@ def addKubernetesServices(self, config: InstanceConfiguration, architecture: str
             ''').format(**values))
 
         # Before we let the kubelet try to start, we have to actually download it (and kubeadm)
+        # We set up this service so it can restart on failure despite not
+        # leaving a process running, see
+        # <https://github.com/openshift/installer/pull/604> and
+        # <https://github.com/litew/droid-config-ham/commit/26601d85d9d972dc1560096db1c419fd6fd9b238>
+        # We use a forking service with RemainAfterExit, since that lets
+        # restarts work if the script fails. We also use a condition which
+        # treats the service as successful and skips it if it made a file to
+        # say it already ran.
         config.addFile("/home/core/install-kubernetes.sh", contents=textwrap.dedent('''\
             #!/usr/bin/env bash
             set -e
+            FLAG_FILE="{SETUP_STATE_DIR}/install-kubernetes.done"
 
             # Make sure we have Docker enabled; Kubeadm later might complain it isn't.
             systemctl enable docker.service
@@ -916,6 +927,9 @@ def addKubernetesServices(self, config: InstanceConfiguration, architecture: str
             cd {DOWNLOAD_DIR}
             curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/{KUBERNETES_VERSION}/bin/linux/{ARCHITECTURE}/{{kubeadm,kubelet,kubectl}}
             chmod +x {{kubeadm,kubelet,kubectl}}
+
+            mkdir -p "{SETUP_STATE_DIR}"
+            touch "$FLAG_FILE"
             ''').format(**values))
         config.addUnit("install-kubernetes.service", contents=textwrap.dedent('''\
             [Unit]
@@ -923,15 +937,19 @@ def addKubernetesServices(self, config: InstanceConfiguration, architecture: str
             Wants=network-online.target
             After=network-online.target
             Before=kubelet.service
+            ConditionPathExists=!{SETUP_STATE_DIR}/install-kubernetes.done
 
             [Service]
-            Type=oneshot
-            Restart=no
             ExecStart=/usr/bin/bash /home/core/install-kubernetes.sh
+            Type=forking
+            RemainAfterExit=yes
+            Restart=on-failure
+            RestartSec=5s
 
             [Install]
             WantedBy=multi-user.target
-            '''))
+            RequiredBy=kubelet.service
+            ''').format(**values))
 
         # Now we should have the kubeadm command, and the bootlooping kubelet
         # waiting for kubeadm to configure it.
@@ -1031,6 +1049,8 @@ def addKubernetesLeader(self, config: InstanceConfiguration):
             #!/usr/bin/env bash
             set -e
 
+            FLAG_FILE="{SETUP_STATE_DIR}/create-kubernetes-cluster.done"
+
             export PATH="$PATH:{DOWNLOAD_DIR}"
 
             # We need the kubelet being restarted constantly by systemd while kubeadm is setting up.
@@ -1040,7 +1060,10 @@ def addKubernetesLeader(self, config: InstanceConfiguration):
             # We also need to set the hostname for 'kubeadm init' to work properly.
             /bin/sh -c "/usr/bin/hostnamectl set-hostname $(curl -s http://169.254.169.254/latest/meta-data/hostname)"
 
-            kubeadm init --config /home/core/kubernetes-leader.yml
+            if [[ ! -e /etc/kubernetes/admin.conf ]] ; then
+                # Only run this once, it isn't idempotent
+                kubeadm init --config /home/core/kubernetes-leader.yml
+            fi
 
             mkdir -p $HOME/.kube
             cp /etc/kubernetes/admin.conf $HOME/.kube/config
@@ -1066,6 +1089,9 @@ def addKubernetesLeader(self, config: InstanceConfiguration):
             echo "JOIN_TOKEN=$(kubeadm token create --ttl 0)" >>/etc/kubernetes/worker.ini
             echo "JOIN_CERT_HASH=sha256:$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')" >>/etc/kubernetes/worker.ini
             echo "JOIN_ENDPOINT=$(hostname):6443" >>/etc/kubernetes/worker.ini
+
+            mkdir -p "{SETUP_STATE_DIR}"
+            touch "$FLAG_FILE"
             ''').format(**values))
         config.addUnit("create-kubernetes-cluster.service", contents=textwrap.dedent('''\
             [Unit]
@@ -1074,15 +1100,19 @@ def addKubernetesLeader(self, config: InstanceConfiguration):
             After=docker.service
             Before=toil-leader.service
             # Can't be before kubelet.service because Kubelet has to come up as we run this.
+            ConditionPathExists=!{SETUP_STATE_DIR}/create-kubernetes-cluster.done
 
             [Service]
-            Type=oneshot
-            Restart=no
             ExecStart=/usr/bin/bash /home/core/create-kubernetes-cluster.sh
+            Type=forking
+            RemainAfterExit=yes
+            Restart=on-failure
+            RestartSec=5s
 
             [Install]
             WantedBy=multi-user.target
-            '''))
+            RequiredBy=toil-leader.service
+            ''').format(**values))
 
         # We also need a node cleaner service
         config.addFile("/home/core/cleanup-nodes.sh", contents=textwrap.dedent('''\
@@ -1108,7 +1138,8 @@ def addKubernetesLeader(self, config: InstanceConfiguration):
         config.addUnit("cleanup-nodes.service", contents=textwrap.dedent('''\
             [Unit]
             Description=Remove scaled-in nodes
-            After=install-kubernetes.service
+            After=create-kubernetes-cluster.service
+            Requires=create-kubernetes-cluster.service
             [Service]
             ExecStart=/home/core/cleanup-nodes.sh
             Restart=always
@@ -1165,6 +1196,7 @@ def addKubernetesWorker(self, config: InstanceConfiguration, authVars: Dict[str,
         config.addFile("/home/core/join-kubernetes-cluster.sh", contents=textwrap.dedent('''\
             #!/usr/bin/env bash
             set -e
+            FLAG_FILE="{SETUP_STATE_DIR}/join-kubernetes-cluster.done"
 
             export PATH="$PATH:{DOWNLOAD_DIR}"
 
@@ -1173,6 +1205,9 @@ def addKubernetesWorker(self, config: InstanceConfiguration, authVars: Dict[str,
             systemctl start kubelet
 
             kubeadm join {JOIN_ENDPOINT} --config /home/core/kubernetes-worker.yml
+
+            mkdir -p "{SETUP_STATE_DIR}"
+            touch "$FLAG_FILE"
             ''').format(**values))
 
         config.addUnit("join-kubernetes-cluster.service", contents=textwrap.dedent('''\
@@ -1181,15 +1216,19 @@ def addKubernetesWorker(self, config: InstanceConfiguration, authVars: Dict[str,
             After=install-kubernetes.service
             After=docker.service
             # Can't be before kubelet.service because Kubelet has to come up as we run this.
+            Requires=install-kubernetes.service
+            ConditionPathExists=!{SETUP_STATE_DIR}/join-kubernetes-cluster.done
 
             [Service]
-            Type=oneshot
-            Restart=no
             ExecStart=/usr/bin/bash /home/core/join-kubernetes-cluster.sh
+            Type=forking
+            RemainAfterExit=yes
+            Restart=on-failure
+            RestartSec=5s
 
             [Install]
             WantedBy=multi-user.target
-            '''))
+            ''').format(**values))
 
     def _getIgnitionUserData(self, role, keyPath=None, preemptable=False, architecture='amd64'):
         """


Test patch:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -132,6 +132,7 @@ def run_conformance_tests(
             "--clean=always",
             "--logDebug",
             "--statusWait=10",
+            "--retryCount=2"
         ]
         if not caching:
             # Turn off caching for the run
@@ -668,6 +669,7 @@ def test_parasol_cwl_conformance(self, **kwargs):
     def test_kubernetes_cwl_conformance(self, **kwargs):
         return self.test_run_conformance(
             batchSystem="kubernetes",
+            extra_args=["--retryCount=3"],
             # This test doesn't work with
             # Singularity; see
             # https://github.com/common-workflow-language/cwltool/blob/7094ede917c2d5b16d11f9231fe0c05260b51be6/conformance-test.sh#L99-L117
@@ -756,6 +758,7 @@ def test_run_conformance_with_caching(self):
     def test_kubernetes_cwl_conformance(self, **kwargs):
         return self.test_run_conformance(
             batchSystem="kubernetes",
+            extra_args=["--retryCount=3"],
             # These tests don't work with
             # Singularity; see
             # https://github.com/common-workflow-language/cwltool/blob/7094ede917c2d5b16d11f9231fe0c05260b51be6/conformance-test.sh#L99-L117
@@ -825,6 +828,7 @@ def test_kubernetes_cwl_conformance(self, **kwargs):
             )
         return self.test_run_conformance(
             batchSystem="kubernetes",
+            extra_args=["--retryCount=3"],
             # This test doesn't work with
             # Singularity; see
             # https://github.com/common-workflow-language/cwltool/blob/7094ede917c2d5b16d11f9231fe0c05260b51be6/conformance-test.sh#L99-L117