Repository: DataBiosphere/toil

Issue:
Kubernetes Toil clusters can't use ephemeral instance storage
Because Toil isn't moving `/var/lib/kubelet` over to the ephemeral SSD RAID along with `/var/lib/docker`, Kubernetes nodes don't actually see or use that storage, or expose it as allocateable by Kubernetes.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-844)
┆Issue Number: TOIL-844

Toil tries to include /dev/xvda in the RAID on r5d.24xlarge when it shouldn't, leading to cluster nodes coming up without their ephemeral drives
When Toil makes an AWS `r5d.24xlarge` instance, it seems to see the root volume as both /dev/xda and /dev/nvme0 something. When it greps for `/dev/xvda` in the output of `mount`, it doesn't find it, and so it tries to include it in the RAID.

This fails, but the setup service doesn't notice that creating /dev/md0 fails, and so it goes on to successfully boot the node and present the cluster with only the root volume for storage, and not all the ephemeral disks. This in turn will deadlock a Kubernetes autoscaling workflow, as the autoscaler thinks the nodes available are adequate when they aren't.

We may need to check for symlinks when polling all the drive names, or we may need to come up with a better way then grepping mount to see if a drive or a partition on it is mounted.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-827)
┆Issue Number: TOIL-827



Discussion:

Resolving patch:
diff --git a/src/toil/lib/ec2.py b/src/toil/lib/ec2.py
--- a/src/toil/lib/ec2.py
+++ b/src/toil/lib/ec2.py
@@ -554,11 +554,15 @@ def get_flatcar_ami(ec2_client: BaseClient) -> str:
     # Take a user override
     ami = os.environ.get('TOIL_AWS_AMI')
     if not ami:
+        logger.debug("No AMI found in TOIL_AWS_AMI; checking Flatcar release feed")
         ami = official_flatcar_ami_release(ec2_client=ec2_client)
     if not ami:
+        logger.warning("No available AMI found in Flatcar release feed; checking marketplace")
         ami = aws_marketplace_flatcar_ami_search(ec2_client=ec2_client)
     if not ami:
+        logger.critical("No available AMI found in marketplace")
         raise RuntimeError('Unable to fetch the latest flatcar image.')
+    logger.info("Selected Flatcar AMI: %s", ami)
     return ami
 
 
diff --git a/src/toil/provisioners/abstractProvisioner.py b/src/toil/provisioners/abstractProvisioner.py
--- a/src/toil/provisioners/abstractProvisioner.py
+++ b/src/toil/provisioners/abstractProvisioner.py
@@ -517,44 +517,60 @@ def addVolumesService(self, config: InstanceConfiguration):
             #!/bin/bash
             set -x
             ephemeral_count=0
-            drives=""
-            directories="toil mesos docker cwl"
+            drives=()
+            directories=(toil mesos docker kubelet cwl)
             for drive in /dev/xvd{a..z} /dev/nvme{0..26}n1; do
-                echo checking for $drive
+                echo "checking for ${drive}"
                 if [ -b $drive ]; then
-                    echo found it
-                    if mount | grep $drive; then
+                    echo "found it"
+                    while [ "$(readlink -f "${drive}")" != "${drive}" ] ; do
+                        drive="$(readlink -f "${drive}")"
+                        echo "was a symlink to ${drive}"
+                    done
+                    seen=0
+                    for other_drive in "${drives[@]}" ; do
+                        if [ "${other_drive}" == "${drive}" ] ; then
+                            seen=1
+                            break
+                        fi
+                    done
+                    if (( "${seen}" == "1" )) ; then
+                        echo "already discovered via another name"
+                        continue
+                    fi
+                    if mount | grep "^${drive}"; then
                         echo "already mounted, likely a root device"
                     else
                         ephemeral_count=$((ephemeral_count + 1 ))
-                        drives="$drives $drive"
-                        echo increased ephemeral count by one
+                        drives+=("${drive}")
+                        echo "increased ephemeral count by one"
                     fi
                 fi
             done
             if (("$ephemeral_count" == "0" )); then
-                echo no ephemeral drive
-                for directory in $directories; do
+                echo "no ephemeral drive"
+                for directory in "${directories[@]}"; do
                     sudo mkdir -p /var/lib/$directory
                 done
                 exit 0
             fi
             sudo mkdir /mnt/ephemeral
             if (("$ephemeral_count" == "1" )); then
-                echo one ephemeral drive to mount
-                sudo mkfs.ext4 -F $drives
-                sudo mount $drives /mnt/ephemeral
+                echo "one ephemeral drive to mount"
+                sudo mkfs.ext4 -F "${drives[@]}"
+                sudo mount "${drives[@]}" /mnt/ephemeral
             fi
             if (("$ephemeral_count" > "1" )); then
-                echo multiple drives
-                for drive in $drives; do
-                    dd if=/dev/zero of=$drive bs=4096 count=1024
+                echo "multiple drives"
+                for drive in "${drives[@]}"; do
+                    sudo dd if=/dev/zero of=$drive bs=4096 count=1024
                 done
-                sudo mdadm --create -f --verbose /dev/md0 --level=0 --raid-devices=$ephemeral_count $drives # determine force flag
+                # determine force flag
+                sudo mdadm --create -f --verbose /dev/md0 --level=0 --raid-devices=$ephemeral_count "${drives[@]}" 
                 sudo mkfs.ext4 -F /dev/md0
                 sudo mount /dev/md0 /mnt/ephemeral
             fi
-            for directory in $directories; do
+            for directory in "${directories[@]}"; do
                 sudo mkdir -p /mnt/ephemeral/var/lib/$directory
                 sudo mkdir -p /var/lib/$directory
                 sudo mount --bind /mnt/ephemeral/var/lib/$directory /var/lib/$directory


Test patch: