Repository: PrefectHQ/prefect

Issue:
Kubernetes Cluster AutoScaler can result in failed Flow Runs
## Description
This only happens on K8 clusters with [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) to dynamically provision the K8 nodes based on current workload demands in the cluster. In my case, I've observed this on AWS EKS with manual NodeGroups.

The Cluster Autoscaler will evaluate the utilization of each Node in the K8 cluster to determine if it can scale down the number of Nodes. By default this happens every 10 minutes (configurable by `--scale-down-unneeded-time flag`), and is determined by a number of factors, the main one being the sum of cpu and memory requests of all pods running on this node is smaller than 50% of the node's allocatable (configurable by `--scale-down-utilization-threshold`). There's also a back-off period after a Node is first added to the cluster of 10 minutes (configurable by `--scale-down-delay-after-add flag`).

This functionality of K8 assumes that all Pods created by a controller object (deployment, replica set, **job**, stateful set etc) are ephemeral, and can be relocated to a different Node, with zero impact to the functionality of those Pods. The Prefect Kubernetes Agent triggers Flow Runs via K8 Jobs (effectively declaring them as ephemeral), which spawn the Pods to perform the actual execution. 

The problem is intermittent, and only comes into play under the following situation:

1. A new, long-running (i.e. >21 minutes), Prefect Flow Run is scheduled, and picked up by a Prefect Agent running in K8 w/Cluster Autoscaler
2. Prefect Agent creates a K8 Job, but there are no available K8 Nodes to run this Job within 1-5 minutes
3. Cluster Autoscaler scales up the Nodes in K8. The Node it scales up has at least 2x cpu/memory allocatable than what the single Prefect Flow Run would consume.
4. The Prefect Flow Run is assigned to the new Node Cluster Autoscaler just started
5. Capacity frees up on the other Nodes in the K8 cluster, such that no other Pods are scheduled on the new Node the Cluster Autoscaler just started (in step 3)
6. After 20 minutes, the Cluster Autoscaler determines that the Node from step 3 is a candidate to scale down because it meets the [criteria](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node) (i.e. sum of cpu/memory of all pods is smaller than 50%, the pods were created by a controller object, etc.)
7. The Pod (which is executing the Flow Run) is evicted from this Node and rescheduled to a different Node in K8, and the Cluster Autoscaler Node is removed from K8

Now the fun part:

1. Because K8 evicted the Pod, Prefect Cloud will attempt to keep track of the original Pod (which no longer exists)
2. When the evicted Pod starts again, it will begin the Flow Run execution again ... because this was a long-running Flow Run, it'll take it >21 minutes to complete
3. Eventually Prefect will encounter `No heartbeat detected from the remote task; marking the run as failed.` for the initial Pod it created, which was since relocated by the Cluster AutoScaler.
4. You can observe the newly started Pod is actually reporting logs to Prefect Cloud, so it's curious why it detected no heartbeat from that new process. I'm guessing it's looking for some metadata to identify a heartbeat from the original Pod IP.
5. Because no heartbeat was detected from that original Pod, the Lazarus process will attempt to reschedule the Flow Run .. but the Flow Run was already restarted by K8, it's just in a different Pod now
6. This confusion will result in the Flow Run being marked as "Failed"

While less likely, it's feasible for a Flow Run to be scheduled on a Node that has been up for awhile, but is a candidate to be scaled down in the next 1 minute. Then after ~1 minute of executing the Flow Run on this Node, capacity on other Nodes could free up, such that the Cluster AutoScaler can determine to relocate all the running Pods, which would result in the same type of failure. In this case, the failure would happen to any Flow Run, regardless of execution time.

## Expected Behavior

The root of the problem is that the Cluster AutoScaler is moving the Pod created by the Job the Prefect Agent generated for this Flow Run, before that Flow Run had time to complete. It seems the scheduler may not be capable of handling situations where the Flow Run is restarted by processes outside of it's control.

A simple way to prevent the Cluster AutoScaler from attempting to evict a Pod from a Node, would be to add this annotation to it's manifest:

```
"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
```

If the AutoScaler see's this annotation on a Pod, it will not consider the Node for scaling down. In theory you could add this to a `job_spec_file`, but it'd be exhausting to do this for every Flow you want to execute in a K8 w/Cluster AutoScaler.

## Reproduction
There's a public flow that simply sleeps for a few minutes:
https://hub.docker.com/repository/docker/szelenka/long-running-flow

In AWS, I have an EKS cluster with a simple NodeGroup of t3.2xlarge, where the Prefect Agent spawns Jobs with 2 vCPU and 2G RAM. Initially, we only have the Prefect Agent running on a single Node in K8.

Through Prefect Cloud, we start 3 Flow Runs to run at 5 minutes each. This fills up the capacity of the single Node in K8. 

Then we start another Flow Run, with a run time of 25 minutes. This causes the Cluster AutoScaler to scale up a Node in K8, and assign this Job to that new Node.

Because it takes longer than 20 minutes to complete, and the other Flow Runs have completed, it will be evicted and relocated to the other Node that capacity freed up on, and trigger the above failure.


## Environment
```json
{
  "config_overrides": {
    "cloud": {
      "use_local_secrets": true
    },
    "context": {
      "secrets": false
    }
  },
  "env_vars": [],
  "system_information": {
    "platform": "macOS-10.15.6-x86_64-i386-64bit",
    "prefect_version": "0.12.6",
    "python_version": "3.8.1"
  }
}
```



Discussion:
@szelenka Thanks for a well written issue! I think the best bet here would be to add the eviction policy option setting to the agent so you could say something along the lines of:

```
prefect agent install/start kubernetes --safe-to-evict=False
```

And then all created jobs will have the annotation

```
"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
```
I agree, that's probably the best path forward.

Since the user would know if they're using Cluster AutoScaler in the cluster they're attempting to register the agent, it makes sense to have a flag when they generate the YAML to install the agent.

I'm guessing by using that CLI argument, it would add some parameter to the EnvVar of the Agent, similar to the `JOB_*` ones used for the CPU/MEM limit/requests. Then when the agent spawns the Pod, it'd check for that EnvVar to add that annotation to the mainfest?

The Prefect Agent itself can be evicted without problems (that I've observed). However, one thing I have not tested is what happens to Pods that are "Completed" but aren't removed from the namespace (because Resource-Manager isn't being used). Would having that annotation on a "Completed" Pod prevent the Cluster AutoScaler from evicting it? I'll try to run a test later this week or next to figure out (unless someone else knows the answer).
> I'm guessing by using that CLI argument, it would add some parameter to the EnvVar of the Agent, similar to the JOB_* ones used for the CPU/MEM limit/requests. Then when the agent spawns the Pod, it'd check for that EnvVar to add that annotation to the mainfest?

Yep that's exactly how it should be done! For the Completed pods it would be good to test because I think k8s should be deallocating resources for the pods after completion
Presumably this can happen when prefect pod is evicted for other reasons too. For instance, I just migrated pods from one worker group to another in order to install weave-net. Would happen for other issues where node needed to be restarted (e.g. you are using autoscaler with spot nodes, and one gets pulled from you by AWS), or higher priority task needed to be scheduled and the cluster wasn't scalable. 

Besides "not safe to evict", perhaps the prefect pod should have a "preStop" hook that sends notification that it is being shut down?

True, a [preStop hook](https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/) in the Pod manifest which phones home to the scheduler sounds like a great idea (IMO). 

The scheduler would need to have some logic baked in to handle these situations, and I'm not aware of all the other type of agents Prefect has to understand if they'd have similar type of functionality.

Another option could be with setting a [PodDisruptionBudget (PDB)](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) for the Jobs to tell Kubernetes how to handle these `voluntary` disruptions.

Where the `minAvailable` could be set to `1` for the Pods that are spawned by a Prefect Flow Run. Since there's only ever 1 replica running, it will block the evictions until that Pod has completed. 

But then that creates another thing for the Resource Manager to clean-up, since the PDB is it's own manifest and isn't controlled by the Job. Though I'm sure you could apply labels/annotations to it and delete all objects by those.
@szelenka -- these are good ideas, but they aren't mutually exclusive. Whatever we tell kubernetes about not causing disruption, sometimes a node has to go down, and a preStop hook seems as if it is the only way to avoid errors in this situation (I think?).

Though another possibility is that the scheduler has some understanding of rescheduled pods -- if a new pod is rescheduled for whatever reason (e.g. because a node goes down), doesn't it contact the scheduler when it comes up? If so, perhaps the scheduler can simply note that it has been moved.
1) This is a fantastic issue--I can clearly understand the story here and it is a very interesting Tale of Two Schedulers
2) I really like the idea of calling back with an SOS on eviction in a preStop hook!  It would be "more Prefecty" (and my instinct) to call back directly to Server rather than to the Agent.  This would elevate the observability of this kind of failure quite well and allow the orchestration layer to continue handling the orchestration
3) re: PDBs... this would make k8s scaling and similar operations really smooth.  Stop me if I'm going too far here, because this idea is either a pattern or an antipattern, but what if we create a PDB (optionally, based on agent and/ or flow config) and then kill it in the preStop hook?
4) We may want to think about using a ConfigMap for mounting agent configuration so we can turn these features on and off.  I believe this would make it possible to both set and reflect these settings in the UI, if desired

Resolving patch:
diff --git a/src/prefect/environments/storage/docker.py b/src/prefect/environments/storage/docker.py
--- a/src/prefect/environments/storage/docker.py
+++ b/src/prefect/environments/storage/docker.py
@@ -56,7 +56,7 @@ class Docker(Storage):
         - env_vars (dict, optional): a dictionary of environment variables to
             use when building
         - files (dict, optional): a dictionary of files to copy into the image
-            when building
+            when building. Takes the format of `{'src': 'dest'}`
         - prefect_version (str, optional): an optional branch, tag, or commit
             specifying the version of prefect you want installed into the container;
             defaults to the version you are currently using or `"master"` if your


Test patch: