Project: Nomad (Orchestration & Management / Scheduling & Orchestration)
File: architecture-eval-triggers.md

The internals documentation covers the path that an evaluation takes through the leader, worker, and plan applier. This document describes what events within the cluster cause Evaluations to be created. Evaluations have a `TriggeredBy` field which is always one of the values defined in : ``` const ( EvalTriggerJobRegister = "job-register" EvalTriggerJobDeregister = "job-deregister" EvalTriggerPeriodicJob = "periodic-job" EvalTriggerNodeDrain = "node-drain" EvalTriggerNodeUpdate = "node-update" EvalTriggerAllocStop = "alloc-stop" EvalTriggerScheduled = "scheduled" EvalTriggerRollingUpdate = "rolling-update" EvalTriggerDeploymentWatcher = "deployment-watcher" EvalTriggerFailedFollowUp = "failed-follow-up" EvalTriggerMaxPlans = "max-plan-attempts" EvalTriggerRetryFailedAlloc = "alloc-failure" EvalTriggerQueuedAllocs = "queued-allocs" EvalTriggerPreemption = "preemption" EvalTriggerScaling = "job-scaling" EvalTriggerMaxDisconnectTimeout = "max-disconnect-timeout" EvalTriggerReconnect = "reconnect" ) ``` The list below covers each trigger and what can trigger it. job-register*: Creating or updating a Job will result in 1 Evaluation created, plus any follow-up Evaluations associated with scheduling, planning, or deployments. job-deregister*: Stopping a Job will result in 1 Evaluation created, plus any follow-up Evaluations associated with scheduling, planning, or deployments. periodic-job*: A periodic job that hits its timer and dispatches a child job will result in 1 Evaluation created, plus any additional Evaluations associated with scheduling or planning. node-drain*: Draining a node will create 1 Evaluation for each Job on the node that's draining, plus any additional Evaluations associated with scheduling or planning. node-update*: When the fingerprint of a client node has changed or the node has changed state (from up to down), Nomad creates 1 Evaluation for each Job running on the Node, plus 1 Evaluation for each system job that has `datacenters` that include the datacenter for that Node. alloc-stop*: When the API that serves the `nomad alloc stop` command is hit, Nomad creates 1 Evaluation. scheduled*: Nomad's internal housekeeping will periodically create Evaluations for garbage collection. rolling-update*: When a `system` job is updated, the block's `stagger` field controls how many Allocations will be scheduled at a time. The scheduler will create 1 Evaluation to follow-up for the next set. deployment-watcher*: When a `service` job is updated, the block controls how many Allocations will be scheduled at a time. The deployment watcher runs on the leader and monitors Allocation healthy. It will create 1 Evaluation when the Deployment has reached the next step. failed-follow-up*: Evaluations that hit a delivery limit and will not be retried by the eval broker are marked as failed. The leader periodically reaps failed Evaluations and creates 1 new Evaluation for these, with a delay. max-plan-attempts*: The scheduler will retry Evaluations that are rejected by the plan applier with a new cluster state snapshot. If the scheduler exceeds the maximum number of retries, it will create 1 new Evaluation in the `blocked` state. alloc-failure*: If an Allocation fails and exceeds its maximum , Nomad creates 1 new Evaluation. queued-allocs*: When a scheduler processes an Evaluation, it may not be able to place all Allocations. It will create 1 new Evaluation in the `blocked` state to be processed later when node updates arrive. preemption*: When Allocations are preempted, the plan applier creates 1 Evaluation for each Job that has been preempted. job-scaling*: Scaling a Job will result in 1 Evaluation created, plus any follow-up Evaluations associated with scheduling, planning, or deployments. max-disconnect-timeout*: When an Allocation is in the `unknown` state for longer than the window, the scheduler will create 1 Evaluation. reconnect*: When a Node in the `disconnected` state reconnects, Nomad will create 1 Evaluation per job with an allocation on the reconnected

Almost any Evaluation processed by the scheduler can result in additional Evaluations being created, whether because the scheduler needs to follow-up from failed scheduling or because the resulting plan changes the state of the cluster. This can result in a large number of Evaluations when the cluster is in an unstable state with frequent changes. Consider the following example where a node running 1 system job and 2 service jobs misses its heartbeat and is marked lost. The Evaluation for the system job is successfully planned. One of the service jobs no longer meets constraints. The other service job is successfully scheduled but the resulting plan is rejected because the scheduler has fallen behind in raft replication. A total of 6 Evaluations are created. ```mermaid flowchart TD event((Node\nmisses\nheartbeat)) system([system\nnode-update]) service1([service 1\nnode-update]) service2([service 2\nnode-update]) blocked([service 1\nblocked\nqueued-allocs]) failed([service 2\nfailed\nmax-plan-attempts]) followup([service 2\nfailed-follow-up]) %% style classes classDef eval fill:#d5f6ea,stroke-width:4px,stroke:#1d9467 classDef other fill:#d5f6ea,stroke:#1d9467 class event other; class system,service1,service2,blocked,failed,followup eval; event --> system event --> service1 event --> service2 service1 --> blocked service2 --> failed failed --> followup ``` Next, consider this example where a `service` job has been updated. The task group has `count = 3` and the following `update` block: ```hcl update { max_parallel = 1 canary = 1 } ``` After each Evaluation is processed, the Deployment Watcher will be waiting to receive information on updated Allocation health. Then it will emit a new Evaluation for the next step. A total of 4 Evaluations are created. ```mermaid flowchart TD registerEvent((Job\nRegister)) alloc1health((Canary\nHealthy)) alloc2health((Alloc 2\nHealthy)) alloc3health((Alloc 3\nHealthy)) register([job-register]) dwPostCanary([deployment-watcher]) dwPostAlloc2([deployment-watcher]) dwPostAlloc3([deployment-watcher]) %% style classes classDef eval fill:#d5f6ea,stroke-width:4px,stroke:#1d9467 classDef other fill:#d5f6ea,stroke:#1d9467 class registerEvent,alloc1health,alloc2health,alloc3health other class register,dwPostCanary,dwPostAlloc2,dwPostAlloc3 eval registerEvent --> register register --> wait1 alloc1health --> wait1 wait1 --> dwPostCanary dwPostCanary --> wait2 alloc2health --> wait2 wait2 --> dwPostAlloc2 dwPostAlloc2 --> wait3 alloc3health --> wait3 wait3 --> dwPostAlloc3 ``` Lastly, consider this example where 2 nodes each running 5 Allocations that are all for system jobs are "flapping" by missing heartbeats and then re-registering, or frequently changing fingerprints. This diagram will show the results from each node going down once and then coming back up. ```mermaid flowchart TD %% style classes classDef eval fill:#d5f6ea,stroke-width:4px,stroke:#1d9467 classDef other fill:#d5f6ea,stroke:#1d9467 eventAdown((Node A\nmisses\nheartbeat)) eventAup((Node A\nheartbeats)) eventBdown((Node B\nmisses\nheartbeat)) eventBup((Node B\nheartbeats)) eventAdown --> eventAup eventBdown --> eventBup A01down([job 1 node A\nnode-update]) A02down([job 2 node A\nnode-update]) A03down([job 3 node A\nnode-update]) A04down([job 4 node A\nnode-update]) A05down([job 5 node A\nnode-update]) B01down([job 1 node B\nnode-update]) B02down([job 2 node B\nnode-update]) B03down([job 3 node B\nnode-update]) B04down([job 4 node B\nnode-update]) B05down([job 5 node B\nnode-update]) A01up([job 1 node A\nnode-update]) A02up([job 2 node A\nnode-update]) A03up([job 3 node A\nnode-update]) A04up([job 4 node A\nnode-update]) A05up([job 5 node A\nnode-update]) B01up([job 1 node B\nnode-update]) B02up([job 2 node B\nnode-update]) B03up([job 3 node B\nnode-update]) B04up([job 4 node B\nnode-update]) B05up([job 5 node B\nnode-update]) eventAdown:::other --> A01down:::eval eventAdown:::other --> A02down:::eval eventAdown:::other --> A03down:::eval eventAdown:::other --> A04down:::eval eventAdown:::other --> A05down:::eval eventAup:::other --> A01up:::eval eventAup:::other --> A02up:::eval eventAup:::other --> A03up:::eval eventAup:::other --> A04up:::eval eventAup:::other --> A05up:::eval eventBdown:::other --> B01down:::eval eventBdown:::other --> B02down:::eval eventBdown:::other --> B03down:::eval eventBdown:::other --> B04down:::eval eventBdown:::other --> B05down:::eval eventBup:::other --> B01up:::eval eventBup:::other --> B02up:::eval eventBup:::other --> B03up:::eval eventBup:::other --> B04up:::eval eventBup:::other --> B05up:::eval ``` You can extrapolate this example to large clusters: 100 nodes each running 10 system jobs and 40 service jobs that go down once and come back up will result in 100 40 2 == 8000 Evaluations created for the service jobs, which will result in rescheduling of service allocations to new nodes. For the system jobs, 2000 Evaluations will be created and all of these will be no-op Evaluations that will still need to be replicated to all raft peers, canceled by the scheduler, and