Project: Nomad (Orchestration & Management / Scheduling & Orchestration)
File: architecture-eval-states.md

The internals documentation covers the path that an evaluation takes through the leader, worker, and plan applier. But it doesn't cover in any detail the various `Evaluation.Status` values, or where the `PreviousEval`, `NextEval`, or `BlockedEval` ID pointers are set. The state diagram below describes the transitions between `Status` values as solid arrows. The dashed arrows represent when a new evaluation is created. The parenthetical labels on those arrows are the `TriggeredBy` field for the new evaluation. The status values are: `pending` evaluations either are queued to be scheduled, are still being processed in the scheduler, or are being applied by the plan applier and not yet acknowledged. `failed` evaluations have failed to be applied by the plan applier (or are somehow invalid in the scheduler; this is always a bug) `blocked` evaluations are created when an eval has failed too many attempts to have its plan applied by the leader, or when a plan can only be partially applied and there are still more allocations to create. `complete` means the plan was applied successfully (at least partially). `canceled` means the evaluation was superseded by state changes like a new version of the job. ```mermaid flowchart LR event((Cluster\nEvent)) pending([pending]) blocked([blocked]) complete([complete]) failed([failed]) canceled([canceled]) %% style classes classDef status fill:#d5f6ea,stroke-width:4px,stroke:#1d9467 classDef other fill:#d5f6ea,stroke:#1d9467 class event other; class pending,blocked,complete,failed,canceled status; event -. "job-register job-deregister periodic-job node-update node-drain alloc-stop scheduled alloc-failure job-scaling" .-> pending pending -. "new eval\n(rolling-update)" .-> pending pending -. "new eval\n(preemption)" .-> pending pending -. "new eval\n(max-plan-attempts)" .-> blocked pending -- if plan submitted --> complete pending -- if invalid --> failed pending -- if no-op --> canceled failed -- if retried --> blocked failed -- if retried --> complete blocked -- if no-op --> canceled blocked -- if plan submitted --> complete complete -. "new eval\n(deployment-watcher)" .-> pending complete -. "new eval\n(queued-allocs)" .-> blocked failed -. "new eval\n(failed-follow-up)" .-> pending ``` But it's hard to get a full picture of the evaluation lifecycle purely from the `Status` fields, because evaluations have several "quasi-statuses" which aren't represented as explicit statuses in the state store: `scheduling` is the status where an eval is being processed by the scheduler worker. `applying` is the status where the resulting plan for the eval is being applied in the plan applier on the

`delayed` is an enqueued eval that will be dequeued some time in the future. `deleted` is when an eval is removed from the state store entirely. By adding these statuses to the diagram (the dashed nodes), you can see where the same `Status` transition might result in different `PreviousEval`, `NextEval`, or `BlockedEval` set. You can also see where the "chain" of evaluations is broken when new evals are created for preemptions or by the deployment watcher. ```mermaid flowchart LR event((Cluster\nEvent)) %% statuss pending([pending]) blocked([blocked]) complete([complete]) failed([failed]) canceled([canceled]) %% quasi-statuss deleted([deleted]) delayed([delayed]) scheduling([scheduling]) applying([applying]) %% style classes classDef status fill:#d5f6ea,stroke-width:4px,stroke:#1d9467 classDef quasistatus fill:#d5f6ea,stroke-dasharray: 5 5,stroke:#1d9467 classDef other fill:#d5f6ea,stroke:#1d9467 class event other; class pending,blocked,complete,failed,canceled status; class deleted,delayed,scheduling,applying quasistatus; event -- "job-register job-deregister periodic-job node-update node-drain alloc-stop scheduled alloc-failure job-scaling" --> pending pending -- dequeued --> scheduling pending -- if delayed --> delayed delayed -- dequeued --> scheduling scheduling -. "not all allocs placed new eval created by scheduler trigger queued-allocs new has .PreviousEval = old.ID old has .BlockedEval = new.ID" .-> blocked scheduling -. "failed to plan new eval created by scheduler trigger: max-plan-attempts new has .PreviousEval = old.ID old has .BlockedEval = new.ID" .-> blocked scheduling -- "not all allocs placed reuse already-blocked eval" --> blocked blocked -- "unblocked by external state changes" --> scheduling scheduling -- allocs placed --> complete scheduling -- "wrong eval type or max retries exceeded on plan submit" --> failed scheduling -- "canceled by job update/stop" --> canceled failed -- retry --> scheduling scheduling -. "new eval from rolling update (system jobs) created by scheduler trigger: rolling-update new has .PreviousEval = old.ID old has .NextEval = new.ID" .-> pending scheduling -- submit --> applying applying -- failed --> scheduling applying -. "new eval for preempted allocs created by plan applier trigger: preemption new has .PreviousEval = unset! old has .BlockedEval = unset!" .-> pending complete -. "new eval from deployments (service jobs) created by deploymentwatcher trigger: deployment-watcher new has .PreviousEval = unset! old has .NextEval = unset!" .-> pending failed -- "new eval trigger: failed-follow-up new has .PreviousEval = old.ID old has .NextEval = new.ID" --> pending pending -- "undeliverable evals reaped by leader" --> failed blocked -- "duplicate blocked evals reaped by leader" --> canceled canceled -- garbage\ncollection --> deleted failed -- garbage\ncollection --> deleted complete -- garbage\ncollection --> deleted ```